Skip to content

Preserve object dtype on full reduction (e.g. .sum()) of object arrays#11468

Open
agu2347 wants to merge 1 commit into
pydata:mainfrom
agu2347:fix-object-dtype-full-reduction
Open

Preserve object dtype on full reduction (e.g. .sum()) of object arrays#11468
agu2347 wants to merge 1 commit into
pydata:mainfrom
agu2347:fix-object-dtype-full-reduction

Conversation

@agu2347

@agu2347 agu2347 commented Jul 21, 2026

Copy link
Copy Markdown

Fixes #5024.

Summing (or otherwise fully reducing over all axes) an object-dtype DataArray/Variable converted the result to a numpy fixed-width string/unicode dtype instead of preserving the original object dtype -- e.g. summing an all-string object array produced dtype <U9 instead of object. Reducing over a subset of axes already preserved the dtype correctly, as noted in the issue.

Root cause: numpy's own full-array reduction (no axis argument) over an object-dtype array returns a raw Python scalar rather than a 0-d ndarray -- e.g. np.sum() on a 3x3 object array of the string 'a' returns the plain str 'aaaaaaaaa', not an array. NamedArray.reduce() passed this raw scalar on to from_array(), whose final fallback path is a bare np.asarray(data) with no explicit dtype. numpy then infers a dtype from the scalar itself (a unicode dtype for a str), losing the fact that it originated from an object-dtype reduction.

from_array() already has a precedent for exactly this class of problem: tuple inputs are wrapped via to_0d_object_array() specifically to avoid this dtype misinference. That special-casing can't happen inside from_array() itself for raw scalars in general, though, since from_array() is also used for legitimate direct scalar construction (e.g. xr.DataArray("hello") should still get numpy's inferred unicode dtype) and has no way to know a given scalar came from an object-dtype reduction versus user input.

Fix: handle it at the point where that information is still available -- in NamedArray.reduce(), right after computing the reduction result, if the original array's dtype was object and the result has no .dtype attribute (i.e. it's a raw scalar, not an array), wrap it with the same to_0d_object_array() helper before it reaches from_array().

Testing: verified against the exact reproduction in the issue (before: dtype <U9; after: dtype object, with the value intact), confirmed partial-axis reductions (already correct) are unaffected, and confirmed plain numeric reductions (e.g. summing a float64 array) are completely unaffected. Added a regression test (test_reduce_keeps_object_dtype_on_full_reduction) covering all three cases. I confirmed the test fails with the original code (dtype <U9) and passes with the fix.

Ran the full existing test_namedarray.py, test_variable.py, and test_dataarray.py suites (913 passed, 313 skipped, 2 xfailed, 1 xpassed -- 912 baseline + 1 new test, no regressions).

Summing (or otherwise fully reducing over all axes) an object-dtype
DataArray/Variable converted the result to a numpy fixed-width
string/unicode dtype instead of preserving the original object dtype,
e.g. summing an all-string object array produced dtype '<U9' instead
of 'object'. Reducing over a subset of axes already preserved the
dtype correctly, as noted in the issue.

Root cause: numpy's own full-array reduction (no axis argument) over
an object-dtype array returns a raw Python scalar rather than a 0-d
ndarray -- e.g. np.sum() on a 3x3 object array of the string 'a'
returns the plain str 'aaaaaaaaa', not an array. NamedArray.reduce()
passed this raw scalar on to from_array(), whose final fallback path
is a bare np.asarray(data) with no explicit dtype. numpy then infers
a dtype from the scalar itself (a unicode dtype for a str), losing
the fact that it originated from an object-dtype reduction.

from_array() already has a precedent for exactly this class of
problem: tuple inputs are wrapped via to_0d_object_array() specifically
to avoid dtype misinference. That special-casing can't happen inside
from_array() itself for raw scalars in general, though, since
from_array() is also used for legitimate direct scalar construction
(e.g. xr.DataArray("hello") should still get numpy's inferred unicode
dtype) and has no way to know a given scalar came from an object-dtype
reduction versus user input.

Fix it at the point where that information is still available:
in NamedArray.reduce(), right after computing the reduction result,
if the original array's dtype was object and the result has no
.dtype attribute (i.e. it's a raw scalar, not an array), wrap it with
the same to_0d_object_array() helper before it reaches from_array().

Verified against the exact reproduction in the issue (before: dtype
'<U9'; after: dtype 'object', with the value intact), confirmed
partial-axis reductions (already correct) are unaffected, and
confirmed plain numeric reductions (e.g. summing a float64 array)
are completely unaffected. Added a regression test
(test_reduce_keeps_object_dtype_on_full_reduction) covering all
three cases. Confirmed the test fails with the original code
(dtype '<U9') and passes with the fix.

Ran the full existing test_namedarray.py, test_variable.py, and
test_dataarray.py suites (913 passed, 313 skipped, 2 xfailed,
1 xpassed -- 912 baseline + 1 new test, no regressions).

Fixes pydata#5024
@github-actions github-actions Bot added the topic-NamedArray Lightweight version of Variable label Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic-NamedArray Lightweight version of Variable

Projects

None yet

Development

Successfully merging this pull request may close these issues.

xr.DataArray.sum() converts string objects into unicode

1 participant