Preserve object dtype on full reduction (e.g. .sum()) of object arrays#11468
Open
agu2347 wants to merge 1 commit into
Open
Preserve object dtype on full reduction (e.g. .sum()) of object arrays#11468agu2347 wants to merge 1 commit into
agu2347 wants to merge 1 commit into
Conversation
Summing (or otherwise fully reducing over all axes) an object-dtype
DataArray/Variable converted the result to a numpy fixed-width
string/unicode dtype instead of preserving the original object dtype,
e.g. summing an all-string object array produced dtype '<U9' instead
of 'object'. Reducing over a subset of axes already preserved the
dtype correctly, as noted in the issue.
Root cause: numpy's own full-array reduction (no axis argument) over
an object-dtype array returns a raw Python scalar rather than a 0-d
ndarray -- e.g. np.sum() on a 3x3 object array of the string 'a'
returns the plain str 'aaaaaaaaa', not an array. NamedArray.reduce()
passed this raw scalar on to from_array(), whose final fallback path
is a bare np.asarray(data) with no explicit dtype. numpy then infers
a dtype from the scalar itself (a unicode dtype for a str), losing
the fact that it originated from an object-dtype reduction.
from_array() already has a precedent for exactly this class of
problem: tuple inputs are wrapped via to_0d_object_array() specifically
to avoid dtype misinference. That special-casing can't happen inside
from_array() itself for raw scalars in general, though, since
from_array() is also used for legitimate direct scalar construction
(e.g. xr.DataArray("hello") should still get numpy's inferred unicode
dtype) and has no way to know a given scalar came from an object-dtype
reduction versus user input.
Fix it at the point where that information is still available:
in NamedArray.reduce(), right after computing the reduction result,
if the original array's dtype was object and the result has no
.dtype attribute (i.e. it's a raw scalar, not an array), wrap it with
the same to_0d_object_array() helper before it reaches from_array().
Verified against the exact reproduction in the issue (before: dtype
'<U9'; after: dtype 'object', with the value intact), confirmed
partial-axis reductions (already correct) are unaffected, and
confirmed plain numeric reductions (e.g. summing a float64 array)
are completely unaffected. Added a regression test
(test_reduce_keeps_object_dtype_on_full_reduction) covering all
three cases. Confirmed the test fails with the original code
(dtype '<U9') and passes with the fix.
Ran the full existing test_namedarray.py, test_variable.py, and
test_dataarray.py suites (913 passed, 313 skipped, 2 xfailed,
1 xpassed -- 912 baseline + 1 new test, no regressions).
Fixes pydata#5024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5024.
Summing (or otherwise fully reducing over all axes) an object-dtype DataArray/Variable converted the result to a numpy fixed-width string/unicode dtype instead of preserving the original object dtype -- e.g. summing an all-string object array produced dtype
<U9instead ofobject. Reducing over a subset of axes already preserved the dtype correctly, as noted in the issue.Root cause: numpy's own full-array reduction (no
axisargument) over an object-dtype array returns a raw Python scalar rather than a 0-d ndarray -- e.g.np.sum()on a 3x3 object array of the string'a'returns the plainstr'aaaaaaaaa', not an array.NamedArray.reduce()passed this raw scalar on tofrom_array(), whose final fallback path is a barenp.asarray(data)with no explicit dtype. numpy then infers a dtype from the scalar itself (a unicode dtype for astr), losing the fact that it originated from an object-dtype reduction.from_array()already has a precedent for exactly this class of problem:tupleinputs are wrapped viato_0d_object_array()specifically to avoid this dtype misinference. That special-casing can't happen insidefrom_array()itself for raw scalars in general, though, sincefrom_array()is also used for legitimate direct scalar construction (e.g.xr.DataArray("hello")should still get numpy's inferred unicode dtype) and has no way to know a given scalar came from an object-dtype reduction versus user input.Fix: handle it at the point where that information is still available -- in
NamedArray.reduce(), right after computing the reduction result, if the original array's dtype wasobjectand the result has no.dtypeattribute (i.e. it's a raw scalar, not an array), wrap it with the sameto_0d_object_array()helper before it reachesfrom_array().Testing: verified against the exact reproduction in the issue (before: dtype
<U9; after: dtypeobject, with the value intact), confirmed partial-axis reductions (already correct) are unaffected, and confirmed plain numeric reductions (e.g. summing a float64 array) are completely unaffected. Added a regression test (test_reduce_keeps_object_dtype_on_full_reduction) covering all three cases. I confirmed the test fails with the original code (dtype<U9) and passes with the fix.Ran the full existing
test_namedarray.py,test_variable.py, andtest_dataarray.pysuites (913 passed, 313 skipped, 2 xfailed, 1 xpassed -- 912 baseline + 1 new test, no regressions).