You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pearu (via GitHub)" <gi...@apache.org> on 2026/10/09 13:36:45 UTC
[PR] GH-50842: [Python] Convert dictionaries to objects without decoding when the dense array would overflow [arrow]
pearu opened a new pull request, #52547:
URL: https://github.com/apache/arrow/pull/52547
### Rationale for this change
Converting a `DictionaryArray` with `string`/`binary` values to a NumPy object array (`to_numpy`, `np.asarray`), or to pandas as the child of a nested type, decodes the dictionary into a dense array first. When the decoded data exceeds the 32-bit offset limit this fails with `Take operation overflowed binary array capacity` (a segfault or garbage before #50841), although the result, a NumPy object array, has no such limit.
### What changes are included in this PR?
- `DecodingWouldOverflow` predicts from the dictionary offsets and the indices whether decoding a chunk would exceed the limit the `take` kernel enforces. Two cheap bounds settle the common case without reading the rows.
- Only when it would overflow, the dictionary is left undecoded and `ObjectWriterVisitor::Visit(const DictionaryType&)` converts each dictionary value to a Python object once and references it per index (the `deduplicate_objects` idea, without the memo table since the dictionary already holds the unique values).
- Every other conversion takes the unchanged decode path. Measured on 10⁶–10⁷ rows with dictionaries of 10²–10⁶ entries: time and memory are within noise of the current code.
Not covered: map keys and items decode before reaching this check (`ConvertMap`), so a map with an overflowing dictionary child still raises.
### Are these changes tested?
Yes: `test_dictionary_to_numpy_without_decoding` (plus null index, null dictionary value and `binary` values) and `test_list_of_dictionary_without_decoding` (the nested path, also with several chunks). The dense form in the tests is 2.1 GB, built from 16385 references to one 128 KiB value, so the tests themselves need under 1 MB.
### Are there any user-facing changes?
Conversions that raised now succeed. In that regime, repeated values in the result refer to one Python object, as `to_pandas(deduplicate_objects=True)` already does.
### AI usage disclosure
Developed with AI assistance (Claude Code): the approach was worked out and the implementation, tests and benchmarks were generated under my direction and reviewed line by line by me. I own and can debug these changes.
* GitHub Issue: #50842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50842: [Python] Convert dictionaries to objects without decoding when the dense array would overflow [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #52547:
URL: https://github.com/apache/arrow/pull/52547#issuecomment-6082051059
:warning: GitHub issue #50842 **has no components**, please add labels for components.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50842: [Python] Convert dictionaries to objects without decoding when the dense array would overflow [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #52547:
URL: https://github.com/apache/arrow/pull/52547#issuecomment-6082050852
:warning: GitHub issue #50842 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50842: [Python] Convert dictionaries to objects without decoding when the dense array would overflow [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #52547:
URL: https://github.com/apache/arrow/pull/52547#issuecomment-6146906744
I find it very weird to vary the output type depending on the input *size*. Why can't we generate a CATEGORICAL without converting to dense first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50842: [Python] Convert dictionaries to objects without decoding when the dense array would overflow [arrow]
Posted by "pearu (via GitHub)" <gi...@apache.org>.
pearu commented on PR #52547:
URL: https://github.com/apache/arrow/pull/52547#issuecomment-6152421172
> I find it very weird to vary the output type depending on the input _size_. Why can't we generate a CATEGORICAL without converting to dense first?
Agreed, the size gate is a premature optimization. I'll reconsider the approach: convert string/binary dictionaries from the dictionary whenever an object array is requested, with no dense intermediate, and drop the overflow predictor. I'll update the PR once I have the implementation and measurements.
On categorical: the two paths this PR touches produce NumPy arrays (`Array.to_numpy`, and the children of nested types, which are sliced into rows as ndarrays), and NumPy has no categorical dtype. The non-dense NumPy form of a dictionary is an object array built from the dictionary's Python objects, which is what `ChunkedArray.to_numpy` already does. Top-level `to_pandas` keeps producing a Categorical and is untouched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org