You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Kayvan-Zahiri (via GitHub)" <gi...@apache.org> on 2026/08/30 18:02:15 UTC
[PR] GH-XXXXX: [Python][Interchange] Preserve the categorical ordered flag in from_dataframe [arrow]
Kayvan-Zahiri opened a new pull request, #51045:
URL: https://github.com/apache/arrow/pull/51045
### Rationale for this change
`from_dataframe` builds the dictionary array without the `is_ordered` flag the producer reports, so an ordered categorical column comes back unordered. A pyarrow table with `dictionary(int32, string, ordered=True)` does not survive its own round trip, and an ordered pandas `Categorical` loses its ordering on the way in. The pandas consumer passes the flag through (`pd.Categorical(values, categories=categories, ordered=categorical["is_ordered"])`), and our own producer reports it, so pyarrow is the only side dropping it.
### What changes are included in this PR?
`categorical_column_to_dictionary` passes `ordered=categorical["is_ordered"]` to `DictionaryArray.from_arrays`. Indices, dictionary and null handling are untouched.
### Are these changes tested?
Yes. `test_pyarrow_roundtrip_categorical` is now parametrized over ordered True and False, and the ordered cases fail without the change: `assert table.equals(result)` sees `ordered=1` going in and `ordered=0` coming out. I also fixed a copy-paste in `test_pandas_roundtrip_categorical`, which read `describe_categorical` from the result column twice, so its `is_ordered` assertion compared the result with itself and could never fail.
### Are there any user-facing changes?
An ordered dictionary column stays ordered through `from_dataframe`. No API change.
I used an AI assistant while finding and writing this. I checked the behavior and the tests myself before opening the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51046: [Python][Interchange] Preserve the categorical ordered flag in from_dataframe [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #51045:
URL: https://github.com/apache/arrow/pull/51045#issuecomment-5526338938
This looks like a good fix, but note that the interchange protocol is deprecated, and so IMO it is not worth it spending much time on it to update / fix things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51046: [Python][Interchange] Preserve the categorical ordered flag in from_dataframe [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51045:
URL: https://github.com/apache/arrow/pull/51045#issuecomment-5470357255
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
**This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format.**
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
**After updating the title, you can mark the pull request as ready for review.**
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - Contributing Overview](https://arrow.apache.org/docs/developers/overview.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51046: [Python][Interchange] Preserve the categorical ordered flag in from_dataframe [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51045:
URL: https://github.com/apache/arrow/pull/51045#issuecomment-5470359652
:warning: GitHub issue #51046 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-51046: [Python][Interchange] Preserve the categorical ordered flag in from_dataframe [arrow]
Posted by "Kayvan-Zahiri (via GitHub)" <gi...@apache.org>.
Kayvan-Zahiri commented on PR #51045:
URL: https://github.com/apache/arrow/pull/51045#issuecomment-5529804074
Understood, and that is your call to make. Happy to close this if you would rather not
carry it.
One thing worth separating out before I do, because it is independent of the interchange
question. The existing `test_pandas_roundtrip_categorical` compares the result to itself:
```python
desc_cat_table = col_result.describe_categorical
desc_cat_result = col_result.describe_categorical
assert desc_cat_table["is_ordered"] == desc_cat_result["is_ordered"]
assert desc_cat_table["is_dictionary"] == desc_cat_result["is_dictionary"]
```
Both sides are `col_result`, so those two assertions hold for any input and the test cannot
fail on a round-trip mismatch. That is presumably why the dropped `ordered` flag went
unnoticed. The one-word change to `col_table` on the first line makes it an actual
comparison.
So there are two things in this PR: the `ordered=` argument in
`categorical_column_to_dictionary`, which sits squarely in the deprecated protocol, and a
test that silently passes, which does not.
Tell me which you prefer and I will do it:
- Close the whole thing, no hard feelings.
- Keep just the test fix, dropping the `from_dataframe.py` change, so the assertion is real
for as long as the module is around.
- Merge as is, since it is +7/-4 with the parametrized case already written.
Not looking to spend your time on it either way. I raised it because a test that cannot
fail seemed worth mentioning regardless of what happens to the protocol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org