You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2025/12/11 05:06:28 UTC

[PR] GH-48450: [Python][C++] Support unsigned dictionary indices in pandas conversion [arrow]

HyukjinKwon opened a new pull request, #48451:
URL: https://github.com/apache/arrow/pull/48451

   ### Rationale for this change
   
   This is the ticket mentioned in https://github.com/apache/arrow/pull/7659 which implements unsigned dictionary indices in pandas conversion.
   
   The reason why it was not implement was:
   
   pandas Categorical uses -1 to represent null values, which cannot be properly represented in unsigned integer types. Simply adding the unsigned cases would have caused:
   
   - uint8: -1 wraps to 255
   - uint16: -1 wraps to 65535
   
   ### What changes are included in this PR?
   
   Implements unsigned dictionary indices in pandas conversion
   
   ### Are these changes tested?
   
   Yes via:
   
   ```
   pytest -xvs python/pyarrow/tests/test_pandas.py::test_dictionary_with_pandas
   ```
   
   ### Are there any user-facing changes?
   
   Yes, `pd.Categorical.from_codes(indices, categories=dictionary)` with unsigned integers should work now as demonstrated in the tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "Danferno (via GitHub)" <gi...@apache.org>.
Danferno commented on PR #48451:
URL: https://github.com/apache/arrow/pull/48451#issuecomment-4242436194

   For those wondering, this was merged into pyarrow 23.0.0. Thanks a lot to everyone involved!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #48451:
URL: https://github.com/apache/arrow/pull/48451#discussion_r2641466271


##########
python/pyarrow/tests/test_pandas.py:
##########
@@ -4114,24 +4114,40 @@ def test_dictionary_with_pandas():
         d1 = pa.DictionaryArray.from_arrays(indices, dictionary)
         d2 = pa.DictionaryArray.from_arrays(indices, dictionary, mask=mask)
 
-        if index_type[0] == 'u':
-            # TODO: unsigned dictionary indices to pandas
-            with pytest.raises(TypeError):
+        if index_type == 'uint64':
+            # uint64 is not supported due to overflow risk (values > 2^63-1)
+            with pytest.raises(TypeError,
+                               match="Converting unsigned dictionary indices"):

Review Comment:
   This expected error message needs updating now too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48451:
URL: https://github.com/apache/arrow/pull/48451#issuecomment-3640277585

   :warning: GitHub issue #47022 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #48451:
URL: https://github.com/apache/arrow/pull/48451#issuecomment-3644188774

   Should be ready for a look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python][C++] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48451:
URL: https://github.com/apache/arrow/pull/48451#issuecomment-3640228555

   :warning: GitHub issue #47022 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48451:
URL: https://github.com/apache/arrow/pull/48451#issuecomment-3640236048

   :warning: GitHub issue #47022 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #48451:
URL: https://github.com/apache/arrow/pull/48451#issuecomment-3684850814

   Thank you @adamreeve !!!!!!!!!!!!!!!!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-48450: [Python][C++] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48451:
URL: https://github.com/apache/arrow/pull/48451#issuecomment-3640142282

   :warning: GitHub issue #48450 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48451:
URL: https://github.com/apache/arrow/pull/48451#issuecomment-3640288775

   :warning: GitHub issue #47022 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve merged PR #48451:
URL: https://github.com/apache/arrow/pull/48451


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on PR #48451:
URL: https://github.com/apache/arrow/pull/48451#issuecomment-3684841960

   Thanks for adding this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #48451:
URL: https://github.com/apache/arrow/pull/48451#issuecomment-3692880432

   After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit de6eb89dbdcf210802c3aad5d3f1a3d4c64c3582.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/58962202666) has more details. It also includes information about 14 possible false positives for unstable benchmarks that are known to sometimes produce them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #48451:
URL: https://github.com/apache/arrow/pull/48451#discussion_r2621429320


##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -2023,9 +2040,9 @@ Status MakeWriter(const PandasOptions& options, PandasWriter::type writer_type,
         CATEGORICAL_CASE(Int16Type);
         CATEGORICAL_CASE(Int32Type);
         CATEGORICAL_CASE(Int64Type);
-        case Type::UINT8:
-        case Type::UINT16:
-        case Type::UINT32:
+        CATEGORICAL_CASE(UInt8Type);
+        CATEGORICAL_CASE(UInt16Type);
+        CATEGORICAL_CASE(UInt32Type);
         case Type::UINT64:
           return Status::TypeError(
               "Converting unsigned dictionary indices to pandas",

Review Comment:
   Can this error message please be updated? It's no longer correct that unsigned indices are not supported after your change. How about "Converting UInt64 dictionary indices to pandas is not supported".



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #48451:
URL: https://github.com/apache/arrow/pull/48451#discussion_r2634041662


##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -2023,9 +2040,9 @@ Status MakeWriter(const PandasOptions& options, PandasWriter::type writer_type,
         CATEGORICAL_CASE(Int16Type);
         CATEGORICAL_CASE(Int32Type);
         CATEGORICAL_CASE(Int64Type);
-        case Type::UINT8:
-        case Type::UINT16:
-        case Type::UINT32:
+        CATEGORICAL_CASE(UInt8Type);
+        CATEGORICAL_CASE(UInt16Type);
+        CATEGORICAL_CASE(UInt32Type);
         case Type::UINT64:
           return Status::TypeError(
               "Converting unsigned dictionary indices to pandas",

Review Comment:
   Sure!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #48451:
URL: https://github.com/apache/arrow/pull/48451#discussion_r2641561937


##########
python/pyarrow/tests/test_pandas.py:
##########
@@ -4114,24 +4114,40 @@ def test_dictionary_with_pandas():
         d1 = pa.DictionaryArray.from_arrays(indices, dictionary)
         d2 = pa.DictionaryArray.from_arrays(indices, dictionary, mask=mask)
 
-        if index_type[0] == 'u':
-            # TODO: unsigned dictionary indices to pandas
-            with pytest.raises(TypeError):
+        if index_type == 'uint64':
+            # uint64 is not supported due to overflow risk (values > 2^63-1)
+            with pytest.raises(TypeError,
+                               match="Converting unsigned dictionary indices"):

Review Comment:
   Yup yup. let me fix it up!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-47022: [Python] Support unsigned dictionary indices in pandas conversion [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48451:
URL: https://github.com/apache/arrow/pull/48451#issuecomment-3640241113

   :warning: GitHub issue #47022 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org