You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "ben-freist (via GitHub)" <gi...@apache.org> on 2025/01/20 12:14:37 UTC

[I] Support inference of unsigned integer types [arrow]

ben-freist opened a new issue, #45312:
URL: https://github.com/apache/arrow/issues/45312

   ### Describe the enhancement requested
   
   The type inference for schema detection that's implemented here https://github.com/apache/arrow/blob/9801801df3f75339f705264252c0eac189f5f2a3/python/pyarrow/src/arrow/python/inference.cc#L493 does not distinguish between signed and unsigned integer types.
   
   This leads to the following behaviour, I think it would be nice if that was more consistent.
   ```
   import pyarrow as pa
   import pandas as pd
   
   data_1 = [{"a": pow(2, 63) - 1}]
   schema_1 = pa.Schema.from_pandas(pd.DataFrame(data_1))
   print(schema_1) # takes a different codepath, correctly infers uint64
   data_2 = [{"a": [pow(2, 63) - 1]}]
   schema_2 = pa.Schema.from_pandas(pd.DataFrame(data_2)) # crashes
   ```
   
   Here's the backtrace that you get when trying to compute `schema_2`.
   ```
   Traceback (most recent call last):
     File "/work/arrow/foo.py", line 5, in <module>
       schema = pa.Schema.from_pandas(pd.DataFrame(data))
     File "pyarrow/types.pxi", line 3104, in pyarrow.lib.Schema.from_pandas
     File "/work/arrow/pyarrow-dev/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 562, in dataframe_to_types
       type_ = pa.array(c, from_pandas=True).type
     File "pyarrow/array.pxi", line 360, in pyarrow.lib.array
     File "pyarrow/array.pxi", line 87, in pyarrow.lib._ndarray_to_array
     File "pyarrow/error.pxi", line 89, in pyarrow.lib.check_status
   OverflowError: Python int too large to convert to C long
   ```
   Is that something that can be changed or would that likely have too many unintended consequences?
   
   I've tested this with pyarrow version 19.0.0 on ubuntu 24.04.
   
   ### Component(s)
   
   C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Support inference of unsigned integer types [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #45312:
URL: https://github.com/apache/arrow/issues/45312#issuecomment-4440643669

   This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org