You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "stephen-a-stc (via GitHub)" <gi...@apache.org> on 2025/01/17 15:30:03 UTC

[I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]

stephen-a-stc opened a new issue, #45296:
URL: https://github.com/apache/arrow/issues/45296

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   ## Summary
   
   Using the pyarrow table method `to_pandas()` results in an exception if `pandas.set_option('future.infer_string', True)` has been set.  This seems related to handling of string data.  
   
   ## Environment
   
   Environments: Windows 11, and Linux (docker image "python")
   Python versions tested: 3.10, 3.11, 3.12
   
   Python packages: `pyarrow==19.0.0`, `pandas==2.2.3`
   
   
   ## Example
   ```python
   import pandas
   import pyarrow
   
   print("Create pyarrow table")
   pat=pyarrow.Table.from_pydict({"foo":["bar","baz"]})
   print(pat)
   
   print("convert to pandas")
   df1 = pat.to_pandas()
   print(df1)
   
   print("Set 'future.infer_string' to True")
   pandas.set_option('future.infer_string', True)
   
   print("exception during convert to pandas")
   df2 = pat.to_pandas()
   print(df2)
   ```
   
   ## Example's output
   ```plaintext
   Create pyarrow table
   pyarrow.Table
   foo: string
   ----
   foo: [["bar","baz"]]
   convert to pandas
      foo
   0  bar
   1  baz
   Set 'future.infer_string' to True
   convert to pandas again
   Traceback (most recent call last):
     File "C:\temp\foo.py", line 16, in <module>
       df2 = pat.to_pandas()
     File "pyarrow\\array.pxi", line 889, in pyarrow.lib._PandasConvertible.to_pandas
     File "pyarrow\\table.pxi", line 5132, in pyarrow.lib.Table._to_pandas
     File "C:\penv\brd_test\lib\site-packages\pyarrow\pandas_compat.py", line 800, in table_to_dataframe
       ext_columns_dtypes = _get_extension_dtypes(
     File "C:\penv\brd_test\lib\site-packages\pyarrow\pandas_compat.py", line 925, in _get_extension_dtypes
       ext_columns[field.name] = _pandas_api.pd.StringDtype(na_value=np.nan)
   TypeError: StringDtype.__init__() got an unexpected keyword argument 'na_value'
   ```
   
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2598642144

   This is related to:
   - https://github.com/apache/arrow/issues/43683
   
   It seems the `na_value` was [merged on pandas main](https://github.com/pandas-dev/pandas/commit/f25a09eacee843f643604a7406a5bb6bcc4361e5) but is not available on pandas 2.2 (where `future.infer_string` was added)
   
   @jorisvandenbossche @WillAyd  do we have to check this is specifically for Pandas 3 or should we specialize the call based on pandas 2 vs 3?
   
   https://github.com/apache/arrow/blob/5181c24f6d780591426e0af0d1b72c68f190484c/python/pyarrow/pandas_compat.py#L917-L925


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2622275706

   Opened a PR with my suggestion above so it can be included in the 19.0.1 release: https://github.com/apache/arrow/pull/45383


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd closed issue #45296: [Python] to_pandas() fails when pandas option 'future.infer_string' is True
URL: https://github.com/apache/arrow/issues/45296


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]

Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2598670892

   In the meantime @stephen-a-stc you can either downgrade Arrow to anything less than version 19, or use a development version of pandas with Arrow 19+, and I think you will get the result you are after


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]

Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2622430866

   Thanks @jorisvandenbossche!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2598677632

   Ah, yes, the current code is assuming that the user only uses the `'future.infer_string'` option when having pandas 2.3+ .. I should have included that in the check at:
   
   https://github.com/apache/arrow/blob/c9f417f1a58663351692433559e85ff2a7ec2e19/python/pyarrow/pandas-shim.pxi#L182-L187
   
   (checking for that option `and pd.__version__ >= 2.3.0)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2624348506

   Issue resolved by pull request 45383
   https://github.com/apache/arrow/pull/45383


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]

Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2598664937

   Yea this is not expected to work with that option in pandas 2.2. The 2.3 release should resolve it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2598688797

   > Yea this is not expected to work with that option in pandas 2.2. The 2.3 release should resolve it
   
   Ah, ok, I wasn't sure this was going to be added to pandas 2.3. @jorisvandenbossche solution makes sense to me for future cases of people using pandas 2.2 and setting `infer_string` but I don't think we are currently planning a patch release for Arrow at the moment so as @WillAyd  suggests using pyarrow < 19 until pandas 2.3 is released or dev pandas is the only solution.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]

Posted by "stephen-a-stc (via GitHub)" <gi...@apache.org>.
stephen-a-stc commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2598683765

   Okay, thanks for the information.  
   I can confirm that this wasn't an issue with earlier versions of pyarrow, for instance it works fine with `pyarrow==18.1.0`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org