You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "attwelveDev (via GitHub)" <gi...@apache.org> on 2024/10/26 13:41:22 UTC
[PR] GH-43112: [Python] Set nullable `Int64` `dtype` for integer columns with `None` values when converting to pandas [arrow]
attwelveDev opened a new pull request, #44538:
URL: https://github.com/apache/arrow/pull/44538
### Rationale for this change
When calling `to_pandas` method on a `Table` object, integer columns with at least one `None` value are converted to `float64` in the resultant pandas `DataFrame`. For example, using the `from_pydict` method to return a `Table` from a Python `Dictionary`, say, `{“col_name”: [1, None]}`, the `to_pandas` method returns a `DataFrame` where the `dtype` of `“col_name”` is `float64`, and whose values are `[1.0, NaN]`. This may cause precision issues when certain integers cannot be precisely converted to a float.
### What changes are included in this PR?
In the `table_to_dataframe` method, columns with the `int64` `dtype` and have at least one `None` value now have the `Int64` `dtype` in `ext_columns_dtypes`.
Various existing tests were modified to reflect this new behaviour.
### Are these changes tested?
Tests have been added as part of `test_pandas_dtype_conversions` in `/python/pyarrow/tests/test_pandas.py`.
### Are there any user-facing changes?
This may affect instances where integer columns with `None` values are expected to be converted to `float64`.
### Additional Notes
There are failing CI tests for other languages, possibly due to unrelated issues from the upstream main branch. All CI tests for Python have been verified to pass.
This is my first contribution to an open source project, and I appreciate any feedback.
- GitHub issue: #43112
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43112: [Python] Set nullable `Int64` `dtype` for integer columns with `None` values when converting to pandas [arrow]
Posted by "avm19 (via GitHub)" <gi...@apache.org>.
avm19 commented on PR #44538:
URL: https://github.com/apache/arrow/pull/44538#issuecomment-3237432695
Please include a clear changelog entry. For me as a user, managing nullable vs non-nullable ints in Pandas has been a huge headache. Some user code may break or become redundant simply because you fixed this bug.
To maintainers: it would be great if Pandas' `arrow_table_to_pandas()` and Arrow's `Table.to_pandas()` were always consistent. I wonder if it is possible to steer this way in the long term? This is desirable because Arrow's `pyarrow.parquet.read_pandas(...).to_pandas()` and Pandas' `pd.read_parquet(..., dtype_backend="pyarrow")` do essentially the same, and from a naive user's perspective, they must be identical. Until this PR is merged, Pandas treats converts to nullable integers correctly, but Pyarrow returns floats as described in the issue. See [`pandas.io._util.arrow_table_to_pandas()`](https://github.com/pandas-dev/pandas/blob/3e1d6d5bc853bd8bc983291b12aec2dbf477dde6/pandas/io/_util.py#L71) for how they do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43112: [Python] Set nullable `Int64` `dtype` for integer columns with `None` values when converting to pandas [arrow]
Posted by "attwelveDev (via GitHub)" <gi...@apache.org>.
attwelveDev commented on PR #44538:
URL: https://github.com/apache/arrow/pull/44538#issuecomment-2439833655
> Everything looks good. Why is the CI failing?
I'm not actually sure. It appears `TestConvertPrimitiveTypes.test_integer_with_nulls` is failing, but I did change this test method after which the CI in my fork passed. I'll look into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43112: [Python] Set nullable `Int64` `dtype` for integer columns with `None` values when converting to pandas [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #44538:
URL: https://github.com/apache/arrow/pull/44538#issuecomment-5468354429
Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43112: [Python] Set nullable `Int64` `dtype` for integer columns with `None` values when converting to pandas [arrow]
Posted by "Isaac7777-cpu-school (via GitHub)" <gi...@apache.org>.
Isaac7777-cpu-school commented on PR #44538:
URL: https://github.com/apache/arrow/pull/44538#issuecomment-2439800095
Everything looks good. Why is the CI failing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43112: [Python] Set nullable `Int64` `dtype` for integer columns with `None` values when converting to pandas [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #44538: GH-43112: [Python] Set nullable `Int64` `dtype` for integer columns with `None` values when converting to pandas
URL: https://github.com/apache/arrow/pull/44538
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org