You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "tadeja (via GitHub)" <gi...@apache.org> on 2026/01/25 12:56:02 UTC
[PR] GH-48978: [Python] test failures on pandas 3.0 (currently CI on 2.3.3) [arrow]
tadeja opened a new pull request, #48979:
URL: https://github.com/apache/arrow/pull/48979
### Rationale for this change
See #48978
### What changes are included in this PR?
Attempt to run tests on CI pandas latest not with pandas 2.3.3 but actually with 3.0 in order to get the locally replicated test failures to show up, and later introduce the fix for those.
### Are these changes tested?
Upgrade of pandas and fix for failures `ValueError: fromutc: dt.tzinfo is not self` already tested locally.
### Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 for fastparquet and for zoneinfo w/o pytz [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #48979:
URL: https://github.com/apache/arrow/pull/48979#discussion_r2804125504
##########
python/pyarrow/tests/parquet/test_basic.py:
##########
@@ -751,17 +752,22 @@ def test_fastparquet_cross_compatibility(tempdir):
fp_file = fp.ParquetFile(file_arrow)
df_fp = fp_file.to_pandas()
- tm.assert_frame_equal(df, df_fp)
+ # TODO: once fastparquet supports pandas 3.0 dtypes revert string/categorical test
+ # check_dtype=False: string/categorical dtype handling differs between libraries
Review Comment:
```suggestion
# TODO: once fastparquet supports pandas 3 dtypes revert string and categorical
# test by removing `check_dtype=False` so type equality is asserted again
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 for fastparquet and for zoneinfo w/o pytz [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #48979:
URL: https://github.com/apache/arrow/pull/48979#issuecomment-3936687932
After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit ba612971b8d421c4daf9629eadb6d1e425efa294.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/64334507099) has more details. It also includes information about 9 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 for fastparquet and for zoneinfo w/o pytz [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #48979:
URL: https://github.com/apache/arrow/pull/48979#discussion_r2828567128
##########
python/pyarrow/tests/parquet/test_basic.py:
##########
@@ -759,17 +760,23 @@ def test_fastparquet_cross_compatibility(tempdir):
fp_file = fp.ParquetFile(file_arrow)
df_fp = fp_file.to_pandas()
- tm.assert_frame_equal(df, df_fp)
+ # TODO: once fastparquet supports pandas 3 dtypes revert string and categorical
+ # tests by removing `check_dtype=False` and `check_categorical=False` so type
+ # equality is asserted again
+ tm.assert_frame_equal(df, df_fp, check_dtype=False, check_categorical=False)
# Fastparquet -> arrow
file_fastparquet = str(tempdir / "cross_compat_fastparquet.parquet")
- fp.write(file_fastparquet, df)
+ # fastparquet can't write pandas 3.0 StringDtype
+ df_for_fp = df.copy()
+ df_for_fp['a'] = df_for_fp['a'].astype(object)
+ fp.write(file_fastparquet, df_for_fp)
table_fp = pq.read_pandas(file_fastparquet)
# for fastparquet written file, categoricals comes back as strings
# (no arrow schema in parquet metadata)
- df['f'] = df['f'].astype(object)
- tm.assert_frame_equal(table_fp.to_pandas(), df)
+ tm.assert_frame_equal(table_fp.to_pandas(), df_for_fp, check_dtype=False,
+ check_categorical=False)
Review Comment:
Maybe we can reuse `expected_types`?
```suggestion
tm.assert_frame_equal(table_fp.to_pandas(), df.astype(expected_types))
```
##########
python/pyarrow/tests/parquet/test_basic.py:
##########
@@ -759,17 +760,23 @@ def test_fastparquet_cross_compatibility(tempdir):
fp_file = fp.ParquetFile(file_arrow)
df_fp = fp_file.to_pandas()
- tm.assert_frame_equal(df, df_fp)
+ # TODO: once fastparquet supports pandas 3 dtypes revert string and categorical
+ # tests by removing `check_dtype=False` and `check_categorical=False` so type
+ # equality is asserted again
+ tm.assert_frame_equal(df, df_fp, check_dtype=False, check_categorical=False)
Review Comment:
Would this work instead? Since underlying type of categorical is not `int` it feels like it should?
```suggestion
# TODO: once fastparquet supports pandas 3's pd.StringDtype() remove casting
expected_types = {"a": pd.StringDtype()}
tm.assert_frame_equal(df, df_fp.astype(expected_types))
```
##########
python/pyarrow/tests/parquet/test_basic.py:
##########
@@ -759,17 +760,23 @@ def test_fastparquet_cross_compatibility(tempdir):
fp_file = fp.ParquetFile(file_arrow)
df_fp = fp_file.to_pandas()
- tm.assert_frame_equal(df, df_fp)
+ # TODO: once fastparquet supports pandas 3 dtypes revert string and categorical
+ # tests by removing `check_dtype=False` and `check_categorical=False` so type
+ # equality is asserted again
+ tm.assert_frame_equal(df, df_fp, check_dtype=False, check_categorical=False)
# Fastparquet -> arrow
file_fastparquet = str(tempdir / "cross_compat_fastparquet.parquet")
- fp.write(file_fastparquet, df)
+ # fastparquet can't write pandas 3.0 StringDtype
+ df_for_fp = df.copy()
+ df_for_fp['a'] = df_for_fp['a'].astype(object)
+ fp.write(file_fastparquet, df_for_fp)
Review Comment:
Perhaps we can avoid creating a new dataframe and only non-destructively cast at write-time?
```suggestion
# fastparquet can't write pandas 3.0 StringDtype
fp_compatible_types = {"a": object}
fp.write(file_fastparquet, df.astype(fp_compatible_types))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 for fastparquet and for zoneinfo w/o pytz [arrow]
Posted by "tadeja (via GitHub)" <gi...@apache.org>.
tadeja commented on PR #48979:
URL: https://github.com/apache/arrow/pull/48979#issuecomment-3897081995
@AlenkaF, @rok this now includes comments and [# TODO: once fastparquet supports pandas 3.0 dtypes revert string/categorical test](https://github.com/apache/arrow/pull/48979/changes#diff-daf07f9d0d68988737c70f2318af3751c2be271bb5d3ad0bcebbb1aa2f9349eaR755) - also according to https://github.com/apache/arrow/issues/48978#issuecomment-3824206089
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 (currently CI on 2.3.3) [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #48979:
URL: https://github.com/apache/arrow/pull/48979#issuecomment-3796623745
:warning: GitHub issue #48978 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 (currently CI on 2.3.3) [arrow]
Posted by "tadeja (via GitHub)" <gi...@apache.org>.
tadeja commented on PR #48979:
URL: https://github.com/apache/arrow/pull/48979#issuecomment-3797120305
[Proposed fixes](https://github.com/apache/arrow/pull/48979/changes/6f778a4aea15df7ea038ea7ab158a014d200f06c) complete previously failing tests with success, you can see it is with pandas 3.0, with `fastparquet` and without `pytz` here:
[AMD64 Conda Python 3.13 Pandas latest](https://github.com/apache/arrow/actions/runs/21337196433/job/61410913145?pr=48979#logs) **succeeded**
- [fastparquet 2025.12.0 pypi_0 pypi](https://github.com/apache/arrow/actions/runs/21337196433/job/61410913145?pr=48979#step:6:2652)
- [pandas 3.0.0 pypi_0 pypi](https://github.com/apache/arrow/actions/runs/21337196433/job/61410913145?pr=48979#step:6:2652)
- ...
- [= 7784 passed, 283 skipped, 22 xfailed, 2 xpassed, 54 warnings in 241.86s (0:04:01) =](https://github.com/apache/arrow/actions/runs/21337196433/job/61410913145?pr=48979#step:6:6593)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 for fastparquet and for zoneinfo w/o pytz [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on PR #48979:
URL: https://github.com/apache/arrow/pull/48979#issuecomment-3937316347
Thanks for the heads-up @tadeja; I was following along here, and will pick up the changes on the feedstock when I get the chance (but since it was really just about making tests pass with pandas v3, the details don't matter all that much).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 for fastparquet and for zoneinfo w/o pytz [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #48979:
URL: https://github.com/apache/arrow/pull/48979#discussion_r2804125504
##########
python/pyarrow/tests/parquet/test_basic.py:
##########
@@ -751,17 +752,22 @@ def test_fastparquet_cross_compatibility(tempdir):
fp_file = fp.ParquetFile(file_arrow)
df_fp = fp_file.to_pandas()
- tm.assert_frame_equal(df, df_fp)
+ # TODO: once fastparquet supports pandas 3.0 dtypes revert string/categorical test
+ # check_dtype=False: string/categorical dtype handling differs between libraries
Review Comment:
```suggestion
# TODO: once fastparquet supports pandas 3 dtypes revert string and categorical
# tests by removing `check_dtype=False` and `check_categorical=False` so type
# equality is asserted again
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 (currently CI on 2.3.3) [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on PR #48979:
URL: https://github.com/apache/arrow/pull/48979#issuecomment-3863526901
I backported this to the conda-forge feedstock in https://github.com/conda-forge/pyarrow-feedstock/pull/169, and can confirm that it works! Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 (currently CI on 2.3.3) [arrow]
Posted by "tadeja (via GitHub)" <gi...@apache.org>.
tadeja commented on code in PR #48979:
URL: https://github.com/apache/arrow/pull/48979#discussion_r2794047286
##########
python/pyarrow/tests/parquet/test_basic.py:
##########
Review Comment:
Ok, @AlenkaF here's the new approach with
[ "f": pd.Categorical([5, 6, 7]),](https://github.com/apache/arrow/pull/48979/changes/8e90276f94148e599a904465535fb8bff3149516..7bd8eb39a09e5f48b753f4085bda761b8432b916#diff-daf07f9d0d68988737c70f2318af3751c2be271bb5d3ad0bcebbb1aa2f9349eaL742-R742)
This fails the test at
```
tm.assert_frame_equal(table_fp.to_pandas(), df_for_fp, check_dtype=False)
pyarrow/tests/parquet/test_basic.py:769:
...
...
E AssertionError: Categorical Expected type <class 'pandas.Categorical'>, found <class 'numpy.ndarray'> instead
```
so we have to add another [check_categorical=False on line 769](https://github.com/apache/arrow/pull/48979/changes/8e90276f94148e599a904465535fb8bff3149516..7bd8eb39a09e5f48b753f4085bda761b8432b916#diff-daf07f9d0d68988737c70f2318af3751c2be271bb5d3ad0bcebbb1aa2f9349eaR770)
as already done before at [line 756 for the same reason.](https://github.com/apache/arrow/pull/48979/changes/8e90276f94148e599a904465535fb8bff3149516..7bd8eb39a09e5f48b753f4085bda761b8432b916#diff-daf07f9d0d68988737c70f2318af3751c2be271bb5d3ad0bcebbb1aa2f9349eaL756)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 for fastparquet and for zoneinfo w/o pytz [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #48979:
URL: https://github.com/apache/arrow/pull/48979#discussion_r2829923701
##########
python/pyarrow/tests/parquet/test_basic.py:
##########
@@ -746,7 +747,7 @@ def test_fastparquet_cross_compatibility(tempdir):
"c": np.arange(4.0, 7.0, dtype="float64"),
"d": [True, False, True],
"e": pd.date_range("20130101", periods=3),
- "f": pd.Categorical(["a", "b", "a"]),
+ "f": pd.Categorical(["a", "b", "c"]),
Review Comment:
```suggestion
"f": pd.Categorical(["a", "b", "a"]),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 for fastparquet and for zoneinfo w/o pytz [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #48979:
URL: https://github.com/apache/arrow/pull/48979#discussion_r2828567128
##########
python/pyarrow/tests/parquet/test_basic.py:
##########
@@ -759,17 +760,23 @@ def test_fastparquet_cross_compatibility(tempdir):
fp_file = fp.ParquetFile(file_arrow)
df_fp = fp_file.to_pandas()
- tm.assert_frame_equal(df, df_fp)
+ # TODO: once fastparquet supports pandas 3 dtypes revert string and categorical
+ # tests by removing `check_dtype=False` and `check_categorical=False` so type
+ # equality is asserted again
+ tm.assert_frame_equal(df, df_fp, check_dtype=False, check_categorical=False)
# Fastparquet -> arrow
file_fastparquet = str(tempdir / "cross_compat_fastparquet.parquet")
- fp.write(file_fastparquet, df)
+ # fastparquet can't write pandas 3.0 StringDtype
+ df_for_fp = df.copy()
+ df_for_fp['a'] = df_for_fp['a'].astype(object)
+ fp.write(file_fastparquet, df_for_fp)
table_fp = pq.read_pandas(file_fastparquet)
# for fastparquet written file, categoricals comes back as strings
# (no arrow schema in parquet metadata)
- df['f'] = df['f'].astype(object)
- tm.assert_frame_equal(table_fp.to_pandas(), df)
+ tm.assert_frame_equal(table_fp.to_pandas(), df_for_fp, check_dtype=False,
+ check_categorical=False)
Review Comment:
Maybe:
```suggestion
expected_types = {"a": pd.StringDtype(), "f": object}
tm.assert_frame_equal(table_fp.to_pandas(), df.astype(expected_types))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 for fastparquet and for zoneinfo w/o pytz [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok merged PR #48979:
URL: https://github.com/apache/arrow/pull/48979
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 for fastparquet and for zoneinfo w/o pytz [arrow]
Posted by "tadeja (via GitHub)" <gi...@apache.org>.
tadeja commented on PR #48979:
URL: https://github.com/apache/arrow/pull/48979#issuecomment-3935200475
> I backported this to the conda-forge feedstock in [conda-forge/pyarrow-feedstock#169](https://github.com/conda-forge/pyarrow-feedstock/pull/169), and can confirm that it works! Thanks!
@h-vetinari, just a ping to let you know that slightly different fixes for [tests/test_pandas.py‎](https://github.com/apache/arrow/pull/48979/changes#diff-a90dace803514318979968ba54f0c16c0439233e2261c13beb89d6e80610bb2b) and [tests/parquet/test_basic.py](https://github.com/apache/arrow/pull/48979/changes#diff-daf07f9d0d68988737c70f2318af3751c2be271bb5d3ad0bcebbb1aa2f9349ea) have just been merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 (currently CI on 2.3.3) [arrow]
Posted by "tadeja (via GitHub)" <gi...@apache.org>.
tadeja commented on PR #48979:
URL: https://github.com/apache/arrow/pull/48979#issuecomment-3796965011
Failures replicated [here on CI job AMD64 Conda Python 3.13 Pandas latest](https://github.com/apache/arrow/actions/runs/21334308997/job/61403809605?pr=48979#step:6:6518)
with temporary CI modification as follows:
pip install [--upgrade pandas fastparquet
pip uninstall -q -y pytz](https://github.com/apache/arrow/pull/48979/changes#diff-1efc86e928fc957fcfbae907ffc712837e50e173a3d3e5a2c0f2164b551949a3R43-R44)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-48978: [Python] test failures on pandas 3.0 (currently CI on 2.3.3) [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #48979:
URL: https://github.com/apache/arrow/pull/48979#discussion_r2791866577
##########
python/pyarrow/tests/parquet/test_basic.py:
##########
Review Comment:
Would it make sense to use integers for categories until fastparquet fully supports pandas `3.0`? That way we can also check the dtype roundtrip here.
##########
python/pyarrow/tests/test_pandas.py:
##########
@@ -4956,7 +4956,7 @@ def test_timestamp_as_object_non_nanosecond(resolution, tz, dt):
assert isinstance(result[0], datetime)
if tz:
assert result[0].tzinfo is not None
- expected = result[0].tzinfo.fromutc(dt)
+ expected = dt.replace(tzinfo=timezone.utc).astimezone(result[0].tzinfo)
Review Comment:
This is totally legit, but we might want to use `tz` just to make it cleaner:
```suggestion
expected = dt.replace(tzinfo=timezone.utc).astimezone(ZoneInfo(tz))
```
and we would need `from zoneinfo import ZoneInfo` on top of the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org