You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "ghaarsma (via GitHub)" <gi...@apache.org> on 2026/07/27 20:58:02 UTC
[I] Significant performance degradation after pyarrow 24.0.0 to 25.0.0 upgrade [arrow]
ghaarsma opened a new issue, #50667:
URL: https://github.com/apache/arrow/issues/50667
### Describe the bug, including details regarding any error messages, version, and platform.
In our application we implement a local TimeSeries caching implementation that stores TimeSeries data into Parquet files. We heavily read/write parquet files via the pandas to_parquet(engine="pyarrow") and read_parquet. We also use the pyarrow.read_metadata directly from the pyarrows package. We use default (snappy) pandas compression when we write out our files, which consist of a single column DataFrame with the index being the timestamps (DateTimeIndex) and the single column representing the TimeSeries values (mostly Float32/64).
After installing pyarrow 25.0.0 we saw a significant reduction in performance (see trend below). When we downgraded back to 24.0.0 the problem went away. No other changes were made and we kept the rest of our dependencies the same.
<img width="1923" height="889" alt="Image" src="https://github.com/user-attachments/assets/34101978-32b4-44b1-8dc0-924643466158" />
The Dev/Prod stack Azure App Service on `Linux 779de6246efc 6.6.139.1-1.azl3`:
```
root@779de6246efc:/home/site/wwwroot# python
Python 3.13.13 (main, May 4 2026, 08:38:58) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow as pa
>>> pa.show_info()
pyarrow version info
--------------------
Package kind : python-wheel-manylinux228
Arrow C++ library version : 24.0.0
Arrow C++ compiler : GNU 14.2.1
Arrow C++ compiler flags : -Wno-noexcept-type -Wno-self-move -Wno-subobject-linkage -fdiagnostics-color=always -Wall -fno-semantic-interposition -msse4.2
Arrow C++ git revision :
Arrow C++ git description :
Arrow C++ build type : release
PyArrow build type : release
Platform:
OS / Arch : Linux x86_64
SIMD Level : avx2
Detected SIMD Level : avx2
Memory:
Default backend : mimalloc
Bytes allocated : 0 bytes
Max memory : 0 bytes
Supported Backends : mimalloc, jemalloc, system
Optional modules:
csv : Enabled
cuda : -
dataset : Enabled
feather : Enabled
flight : Enabled
fs : Enabled
gandiva : -
json : Enabled
orc : Enabled
parquet : Enabled
opentelemetry : Enabled
Filesystems:
AzureFileSystem : Enabled
GcsFileSystem : Enabled
HadoopFileSystem : Enabled
S3FileSystem : Enabled
Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4_frame : Enabled
lz4 : Enabled
snappy : Enabled
zstd : Enabled
```
### Component(s)
C++, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Significant performance degradation after pyarrow 24.0.0 to 25.0.0 upgrade [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #50667:
URL: https://github.com/apache/arrow/issues/50667#issuecomment-5207567806
I tried to reproduce based on your description of a DataFrame with a single float column and a timeseries index:
```python
import time
import numpy as np
import pandas as pd
N = 10_000_000
dates = pd.date_range("2020-01-01", periods=N, freq="h", tz="UTC")
df = pd.DataFrame({"col": np.random.randn(N)}, index=dates)
write_times = []
read_times = []
for _ in range(10):
t0 = time.time()
df.to_parquet("test.parquet")
t1 = time.time()
df2 = pd.read_parquet("test.parquet")
t2 = time.time()
write_times.append(t1 - t0)
read_times.append(t2 - t1)
print("Writing took :", sum(write_times))
print("Reading took :", sum(read_times))
```
and compared pyarrow 24 to 25 (but using conda packages, not pypi packages), and don't see any significant difference:
```
$ pixi exec --with pyarrow=24 --with pandas=3.0.5 python test_pyarrow_parquet_perf.py
Writing took : 5.342243432998657
Reading took : 1.740952730178833
$ pixi exec --with pyarrow=25 --with pandas=3.0.5 python test_pyarrow_parquet_perf.py
Writing took : 5.0998406410217285
Reading took : 1.74660325050354
```
@ghaarsma could you run this code on your end to see if that reproduces the slowdown?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Significant performance degradation after pyarrow 24.0.0 to 25.0.0 upgrade [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #50667:
URL: https://github.com/apache/arrow/issues/50667#issuecomment-5102500436
This is a large regression. Could you help narrow it down by:
- Trying pd.read_parquet(path, engine="pyarrow", pre_buffer=False) and pre_buffer=True, and reporting the timings? The ParquetFile default changed between 24 and 25, although pd.read_parquet ultimately uses pq.read_table, whose default was already True in Arrow 24.
- Reporting the pandas version and whether the DateTimeIndex is timezone-aware? Arrow 25 changed reconstructed named timezones from pytz to zoneinfo.ZoneInfo with pandas 3. If the cache compares timezone objects directly, cached ranges could be rejected even though the timestamps are equivalent. It would be useful to print type(df.index.tz), repr(df.index.tz), and the corresponding values for the requested timezone.
- Clarifying whether “Query Length (s)” is operation duration or the time range requested from the upstream API? If it is the latter, this may indicate changed cache-hit behavior rather than slower Parquet I/O.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Significant performance degradation after pyarrow 24.0.0 to 25.0.0 upgrade [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #50667:
URL: https://github.com/apache/arrow/issues/50667#issuecomment-5128540941
Thanks for running these!
Perhaps we can narrow it down - could you try normalizing the index to one timezone immediately after every Parquet read and rerun your benchmark? E.g.:
```python
from datetime import timezone
df = pd.read_parquet(path, engine="pyarrow", pre_buffer=False)
df.index = df.index.tz_convert(timezone.utc)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Significant performance degradation after pyarrow 24.0.0 to 25.0.0 upgrade [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #50667:
URL: https://github.com/apache/arrow/issues/50667#issuecomment-5219654648
> But I don't fully understand when you would end up with a `datetime.UTC` timezone.
I figured out this part: in my test I was manually converting some pyarrow data to pandas, but actually when using `pd.read_parquet`, it does some post-processing after converting the pyarrow table to a pandas dataframe. And one of those things is to convert pytz timezones, meaning if you did end up with a pytz UTC timezone, it gets converted to datetime.UTC, but if you already have zoneinfo UTC timezone, we leave it alone (that's potentially something to fix in pandas to make more consistent, to always normalize the UTC timezones).
```python
# creating a small parquet file
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
table = pa.table({"col": pd.date_range("2020-01-01", periods=3, tz="UTC")})
pq.write_table(table, "test_utc.parquet")
# reading it with pandas 3.0.5 and pyarrow 25.0
>>> pd.read_parquet("test_utc.parquet")["col"].dtype.tz
zoneinfo.ZoneInfo(key='UTC')
# reading it with pandas 3.0.5 and pyarrow 24.0
>>> pd.read_parquet("test_utc.parquet")["col"].dtype.tz
datetime.timezone.utc
# unless pytz is not installed, then pyarrow already gives a zoneinfo zone and pandas does not convert it to datetime.UTC
>>> import sys
>>> sys.modules["pytz"] = None
>>> pd.read_parquet("test_utc.parquet")["col"].dtype.tz
zoneinfo.ZoneInfo(key='UTC')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Significant performance degradation after pyarrow 24.0.0 to 25.0.0 upgrade [arrow]
Posted by "ghaarsma (via GitHub)" <gi...@apache.org>.
ghaarsma commented on issue #50667:
URL: https://github.com/apache/arrow/issues/50667#issuecomment-5218934890
I have not yet been able to generate a fully syntactic example (that I can share) to demonstrate the slowdown. But I have made progress in determining where the slowdown is. I have also reproduced the problem on Windows (our local Dev env).
The actual slowdown is not in the pandas read_parquet or to_parquet (via engine="pyarrow") as can be seen here:
<img width="2417" height="1328" alt="Image" src="https://github.com/user-attachments/assets/471f2e64-60f9-4e55-a071-2548e7d6b424" />
All the slowdown is caused in the pandas concat which concatenates the old and new dataframe together before writing it back out
<img width="2420" height="1242" alt="Image" src="https://github.com/user-attachments/assets/ef59340d-b120-43ce-9e01-d07c35d50b96" />
I can confirm that the slowdown goes away if both dataframes have identical DatetimeIndex timezones. Either both datetime.UTC or both ZoneInfo("UTC"). So I think it would be fair to say that the bug is more in Pandas, but that it has been induced by the Pyarrow 25.0.0 change to use ZoneInfo("UTC") instead of datetime.UTC.
The slowdown is quite significant, about ~ 365 times slower, when comparing the slopes of the scatter plots in the 2nd figure.
- Would it be possible to go back to datetime.UTC as was in pyarrow 24.0.0? BTW I could not find the change in the [release notes](https://arrow.apache.org/release/25.0.0.html)
- What I still don't understand is that out of my 831 cached TimeSeries / parquet files, in pyarrow 25.0.0, when concatenating with a second datetime.UTC indexed dataframe, 56 will result in a ZoneInfo("UTC"), without any slowdown. The other 775 will result in a datetime.UTC index and this is where all the slowdown happens.
I'll continue to investigate the 2nd point, because I think this is the key to a reproducible example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Significant performance degradation after pyarrow 24.0.0 to 25.0.0 upgrade [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #50667:
URL: https://github.com/apache/arrow/issues/50667#issuecomment-5101894401
Just to confirm you are using the provided wheels from PyPI, not building PyArrow yourselves, right?
From what I can see Py3.13 on Linux amd64 (wheel-manylinux-2-28-cp313).
Is there any possibility you could share a synthetic Parquet file where we could reproduce or a minimal reproducer? I don't think we've seen any report like this so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Significant performance degradation after pyarrow 24.0.0 to 25.0.0 upgrade [arrow]
Posted by "ghaarsma (via GitHub)" <gi...@apache.org>.
ghaarsma commented on issue #50667:
URL: https://github.com/apache/arrow/issues/50667#issuecomment-5239013029
@jorisvandenbossche and others,
I have finally been able to produce a **MRE** (see attached [timeseries_test.py](https://github.com/user-attachments/files/30893842/timeseries_test.py)). Whether the bug/performance degradation shows up depends on 3 factors:
1. Do the existing parquet files have nanosecond timestamps (holdover from Pandas < 3.0 default)
2. Is python pytz dependency installed (this was always the case for Pandas < 3.0)
3. Do you run pyarrow 24.0.0 or pyarrow 25.0.0
- With pyarrow 24.0.0 all is good, provided that you have pytz installed (no need to import). Both `us` and `ns` files work
- With pyarrow 24.0.0, but no pytz, concatenating a 'ns' with a new 'us' dataframe becomes very slow (300~400 times slower for large timeseries)
- With pyarrow 25.0.0, (irrespective of pytz installed), concatenating a 'ns' dataframe with a new 'us' dataframe becomes very slow
```
pyarrow 24.0.0 stats: -------------------------------------------------
read append write unit_in unit_out tz_in tz_out
file
syn_data_us.parquet 0.501423 0.081193 1.506878 us us datetime.timezone.utc datetime.timezone.utc
syn_data_ns.parquet 0.504985 0.079682 1.768292 ns ns datetime.timezone.utc datetime.timezone.utc
pyarrow 25.0.0 stats: -------------------------------------------------
read append write unit_in unit_out tz_in tz_out
file
syn_data_us.parquet 0.480525
0.078757 1.589404 us us zoneinfo.ZoneInfo(key='UTC') zoneinfo.ZoneInfo(key='UTC')
syn_data_ns.parquet 0.492546 28.473683 1.583237 ns ns zoneinfo.ZoneInfo(key='UTC') datetime.timezone.utc
```
This was all run with the latest packages (where only pyarrow was changed from 24.0.0 to 25.0.0):
```
(venv) PS C:\Users\gabe\Python\pyarrow> pip list
Package Version
--------------- ------------
numpy 2.5.2
pandas 3.0.5
pip 26.1.2
pyarrow 24.0.0 <-> 25.0.0
python-dateutil 2.9.0.post0
pytz 2026.3.post1
six 1.17.0
tzdata 2026.3
```
To be clear, the majority of the Parquet files were originally created under Pandas 2.x so it had `ns` timestamps. Concatenating these with a newer pandas 3.0 dataframe with `us` timestamps and writing back to parquet, retained the ns timestamps. Because I occasionally get some cache poisoning, I manually over time deleted 56 (see prior post) parquet files. This will now generate a brand new parquet file from scratch with only us timestamps.
I'm still trying to wrap my mind around all of this, but at this time, I'm not convinced that the newer pyarrow 25.0.0 behavior of UTC timezone handling is an improvement over 24.0.0
I'm still trying to wrap my head around all of this. I think the majority of the fixes/cleanup will/should be on the pandas side of things. However
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Significant performance degradation after pyarrow 24.0.0 to 25.0.0 upgrade [arrow]
Posted by "ghaarsma (via GitHub)" <gi...@apache.org>.
ghaarsma commented on issue #50667:
URL: https://github.com/apache/arrow/issues/50667#issuecomment-5124337548
Thank you @raulcd and @rok for helping to investigate.
- I can confirm that we are using provided wheels from PyPi. We do not build PyArrow ourself.
- I will try to work on a MRE. This will likely have to be pretty synthetic and it might take me a while to extract the core behavior and try to reproduce.
- I have run pd.read_parquet timings with both pre_buffer settings on both pyarrow 24 & 25. A total of 834 parquet files (~30 GB). Times are total seconds to read all 834 files. Each timing is repeated 3 times.
```
pyarrow 24.0.0 --------------------------------------
1st 2nd 3rd
pd.read_parquet(prebuffer=False) 592.1 606.4 607.1
pd.read_parquet(prebuffer=True) 609.4 635.3 623.3
pyarrow 25.0.0 --------------------------------------
1st 2nd 3rd
pd.read_parquet(prebuffer=False) 634.2 623.4 628.6
pd.read_parquet(prebuffer=True) 663.5 641.8 665.2
```
- Pandas version is 3.0.5 and has not changed. We use reproducible builds with a lock file. All DateTimeIndex are timezone-aware and all should be UTC. I have checked all files and here is where it gets perhaps a bit interesting:
```
pyarrow 24.0.0 reports: <class 'datetime.timezone'>, 'datetime.timezone.utc'
pyarrow 25.0.0 reports: <class 'zoneinfo.ZoneInfo'>, "zoneinfo.ZoneInfo(key='UTC')"
```
- The query length (s) on the figure is the time in seconds to run a `warming the cache` loop. Cache hit/miss behavior should not impact the results more then a few %.
Hoping this helps for know. Could it be that the timezone awareness difference between pyarrow 24/25 as datetime.utc and ZoneInfo("UTC") be the cause? What if we have to stich `pd.concat` timeseries data together all based on UTC but differ between datetime and ZoneInfo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Significant performance degradation after pyarrow 24.0.0 to 25.0.0 upgrade [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #50667:
URL: https://github.com/apache/arrow/issues/50667#issuecomment-5219558937
Thanks for the further research, that is interesting!
> BTW I could not find the change in the [release notes](https://arrow.apache.org/release/25.0.0.html)
The actual change was in https://github.com/apache/arrow/pull/49694 (I think it didn't how up properly in the release notes because I did not mark / close the issue for the 25.0 milestone ..)
> What I still don't understand is that out of my 831 cached TimeSeries / parquet files, in pyarrow 25.0.0, when concatenating with a second datetime.UTC indexed dataframe, 56 will result in a ZoneInfo("UTC"), without any slowdown. The other 775 will result in a datetime.UTC index and this is where all the slowdown happens.
That is indeed strange ..
The logic should be (in pyarrow 25) to first try to convert the tz string to `zoneinfo.ZoneInfo`, and if that does not succeed, fallback with trying to convert it to `pytz`. In pyarrow 24, it just always first converted to `pytz`, and only if that was not available, try `zoneinfo`.
But I don't fully understand when you would end up with a `datetime.UTC` timezone.
Something that might be useful to check: could you verify in the Parquet files which exact timezone is being used? If you read the data with pyarrow and check for the schema of the resulting table, i.e. `pyarrow.parquet.read_table(..).schema`, which timezone does it show for the timeseries column?
(e.g. I am wondering if capitalization could trigger some different behaviour)
---
Regardless of figuring out how you get the different timezones, I am also not actually seeing a slowdown in `pd.concat` in this case:
```python
>>> N = 10_000_000
>>> import zoneinfo
>>> dates = pd.date_range("2020-01-01", periods=N, freq="min", tz="UTC")
... df = pd.DataFrame({"col": np.random.randn(N)}, index=dates)
>>> df.index.dtype.tz
datetime.timezone.utc
>>> dates = pd.date_range("2020-01-01", periods=N, freq="min", tz=zoneinfo.ZoneInfo("UTC"))
... df2 = pd.DataFrame({"col": np.random.randn(N)}, index=dates)
>>> df2.index.dtype.tz
zoneinfo.ZoneInfo(key='UTC')
>>> df_copy = df.copy()
# concatting two dataframes with `datetime.timezone.utc`
>>> %timeit pd.concat([df, df_copy])
77.8 ms ± 3.64 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
# concatting dataframe with `datetime.timezone.utc` and `zoneinfo.ZoneInfo(key='UTC')`
>>> %timeit pd.concat([df, df2])
79.5 ms ± 4.54 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org