[jira] [Created] (ARROW-11469) Performance degradation wide dataframes

["Axel G (Jira)" <ji...@apache.org>]

Axel G created ARROW-11469:
------------------------------

             Summary: Performance degradation wide dataframes
                 Key: ARROW-11469
                 URL: https://issues.apache.org/jira/browse/ARROW-11469
             Project: Apache Arrow
          Issue Type: Bug
          Components: Python
    Affects Versions: 3.0.0, 2.0.0, 1.0.1, 1.0.0
            Reporter: Axel G


I noticed a relatively big performance degradation in version 1.0.0+ when trying to load wide dataframes.

For example you should be able to reproduce by doing:
{code:java}
import numpy as np
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

df = pd.DataFrame(np.random.rand(100, 10000))
table = pa.Table.from_pandas(df)
pd.write_table(table, "temp.parquet")

%timeit pd.read_parquet("temp.parquet"){code}
In version 0.17.0, this takes about 300-400 ms and for anything above and including 1.0.0, this suddenly takes around 2 seconds.

 

Thanks for looking into this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)