You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Elena Henderson (Jira)" <ji...@apache.org> on 2021/05/03 21:41:00 UTC
[jira] [Comment Edited] (ARROW-11469) [Python] Performance
degradation parquet reading of wide dataframes
[ https://issues.apache.org/jira/browse/ARROW-11469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17338622#comment-17338622 ]
Elena Henderson edited comment on ARROW-11469 at 5/3/21, 9:40 PM:
------------------------------------------------------------------
Performance of reading wide dataframes degraded farther in pyarrow 4.0.0:
[https://conbench.ursa.dev/compare/runs/afbbfc387b314fb6886168225c29d3af...c30d6d188bc24aa2b319808446b8d0aa/]
[https://conbench.ursa.dev/compare/runs/87781efdd9f940098f91cc8b50a32ca1...a54fde45cfa2406bab2ff0f5080f274d/]
Baseline = pyarrow 3.0.0
Contender = pyarrow 4.0.0
!image-2021-05-03-14-39-59-485.png!
We track down regression to this commit: [https://github.com/apache/arrow/commit/c5c583b5d4290563332c653a0084c666ef232f0c]
[https://conbench.ursa.dev/compare/runs/d731c9100c6190566354a83dee2239ddb7be21d6...c5c583b5d4290563332c653a0084c666ef232f0c/]
Baseline = [https://github.com/apache/arrow/commit/d731c9100c6190566354a83dee2239ddb7be21d6]
Contender = [https://github.com/apache/arrow/commit/c5c583b5d4290563332c653a0084c666ef232f0c]
!image-2021-05-03-14-40-09-520.png!
was (Author: elena@ursacomputing.com):
Performance of reading wide dataframes degraded farther in pyarrow 4.0.0:
[https://conbench.ursa.dev/compare/runs/afbbfc387b314fb6886168225c29d3af...c30d6d188bc24aa2b319808446b8d0aa/]
baseline = pyarrow 3.0.0 and contender = pyarrow 4.0.0
!image-2021-05-03-14-31-41-260.png|width=942,height=501!
h1.
> [Python] Performance degradation parquet reading of wide dataframes
> -------------------------------------------------------------------
>
> Key: ARROW-11469
> URL: https://issues.apache.org/jira/browse/ARROW-11469
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 1.0.0, 1.0.1, 2.0.0, 3.0.0
> Reporter: Axel G
> Priority: Minor
> Attachments: image-2021-05-03-14-31-41-260.png, image-2021-05-03-14-39-59-485.png, image-2021-05-03-14-40-09-520.png, profile_wide300.svg
>
>
> I noticed a relatively big performance degradation in version 1.0.0+ when trying to load wide dataframes.
> For example you should be able to reproduce by doing:
> {code:java}
> import numpy as np
> import pandas as pd
> import pyarrow as pa
> import pyarrow.parquet as pq
> df = pd.DataFrame(np.random.rand(100, 10000))
> table = pa.Table.from_pandas(df)
> pq.write_table(table, "temp.parquet")
> %timeit pd.read_parquet("temp.parquet"){code}
> In version 0.17.0, this takes about 300-400 ms and for anything above and including 1.0.0, this suddenly takes around 2 seconds.
>
> Thanks for looking into this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)