You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "mimoune djouallah (Jira)" <ji...@apache.org> on 2022/09/12 05:15:00 UTC
[jira] [Updated] (ARROW-17679) slow performance when reading data from GCP
[ https://issues.apache.org/jira/browse/ARROW-17679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
mimoune djouallah updated ARROW-17679:
--------------------------------------
Description:
I am using pyarrow and duckdb to query some parquet files in GCP, thanks for making the experience so smooth, but I have an issue with the performance, see code used.
import pyarrow.dataset as ds
import duckdb
import json
lineitem = ds.dataset("gs://xxxxx/lineitem")
lineitem_partition = ds.dataset("gs://xxxx/yyy",format="parquet", partitioning="hive")
lineitem_180 = ds.dataset("gs://xxxxx/lineitem_180",format="parquet", partitioning="hive")
con = duckdb.connect()
con.register("lineitem", lineitem)
con.register("lineitem_partition", lineitem_partition)
con.register("lineitem_180", lineitem_180)
def Query(request):
SQL = request.get_json().get('name')
df = con.execute(SQL).df()
return json.dumps(df.to_json(orient="records")), 200, \{'Content-Type': 'application/json'}
the issue is I am getting slow some extremely slow throughput performance, around 30 MBper second, the same files using local ssd laptop is extremely fast.
I am not sure what's the issue, I tried using pyarrow compute Query and it is the same performance
was:
I am using pyarrow and duckdb to query some parquet files in GCP, thanks for making the experience so smooth, but I have an issue with the performance, see code used.
import pyarrow.dataset as ds
import duckdb
import json
lineitem = ds.dataset("gs://duckddelta/lineitem")
lineitem_partition = ds.dataset("gs://duckddelta/delta2",format="parquet", partitioning="hive")
lineitem_180 = ds.dataset("gs://duckddelta/lineitem_180",format="parquet", partitioning="hive")
con = duckdb.connect()
con.register("lineitem", lineitem)
con.register("lineitem_partition", lineitem_partition)
con.register("lineitem_180", lineitem_180)
def Query(request):
SQL = request.get_json().get('name')
df = con.execute(SQL).df()
return json.dumps(df.to_json(orient="records")), 200, \{'Content-Type': 'application/json'}
the issue is I am getting slow some extremely slow throughput performance, around 30 MBper second, the same files using local ssd laptop is extremely fast.
I am not sure what's the issue, I tried using pyarrow compute Query and it is the same performance
> slow performance when reading data from GCP
> -------------------------------------------
>
> Key: ARROW-17679
> URL: https://issues.apache.org/jira/browse/ARROW-17679
> Project: Apache Arrow
> Issue Type: Bug
> Components: Parquet, Python
> Affects Versions: 9.0.0
> Reporter: mimoune djouallah
> Priority: Major
>
> I am using pyarrow and duckdb to query some parquet files in GCP, thanks for making the experience so smooth, but I have an issue with the performance, see code used.
> import pyarrow.dataset as ds
> import duckdb
> import json
> lineitem = ds.dataset("gs://xxxxx/lineitem")
> lineitem_partition = ds.dataset("gs://xxxx/yyy",format="parquet", partitioning="hive")
> lineitem_180 = ds.dataset("gs://xxxxx/lineitem_180",format="parquet", partitioning="hive")
> con = duckdb.connect()
> con.register("lineitem", lineitem)
> con.register("lineitem_partition", lineitem_partition)
> con.register("lineitem_180", lineitem_180)
> def Query(request):
> SQL = request.get_json().get('name')
> df = con.execute(SQL).df()
> return json.dumps(df.to_json(orient="records")), 200, \{'Content-Type': 'application/json'}
>
> the issue is I am getting slow some extremely slow throughput performance, around 30 MBper second, the same files using local ssd laptop is extremely fast.
> I am not sure what's the issue, I tried using pyarrow compute Query and it is the same performance
--
This message was sent by Atlassian Jira
(v8.20.10#820010)