You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/12/21 17:54:15 UTC
[GitHub] [iceberg] Fokko opened a new issue, #6475: Python: Improve PyArrow performance
Fokko opened a new issue, #6475:
URL: https://github.com/apache/iceberg/issues/6475
### Apache Iceberg version
None
### Query engine
None
### Please describe the bug ๐
I noticed that s3fs is much faster than PyArrow. @rdblue also noticed this and added a buffer in https://github.com/apache/iceberg/pull/6283
First, we want to have a benchmark to get a baseline and then check what's going on. We could check locally against minio to see how many requests are being made by enabling audit logging.
Also noticed that we open files using `open_input_file` instead of `open_input_stream`. The latter also allows us to buffer as well and makes much more sense for our use case where we sequentially run through the Avro files.
Using the benchmark we can check what the differences are with using `open_input_stream`, and see if we need to wrap a buffered reader on our side. We can validate this using the benchmark.
https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html#pyarrow.fs.S3FileSystem.open_input_stream
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #6475: Python: Improve PyArrow performance
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #6475:
URL: https://github.com/apache/iceberg/issues/6475#issuecomment-1362038098
@Fokko, I already tested open_input_stream and it didn't perform better than Python buffering and open_input_file. In my testing, opening the file blocked for about 500ms, which seems just way too high and didn't change across the two methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #6475: Python: Improve PyArrow performance
Posted by GitBox <gi...@apache.org>.
Fokko commented on issue #6475:
URL: https://github.com/apache/iceberg/issues/6475#issuecomment-1365733458
It looks like PyArrow is still doing more requests than s3fs.
I've created a local table of taxis in PyArrow:
```sql
%%sql
CREATE DATABASE nyc;
%%sql
CREATE TABLE nyc.taxis (
VendorID bigint,
tpep_pickup_datetime timestamp,
tpep_dropoff_datetime timestamp,
passenger_count double,
trip_distance double,
RatecodeID double,
store_and_fwd_flag string,
PULocationID bigint,
DOLocationID bigint,
payment_type bigint,
fare_amount double,
extra double,
mta_tax double,
tip_amount double,
tolls_amount double,
improvement_surcharge double,
total_amount double,
congestion_surcharge double,
airport_fee double
)
USING iceberg
PARTITIONED BY (days(tpep_pickup_datetime))
```
```python
%%python
# Loop over it to avoid OOM, otherwise *.parquet would also work (and be more efficient)
for filename in [
"yellow_tripdata_2022-04.parquet",
"yellow_tripdata_2022-03.parquet",
"yellow_tripdata_2022-02.parquet",
"yellow_tripdata_2022-01.parquet",
"yellow_tripdata_2021-12.parquet",
"yellow_tripdata_2021-11.parquet",
"yellow_tripdata_2021-10.parquet",
"yellow_tripdata_2021-09.parquet",
"yellow_tripdata_2021-08.parquet"
]:
df = spark.read.parquet(f"/home/iceberg/data/{filename}")
df.write.mode("append").saveAsTable("nyc.taxis")
```
Looking at the minio requests when running `pyiceberg --catalog local files nyc.taxis`:
```
Snapshots: local.nyc.taxis
โโโ Snapshot 6682082212753545990, schema 0: s3a://warehouse/wh/nyc/taxis/metadata/snap-6682082212753545990-1-9f900c62-59ca-4ea1-ae75-e373d113b036.avro
โโโ Manifest: s3a://warehouse/wh/nyc/taxis/metadata/9f900c62-59ca-4ea1-ae75-e373d113b036-m0.avro
โ ...
โโโ Manifest: s3a://warehouse/wh/nyc/taxis/metadata/f4d30f09-ebf1-4339-ae9f-652739d4e0bc-m0.avro
โ ...
โโโ Manifest: s3a://warehouse/wh/nyc/taxis/metadata/3fb110ad-0b08-40d7-8e02-ba86192cfc8d-m0.avro
โ ...
โโโ Manifest: s3a://warehouse/wh/nyc/taxis/metadata/6300efd8-fa3a-40ea-bc77-8c128e04d83d-m0.avro
โ ...
โโโ Manifest: s3a://warehouse/wh/nyc/taxis/metadata/27187379-514d-42f4-aada-ade2b239b41a-m0.avro
โ ...
โโโ Manifest: s3a://warehouse/wh/nyc/taxis/metadata/29b8cfd9-38a8-48ac-aabd-ede8b4118d3d-m0.avro
โ ...
โโโ Manifest: s3a://warehouse/wh/nyc/taxis/metadata/37f92724-8248-43c2-b8c7-cd40e2da21cf-m0.avro
โ ...
โโโ Manifest: s3a://warehouse/wh/nyc/taxis/metadata/02096f9b-1056-43ed-85af-871a47ad87f8-m0.avro
โ ...
โโโ Manifest: s3a://warehouse/wh/nyc/taxis/metadata/5a44ffb0-64ac-4b4d-9896-a53c52d6c30b-m0.avro
...
```
### PyArrow
```
2022-12-27T08:45:32.822 [206 Partial Content] s3.GetObject minio:9000/warehouse/wh/nyc/taxis/metadata/00009-3dd7ff4a-a81f-4780-bee1-de9335490c20.metadata.json 172.18.0.3 1.142ms โ 169 B โ 14 KiB
2022-12-27T08:45:32.913 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/snap-6682082212753545990-1-9f900c62-59ca-4ea1-ae75-e373d113b036.avro 172.18.0.1 867ยตs โ 153 B โ 412 B
2022-12-27T08:45:32.925 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/snap-6682082212753545990-1-9f900c62-59ca-4ea1-ae75-e373d113b036.avro 172.18.0.1 1.626ms โ 159 B โ 4.6 KiB
2022-12-27T08:45:32.973 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/9f900c62-59ca-4ea1-ae75-e373d113b036-m0.avro 172.18.0.1 1.216ms โ 153 B โ 413 B
2022-12-27T08:45:32.989 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/9f900c62-59ca-4ea1-ae75-e373d113b036-m0.avro 172.18.0.1 3.719ms โ 159 B โ 8.5 KiB
2022-12-27T08:45:33.020 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/9f900c62-59ca-4ea1-ae75-e373d113b036-m0.avro 172.18.0.1 3.904ms โ 159 B โ 8.5 KiB
2022-12-27T08:45:33.042 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/9f900c62-59ca-4ea1-ae75-e373d113b036-m0.avro 172.18.0.1 1.903ms โ 159 B โ 1.7 KiB
2022-12-27T08:45:33.104 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/f4d30f09-ebf1-4339-ae9f-652739d4e0bc-m0.avro 172.18.0.1 1.232ms โ 153 B โ 413 B
2022-12-27T08:45:33.113 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/f4d30f09-ebf1-4339-ae9f-652739d4e0bc-m0.avro 172.18.0.1 683ยตs โ 159 B โ 8.5 KiB
2022-12-27T08:45:33.120 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/f4d30f09-ebf1-4339-ae9f-652739d4e0bc-m0.avro 172.18.0.1 975ยตs โ 159 B โ 7.0 KiB
2022-12-27T08:45:33.141 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/3fb110ad-0b08-40d7-8e02-ba86192cfc8d-m0.avro 172.18.0.1 383ยตs โ 153 B โ 413 B
2022-12-27T08:45:33.144 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/3fb110ad-0b08-40d7-8e02-ba86192cfc8d-m0.avro 172.18.0.1 774ยตs โ 159 B โ 8.5 KiB
2022-12-27T08:45:33.148 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/3fb110ad-0b08-40d7-8e02-ba86192cfc8d-m0.avro 172.18.0.1 833ยตs โ 159 B โ 7.4 KiB
2022-12-27T08:45:33.170 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/6300efd8-fa3a-40ea-bc77-8c128e04d83d-m0.avro 172.18.0.1 432ยตs โ 153 B โ 413 B
2022-12-27T08:45:33.173 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/6300efd8-fa3a-40ea-bc77-8c128e04d83d-m0.avro 172.18.0.1 1.208ms โ 159 B โ 8.5 KiB
2022-12-27T08:45:33.178 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/6300efd8-fa3a-40ea-bc77-8c128e04d83d-m0.avro 172.18.0.1 814ยตs โ 159 B โ 8.2 KiB
2022-12-27T08:45:33.202 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/27187379-514d-42f4-aada-ade2b239b41a-m0.avro 172.18.0.1 427ยตs โ 153 B โ 413 B
2022-12-27T08:45:33.205 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/27187379-514d-42f4-aada-ade2b239b41a-m0.avro 172.18.0.1 671ยตs โ 159 B โ 8.5 KiB
2022-12-27T08:45:33.209 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/27187379-514d-42f4-aada-ade2b239b41a-m0.avro 172.18.0.1 502ยตs โ 159 B โ 7.9 KiB
2022-12-27T08:45:33.233 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/29b8cfd9-38a8-48ac-aabd-ede8b4118d3d-m0.avro 172.18.0.1 616ยตs โ 153 B โ 413 B
2022-12-27T08:45:33.236 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/29b8cfd9-38a8-48ac-aabd-ede8b4118d3d-m0.avro 172.18.0.1 955ยตs โ 159 B โ 8.5 KiB
2022-12-27T08:45:33.240 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/29b8cfd9-38a8-48ac-aabd-ede8b4118d3d-m0.avro 172.18.0.1 934ยตs โ 159 B โ 7.4 KiB
2022-12-27T08:45:33.262 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/37f92724-8248-43c2-b8c7-cd40e2da21cf-m0.avro 172.18.0.1 308ยตs โ 153 B โ 413 B
2022-12-27T08:45:33.265 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/37f92724-8248-43c2-b8c7-cd40e2da21cf-m0.avro 172.18.0.1 641ยตs โ 159 B โ 8.5 KiB
2022-12-27T08:45:33.269 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/37f92724-8248-43c2-b8c7-cd40e2da21cf-m0.avro 172.18.0.1 831ยตs โ 159 B โ 7.6 KiB
2022-12-27T08:45:33.295 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/02096f9b-1056-43ed-85af-871a47ad87f8-m0.avro 172.18.0.1 625ยตs โ 153 B โ 413 B
2022-12-27T08:45:33.298 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/02096f9b-1056-43ed-85af-871a47ad87f8-m0.avro 172.18.0.1 828ยตs โ 159 B โ 8.5 KiB
2022-12-27T08:45:33.302 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/02096f9b-1056-43ed-85af-871a47ad87f8-m0.avro 172.18.0.1 897ยตs โ 159 B โ 7.8 KiB
2022-12-27T08:45:33.324 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/5a44ffb0-64ac-4b4d-9896-a53c52d6c30b-m0.avro 172.18.0.1 474ยตs โ 153 B โ 413 B
2022-12-27T08:45:33.326 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/5a44ffb0-64ac-4b4d-9896-a53c52d6c30b-m0.avro 172.18.0.1 644ยตs โ 159 B โ 8.5 KiB
2022-12-27T08:45:33.330 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/5a44ffb0-64ac-4b4d-9896-a53c52d6c30b-m0.avro 172.18.0.1 904ยตs โ 159 B โ 7.1 KiB
```
### S3FS
```
2022-12-27T09:05:13.127 [206 Partial Content] s3.GetObject minio:9000/warehouse/wh/nyc/taxis/metadata/00009-3dd7ff4a-a81f-4780-bee1-de9335490c20.metadata.json 172.18.0.3 1.167ms โ 169 B โ 14 KiB
2022-12-27T09:05:13.245 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/snap-6682082212753545990-1-9f900c62-59ca-4ea1-ae75-e373d113b036.avro 172.18.0.1 4.86ms โ 138 B โ 412 B
2022-12-27T09:05:13.258 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/snap-6682082212753545990-1-9f900c62-59ca-4ea1-ae75-e373d113b036.avro 172.18.0.1 860ยตs โ 153 B โ 4.6 KiB
2022-12-27T09:05:13.265 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/9f900c62-59ca-4ea1-ae75-e373d113b036-m0.avro 172.18.0.1 282ยตs โ 138 B โ 413 B
2022-12-27T09:05:13.268 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/9f900c62-59ca-4ea1-ae75-e373d113b036-m0.avro 172.18.0.1 943ยตs โ 153 B โ 18 KiB
2022-12-27T09:05:13.297 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/f4d30f09-ebf1-4339-ae9f-652739d4e0bc-m0.avro 172.18.0.1 472ยตs โ 138 B โ 413 B
2022-12-27T09:05:13.300 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/f4d30f09-ebf1-4339-ae9f-652739d4e0bc-m0.avro 172.18.0.1 907ยตs โ 153 B โ 15 KiB
2022-12-27T09:05:13.320 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/3fb110ad-0b08-40d7-8e02-ba86192cfc8d-m0.avro 172.18.0.1 349ยตs โ 138 B โ 413 B
2022-12-27T09:05:13.323 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/3fb110ad-0b08-40d7-8e02-ba86192cfc8d-m0.avro 172.18.0.1 779ยตs โ 153 B โ 15 KiB
2022-12-27T09:05:13.348 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/6300efd8-fa3a-40ea-bc77-8c128e04d83d-m0.avro 172.18.0.1 346ยตs โ 138 B โ 413 B
2022-12-27T09:05:13.351 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/6300efd8-fa3a-40ea-bc77-8c128e04d83d-m0.avro 172.18.0.1 681ยตs โ 153 B โ 16 KiB
2022-12-27T09:05:13.374 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/27187379-514d-42f4-aada-ade2b239b41a-m0.avro 172.18.0.1 291ยตs โ 138 B โ 413 B
2022-12-27T09:05:13.377 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/27187379-514d-42f4-aada-ade2b239b41a-m0.avro 172.18.0.1 771ยตs โ 153 B โ 16 KiB
2022-12-27T09:05:13.399 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/29b8cfd9-38a8-48ac-aabd-ede8b4118d3d-m0.avro 172.18.0.1 375ยตs โ 138 B โ 413 B
2022-12-27T09:05:13.403 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/29b8cfd9-38a8-48ac-aabd-ede8b4118d3d-m0.avro 172.18.0.1 789ยตs โ 153 B โ 15 KiB
2022-12-27T09:05:13.431 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/37f92724-8248-43c2-b8c7-cd40e2da21cf-m0.avro 172.18.0.1 373ยตs โ 138 B โ 413 B
2022-12-27T09:05:13.434 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/37f92724-8248-43c2-b8c7-cd40e2da21cf-m0.avro 172.18.0.1 650ยตs โ 153 B โ 16 KiB
2022-12-27T09:05:13.457 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/02096f9b-1056-43ed-85af-871a47ad87f8-m0.avro 172.18.0.1 308ยตs โ 138 B โ 413 B
2022-12-27T09:05:13.460 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/02096f9b-1056-43ed-85af-871a47ad87f8-m0.avro 172.18.0.1 753ยตs โ 153 B โ 16 KiB
2022-12-27T09:05:13.482 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/5a44ffb0-64ac-4b4d-9896-a53c52d6c30b-m0.avro 172.18.0.1 279ยตs โ 138 B โ 413 B
2022-12-27T09:05:13.488 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/5a44ffb0-64ac-4b4d-9896-a53c52d6c30b-m0.avro 172.18.0.1 885ยตs โ 153 B โ 15 KiB
```
We can observe that PyArrow does two calls call to the Avro file, and s3fs just one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #6475: Python: Improve PyArrow performance
Posted by GitBox <gi...@apache.org>.
Fokko commented on issue #6475:
URL: https://github.com/apache/iceberg/issues/6475#issuecomment-1362157332
@rdblue Thanks for letting me know, I was hoping that this was missed somewhere. Let's reserve this issue for looking why it is blocked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #6475: Python: Improve PyArrow performance
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #6475:
URL: https://github.com/apache/iceberg/issues/6475#issuecomment-1527302366
I'll close this one since we're waiting on an Arrow release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #6475: Python: Improve PyArrow performance
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #6475:
URL: https://github.com/apache/iceberg/issues/6475#issuecomment-1416747817
https://github.com/apache/arrow/pull/34015 Has just been merged. This will reduce the IO overhead by removing an unnecessary call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on issue #6475: Python: Improve PyArrow performance
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #6475:
URL: https://github.com/apache/iceberg/issues/6475#issuecomment-1404692576
Looking at the recent improvements by @rdblue
Before 887 requests, after the change 203 requests, which is a great improvement, but still more to be done!
After:
```
โ ~ time python3 /tmp/vo.py
python3 /tmp/vo.py 2.10s user 2.02s system 77% cpu 5.294 total
python3 /tmp/vo.py 2.56s user 1.27s system 68% cpu 5.558 total
python3 /tmp/vo.py 1.96s user 1.94s system 68% cpu 5.729 total
```
Before:
```
python3 /tmp/vo.py 2.15s user 1.86s system 64% cpu 6.232 total
python3 /tmp/vo.py 2.61s user 1.53s system 65% cpu 6.315 total
python3 /tmp/vo.py 2.48s user 1.58s system 66% cpu 6.081 total
```
Keep in mind that I'm testing against a local minio, so when using high-latency object storage such as s3 will make it much worse.
Before: https://gist.github.com/Fokko/495dce8234148be74eb312d1af93e694
After: https://gist.github.com/Fokko/a0cc4ac4ec6a7701a145a4a3da3a6bc6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko closed issue #6475: Python: Improve PyArrow performance
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #6475: Python: Improve PyArrow performance
URL: https://github.com/apache/iceberg/issues/6475
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org