You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jonashaag (via GitHub)" <gi...@apache.org> on 2024/02/01 16:09:59 UTC
[I] Faster ingestion from Parquet [iceberg-python]
jonashaag opened a new issue, #346:
URL: https://github.com/apache/iceberg-python/issues/346
### Question
I'm using the new pyiceberg write functionality. I wonder if there is any way to make it faster in my scenario:
I have around 1 TiB of Parquet files (zstd 3 compressed) that I want to ingest into Iceberg.
Table sizes are ~ power law distributed: The largest table is 25 % of total size, and there are ~ 100 tables.
Since Iceberg wants to repartition data I don't see a way to have it use my Parquet files without rewriting them.
Is it possible to use multiple cores for writing the Parquet files? I don't think that's something that PyArrow supports natively but it might be possible to run multiple PyArrow writers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Faster ingestion from Parquet [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #346:
URL: https://github.com/apache/iceberg-python/issues/346#issuecomment-1924198159
Hello! As of right now, the Pyiceberg write API requires the input to be read as pyarrow tables and can only write to unpartitioned tables. Partitioned write support is currently worked on in #208.
With that said, I wonder if there's a way to create Iceberg tables without rewriting the data files. Technically, Iceberg is a metadata layer on top of the actual data files. The metadata is created from table file stats.
https://github.com/apache/iceberg-python/blob/a4856bc2eadf90ac85dec96d4502ca3517bb1bb5/pyiceberg/table/__init__.py#L2432-L2465
It would be super fast to just read the input files, collect metadata, and create the necessary metadata files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Faster ingestion from Parquet [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #346:
URL: https://github.com/apache/iceberg-python/issues/346#issuecomment-1924214736
Another avenue to look into. On the Spark side of Iceberg, there's a [`add_files`](https://iceberg.apache.org/docs/1.3.0/spark-procedures/#add_files) procedure that can create an Iceberg table without rewriting parquet files.
> This command will create metadata for the new files and will not move them.
And it supports table partitioning!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Faster ingestion from Parquet [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #346: Faster ingestion from Parquet
URL: https://github.com/apache/iceberg-python/issues/346
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Faster ingestion from Parquet [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #346:
URL: https://github.com/apache/iceberg-python/issues/346#issuecomment-2106799882
Fixed in #444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org