You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "vtk9 (via GitHub)" <gi...@apache.org> on 2024/11/18 19:57:36 UTC
[I] improve performance of Table.add_files by parallelizing [iceberg-python]
vtk9 opened a new issue, #1335:
URL: https://github.com/apache/iceberg-python/issues/1335
### Feature Request / Improvement
`Table.add_files()` processes the list of files in sequential order. Part of this flow can be parallelized, particularly https://github.com/apache/iceberg-python/blob/3ccdc44735d70bd3ef6ed18b60b3eba43c4b3b44/pyiceberg/table/__init__.py#L591-L593.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] improve performance of Table.add_files by parallelizing [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #1335:
URL: https://github.com/apache/iceberg-python/issues/1335#issuecomment-2489383016
Yes, looks like this shouldn't be too hard. I think it would be good to [re-use the `ExecutorFactory`](https://github.com/apache/iceberg-python/blob/main/pyiceberg/utils/concurrent.py):
I would refactor `parquet_files_to_data_files` to let it take a single file instead of an `Iterator`, and then call it `parquet_file_to_data_file`.
```python
def _parquet_files_to_data_files(table_metadata: TableMetadata, file_paths: List[str], io: FileIO) -> Iterable[DataFile]:
"""Convert a list files into DataFiles.
Returns:
An iterable that supplies DataFiles that describe the parquet files.
"""
from pyiceberg.io.pyarrow import parquet_files_to_data_files
executor = ExecutorFactory.get_or_create()
futures = [
executor.submit(
parquet_file_to_data_file,
io,
table_metadata,
file_path
)
for file_path in file_paths
]
return [f.result() for f in futures if f.result()]
```
@kevinjqliu I would not classify this as a bugfix, so I'm not sure if this is appropriate for 0.8.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] improve performance of Table.add_files by parallelizing [iceberg-python]
Posted by "vtk9 (via GitHub)" <gi...@apache.org>.
vtk9 commented on issue #1335:
URL: https://github.com/apache/iceberg-python/issues/1335#issuecomment-2487563360
@kevinjqliu when i find time, yes.
I would definitely love this feature in the next release of `pyiceberg` and will prioritize this with enough heads up (if possible) before a release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] improve performance of Table.add_files by parallelizing [iceberg-python]
Posted by "vtk9 (via GitHub)" <gi...@apache.org>.
vtk9 commented on issue #1335:
URL: https://github.com/apache/iceberg-python/issues/1335#issuecomment-2487544116
Apologies @kevinjqliu , i forgot to link the relevant slack thread https://apache-iceberg.slack.com/archives/C029EE6HQ5D/p1731611943890879
Exactly, thank you @bigluck!
I tried something like this and there's a noticeable improvement even when `add_files` contains ~30 files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] improve performance of Table.add_files by parallelizing [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1335:
URL: https://github.com/apache/iceberg-python/issues/1335#issuecomment-2487549044
@vtk9 thanks for the context from slack, I must have missed that thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] improve performance of Table.add_files by parallelizing [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1335:
URL: https://github.com/apache/iceberg-python/issues/1335#issuecomment-2487569620
sounds good! Feel free to ping me for review. I'll add this issue to the 0.8.1 milestone for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] improve performance of Table.add_files by parallelizing [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1335:
URL: https://github.com/apache/iceberg-python/issues/1335#issuecomment-2487547946
thanks @bigluck that makes sense! I think `_parquet_files_to_data_files` might be a good place to add the parallelism
@vtk9 is this something you would like to contribute?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] improve performance of Table.add_files by parallelizing [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1335:
URL: https://github.com/apache/iceberg-python/issues/1335#issuecomment-2489493449
make sense, this is a feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] improve performance of Table.add_files by parallelizing [iceberg-python]
Posted by "bigluck (via GitHub)" <gi...@apache.org>.
bigluck commented on issue #1335:
URL: https://github.com/apache/iceberg-python/issues/1335#issuecomment-2487492436
I believe @vtk9 is suggesting the files to be read in parallel rather than sequentially.
I could be mistaken, but it seems that if you have 10,000 files, each one is being read one after the other. This approach can be quite time-consuming, even though I understand that we are only reading the metadata of each parquet file.
One option could be to have something like (pseudo-code alert):
```python
def parquet_files_to_data_files(io: FileIO, table_metadata: TableMetadata, file_paths: Iterator[str]) -> Iterator[DataFile]:
futures = []
with concurrent.futures.ThreadPoolExecutor() as executor:
for file_path in file_paths:
futures.append(executor.submit(scan_file, file_path))
for future in concurrent.futures.as_completed(futures):
yield future.result()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] improve performance of Table.add_files by parallelizing [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1335:
URL: https://github.com/apache/iceberg-python/issues/1335#issuecomment-2487476101
[`_parquet_files_to_data_files` is a generator](https://github.com/apache/iceberg-python/blob/3ccdc44735d70bd3ef6ed18b60b3eba43c4b3b44/pyiceberg/table/__init__.py#L1529-L1537) and uses [`parquet_files_to_data_files` which is also a generator](https://github.com/apache/iceberg-python/blob/93ebd39e3c457dcb86cd053c60d2d13f0713a637/pyiceberg/io/pyarrow.py#L2522)
what do you have in mind to parallelize this part of the code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] improve performance of Table.add_files by parallelizing [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #1335: improve performance of Table.add_files by parallelizing
URL: https://github.com/apache/iceberg-python/issues/1335
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] improve performance of Table.add_files by parallelizing [iceberg-python]
Posted by "dannypage (via GitHub)" <gi...@apache.org>.
dannypage commented on issue #1335:
URL: https://github.com/apache/iceberg-python/issues/1335#issuecomment-2936111497
Hi folks! We are loving Iceberg and PyIceberg makes it a lot more accessible. We are currently doing a massive backfill (50k files per table) and seeing ~100-500 files per minute in S3 when working with batches of 1000 files. Going to test with the nightly build to see if there will be a performance, but I was curious about two things:
- Will this Issue be included in the `0.10` release?
- Is there a science to the ideal batch size for `add_files`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org