You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "gitzwz (via GitHub)" <gi...@apache.org> on 2024/12/31 09:42:19 UTC
[I] [Question] Why does plan_files not seem to get multi-threading improvement [iceberg-python]
gitzwz opened a new issue, #1479:
URL: https://github.com/apache/iceberg-python/issues/1479
### Question
I encountered a problem with table.scan.plan_files() where there is no noticeable time difference between single-threaded and multi-threaded execution. The total time is directly proportional to the number of manifest entries. The table I used for testing has 6 manifest files, and each manifest file contains around 70,000 entries. The most time-consuming process is _open_manifest in the DataScan.plan_files() function, and it performs similarly whether using a thread pool or not. Could someone help me investigate if there might be an issue?
Here is my test code:
`from pyiceberg.catalog import load_catalog
from pyspark.sql import SparkSession
from pyiceberg import expressions as pyi_expr
import time
from line_profiler import LineProfiler
catalog = load_catalog("default")
table = catalog.load_table('b_ods.pyiceberg_test2')
def scan_plan_files(key, values):
row_filter=pyi_expr.In(key, values)
files = table.scan(
row_filter=row_filter,
limit=1000
).plan_files()
print(f"total plans {len(files)}")
for file in files:
print(file.file.file_path)
start_time = time.perf_counter()
scan_plan_files("cid", {'844'})
print(f"Time consumed：{time.perf_counter() - start_time:.3f} seconds")
`
I also modified the ~/.pyiceberg.yaml file, changing *max-workers: 1* to *max-workers: 32*, but the total time is still around 64 seconds with little to no change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Question] Why does plan_files not seem to get multi-threading improvement [iceberg-python]
Posted by "gitzwz (via GitHub)" <gi...@apache.org>.
gitzwz commented on issue #1479:
URL: https://github.com/apache/iceberg-python/issues/1479#issuecomment-2567363410
The most time-consuming process is this :
```Python
for manifest_entry in chain(
*executor.map(
lambda args: _open_manifest(*args),
[
(
self.io,
manifest,
partition_evaluators[manifest.partition_spec_id],
metrics_evaluator,
)
for manifest in manifests
if self._check_sequence_number(min_sequence_number, manifest)
],
)
):
...
```
For instance, consider a scenario with 6 manifest files, each containing 7,000 entries. With **max-workers=32**, the code spawns 6 threads, each completing after approximately 30 seconds concurrently. In contrast, with **max-workers=1**, the code processes the manifest files sequentially, yet still finishes in roughly 30 seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Question] Why does plan_files not seem to get multi-threading improvement [iceberg-python]
Posted by "gitzwz (via GitHub)" <gi...@apache.org>.
gitzwz commented on issue #1479:
URL: https://github.com/apache/iceberg-python/issues/1479#issuecomment-2566291942
Here is my test code:
`
from pyiceberg.catalog import load_catalog
from pyspark.sql import SparkSession
from pyiceberg import expressions as pyi_expr
import time
from line_profiler import LineProfiler
catalog = load_catalog("default")
table = catalog.load_table('b_ods.pyiceberg_test2')
def scan_plan_files(key, values):
row_filter=pyi_expr.In(key, values)
files = table.scan(
row_filter=row_filter,
limit=1000
).plan_files()
print(f"total plans {len(files)}")
for file in files:
print(file.file.file_path)
start_time = time.perf_counter()
scan_plan_files("cid", {'844'})
print(f"Time consumed：{time.perf_counter() - start_time:.3f} seconds")
`
I also modified the ~/.pyiceberg.yaml file, changing *max-workers: 1* to *max-workers: 32*, but the total time is still around 64 seconds with little to no change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Question] Why does plan_files not seem to get multi-threading improvement [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1479:
URL: https://github.com/apache/iceberg-python/issues/1479#issuecomment-2568436698
> there is no noticeable time difference between single-threaded and multi-threaded execution. The total time is directly proportional to the number of manifest entries.
Could you print out `ExecutorFactory.max_workers()` to double check the value?
> For instance, consider a scenario with 6 manifest files, each containing 7,000 entries. With max-workers=32, the code spawns 6 threads, each completing after approximately 30 seconds concurrently. In contrast, with max-workers=1, the code processes the manifest files sequentially, yet still finishes in roughly 30 seconds.
Theres already some discussions around this in #1229. The issue might be with I/O bound tasks and the python GIL. Can you give `ProcessPoolExecutor` a try?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Question] Why does plan_files not seem to get multi-threading improvement [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed issue #1479: [Question] Why does plan_files not seem to get multi-threading improvement
URL: https://github.com/apache/iceberg-python/issues/1479
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Question] Why does plan_files not seem to get multi-threading improvement [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1479:
URL: https://github.com/apache/iceberg-python/issues/1479#issuecomment-2569711529
sounds good! Lets close this issue and move the discussion to #1229
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [Question] Why does plan_files not seem to get multi-threading improvement [iceberg-python]
Posted by "gitzwz (via GitHub)" <gi...@apache.org>.
gitzwz commented on issue #1479:
URL: https://github.com/apache/iceberg-python/issues/1479#issuecomment-2568858562
> Could you print out `ExecutorFactory.max_workers()` to double check the value?
I hacked the pyIceberg code to print executing thread name, so I can see the thread difference.
> Theres already some discussions around this in https://github.com/apache/iceberg-python/issues/1229. The issue might be with I/O bound tasks and the python GIL. Can you give ProcessPoolExecutor a try?
Thank you so much for pointing this out. That's same issue. I will try out and reply in https://github.com/apache/iceberg-python/issues/1229.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org