You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "frankliee (via GitHub)" <gi...@apache.org> on 2024/03/25 13:20:50 UTC
[I] How to load a table with bucket partitioned [iceberg-python]
frankliee opened a new issue, #548:
URL: https://github.com/apache/iceberg-python/issues/548
### Question
The table DDL is
```
create table test1 (
id int,
data int
)
using iceberg
partitioned by (bucket(4, id));
```
How to write the row_filter to load one of bucket?
table.scan(row_filter=xxx)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to load a table with bucket partition [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #548:
URL: https://github.com/apache/iceberg-python/issues/548#issuecomment-2018364320
Ah, I misread what you're looking for. Have you considered the `.plan_files()` API where you just get a list of tasks to read?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to load a table with bucket partition [iceberg-python]
Posted by "frankliee (via GitHub)" <gi...@apache.org>.
frankliee commented on issue #548:
URL: https://github.com/apache/iceberg-python/issues/548#issuecomment-2019248895
I have considered this, `.plan_files()` will get all files, and cannot distinguish which files are in the same bucket.
For Iceberg-Spark, there is a helpful system function expression like this,
`SELECT * FROM test1 WHERE system.bucket(4, id) == 2`
Iceberg-Python seems to be missing some things like `bucket(n, ref)`.
@Fokko
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to load a table with bucket partition [iceberg-python]
Posted by "goalzz85 (via GitHub)" <gi...@apache.org>.
goalzz85 commented on issue #548:
URL: https://github.com/apache/iceberg-python/issues/548#issuecomment-2428665724
```python
from pyiceberg.transforms import BucketTransform
from pyiceberg.types import IntegerType
id = 50
t = BucketTransform(50)
bucket_int_func = t.transform(IntegerType())
bucket_num = bucket_int_func(id)
print(bucket_num)
```
I looked into the source code again and found a way to calculate the bucket number from the ID.
@frankliee
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to load a table with bucket partition [iceberg-python]
Posted by "frankliee (via GitHub)" <gi...@apache.org>.
frankliee commented on issue #548:
URL: https://github.com/apache/iceberg-python/issues/548#issuecomment-2018198305
> @frankliee PyIceberg will do the filtering automatically, so when you filter on the id column, it will automatically use the bucketing to filter down to the correct bucket:
>
> ```python
> table.scan(row_filter='id = 123')
> ```
However, 'id=123' cannot control which bucket to load in a fine-grained manner.
For example, if we have four readers, and each reader loads one of bucket independently, it is difficult to write the row_filter expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to load a table with bucket partition [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #548:
URL: https://github.com/apache/iceberg-python/issues/548#issuecomment-2367035408
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to load a table with bucket partition [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #548:
URL: https://github.com/apache/iceberg-python/issues/548#issuecomment-2018051402
@frankliee PyIceberg will do the filtering automatically, so when you filter on the id column, it will automatically use the bucketing to filter down to the correct bucket:
```python
table.scan(row_filter='id = 123')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to load a table with bucket partition [iceberg-python]
Posted by "goalzz85 (via GitHub)" <gi...@apache.org>.
goalzz85 commented on issue #548:
URL: https://github.com/apache/iceberg-python/issues/548#issuecomment-2426506481
```python
table = catalog.load_table(("default", "mq_log10"))
scan = table.scan()
file_scan_tasks = scan.plan_files()
partition_num = 20
new_file_scan_tasks = []
for task in file_scan_tasks:
if task.file.partition[0] == partition_num:
new_file_scan_tasks.append(task)
# referenceļ¼
# https://github.com/apache/iceberg-python/blob/pyiceberg-0.7.1/pyiceberg/table/__init__.py#L2014
# @deprecated(
# deprecated_in="0.8.0",
# removed_in="0.9.0",
# help_message="project_table is deprecated. Use ArrowScan.to_table instead.",
# )
from pyiceberg.io.pyarrow import project_table
pa_data_table = project_table(new_file_scan_tasks, scan.table_metadata, scan.io, scan.row_filter, scan.projection(), case_sensitive=scan.case_sensitive, limit=scan.limit)
```
I use this code to load a table with a bucketed partition, which may be useful for you. You should note that the project_table function will be removed in version 0.9.0, but you can refer to the new implementation and rewrite this code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to load a table with bucket partition [iceberg-python]
Posted by "jaimeferj (via GitHub)" <gi...@apache.org>.
jaimeferj commented on issue #548:
URL: https://github.com/apache/iceberg-python/issues/548#issuecomment-3411533044
Hi there! I'm encountering a similar use case and would like to revive this discussion.
## Use Case: Efficient Partition-Aware Joins in Polars
I have multiple large Iceberg tables that are all partitioned identically using `bucket(user_id, N)`. I need to perform joins between these tables using Polars, but currently, I cannot load all tables into memory at once.
**What I want to achieve:**
- Join tables partition-by-partition to avoid loading all data into memory
- Leverage the fact that both tables share the same bucketing scheme on the join key (`user_id`)
- Process each bucket independently, which should be much more memory-efficient
**Current workaround:**
Based on @goalzz85's solution above, I can manually filter files by partition:
```python
import polars as pl
from collections import defaultdict
# Group files by partition value
def get_files_by_partition(table):
files_by_partition = defaultdict(list)
for task in table.scan().plan_files():
partition_key = tuple(task.file.partition) if task.file.partition else ()
files_by_partition[partition_key].append(task.file.file_path)
return files_by_partition
# Get partitioned files
files1_by_partition = get_files_by_partition(table1)
files2_by_partition = get_files_by_partition(table2)
# Process partition by partition
results = []
for partition_key in files1_by_partition.keys():
if partition_key in files2_by_partition:
lf1 = pl.scan_parquet(files1_by_partition[partition_key])
lf2 = pl.scan_parquet(files2_by_partition[partition_key])
result = lf1.join(lf2, on="user_id").collect()
results.append(result)
final = pl.concat(results)
```
**The problem:**
This workaround requires manually extracting file paths and bypassing PyIceberg's scan interface entirely. It would be much cleaner to have native support for partition-aware scans.
## Feature Request
It would be extremely valuable to have one of the following:
1. **Expose partition values in row filters**: Allow filtering by hidden partition columns, e.g.:
```python
table.scan(row_filter=PartitionEquals("user_id_bucket", 5))
```
2. **Add a method to scan specific partitions directly**:
```python
table.scan(partition_filter={"user_id_bucket": 5}).to_polars()
```
3. **Add an iterator over partitions**:
```python
for partition_value, scan in table.scan_partitions():
df = scan.to_polars()
# process partition
```
This would enable memory-efficient partition-aware joins and other partition-level processing patterns that are common in distributed systems like Spark, but currently difficult to implement efficiently in PyIceberg + Polars.
Would love to hear thoughts on this! Happy to contribute if there's interest in adding this functionality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to load a table with bucket partition [iceberg-python]
Posted by "sam-leonid (via GitHub)" <gi...@apache.org>.
sam-leonid commented on issue #548:
URL: https://github.com/apache/iceberg-python/issues/548#issuecomment-3696719225
Variant of https://github.com/apache/iceberg-python/issues/548#issuecomment-2426506481 worked using new version pyiceberg (v 0.9.1):
```python
table = catalog.load_table(("default", "mq_log10"))
scan = table.scan()
file_scan_tasks = scan.plan_files()
partition_num = 20
new_file_scan_tasks = []
for task in file_scan_tasks:
if task.file.partition[0] == partition_num:
new_file_scan_tasks.append(task)
arrow_scan = ArrowScan(
table_metadata=scan.table_metadata,
io=scan.io,
row_filter=scan.row_filter,
projected_schema=scan.table_metadata.schema(),
case_sensitive=scan.case_sensitive,
limit=scan.limit,
)
pa_data_table = arrow_scan.to_table(new_file_scan_tasks)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org