You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "C-Loftus (via GitHub)" <gi...@apache.org> on 2026/05/14 15:47:32 UTC
[I] Question about partitioning for string scans [iceberg-go]
C-Loftus opened a new issue, #1080:
URL: https://github.com/apache/iceberg-go/issues/1080
### Question
Thank you for the great work on iceberg-go!
## Question
Are there any best practices for writing columns of string data where there is high cardinality but a finite set of values? (i.e. to improve parquet row group metadata and scan performance?)
## Context
I want to write a column like `project_identifier` via iceberg-go that has many string values like `foo_123`, `bar_123`, `foo_baz_123`, `foo_bar_123`, `foo_test_123` .... etc. This is a finite set but it is high cardinality (say 1000+ values)
I don't want to transform the data and separate it into more tables if possible.
However, since these are all string values, scans can be rather slow (as I saw here with `DELETE`s https://github.com/apache/iceberg-go/issues/1077) since strings provide less useful row group statistics metadata to my understanding. However, I was thinking that if the underlying parquet files were partitioned by the value of `project_identifier` (for instance, parquet file 1 contains all rows with `foo_*` and parquet file 2 contains all rows with `bar_*`) then the row group statistics would be much more useful.
However, I was unclear how to accomplish this (i.e. is it possible to partition on a substring?) and how https://github.com/apache/iceberg-go/pull/931 might affect this when dictionary encoding is added.
Thank you very much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Question about partitioning for string scans [iceberg-go]
Posted by "C-Loftus (via GitHub)" <gi...@apache.org>.
C-Loftus commented on issue #1080:
URL: https://github.com/apache/iceberg-go/issues/1080#issuecomment-4455639695
Awesome, thank you so much for that clarification, I really appreciate it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Question about partitioning for string scans [iceberg-go]
Posted by "C-Loftus (via GitHub)" <gi...@apache.org>.
C-Loftus closed issue #1080: Question about partitioning for string scans
URL: https://github.com/apache/iceberg-go/issues/1080
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Question about partitioning for string scans [iceberg-go]
Posted by "C-Loftus (via GitHub)" <gi...@apache.org>.
C-Loftus commented on issue #1080:
URL: https://github.com/apache/iceberg-go/issues/1080#issuecomment-4453604242
Thank you so much @tanmayrauth that makes sense! As one minor follow up if you don't mind, I see that upon table creation you can pass the `iceberg.TruncateTransform` in two different places. Does it matter where this is specified? Is one a better practice?
1. Within an `iceberg.PartitionSpec` via `catalog.WithPartitionSpec`. That struct has a `iceberg.PartitionField` field which could take in a `iceberg.TruncateTransform`
2. Within a `table.SortOrder` via `catalog.WithSortOrder`. That struct has a sort field which could take in `iceberg.TruncateTransform`
## Example
```go
partitionSpec := iceberg.NewPartitionSpec(
iceberg.PartitionField{SourceIDs: []int{2}, Transform: iceberg.TruncateTransform{Width: 20}, Name: "project_partition"},
)
// this could also take in a Transform struct. Is that better to do? vvvvvvv
sortField := table.SortField{SourceIDs: []int{2}, Direction: table.SortASC}
sortOrder, err := table.NewSortOrder(table.InitialSortOrderID, []table.SortField{sortField})
tbl, err := cat.CreateTable(ctx, tableIdent, icebergSchema,
catalog.WithPartitionSpec(&partitionSpec),
catalog.WithSortOrder(sortOrder),
// abbreviated for clarity...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Question about partitioning for string scans [iceberg-go]
Posted by "tanmayrauth (via GitHub)" <gi...@apache.org>.
tanmayrauth commented on issue #1080:
URL: https://github.com/apache/iceberg-go/issues/1080#issuecomment-4453374782
Yes, you can partition on a substring, the truncate[N] partition transform does exactly this for strings. It takes the first N characters as the partition value, so all rows sharing a prefix land in the same data files.
For your pattern (foo_123, foo_baz_123, bar_123, etc.), something like truncate[4] would group all foo_ rows into one partition and bar_ into another. When scanning with a predicate on project_identifier, the engine prunes partitions whose truncated prefix can't match - eliminating files without reading them.
-- In Spark SQL equivalent terms:
PARTITION BY truncate(4, project_identifier)
In iceberg-go, this is iceberg.TruncateTransform{Width: N} on the partition spec. Pick a width that captures the meaningful prefix structure of your values.
Why this helps row group stats: Since truncate preserves order, all values within a partition file share a common prefix, making Parquet min/max statistics much tighter. You'll get effective row-group skipping even within partition files.
Complementary strategy - sort order: Setting a sort order on project_identifier ensures rows are physically sorted within each file. This makes min/max column chunk stats as tight as possible. The combination of truncate partitioning + sorted writes is the standard approach for this scenario.
Alternative - bucket[N]: If your prefixes aren't uniform length or you just want even distribution, bucket[N] hashes values into N buckets. Trade-off: only equality/IN predicates benefit from pruning (no range scan benefit since bucket doesn't preserve order). With ~1000 distinct values, 16–64 buckets works well.
Regarding #931 (dictionary encoding): Dictionary encoding is orthogonal - it operates within a single Parquet file, replacing repeated strings with integer codes. It helps most when cardinality within a row group is low, which sorting naturally achieves. So these strategies stack: partition → sort → dictionary encode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Question about partitioning for string scans [iceberg-go]
Posted by "tanmayrauth (via GitHub)" <gi...@apache.org>.
tanmayrauth commented on issue #1080:
URL: https://github.com/apache/iceberg-go/issues/1080#issuecomment-4455030135
PartitionSpec (TruncateTransform) : controls which rows go into which files. All rows sharing the same truncated prefix land in the same partition's data files. This is what enables file-level pruning during scans.
SortOrder (IdentityTransform) : controls the physical row ordering within each data file. This is what makes Parquet column chunk min/max statistics tight within a file, enabling row-group skipping.
Best practice for your case:
```
partitionSpec := iceberg.NewPartitionSpec(
iceberg.PartitionField{
SourceIDs: []int{2},
Transform: iceberg.TruncateTransform{Width: 20},
Name: "project_partition",
},
)
sortField := table.SortField{
SourceIDs: []int{2},
Transform: iceberg.IdentityTransform{},
Direction: table.SortASC,
NullOrder: table.NullsLast,
}
sortOrder, err := table.NewSortOrder(table.InitialSortOrderID, []table.SortField{sortField})
tbl, err := cat.CreateTable(ctx, tableIdent, icebergSchema,
catalog.WithPartitionSpec(&partitionSpec),
catalog.WithSortOrder(sortOrder),
)
```
You don't want TruncateTransform in the sort order — that would sort by the truncated value, making all rows within a partition appear equivalently ordered (since they already share the same truncated prefix). Instead use IdentityTransform{} to sort by the full raw value, giving you the tightest possible min/max stats within each file.
In short:
partition spec = which file a row lands in, sort order = row arrangement within that file. Use truncate for the former, identity for the latter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org