[I] Proposal: Collation support for string types [iceberg]

["laskoviymishka (via GitHub)" <gi...@apache.org>]

laskoviymishka opened a new issue, #17620:
URL: https://github.com/apache/iceberg/issues/17620

   ## Summary
   
   Add column-level collation to the Iceberg spec: a `string` field can carry a provider-qualified collation (e.g. `icu.en_US-ci`) that defines case-insensitive, accent-insensitive, or locale-aware comparison and ordering. Storage stays UTF-8 — only comparison changes. Collation-aware min/max are stored so collated columns stay prunable.
   
   ## Motivation
   
   Iceberg strings compare by UTF-8 bytes only — there is no way to declare a column case-/accent-insensitive or locale-sorted. Snowflake and Databricks users cite this as a migration blocker. Query-rewrite workarounds (`UPPER()`/`LOWER()`, `ILIKE`) don't scale and defeat min/max pruning.
   
   ## Design (current direction)
   
   - A `collation` annotation on the string field, stored **unversioned** in the schema so any compatible engine can read the table.
   - Collation-aware bounds are stored as ordinary `content_stats` under schema-declared collation metric fields (`collation-metrics`: `(collation, version) -> metric field id`); original values (not sort keys), tight, tagged with the collation implementation version they were selected under.
   - Byte-order `lower_bounds`/`upper_bounds` are still written for collation-unaware engines. Readers prune with a collation metric field's bounds only on an exact collation + version match; otherwise they scan. A version mismatch degrades pruning, never correctness.
   - Collation is comparison-only: partition/bucket transforms, equality-delete matching, and identifier-field equality stay binary. Providers are a registered set (like geo), `icu` to start.
   
   ## Artifacts
   
   - Spec PR: https://github.com/apache/iceberg/pull/16972
   - Reference implementation (iceberg-go): https://github.com/apache/iceberg-go/pull/1318
   - Java POC: https://github.com/laskoviymishka/iceberg/tree/prototype/collation-support
   - Proposal write-up: https://github.com/laskoviymishka/iceberg/blob/collation-support-proposal/format/collation-support-proposal.md
   - Original proposal doc: https://docs.google.com/document/d/1m8b7u97uteHYjXk-4DNglJSpQO8OcZOCzW2tApCNTW4/edit
   - Dev-list thread: https://lists.apache.org/thread/44todz4x460g8pb89y8rpozlnmo8vdhc
   
   ## Open decisions (working session Aug 5, 2026 · 5PM CET / 8AM PST)
   
   1. Do we pursue collation in the spec at all?
   2. Who owns the ICU version — the format (pinned, deterministic across engines) or the engine (versionless, independent upgrades)? May differ for pruning (performance) vs execution semantics (correctness).
   3. Equality deletes on collated columns — allow, or disallow in v1?
   4. v1 scope — annotation + file pruning only, or execution semantics too?
   5. Provider model — registered set vs open namespace.
   
   This binds every engine that implements collations, so implementer input (Spark, Trino, Flink, PyIceberg, iceberg-rust, iceberg-go) is welcome before we fix field IDs.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org