You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "goel-skd (via GitHub)" <gi...@apache.org> on 2026/07/03 20:21:20 UTC

[I] Full Unicode case-mapping parity for case-insensitive field matching [iceberg-cpp]

goel-skd opened a new issue, #808:
URL: https://github.com/apache/iceberg-cpp/issues/808

   ### Context
   
   Issue #613 asks for case-insensitive field matching consistent with iceberg-java and iceberg-python (both Unicode-aware), with `İ` (U+0130) as the example. PR #760 (Part of #613) made `StringUtils::ToLower` Unicode-aware using utf8proc **simple (1:1)** case mapping and added allocation-free ASCII fast paths.
   
   This issue captures the **design and remaining plan** to reach full parity and tracks the follow-up PRs.
   
   ### Remaining gap
   
   utf8proc's simple mapping still diverges from java for the few code points where simple ≠ full case mapping — chiefly `İ`:
   
   | input | iceberg-cpp (simple) | iceberg-java `toLowerCase(Locale.ROOT)` / Python `str.lower()` |
   |---|---|---|
   | `İ` (U+0130) | `i` (U+0069) | `i̇` (U+0069 U+0307) |
   
   So `EqualsIgnoreCase("İD", "id")` is **true** in iceberg-cpp but **false** in java/python — the inconsistency #613 is about.
   
   ### Design questions
   
   - Match iceberg-java `toLowerCase(Locale.ROOT)` exactly; confirm the operation PyIceberg uses for matching and that it agrees.
   - Full lowercase mapping vs. Unicode case folding: utf8proc offers full case folding (`utf8proc_map` + `UTF8PROC_CASEFOLD`); verify it reproduces the java/python result, or add a small explicit mapping.
   - Keep the ASCII fast path; stream the non-ASCII path rather than materialize.
   
   ### Work Items
   
   - [ ] Full case mapping to close the `İ` / java-parity gap
   - [ ] Streaming / allocation-free non-ASCII comparison in `EqualsIgnoreCase` / `StartsWithIgnoreCase` (deferred from #760)
   
   ### References
   
   - Issue: #613 (origin)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Full Unicode case-mapping parity for case-insensitive field matching [iceberg-cpp]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #808:
URL: https://github.com/apache/iceberg-cpp/issues/808#issuecomment-5226934795

   @goel-skd Do you want to continue working on Java parity for this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org