You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "badalprasadsingh (via GitHub)" <gi...@apache.org> on 2026/06/17 20:52:48 UTC

[I] transforms: truncate produces invalid UTF-8 for non-ASCII strings [iceberg-go]

badalprasadsingh opened a new issue, #1222:
URL: https://github.com/apache/iceberg-go/issues/1222

   ### Apache Iceberg version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   ### Description
   
   In Iceberg-Go, `iceberg.TruncateTransform.Transformer`for `StringType` slices the input by raw UTF-8 byte length instead of Unicode code points. The [**Iceberg spec**](https://iceberg.apache.org/spec/#truncate-transform-details) defines string truncate as truncation to *N Unicode characters*.
   
   For any non-ASCII input (CJK, emoji, accented Latin), Iceberg-Go:
   
   - produces output that may not be valid UTF-8 because the slice cuts mid-codepoint, and
   - routes rows to a different partition directory
   
   Example: `truncate(5, "δΈ–η•Œζ‚¨ε₯½δΈ–η•Œ")`
   
   ### Proposed fix
   
   Walk the input by Unicode code points using `unicode/utf8`. Keep binary truncation byte-based.
   
   Something similar to what Iceberg-Java does in [**Truncate Transform**](https://github.com/apache/iceberg/blob/ccfec2b771c31ad7b9bd893fb3dc9630c3cf982f/api/src/main/java/org/apache/iceberg/transforms/Truncate.java).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] transforms: truncate produces invalid UTF-8 for non-ASCII strings [iceberg-go]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka closed issue #1222: transforms: truncate produces invalid UTF-8 for non-ASCII strings
URL: https://github.com/apache/iceberg-go/issues/1222


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org