[PR] KAFKA-12443: Add TranslateSurrogates SMT to Kafka Connect [kafka]

["akhil-sabu-ibm (via GitHub)" <gi...@apache.org>]

akhil-sabu-ibm opened a new pull request, #23084:
URL: https://github.com/apache/kafka/pull/23084

   ## Summary
   
   This PR implements a new `TranslateSurrogates` transformation to handle UTF-16 surrogate pairs in string fields.
   
   ## Changes
   
   **Files:**
   - `connect/transforms/src/main/java/org/apache/kafka/connect/transforms/TranslateSurrogates.java` (new)
   - `connect/transforms/src/main/resources/META-INF/services/org.apache.kafka.connect.transforms.Transformation` (updated)
   - `connect/transforms/src/test/java/org/apache/kafka/connect/transforms/TranslateSurrogatesTest.java` (new)
   
   **Motivation:** Some data sources produce UTF-16 surrogate pairs in string fields that cannot be properly serialized or processed by downstream systems. This transformation provides configurable handling of such characters.
   
   **Features:**
   - Supports three encoding modes:
     - `url-encode` (default): Percent-encodes using UTF-8 byte sequence (e.g., U+23A3A → `%F0%A3%98%BA`)
     - `java-encode`: Java-style surrogate pair notation (e.g., `\UD84D\UDE3A`)
     - `replace`: Replaces with configurable string (defaults to Unicode replacement character U+FFFD)
   - Processes string fields at any depth in structs, maps, and arrays
   - Supports optional field filtering via `fields` config
   - Available as both `Key` and `Value` transformation variants
   - Validates that replacement strings do not themselves contain surrogate pairs
   
   ## Testing Strategy
   
   Comprehensive test suite covering:
   - **Configuration validation**: Invalid modes, replacement strings containing surrogates
   - **Encoding behaviour per mode**: URL encoding, Java encoding, custom replacements
   - **String translation**: ASCII passthrough, single/multiple surrogate pairs
   - **Schemaless records**: String values, maps (all/filtered fields), arrays, null handling
   - **Schema-based records**: Structs with selective field translation, nested structures
   - **Key vs. Value transforms**: Correct record updates for each variant
   - **Edge cases**: Unsupported types, multiple consecutive surrogate pairs
   
   ## Key improvements over earlier draft (PR #10287)
   - Use `ConfigDef.LambdaValidator` instead of anonymous `Validator` classes
   - `URLEncoder.encode(str, Charset)` instead of `(str, String)` — no checked exception
   - Fixed `Key.newRecord()` — original had key/value schemas swapped
   - Null-value guard in `apply()` — original would NPE on null values
   - `fields` config respected consistently for both MAP and STRUCT paths
   
   JIRA: https://issues.apache.org/jira/browse/KAFKA-12443
   
   Reviewers: Mickael Maison <mi...@gmail.com>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] KAFKA-12443: Add TranslateSurrogates SMT to Kafka Connect [kafka]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on PR #23084:
URL: https://github.com/apache/kafka/pull/23084#issuecomment-5612147541

   A label of 'needs-attention' was automatically added to this PR in order to raise the
   attention of the committers. Once this issue has been triaged, the `triage` label
   should be removed to prevent this automation from happening again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org