You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@datasketches.apache.org by "proost (via GitHub)" <gi...@apache.org> on 2026/10/16 08:40:25 UTC
[PR] fix: replace invalid UTF-8 in AoS hash keys as Java does [datasketches-go]
proost opened a new pull request, #197:
URL: https://github.com/apache/datasketches-go/pull/197
## Summary
Follow-up to #196 for #191.
#196 made `GenerateHashKeyFromStrings` hash keys as UTF-16LE, as Java does, but it decoded each string by ranging over it. That gives one U+FFFD per invalid byte. Java only hashes invalid UTF-8 after its UTF-8 decoder (`new String(bytes, UTF_8)`) has turned the bytes into a `String`, and that decoder replaces them differently in two cases:
| Input | Java | #196 |
|---|---|---|
| `E2 82 41` (cut-off 3-byte character) | `FFFD 0041` | `FFFD FFFD 0041` |
| `F0 9F 94 41` (cut-off 4-byte character) | `FFFD 0041` | `FFFD FFFD FFFD 0041` |
| `ED A0 80` (encoded surrogate) | `FFFD` | `FFFD FFFD FFFD` |
This PR decodes the bytes with the same rules as the JDK's `String.decodeUTF8_UTF16`: the prefix lengths of `malformed3` and `malformed4`, the surrogate check, and the end-of-input handling. Valid UTF-8 hashes exactly as before.
## Verification
- **Differential test against real JDKs:** every 1–4 byte sequence over 27 boundary bytes (one or two from each byte class a UTF-8 decoder distinguishes), plus 200,000 random 5–10 byte sequences, for 751,880 inputs in total.
- JDK 8, 11, 17, 21, 25 and 26 all produce identical output.
- `new String(b, UTF_8)` and `CharsetDecoder` with `REPLACE` agree on every input.
- This PR has 0 mismatches. #196 has 167,194.
- **`TestGenerateHashKeyFromStrings_JavaDecoding`:** 20 cases with expected values taken from Java, including inputs cut off at the end.
- `make test` (with the TCK snapshots) and `make lint` pass.
## Compatibility
#196 is not in any release, so this adds no break beyond the one v0.3.0 already has for AoS key hashing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] fix: replace invalid UTF-8 in AoS hash keys as Java does [datasketches-go]
Posted by "proost (via GitHub)" <gi...@apache.org>.
proost commented on PR #197:
URL: https://github.com/apache/datasketches-go/pull/197#issuecomment-6211635394
I got help from claude for comparison binaries between go and java.
And Claude found invalid utf8 input cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: [PR] fix: replace invalid UTF-8 in AoS hash keys as Java does [datasketches-go]
Posted by "proost (via GitHub)" <gi...@apache.org>.
proost merged PR #197:
URL: https://github.com/apache/datasketches-go/pull/197
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org