You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "lenamonj (via GitHub)" <gi...@apache.org> on 2026/09/06 02:26:24 UTC
[PR] GitIdentifiers: order tree entries by UTF-8 bytes as Git does [commons-codec]
lenamonj opened a new pull request, #443:
URL: https://github.com/apache/commons-codec/pull/443
`DirectoryEntry.compareTo` used `String.compareTo`, which orders UTF-16 code units. Git orders raw UTF-8 bytes; outside the Basic Multilingual Plane the two differ, so `treeId` disagreed with `git write-tree` for a tree holding U+FF21 and U+1F600.
The sort key is now the name's UTF-8 bytes, compared unsigned. Names below U+D800 keep their order, so the existing constants are unchanged. The new test pins the id `git write-tree` produces and fails on the old comparator.
Checklist: guidelines and the ASF generative-tooling guidance read. AI was used: Claude (Anthropic) found the defect and drafted the fix and test in an audit loop I run; I reviewed it, reproduced the id with `git write-tree`, and ran the default Maven goal, which passes. changes.xml updated.
Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01BzeiNYQSD3LEBxsHTKRxQt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GitIdentifiers: order tree entries by UTF-8 bytes as Git does [commons-codec]
Posted by "garydgregory (via GitHub)" <gi...@apache.org>.
garydgregory commented on PR #443:
URL: https://github.com/apache/commons-codec/pull/443#issuecomment-5647121189
@lenamonj
Please review copilot's comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GitIdentifiers: order tree entries by UTF-8 bytes as Git does [commons-codec]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #443:
URL: https://github.com/apache/commons-codec/pull/443#discussion_r3996759697
##########
src/main/java/org/apache/commons/codec/digest/GitIdentifiers.java:
##########
@@ -100,13 +101,22 @@ static class DirectoryEntry implements Comparable<DirectoryEntry> {
}
this.name = name;
this.type = Objects.requireNonNull(type, "type");
- this.sortKey = type == FileMode.DIRECTORY ? name + "/" : name;
+ this.sortKey = (type == FileMode.DIRECTORY ? name + "/" : name).getBytes(StandardCharsets.UTF_8);
Review Comment:
`String.getBytes(StandardCharsets.UTF_8)` replaces malformed UTF-16 (for example, a lone surrogate) with the encoder's replacement bytes. Consequently, two distinct names can produce the same `sortKey`, `compareTo` returns 0, and the `TreeSet` in `TreeIdBuilder.get()` silently drops one entry. Reject malformed names (or otherwise enforce a one-to-one sort key) before relying on this comparator so every accepted entry is retained.
##########
src/main/java/org/apache/commons/codec/digest/GitIdentifiers.java:
##########
@@ -57,8 +57,9 @@ public class GitIdentifiers {
* <li>the raw object id of the referenced blob or sub-tree.</li>
* </ul>
*
- * <p>Entries are ordered by {@link #compareTo} using Git's tree-sort rule: directory names are compared as if they ended with {@code '/'}, so that {@code foo/}
- * sorts after {@code foobar}.</p>
+ * <p>Entries are ordered by {@link #compareTo} using Git's tree-sort rule: names are compared as unsigned UTF-8 bytes, and directory names are compared as if
+ * they ended with {@code '/'}, so that {@code foo/} sorts after {@code foobar}. Comparing the UTF-8 bytes rather than the Java {@link String} is what keeps
+ * the order Git's for names outside the Basic Multilingual Plane, whose UTF-16 code units do not sort in code point order.</p>
Review Comment:
The phrase `the order Git's for names` is grammatically malformed; rephrase it so the Javadoc clearly explains that the byte comparison preserves Git's ordering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GitIdentifiers: order tree entries by UTF-8 bytes as Git does [commons-codec]
Posted by "garydgregory (via GitHub)" <gi...@apache.org>.
garydgregory commented on PR #443:
URL: https://github.com/apache/commons-codec/pull/443#issuecomment-5656966950
@ppkarwasz any thoughts on merging this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GitIdentifiers: order tree entries by UTF-8 bytes as Git does [commons-codec]
Posted by "ppkarwasz (via GitHub)" <gi...@apache.org>.
ppkarwasz commented on PR #443:
URL: https://github.com/apache/commons-codec/pull/443#issuecomment-5654527926
@lenamonj,
Nice catch! :100:
I completely overlooked that both Git and SWHID (https://www.swhid.org/) treat file names as plain byte sequences.
That raises a question, though: did we miss a `Charset` parameter in the API? Encoding file names in a Git repo with anything other than `UTF-8` is technically possible, but it's a recipe for collaboration headaches.
I'm okay with just assuming UTF-8 for file names, but we should spell that out somewhere:
> Git and SWHID treat file names as opaque byte sequences with no defined encoding. The identifiers produced here coincide with Git's or SWHID's own identifiers only if the original file names were UTF-8 encoded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GitIdentifiers: order tree entries by UTF-8 bytes as Git does [commons-codec]
Posted by "lenamonj (via GitHub)" <gi...@apache.org>.
lenamonj commented on PR #443:
URL: https://github.com/apache/commons-codec/pull/443#issuecomment-5654869305
`getBytes(UTF_8)` turns a lone surrogate into `?`, so a name holding one shared its sort key with `?` and the `TreeSet` kept only one of them; `compareTo` now falls back to the `String` order when the bytes tie, which keeps both as the `String` comparator did, and never fires for well-formed names, whose UTF-8 encoding is unique; `testTreeIdKeepsNamesWithTheSameUtf8Bytes` covers it. Both Javadoc sentences are reworded. Two such names still serialize to the same byte, and `git hash-object -t tree -w` refuses that tree with `duplicateEntries`, so rejecting names UTF-8 cannot encode would stop `treeId` from hashing a tree git cannot store; that changes what it accepts, so it is not in this change unless you want it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GitIdentifiers: order tree entries by UTF-8 bytes as Git does [commons-codec]
Posted by "ppkarwasz (via GitHub)" <gi...@apache.org>.
ppkarwasz merged PR #443:
URL: https://github.com/apache/commons-codec/pull/443
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GitIdentifiers: order tree entries by UTF-8 bytes as Git does [commons-codec]
Posted by "lenamonj (via GitHub)" <gi...@apache.org>.
lenamonj commented on PR #443:
URL: https://github.com/apache/commons-codec/pull/443#issuecomment-5654801711
The class Javadoc now carries that paragraph, with symbolic link targets added, since Git also stores a link target as bytes with no defined encoding. Every name and link target has been encoded with `StandardCharsets.UTF_8` since #428, so the paragraph describes what the code already does and the API does not change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GitIdentifiers: order tree entries by UTF-8 bytes as Git does [commons-codec]
Posted by "ppkarwasz (via GitHub)" <gi...@apache.org>.
ppkarwasz commented on PR #443:
URL: https://github.com/apache/commons-codec/pull/443#issuecomment-5668330004
Let's merge it.
@lenamonj, thanks again! :100:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GitIdentifiers: order tree entries by UTF-8 bytes as Git does [commons-codec]
Posted by "lenamonj (via GitHub)" <gi...@apache.org>.
lenamonj commented on PR #443:
URL: https://github.com/apache/commons-codec/pull/443#issuecomment-5656767220
None, and Git's object model enforces none either: a tree object stores the name bytes it was given, so the NFC and NFD spellings of one name are two entries with two different ids. On git 2.53.0 on Linux, `git mktree` accepts a tree holding both `caf\303\251` and `cafe\314\201`, `git fsck --strict` passes it, and `git write-tree` over a directory holding both files gives `79bc87b55119be79977a4f03942118bf8f06fadc`; `treeId` at 7ed2bae over the same directory gives the same id. The NFC that Google reports is `core.precomposeUnicode`, a macOS-only option: `compat/precompose_utf8.c` wraps `readdir` and converts every non-ASCII name to NFC through `iconv` before it reaches the index, and `git init` on macOS turns it on when the filesystem treats the two spellings as one file. So on a Mac, `git write-tree` hashes NFC names while `treeId` hashes the names as the JDK returns them, and the two agree there only for names that are already NFC. The Javadoc paragraph can carry both facts, tha
t names are hashed with no normalization and that git's macOS build precomposes first, if you would like them spelled out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GitIdentifiers: order tree entries by UTF-8 bytes as Git does [commons-codec]
Posted by "garydgregory (via GitHub)" <gi...@apache.org>.
garydgregory commented on PR #443:
URL: https://github.com/apache/commons-codec/pull/443#issuecomment-5656665184
Hi @lenamonj and @ppkarwasz
Which UTF-8 Normalization does this PR enforce? Google tells me git uses Form C (NFC).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org