You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "abstractdog (via GitHub)" <gi...@apache.org> on 2026/09/07 12:29:50 UTC
[PR] Use JDK intrinsics for Binary equality/comparison/hashing [parquet-java]
abstractdog opened a new pull request, #3782:
URL: https://github.com/apache/parquet-java/pull/3782
### Rationale for this change
`Binary#equals`, `Binary#lexicographicCompare`, and `Binary#hashCode` sit on hot paths for statistics min/max maintenance, dictionary hash-map probing, predicate evaluation, and bloom-filter build — so any per-byte overhead is amortized across every reader and writer. The current implementations use hand-rolled scalar byte loops; the JDK 9+ range overloads of `Arrays.equals` / `Arrays.compareUnsigned` / `Arrays.hashCode` / `ByteBuffer.mismatch` route through `ArraysSupport.vectorizedMismatch`, an `@IntrinsicCandidate` helper HotSpot substitutes with a SIMD byte-scan (SSE / AVX2 / NEON).
### What changes are included in this PR?
- Replace the scalar loops in `Binary.java` for `equals`, `lexicographicCompare`, and `hashCode` with the intrinsic-backed `Arrays.*` / `ByteBuffer.mismatch` APIs across all four `Binary` shape pairs (`byte[]`/`byte[]`, `byte[]`/`ByteBuffer`, `ByteBuffer`/`ByteBuffer`).
- Add a JMH suite `parquet-benchmarks/BinaryComparisonBenchmark` covering `equals` / `compareTo` / `hashCode` at length regimes 8, 64, and 512, both worst-case (full match) and average-case (mid-length mismatch).
Follows the same technique already accepted for `DeltaByteArrayWriter` in #3465. Complementary to the `Binary#hashCode` caching in #3566 — caching reduces call frequency, intrinsics speed up the calls that remain plus `equals` / `compareTo` which caching does not address.
### Are these changes tested?
Yes. Covered by the existing `Binary` unit tests (equality, comparison, hashing across all shape pairs) plus the new JMH suite. `compareTo` semantics are preserved — `Arrays.compareUnsigned` matches the contract of the previous loop (unsigned byte order, shorter-runs-first on prefix match).
### Are there any user-facing changes?
No API changes. Behavior is identical; only the implementation is faster.
**Benchmark** — JDK 17, 1 fork, 3×1s warmup, 5×1s measurement, throughput (ops/s):
| bench | len | before | after | speedup |
| --------------------------- | --: | -----: | ------: | ------: |
| `equalsMatch_bytesBytes` | 512 | 9.1 M | 39.1 M | 4.30× |
| `equalsMatch_bytesBytes` | 64 | 41.4 M | 160.9 M | 3.89× |
| `equalsMismatch_bytesBytes` | 512 | 14.6 M | 55.5 M | 3.79× |
| `compareTo_bytesBytes` | 512 | 15.1 M | 58.8 M | 3.91× |
| `compareTo_bytesBytes` | 64 | 59.0 M | 199.4 M | 3.38× |
| `equalsMismatch_bytesBuf` | 64 | 63.8 M | 190.2 M | 2.98× |
Wins scale with length — at len=512 the SIMD lane count amortizes fully (≈4×); at len=64 it's ≈3–4×; len=8 sees only 1.1–1.7× because there's barely enough work for one SIMD lane. `hashCode` is unchanged in JDK 17 numbers — the `31*h+b` polynomial has a serial cross-iteration dependency; HotSpot only gained a lane-split intrinsic (`ArraysSupport.vectorizedHashCode`) in JDK 21, so the new `Arrays.hashCode` call is equivalent on 17 and picks up the vectorization automatically on 21+.
<!-- Closes #${GITHUB_ISSUE_ID} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Use JDK intrinsics for Binary equality/comparison/hashing [parquet-java]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #3782:
URL: https://github.com/apache/parquet-java/pull/3782#issuecomment-5580939695
> Thanks for raising this PR @abstractdog, very interesting. Any way to verify using tests that we don't change hashes?
good catch! addressed in https://github.com/apache/parquet-java/pull/3782/changes/f94e189a52e9d2f9598131ec8c18bb32ab9f13dd
the test cases now validate against a fixed truth (generated by `scalarHashCode`), also checked pre/post patch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Use JDK intrinsics for Binary equality/comparison/hashing [parquet-java]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #3782:
URL: https://github.com/apache/parquet-java/pull/3782#issuecomment-5574714420
Thanks for raising this PR @abstractdog, very interesting. Any way to verify using tests that we don't change hashes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org