You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by "quantranhong1999 (via GitHub)" <gi...@apache.org> on 2025/07/22 10:18:39 UTC
[PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]
quantranhong1999 opened a new pull request, #2770:
URL: https://github.com/apache/james-project/pull/2770
Defaults to the current sha256 hashing algorithm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on PR #2770:
URL: https://github.com/apache/james-project/pull/2770#issuecomment-3104859163
I played a bit around and the java code for blake 3 seems indeed to suck.
I played for fun with a JNI lib https://github.com/semantalytics/BLAKE3jni
Results:
```
Blake3Bench.measure_hashing_blake3_jni sample 1115459 8.976 ± 0.014 us/op
Blake3Bench.measure_hashing_sha256 sample 545709 36.512 ± 0.042 us/op
```
=> blake3 shines
However their JNI binding looks crappy and output a constant performance degradation over time...
```
Iteration 1: 9.693 ±(99.9%) 0.052 us/op
p0.00: 5.080 us/op
p0.50: 8.928 us/op
p0.90: 12.624 us/op
p0.95: 14.176 us/op
p0.99: 18.560 us/op
p0.999: 32.192 us/op
p0.9999: 119.298 us/op
p1.00: 5275.648 us/op
Iteration 2: 18.870 ±(99.9%) 7.045 us/op
p0.00: 5.008 us/op
p0.50: 8.992 us/op
p0.90: 16.384 us/op
p0.95: 25.856 us/op
p0.99: 48.768 us/op
p0.999: 189.400 us/op
p0.9999: 1657.293 us/op
p1.00: 533725.184 us/op
Iteration 3: 64.233 ±(99.9%) 26.584 us/op
p0.00: 5.000 us/op
p0.50: 9.312 us/op
p0.90: 31.232 us/op
p0.95: 37.568 us/op
p0.99: 67.456 us/op
p0.999: 206.109 us/op
p0.9999: 98005.051 us/op
p1.00: 723517.440 us/op
Iteration 4: 64.656 ±(99.9%) 21.634 us/op
p0.00: 4.984 us/op
p0.50: 11.440 us/op
p0.90: 35.136 us/op
p0.95: 41.600 us/op
p0.99: 60.352 us/op
p0.999: 364.042 us/op
p0.9999: 98828.288 us/op
p1.00: 756023.296 us/op
Iteration 5: 90.474 ±(99.9%) 45.996 us/op
p0.00: 5.000 us/op
p0.50: 9.168 us/op
p0.90: 23.094 us/op
p0.95: 35.648 us/op
p0.99: 60.992 us/op
p0.999: 138.337 us/op
p0.9999: 196646.535 us/op
p1.00: 835715.072 us/op
```
They do not even have a bug tracker for me to report it... ... ... ... ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]
Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 commented on PR #2770:
URL: https://github.com/apache/james-project/pull/2770#issuecomment-3102123951
I am doing some JMH to compare Blake3 to SHA256. But the blake3 result (on Java libraries) is worse than the current SHA256 hashing, thus the PR is in draft state.
I used 3 Java libraries that supports Blake3 in my JMH: Apache Common (current lib used in this PR), Bouncy Castle, and [Scala Blake 3](https://github.com/kcrypt/scala-blake3).
JMH setup:
```java
@Test
public void launchBenchmark() throws Exception {
Options opt = new OptionsBuilder()
.include(this.getClass().getName() + ".measure*")
.mode(Mode.SampleTime)
.addProfiler(GCProfiler.class)
.timeUnit(TimeUnit.MICROSECONDS)
.warmupTime(TimeValue.seconds(5))
.warmupIterations(2)
.measurementTime(TimeValue.seconds(5))
.measurementIterations(5)
.threads(4)
.forks(1)
.shouldFailOnError(true)
.shouldDoGC(true)
.build();
new Runner(opt).run();
}
@Benchmark
public void measure_hashing_blake3_apache_common() {
// https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/digest/Blake3.html
Blake3.initHash().update(ELEVEN_KILOBYTES).doFinalize(16);
}
@Benchmark
public void measure_hashing_blake3_bouncy_castle() {
// https://downloads.bouncycastle.org/java/docs/bcprov-jdk14-javadoc/org/bouncycastle/crypto/digests/Blake3Digest.html
byte[] output = new byte[16]; // 128-bit digest
Blake3Digest blake3Digest = new Blake3Digest(128);
blake3Digest.update(ELEVEN_KILOBYTES, 0, ELEVEN_KILOBYTES.length);
blake3Digest.doFinal(output, 0);
}
@Benchmark
public void measure_hashing_blake3_scala_blake3() {
// https://github.com/kcrypt/scala-blake3
byte[] output = new byte[16]; // 128-bit digest
pt.kcry.blake3.Blake3.newHasher().update(ELEVEN_KILOBYTES).done(output);
}
@Benchmark
public void measure_hashing_sha256() {
Hashing.sha256().hashBytes(ELEVEN_KILOBYTES).asBytes();
}
```
Result:
```java
Benchmark Mode Cnt Score Error Units
MemoryBlobStoreTest.measure_hashing_blake3_apache_common sample 552991 180.804 ± 2.772 us/op
MemoryBlobStoreTest.measure_hashing_blake3_apache_common:gc.alloc.rate sample 5 842.119 ± 732.480 MB/sec
MemoryBlobStoreTest.measure_hashing_blake3_apache_common:gc.alloc.rate.norm sample 5 40053.993 ± 21.755 B/op
MemoryBlobStoreTest.measure_hashing_blake3_apache_common:gc.count sample 5 32.000 counts
MemoryBlobStoreTest.measure_hashing_blake3_apache_common:gc.time sample 5 44.000 ms
MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p0.00 sample 34.176 us/op
MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p0.50 sample 138.496 us/op
MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p0.90 sample 173.568 us/op
MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p0.95 sample 357.888 us/op
MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p0.99 sample 833.536 us/op
MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p0.999 sample 5341.250 us/op
MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p0.9999 sample 25716.431 us/op
MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p1.00 sample 129499.136 us/op
MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle sample 874642 114.237 ± 0.354 us/op
MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:gc.alloc.rate sample 5 35.387 ± 13.414 MB/sec
MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:gc.alloc.rate.norm sample 5 1062.509 ± 6.770 B/op
MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:gc.count sample 5 ≈ 0 counts
MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p0.00 sample 52.800 us/op
MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p0.50 sample 101.888 us/op
MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p0.90 sample 121.856 us/op
MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p0.95 sample 136.704 us/op
MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p0.99 sample 391.680 us/op
MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p0.999 sample 1323.008 us/op
MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p0.9999 sample 3381.394 us/op
MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p1.00 sample 20283.392 us/op
MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3 sample 1557152 51.557 ± 0.288 us/op
MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:gc.alloc.rate sample 5 379.618 ± 99.400 MB/sec
MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:gc.alloc.rate.norm sample 5 5043.500 ± 3.057 B/op
MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:gc.count sample 5 12.000 counts
MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:gc.time sample 5 20.000 ms
MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p0.00 sample 22.656 us/op
MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p0.50 sample 44.928 us/op
MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p0.90 sample 48.832 us/op
MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p0.95 sample 68.480 us/op
MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p0.99 sample 210.176 us/op
MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p0.999 sample 623.616 us/op
MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p0.9999 sample 4578.133 us/op
MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p1.00 sample 26345.472 us/op
MemoryBlobStoreTest.measure_hashing_sha256 sample 2430830 5.222 ± 0.019 us/op
MemoryBlobStoreTest.measure_hashing_sha256:gc.alloc.rate sample 5 331.547 ± 30.856 MB/sec
MemoryBlobStoreTest.measure_hashing_sha256:gc.alloc.rate.norm sample 5 448.269 ± 0.206 B/op
MemoryBlobStoreTest.measure_hashing_sha256:gc.count sample 5 10.000 counts
MemoryBlobStoreTest.measure_hashing_sha256:gc.time sample 5 12.000 ms
MemoryBlobStoreTest.measure_hashing_sha256:p0.00 sample 4.624 us/op
MemoryBlobStoreTest.measure_hashing_sha256:p0.50 sample 4.952 us/op
MemoryBlobStoreTest.measure_hashing_sha256:p0.90 sample 5.120 us/op
MemoryBlobStoreTest.measure_hashing_sha256:p0.95 sample 5.328 us/op
MemoryBlobStoreTest.measure_hashing_sha256:p0.99 sample 9.664 us/op
MemoryBlobStoreTest.measure_hashing_sha256:p0.999 sample 32.576 us/op
MemoryBlobStoreTest.measure_hashing_sha256:p0.9999 sample 171.499 us/op
MemoryBlobStoreTest.measure_hashing_sha256:p1.00 sample 6193.152 us/op
```
The result ordering in terms of performance (response time, throughput, memory allocation pressure onto GC): Current SHA256 > Scala Blake 3 > Bouncy Castle > Apache Common.
Blake3 claimed to be faster than SHA256. I do not fully understand why it is slower than SHA256 in our Java case. My guess is that Java code does not access CPU instructions directly like other languages e.g. C or Rust ...(Blake3 seems to rely on deeply on SIMD instructions).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on PR #2770:
URL: https://github.com/apache/james-project/pull/2770#issuecomment-3104758692
Just saw the JMH benchmarks.
Damn.
> Blake3 claimed to be faster than SHA256. I do not fully understand why it is slower than SHA256 in our Java case. My guess is that Java code does not access CPU instructions directly like other languages e.g. C or Rust ...(Blake3 seems to rely on deeply on SIMD instructions).
sha256 is litterally implemented by the VM with native code. It's thus heavily optimized.
FYI I found https://github.com/zalopay-oss/blake3-jni/ which tend to reflect this is a common issue.
Ideas to move this bench forward:
- wamup time is rather low. Have you tried warming up longer? Does it impact the results?
- use a blackhole in order to consume the hashed value. Or the JVM is likely to optimize stuff away..
- maybe one run with longer hashes on 11MB ?
- ideally we should do computation on distinct arrays. Can we append something random ' iteration count at the end of the array just to be sure there's no cache effect?
- Apparently using small buffers for blake3 can harm paralelism. Can we try with 256 size like sha ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]
Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 closed pull request #2770: JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore
URL: https://github.com/apache/james-project/pull/2770
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on PR #2770:
URL: https://github.com/apache/james-project/pull/2770#issuecomment-3104878618
IMO we can move on.
JNI pain is not worth for minor gains in hash computation.
(dispointment...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]
Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2770:
URL: https://github.com/apache/james-project/pull/2770#discussion_r2222321687
##########
docs/modules/servers/partials/configure/jvm.adoc:
##########
@@ -212,6 +212,18 @@ james.deduplicating.blobstore.thread.switch.threshold=32768
james.deduplicating.blobstore.file.threshold=10240
----
+== Change hashing algorithm of deduplication blob store
+
+The property `james.deduplicating.blobstore.hash.algorithm` allows to change the hashing algorithm.
+The support value are: `sha256`, `blake3`.
+
+Ex in `jvm.properties`
+----
+james.deduplicating.blobstore.hash.algorithm=sha256
+----
+
+Optional. String. By default, the deduplication blob store uses SHA-256 as the hashing algorithm.
Review Comment:
Document other options
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org
Re: [PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]
Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 commented on PR #2770:
URL: https://github.com/apache/james-project/pull/2770#issuecomment-3106275166
> IMO we can move on.
JNI pain is not worth for minor gains in hash computation.
I agree. We can close this PR IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org