You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by "quantranhong1999 (via GitHub)" <gi...@apache.org> on 2025/07/22 10:18:39 UTC

[PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]

quantranhong1999 opened a new pull request, #2770:
URL: https://github.com/apache/james-project/pull/2770

   Defaults to the current sha256 hashing algorithm.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]

Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on PR #2770:
URL: https://github.com/apache/james-project/pull/2770#issuecomment-3104859163

   I played a bit around and the java code for blake 3 seems indeed to suck.
   
   I played for fun with a JNI lib https://github.com/semantalytics/BLAKE3jni
   
   Results:
   
   ```
   Blake3Bench.measure_hashing_blake3_jni          sample  1115459     8.976 ± 0.014  us/op
   Blake3Bench.measure_hashing_sha256              sample   545709    36.512 ± 0.042  us/op
   ```
   
   => blake3 shines
   
   However their JNI binding looks crappy and output a constant performance degradation over time...
   
   ```
   Iteration   1: 9.693 ±(99.9%) 0.052 us/op
                    p0.00:   5.080 us/op
                    p0.50:   8.928 us/op
                    p0.90:   12.624 us/op
                    p0.95:   14.176 us/op
                    p0.99:   18.560 us/op
                    p0.999:  32.192 us/op
                    p0.9999: 119.298 us/op
                    p1.00:   5275.648 us/op
   
   Iteration   2: 18.870 ±(99.9%) 7.045 us/op
                    p0.00:   5.008 us/op
                    p0.50:   8.992 us/op
                    p0.90:   16.384 us/op
                    p0.95:   25.856 us/op
                    p0.99:   48.768 us/op
                    p0.999:  189.400 us/op
                    p0.9999: 1657.293 us/op
                    p1.00:   533725.184 us/op
   
   Iteration   3: 64.233 ±(99.9%) 26.584 us/op
                    p0.00:   5.000 us/op
                    p0.50:   9.312 us/op
                    p0.90:   31.232 us/op
                    p0.95:   37.568 us/op
                    p0.99:   67.456 us/op
                    p0.999:  206.109 us/op
                    p0.9999: 98005.051 us/op
                    p1.00:   723517.440 us/op
   
   Iteration   4: 64.656 ±(99.9%) 21.634 us/op
                    p0.00:   4.984 us/op
                    p0.50:   11.440 us/op
                    p0.90:   35.136 us/op
                    p0.95:   41.600 us/op
                    p0.99:   60.352 us/op
                    p0.999:  364.042 us/op
                    p0.9999: 98828.288 us/op
                    p1.00:   756023.296 us/op
   
   Iteration   5: 90.474 ±(99.9%) 45.996 us/op
                    p0.00:   5.000 us/op
                    p0.50:   9.168 us/op
                    p0.90:   23.094 us/op
                    p0.95:   35.648 us/op
                    p0.99:   60.992 us/op
                    p0.999:  138.337 us/op
                    p0.9999: 196646.535 us/op
                    p1.00:   835715.072 us/op
   ```
   
   They do not even have a bug tracker for me to report it... ... ... ... ...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]

Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 commented on PR #2770:
URL: https://github.com/apache/james-project/pull/2770#issuecomment-3102123951

   I am doing some JMH to compare Blake3 to SHA256. But the blake3 result (on Java libraries) is worse than the current SHA256 hashing, thus the PR is in draft state.
   
   I used 3 Java libraries that supports Blake3 in my JMH: Apache Common (current lib used in this PR), Bouncy Castle, and [Scala Blake 3](https://github.com/kcrypt/scala-blake3).
   
   JMH setup:
   ```java
       @Test
       public void launchBenchmark() throws Exception {
           Options opt = new OptionsBuilder()
               .include(this.getClass().getName() + ".measure*")
               .mode(Mode.SampleTime)
               .addProfiler(GCProfiler.class)
               .timeUnit(TimeUnit.MICROSECONDS)
               .warmupTime(TimeValue.seconds(5))
               .warmupIterations(2)
               .measurementTime(TimeValue.seconds(5))
               .measurementIterations(5)
               .threads(4)
               .forks(1)
               .shouldFailOnError(true)
               .shouldDoGC(true)
               .build();
   
           new Runner(opt).run();
       }
   
       @Benchmark
       public void measure_hashing_blake3_apache_common() {
           // https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/digest/Blake3.html
           Blake3.initHash().update(ELEVEN_KILOBYTES).doFinalize(16);
       }
   
       @Benchmark
       public void measure_hashing_blake3_bouncy_castle() {
           // https://downloads.bouncycastle.org/java/docs/bcprov-jdk14-javadoc/org/bouncycastle/crypto/digests/Blake3Digest.html
           byte[] output = new byte[16]; // 128-bit digest
           Blake3Digest blake3Digest = new Blake3Digest(128);
           blake3Digest.update(ELEVEN_KILOBYTES, 0, ELEVEN_KILOBYTES.length);
           blake3Digest.doFinal(output, 0);
       }
   
       @Benchmark
       public void measure_hashing_blake3_scala_blake3() {
           // https://github.com/kcrypt/scala-blake3
           byte[] output = new byte[16]; // 128-bit digest
           pt.kcry.blake3.Blake3.newHasher().update(ELEVEN_KILOBYTES).done(output);
       }
   
       @Benchmark
       public void measure_hashing_sha256() {
           Hashing.sha256().hashBytes(ELEVEN_KILOBYTES).asBytes();
       }
   ```
   
   Result:
   ```java
   Benchmark                                                                      Mode      Cnt       Score     Error   Units
   MemoryBlobStoreTest.measure_hashing_blake3_apache_common                     sample   552991     180.804 ±   2.772   us/op
   MemoryBlobStoreTest.measure_hashing_blake3_apache_common:gc.alloc.rate       sample        5     842.119 ± 732.480  MB/sec
   MemoryBlobStoreTest.measure_hashing_blake3_apache_common:gc.alloc.rate.norm  sample        5   40053.993 ±  21.755    B/op
   MemoryBlobStoreTest.measure_hashing_blake3_apache_common:gc.count            sample        5      32.000            counts
   MemoryBlobStoreTest.measure_hashing_blake3_apache_common:gc.time             sample        5      44.000                ms
   MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p0.00               sample               34.176             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p0.50               sample              138.496             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p0.90               sample              173.568             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p0.95               sample              357.888             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p0.99               sample              833.536             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p0.999              sample             5341.250             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p0.9999             sample            25716.431             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_apache_common:p1.00               sample           129499.136             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle                     sample   874642     114.237 ±   0.354   us/op
   MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:gc.alloc.rate       sample        5      35.387 ±  13.414  MB/sec
   MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:gc.alloc.rate.norm  sample        5    1062.509 ±   6.770    B/op
   MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:gc.count            sample        5         ≈ 0            counts
   MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p0.00               sample               52.800             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p0.50               sample              101.888             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p0.90               sample              121.856             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p0.95               sample              136.704             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p0.99               sample              391.680             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p0.999              sample             1323.008             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p0.9999             sample             3381.394             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_bouncy_castle:p1.00               sample            20283.392             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3                      sample  1557152      51.557 ±   0.288   us/op
   MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:gc.alloc.rate        sample        5     379.618 ±  99.400  MB/sec
   MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:gc.alloc.rate.norm   sample        5    5043.500 ±   3.057    B/op
   MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:gc.count             sample        5      12.000            counts
   MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:gc.time              sample        5      20.000                ms
   MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p0.00                sample               22.656             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p0.50                sample               44.928             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p0.90                sample               48.832             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p0.95                sample               68.480             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p0.99                sample              210.176             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p0.999               sample              623.616             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p0.9999              sample             4578.133             us/op
   MemoryBlobStoreTest.measure_hashing_blake3_scala_blake3:p1.00                sample            26345.472             us/op
   MemoryBlobStoreTest.measure_hashing_sha256                                   sample  2430830       5.222 ±   0.019   us/op
   MemoryBlobStoreTest.measure_hashing_sha256:gc.alloc.rate                     sample        5     331.547 ±  30.856  MB/sec
   MemoryBlobStoreTest.measure_hashing_sha256:gc.alloc.rate.norm                sample        5     448.269 ±   0.206    B/op
   MemoryBlobStoreTest.measure_hashing_sha256:gc.count                          sample        5      10.000            counts
   MemoryBlobStoreTest.measure_hashing_sha256:gc.time                           sample        5      12.000                ms
   MemoryBlobStoreTest.measure_hashing_sha256:p0.00                             sample                4.624             us/op
   MemoryBlobStoreTest.measure_hashing_sha256:p0.50                             sample                4.952             us/op
   MemoryBlobStoreTest.measure_hashing_sha256:p0.90                             sample                5.120             us/op
   MemoryBlobStoreTest.measure_hashing_sha256:p0.95                             sample                5.328             us/op
   MemoryBlobStoreTest.measure_hashing_sha256:p0.99                             sample                9.664             us/op
   MemoryBlobStoreTest.measure_hashing_sha256:p0.999                            sample               32.576             us/op
   MemoryBlobStoreTest.measure_hashing_sha256:p0.9999                           sample              171.499             us/op
   MemoryBlobStoreTest.measure_hashing_sha256:p1.00                             sample             6193.152             us/op
   ```
   
   The result ordering in terms of performance (response time, throughput, memory allocation pressure onto GC): Current SHA256 > Scala Blake 3 > Bouncy Castle > Apache Common.
   
   Blake3 claimed to be faster than SHA256. I do not fully understand why it is slower than SHA256 in our Java case. My guess is that Java code does not access CPU instructions directly like other languages e.g. C or Rust ...(Blake3 seems to rely on deeply on SIMD instructions).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]

Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on PR #2770:
URL: https://github.com/apache/james-project/pull/2770#issuecomment-3104758692

   Just saw the JMH benchmarks.
   
   Damn.
   
   > Blake3 claimed to be faster than SHA256. I do not fully understand why it is slower than SHA256 in our Java case. My guess is that Java code does not access CPU instructions directly like other languages e.g. C or Rust ...(Blake3 seems to rely on deeply on SIMD instructions).
   
   sha256 is litterally implemented by the VM with native code. It's thus heavily optimized. 
   
   FYI I found https://github.com/zalopay-oss/blake3-jni/ which tend to reflect this is a common issue.
   
   Ideas to move this bench forward:
   
    - wamup time is rather low. Have you tried warming up longer? Does it impact the results?
    - use a blackhole in order to consume the hashed value. Or the JVM is likely to optimize stuff away..
    - maybe one run with longer hashes on 11MB ?
    - ideally we should do computation on distinct arrays. Can we append something random ' iteration count at the end of the array just to be sure there's no cache effect?
    - Apparently using small buffers for blake3 can harm paralelism. Can we try with 256 size like sha ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]

Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 closed pull request #2770: JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore
URL: https://github.com/apache/james-project/pull/2770


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]

Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on PR #2770:
URL: https://github.com/apache/james-project/pull/2770#issuecomment-3104878618

   IMO we can move on.
   
   JNI pain is not worth for minor gains in hash computation.
   
   (dispointment...)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]

Posted by "chibenwa (via GitHub)" <gi...@apache.org>.
chibenwa commented on code in PR #2770:
URL: https://github.com/apache/james-project/pull/2770#discussion_r2222321687


##########
docs/modules/servers/partials/configure/jvm.adoc:
##########
@@ -212,6 +212,18 @@ james.deduplicating.blobstore.thread.switch.threshold=32768
 james.deduplicating.blobstore.file.threshold=10240
 ----
 
+== Change hashing algorithm of deduplication blob store
+
+The property `james.deduplicating.blobstore.hash.algorithm` allows to change the hashing algorithm.
+The support value are: `sha256`, `blake3`.
+
+Ex in `jvm.properties`
+----
+james.deduplicating.blobstore.hash.algorithm=sha256
+----
+
+Optional. String. By default, the deduplication blob store uses SHA-256 as the hashing algorithm.

Review Comment:
   Document other options 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org


Re: [PR] JAMES-4141 Add blake3 as a hashing option for DeDuplicationBlobStore [james-project]

Posted by "quantranhong1999 (via GitHub)" <gi...@apache.org>.
quantranhong1999 commented on PR #2770:
URL: https://github.com/apache/james-project/pull/2770#issuecomment-3106275166

   > IMO we can move on.
   JNI pain is not worth for minor gains in hash computation.
   
   I agree. We can close this PR IMO.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org