[GitHub] [incubator-druid] richardstartin edited a comment on issue #7408: upgrade to RoaringBitmap 0.8.0 and serialise directly to ByteBuffer

[GitBox <gi...@apache.org>]

richardstartin edited a comment on issue #7408: upgrade to RoaringBitmap 0.8.0 and serialise directly to ByteBuffer
URL: https://github.com/apache/incubator-druid/pull/7408#issuecomment-479694073
 
 
   I doubt druid bottlenecks on serialising bitmaps, but I wrote this new method in RoaringBitmap for an application that does, and it may be useful here. Here's some quick benchmark evidence, taking a look at how long `WrappedRoaringBitmap.serialize` takes before and after the change.
   
   ```java
     @Benchmark
     public void serialize(SerializeBenchmarkState state, Blackhole bh)
     {
       state.bitmap.serialize(state.buffer);
       bh.consume(state.buffer);
       state.buffer.flip();
     }
   
     @State(Scope.Benchmark)
     public static class SerializeBenchmarkState
     {
       WrappedRoaringBitmap bitmap;
       ByteBuffer buffer;
   
       @Setup
       public void setup(BitmapIterationBenchmark state)
       {
         bitmap = new WrappedRoaringBitmap();
         Random random = ThreadLocalRandom.current();
         for (int bit = 0; bit < state.size; bit++) {
           if (random.nextDouble() < state.prob) {
             bitmap.add(bit);
           }
         }
         buffer = ByteBuffer.allocate(bitmap.getSizeInBytes());
       }
     }
   ```
   
   ```
   8c104a115c4e08b2df60d940d6d85a7e79bef752 (incubator-druid:master)
   
   Benchmark                           (bitmapAlgo)  (prob)   (size)  Mode  Cnt       Score       Error  Units
   BitmapIterationBenchmark.serialize       roaring     0.0  1000000  avgt   10      29.110 ±     0.459  ns/op
   BitmapIterationBenchmark.serialize       roaring   0.001  1000000  avgt   10   10624.144 ±   139.993  ns/op
   BitmapIterationBenchmark.serialize       roaring     0.1  1000000  avgt   10  187265.856 ±  5515.013  ns/op
   BitmapIterationBenchmark.serialize       roaring     0.5  1000000  avgt   10  184694.297 ±  8353.848  ns/op
   BitmapIterationBenchmark.serialize       roaring    0.99  1000000  avgt   10  233570.949 ± 67602.280  ns/op
   BitmapIterationBenchmark.serialize       roaring     1.0  1000000  avgt   10    1083.100 ±    38.245  ns/op
   
   
   
   32673c121edea450b84ffef767506b60a49f4a23 (richardstartin:faster-roaring-serialize)
   Benchmark                           (bitmapAlgo)  (prob)   (size)  Mode  Cnt      Score     Error  Units
   BitmapIterationBenchmark.serialize       roaring     0.0  1000000  avgt   10     10.510 ±   0.182  ns/op
   BitmapIterationBenchmark.serialize       roaring   0.001  1000000  avgt   10    957.783 ±  12.215  ns/op
   BitmapIterationBenchmark.serialize       roaring     0.1  1000000  avgt   10  54361.198 ± 655.209  ns/op
   BitmapIterationBenchmark.serialize       roaring     0.5  1000000  avgt   10  53632.341 ± 377.866  ns/op
   BitmapIterationBenchmark.serialize       roaring    0.99  1000000  avgt   10  16405.135 ± 271.594  ns/op
   BitmapIterationBenchmark.serialize       roaring     1.0  1000000  avgt   10    404.527 ±   3.868  ns/op
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org