You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@datasketches.apache.org by "tisonkun (via GitHub)" <gi...@apache.org> on 2026/02/11 01:41:44 UTC

[I] Inconsistent BloomFilter.MAX_SIZE_BITS (datasketches-rust)

tisonkun opened a new issue, #89:
URL: https://github.com/apache/datasketches-rust/issues/89

   In [datasketches-java](https://github.com/apache/datasketches-java/blob/3def49b3ea13c779a3aef81f9ca8c54d8013a62d/src/main/java/org/apache/datasketches/filters/bloomfilter/BloomFilter.java#L65):
   
   ```java
   public static final long MAX_SIZE_BITS = (Integer.MAX_VALUE - Family.BLOOMFILTER.getMaxPreLongs()) * (long) Long.SIZE;
   // result in (Integer.MAX_VALUE - 4) * 64 = 137_438_953_152
   ```
   
   In [datasketches-cpp](https://github.com/apache/datasketches-cpp/blob/fccb2385f3a66416fc34c0d7bd0513696721ecd9/filters/include/bloom_filter.hpp#L577-L581):
   
   ```cpp
   static const uint64_t MAX_HEADER_SIZE_BYTES = 32; // 4 Java Longs
   static const uint64_t MAX_FILTER_SIZE_BITS = (INT32_MAX - MAX_HEADER_SIZE_BYTES) * sizeof(uint64_t);
   // result in (INT32_MAX - 32) * 8 = 17_179_868_920
   ```
   
   In [datasketches-go](https://github.com/apache/datasketches-go/blob/2cb3a8c970d797cf95d1efc4aa28b7cff38a29a8/filters/bloom_filter_builder.go#L62-L65):
   
   ```go
   maxBits := uint64(math.MaxInt32-32) * 64
   // result in 137_438_951_360
   ```
   
   In [datasketches-rust](https://github.com/apache/datasketches-rust/blob/9a830707410a66823e997b04cfd6e48890904957/datasketches/src/bloom/builder.rs#L22):
   
   ```rust
   const MAX_NUM_BITS: u64 = (1u64 << 35) - 64; // ~32 GB - reasonable limit 
   // result in 34_359_738_304
   ```
   
   None of them is equal.
   
   Besides, in datasketches-rust, there is a lower bound (`MIN_NUM_BITS`) = 64, while other impls check only `num_bits > 0`.
   
   @PsiACE where is the bound of bloomfilter comes from in datasketches-rust?
   
   @leerho @AlexanderSaydakov @freakyzoidberg @proost You may take a look and I wonder which one of the upper bound is the correct one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org


Re: [I] Inconsistent BloomFilter.MAX_SIZE_BITS (datasketches-rust)

Posted by "jmalkin (via GitHub)" <gi...@apache.org>.
jmalkin commented on issue #89:
URL: https://github.com/apache/datasketches-rust/issues/89#issuecomment-3887367083

   Sorry, yes, 2^31-1. And Java does indeed check if the serialized size would exceed what `toByteArray()` can handle and throws an exception saying to use `toLongArray()` instead.
   
   The implementations should standardize on Java’s limit. C++'s limit can be safely increased without breaking anyone’s existing filters.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org


Re: [I] Inconsistent BloomFilter.MAX_SIZE_BITS (datasketches-rust)

Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on issue #89:
URL: https://github.com/apache/datasketches-rust/issues/89#issuecomment-3883373028

   > Java is correct. Max array length in Java is 2^32-1, the longest contiguous array is of primitive longs, and there’s a little room taken for the preamble in the actual serialized sketch.
   
   @jmalkin Thanks for your information!
   
   > C++ has the byte count, not bit count.
   
   I can see the variable names are `num_filter_bits` and `MAX_FILTER_SIZE_BITS`. Does it mean the C++ limit is 8x less than Java's?
   
   (However, since in the real world few people would create a filter at that size, it wasn't a real problem in recent years. But I'd like to keep all impls as similar as possible to reduce unnecessary unalignments.)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org


Re: [I] Inconsistent BloomFilter.MAX_SIZE_BITS (datasketches-rust)

Posted by "leerho (via GitHub)" <gi...@apache.org>.
leerho commented on issue #89:
URL: https://github.com/apache/datasketches-rust/issues/89#issuecomment-3887291125

   > Java is correct. Max array length in Java is 2^32-1, the longest contiguous array is of primitive longs, and there’s a little room taken for the preamble in the actual serialized sketch.
   > 
   > C++ has the byte count, not bit count. I dunno the others.
   
   **NIT:** Max array length in Java is 2^31 - 1 :)
   
   This assumes that the BloomFilter is serialized using _toLongArray()_.  Attempting to serialize a filter this large will throw an exception if using _toByteArray()_. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org


Re: [I] Inconsistent BloomFilter.MAX_SIZE_BITS (datasketches-rust)

Posted by "jmalkin (via GitHub)" <gi...@apache.org>.
jmalkin commented on issue #89:
URL: https://github.com/apache/datasketches-rust/issues/89#issuecomment-3882272617

   Java is correct. Max array length in Java is 2^32-1, the longest contiguous array is of primitive longs, and there’s a little room taken for the preamble in the actual serialized sketch.
   
   C++ has the byte count, not bit count. I dunno the others.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org


Re: [I] Inconsistent BloomFilter.MAX_SIZE_BITS (datasketches-rust)

Posted by "proost (via GitHub)" <gi...@apache.org>.
proost commented on issue #89:
URL: https://github.com/apache/datasketches-rust/issues/89#issuecomment-3882272297

   What i remember is go might follow C++ implementation, but there is mistake(8 -> 64). @freakyzoidberg knows better than me.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org


Re: [I] Inconsistent BloomFilter.MAX_SIZE_BITS (datasketches-rust)

Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo commented on issue #89:
URL: https://github.com/apache/datasketches-rust/issues/89#issuecomment-3899048604

   Let's align on the Java limits. Creating a PR...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org


Re: [I] Inconsistent BloomFilter.MAX_SIZE_BITS (datasketches-rust)

Posted by "notfilippo (via GitHub)" <gi...@apache.org>.
notfilippo closed issue #89: Inconsistent BloomFilter.MAX_SIZE_BITS
URL: https://github.com/apache/datasketches-rust/issues/89


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org