You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "aherbert (via GitHub)" <gi...@apache.org> on 2023/07/04 13:26:34 UTC
[GitHub] [commons-collections] aherbert commented on a diff in pull request #402: COLLECTIONS-843: Implement Layered Bloom filter
aherbert commented on code in PR #402:
URL: https://github.com/apache/commons-collections/pull/402#discussion_r1251835820
##########
src/main/java/org/apache/commons/collections4/bloomfilter/Shape.java:
##########
@@ -227,6 +227,15 @@ public double estimateN(final int cardinality) {
return -(m / k) * Math.log1p(-c / m);
}
+ /**
+ * Estimates the maximum number of elements that can be merged into a filter of this shape before
+ * it begins to return excessive false positives.
+ * @return An estimate of max N.
+ */
+ public double estimateMaxN() {
+ return 1.0/(numberOfHashFunctions/LN_2/numberOfBits);
Review Comment:
Whitepace formatting:
```
return 1.0 / (numberOfHashFunctions / LN_2 / numberOfBits);
```
##########
src/test/java/org/apache/commons/collections4/bloomfilter/TestingHashers.java:
##########
@@ -88,4 +90,18 @@ public static <T extends BloomFilter> T populateRange(T filter, int start, int e
});
return filter;
}
+
+ private static Random random;
Review Comment:
I would drop this 'Random'. The class is not a good RNG. It's only merit is being thread safe. So just use `ThreadLocalRandom.current()` where you use the random instance.
##########
src/test/java/org/apache/commons/collections4/bloomfilter/DefaultBloomFilterTest.java:
##########
Review Comment:
These changes are not required. They only make items public. These are not used outside of the package and so should remain package private.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
Review Comment:
Is 'last' redundant here?
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
Review Comment:
Redundant line
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
+ };
+
+ /**
+ * Removes the earliest filters in the list when the the number of filters
+ * exceeds maxSize.
+ *
+ * @param maxSize the maximum number of filters for the list.
+ * @return A Consumer for the LayerManager filterCleanup constructor argument.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> onMaxSize(int maxSize) {
+ return (ll) -> {
+ while (ll.size() > maxSize) {
+ ll.removeFirst();
+ }
+ };
+ }
+ }
+
+ private final LinkedList<BloomFilter> filters = new LinkedList<>();
+ private final Consumer<LinkedList<BloomFilter>> filterCleanup;
+ private final Predicate<LayerManager> extendCheck;
+ private final Supplier<BloomFilter> filterSupplier;
+
+ /**
+ * Creates a new Builder with defaults of NEVER_ADVANCE and NO_CLEANUP
+ *
+ * @return A builder.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param filterSupplier the supplier of new Bloom filters to add the the list
+ * when necessary.
+ * @param extendCheck The predicate that checks if a new filter should be
+ * added to the list.
+ * @param filterCleanup the consumer the removes any old filters from the list.
+ */
+ private LayerManager(Supplier<BloomFilter> filterSupplier, Predicate<LayerManager> extendCheck,
+ Consumer<LinkedList<BloomFilter>> filterCleanup) {
+ this.filterSupplier = filterSupplier;
+ this.extendCheck = extendCheck;
+ this.filterCleanup = filterCleanup;
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Creates a deep copy of this LayerManager.
+ *
+ * @return a copy of this layer Manager.
+ */
+ public LayerManager copy() {
+ LayerManager newMgr = new LayerManager(filterSupplier, extendCheck, filterCleanup);
+ newMgr.filters.clear();
+ for (BloomFilter bf : filters) {
+ newMgr.filters.add(bf.copy());
+ }
+ return newMgr;
+ }
+
+ /**
+ * Forces an advance to the next depth for subsequent merges. Executes the same
+ * logic as when {@code ExtendCheck} returns {@code true}
+ */
+ public void next() {
+ if (!filters.isEmpty() && filters.getLast().cardinality() == 0) {
+ filters.removeLast();
+ }
+ this.filterCleanup.accept(filters);
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Returns the number of filters in the LayerManager.
+ *
+ * @return the current depth.
+ */
+ public final int getDepth() {
+ return filters.size();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth. The filter at depth 0 is the
+ * oldest filter.
+ *
+ * @param depth the depth at which the desired filter is to be found.
+ * @return the filter.
+ * @throws NoSuchElementException if depth is not in the range
+ * [0,filters.size())
+ */
+ public final BloomFilter get(int depth) {
+ if (depth < 0 || depth >= filters.size()) {
+ throw new NoSuchElementException(String.format("Depth must be in the range [0,%s)", filters.size()));
+ }
+ return filters.get(depth);
+ }
+
+ /**
+ * Returns the current target filter. If the a new filter should be created
+ * based on {@code extendCheck} it will be created before this method returns.
+ *
+ * @return the current target filter.
+ */
+ public final BloomFilter target() {
+ if (extendCheck.test(this)) {
+ next();
+ }
+ return filters.peekLast();
+ }
+
+ /**
+ * Clear all the filters in the layer manager, and set up a new one as the
+ * target.
+ */
+ public final void clear() {
+ filters.clear();
+ next();
+ }
+
+ /**
+ * Executes a Bloom filter Predicate on each Bloom filter in the manager in
+ * depth order. Oldest filter first.
+ *
+ * @param bloomFilterPredicate the predicate to evaluate each Bloom filter with.
+ * @return {@code false} when the first filter fails the predicate test. Returns
+ * {@code true} if all filters pass the test.
+ */
+ @Override
+ public boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate) {
+ for (BloomFilter bf : filters) {
+ if (!bloomFilterPredicate.test(bf)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Builder to create Layer Manager
+ */
+ public static class Builder {
+ private Predicate<LayerManager> extendCheck;
+ private Supplier<BloomFilter> supplier;
+ private Consumer<LinkedList<BloomFilter>> cleanup;
+
+ private Builder() {
+ extendCheck = ExtendCheck.NEVER_ADVANCE;
+ cleanup = Cleanup.NO_CLEANUP;
+ }
+
+ public LayerManager build() {
+ if (supplier == null) {
+ throw new IllegalStateException("Supplier must not be null");
+ }
+ if (extendCheck == null) {
+ throw new IllegalStateException("ExtendCheck must not be null");
+ }
+ if (cleanup == null) {
+ throw new IllegalStateException("Cleanup must not be null");
+ }
+ return new LayerManager(supplier, extendCheck, cleanup);
+ }
+
+ public Builder withExtendCheck(Predicate<LayerManager> extendCheck) {
+ this.extendCheck = extendCheck;
+ return this;
+ }
+
+ public Builder withSuplier(Supplier<BloomFilter> supplier) {
Review Comment:
Spelling! `withSupplier`
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
+ };
+
+ /**
+ * Removes the earliest filters in the list when the the number of filters
+ * exceeds maxSize.
+ *
+ * @param maxSize the maximum number of filters for the list.
+ * @return A Consumer for the LayerManager filterCleanup constructor argument.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> onMaxSize(int maxSize) {
+ return (ll) -> {
+ while (ll.size() > maxSize) {
+ ll.removeFirst();
+ }
+ };
+ }
+ }
+
+ private final LinkedList<BloomFilter> filters = new LinkedList<>();
+ private final Consumer<LinkedList<BloomFilter>> filterCleanup;
+ private final Predicate<LayerManager> extendCheck;
+ private final Supplier<BloomFilter> filterSupplier;
+
+ /**
+ * Creates a new Builder with defaults of NEVER_ADVANCE and NO_CLEANUP
+ *
+ * @return A builder.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param filterSupplier the supplier of new Bloom filters to add the the list
+ * when necessary.
+ * @param extendCheck The predicate that checks if a new filter should be
+ * added to the list.
+ * @param filterCleanup the consumer the removes any old filters from the list.
+ */
+ private LayerManager(Supplier<BloomFilter> filterSupplier, Predicate<LayerManager> extendCheck,
+ Consumer<LinkedList<BloomFilter>> filterCleanup) {
+ this.filterSupplier = filterSupplier;
+ this.extendCheck = extendCheck;
+ this.filterCleanup = filterCleanup;
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Creates a deep copy of this LayerManager.
+ *
+ * @return a copy of this layer Manager.
+ */
+ public LayerManager copy() {
+ LayerManager newMgr = new LayerManager(filterSupplier, extendCheck, filterCleanup);
+ newMgr.filters.clear();
+ for (BloomFilter bf : filters) {
+ newMgr.filters.add(bf.copy());
+ }
+ return newMgr;
+ }
+
+ /**
+ * Forces an advance to the next depth for subsequent merges. Executes the same
+ * logic as when {@code ExtendCheck} returns {@code true}
+ */
+ public void next() {
+ if (!filters.isEmpty() && filters.getLast().cardinality() == 0) {
+ filters.removeLast();
+ }
+ this.filterCleanup.accept(filters);
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Returns the number of filters in the LayerManager.
+ *
+ * @return the current depth.
+ */
+ public final int getDepth() {
+ return filters.size();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth. The filter at depth 0 is the
+ * oldest filter.
+ *
+ * @param depth the depth at which the desired filter is to be found.
+ * @return the filter.
+ * @throws NoSuchElementException if depth is not in the range
+ * [0,filters.size())
+ */
+ public final BloomFilter get(int depth) {
+ if (depth < 0 || depth >= filters.size()) {
+ throw new NoSuchElementException(String.format("Depth must be in the range [0,%s)", filters.size()));
+ }
+ return filters.get(depth);
+ }
+
+ /**
+ * Returns the current target filter. If the a new filter should be created
+ * based on {@code extendCheck} it will be created before this method returns.
+ *
+ * @return the current target filter.
+ */
+ public final BloomFilter target() {
+ if (extendCheck.test(this)) {
+ next();
+ }
+ return filters.peekLast();
+ }
+
+ /**
+ * Clear all the filters in the layer manager, and set up a new one as the
+ * target.
+ */
+ public final void clear() {
+ filters.clear();
+ next();
+ }
+
+ /**
+ * Executes a Bloom filter Predicate on each Bloom filter in the manager in
+ * depth order. Oldest filter first.
+ *
+ * @param bloomFilterPredicate the predicate to evaluate each Bloom filter with.
+ * @return {@code false} when the first filter fails the predicate test. Returns
+ * {@code true} if all filters pass the test.
+ */
+ @Override
+ public boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate) {
+ for (BloomFilter bf : filters) {
+ if (!bloomFilterPredicate.test(bf)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Builder to create Layer Manager
+ */
+ public static class Builder {
Review Comment:
public methods in the Builder should have javadoc.
The `with` pattern is typically used when a **new** instance of the same object type will be returned with a changed value for the property targeted by the method. If you are returning the same builder then you can use `setXXX` or for a less verbose builder `XXX` (as in property setters/getters):
```java
LayerManager manager = LayerManager.builder()
.setExtendCheck(LayerManager.ExtendCheck.ADVANCE_ON_POPULATED)
.setCcleanup(LayerManager.Cleanup.onMaxSize(maxDepth))
.setSupplier(() -> new SimpleBloomFilter(shape))
.build();
```
The second style is less java-like so I would use `setXXX`.
Note: I wonder about the utility of this builder. It has 3 properties and all must be set. What is the value of it? Do you wish to reuse the builder several times, possibly changing less than all the properties for each build?
##########
src/main/java/org/apache/commons/collections4/bloomfilter/BloomFilterProducer.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.function.BiPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Produces Bloom filters that are copies of Bloom filters in a collection (e.g.
Review Comment:
The statement `that are copies` is not true. There is no mandate to generate copies here. The only implementation (LayerManager) passes the filters by reference.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/BloomFilterProducer.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.function.BiPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Produces Bloom filters that are copies of Bloom filters in a collection (e.g.
+ * LayerBloomFilter).
+ *
+ * @since 4.5
+ */
+public interface BloomFilterProducer {
+ /**
+ * Executes a Bloom filter Predicate on each Bloom filter in the manager in
+ * depth order. Oldest filter first.
+ *
+ * @param bloomFilterPredicate the predicate to evaluate each Bloom filter with.
+ * @return {@code false} when the first filter fails the predicate test. Returns
+ * {@code true} if all filters pass the test.
+ */
+ boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate);
+
+ /**
+ * Return a deep copy of the BloomFilterProducer data as a Bloom filter array.
+ * <p>
+ * The default implementation of this method is slow. It is recommended that
+ * implementing classes reimplement this method.
+ * </p>
+ *
+ * @return An array of Bloom filters.
+ */
+ default BloomFilter[] asBloomFilterArray() {
+ class Filters {
+ private BloomFilter[] data = new BloomFilter[16];
+ private int size;
+
+ boolean add(final BloomFilter filter) {
+ if (size == data.length) {
+ // This will throw an out-of-memory error if there are too many Bloom filters.
+ data = Arrays.copyOf(data, size * 2);
+ }
+ data[size++] = filter.copy();
+ return true;
+ }
+
+ BloomFilter[] toArray() {
+ // Edge case to avoid a large array copy
+ return size == data.length ? data : Arrays.copyOf(data, size);
+ }
+ }
+ final Filters filters = new Filters();
+ forEachBloomFilter(filters::add);
+ return filters.toArray();
+ }
+
+ /**
+ * Applies the {@code func} to each Bloom filter pair in order. Will apply all
+ * of the Bloom filters from the other BloomFilterProducer to this producer. If
+ * this producer does not have as many BloomFilters it will provide
+ * {@code null} for all excess calls to the BiPredicate.
+ *
+ * @param other The other BloomFilterProducer that provides the y values in the
+ * (x,y) pair.
+ * @param func The function to apply.
+ * @return A LongPredicate that tests this BitMapProducers bitmap values in
Review Comment:
Wrong return comment
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
Review Comment:
This type of functionality makes me think we should have a `BloomFilter.isEmpty` method in the interface. It can use this `forEachBitMap` code as a default. But implementations may know better, e.g. the SparseBloomFilter.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
Review Comment:
Javadoc `{@link }` to the other method.
Do we require two methods? The only difference seems to be that the `int` version uses a rounded version of the double returned from the Shape estimate.
##########
src/test/java/org/apache/commons/collections4/bloomfilter/TestingHashers.java:
##########
@@ -88,4 +90,18 @@ public static <T extends BloomFilter> T populateRange(T filter, int start, int e
});
return filter;
}
+
+ private static Random random;
+
+ static {
+ random = new Random();
+ random.setSeed( System.currentTimeMillis());
Review Comment:
No trailing space after '('. See also the same a few lines below.
##########
src/test/java/org/apache/commons/collections4/bloomfilter/TestingHashers.java:
##########
@@ -79,7 +81,7 @@ public static <T extends BloomFilter> T populateEntireFilter(T filter) {
*/
public static <T extends BloomFilter> T populateRange(T filter, int start, int end) {
filter.merge((IndexProducer) p -> {
- for (int i = start; i <= end; i++) {
+ for (int i=start; i<=end; i++) {
Review Comment:
This formatting change should be reverted
##########
src/main/java/org/apache/commons/collections4/bloomfilter/Shape.java:
##########
@@ -227,6 +227,15 @@ public double estimateN(final int cardinality) {
return -(m / k) * Math.log1p(-c / m);
}
+ /**
+ * Estimates the maximum number of elements that can be merged into a filter of this shape before
Review Comment:
The wording 'excessive false positives' is vague. Is this quantifiable? If this is a standard bloom filter computation then we should add the equation to the docs and some quoted reference as to the 'excessive' value. E.g. Is it half the false positive rate when empty, etc.
##########
src/test/java/org/apache/commons/collections4/bloomfilter/TestingHashers.java:
##########
@@ -88,4 +90,18 @@ public static <T extends BloomFilter> T populateRange(T filter, int start, int e
});
return filter;
}
+
+ private static Random random;
+
+ static {
+ random = new Random();
+ random.setSeed( System.currentTimeMillis());
+ }
+
+ /**
+ * Creates an EnhancedDoubleHasher hasher from 2 random longs.
+ */
+ public static Hasher randomHasher() {
Review Comment:
Since everything is in the same package then this doesn't have to be public.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
Review Comment:
Missing `<p>`
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
Review Comment:
This comment should be reversed. It uses the double estimation from the Shape, ...
##########
src/main/java/org/apache/commons/collections4/bloomfilter/BloomFilterProducer.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.function.BiPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Produces Bloom filters that are copies of Bloom filters in a collection (e.g.
+ * LayerBloomFilter).
+ *
+ * @since 4.5
+ */
+public interface BloomFilterProducer {
+ /**
+ * Executes a Bloom filter Predicate on each Bloom filter in the manager in
Review Comment:
This discusses 'manager' and 'depth order'. This javadoc should be moved to the LayerManager. The generic interface is not restricted to processing in any order.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
+ };
+
+ /**
+ * Removes the earliest filters in the list when the the number of filters
+ * exceeds maxSize.
+ *
+ * @param maxSize the maximum number of filters for the list.
+ * @return A Consumer for the LayerManager filterCleanup constructor argument.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> onMaxSize(int maxSize) {
+ return (ll) -> {
+ while (ll.size() > maxSize) {
+ ll.removeFirst();
+ }
+ };
+ }
+ }
+
+ private final LinkedList<BloomFilter> filters = new LinkedList<>();
+ private final Consumer<LinkedList<BloomFilter>> filterCleanup;
+ private final Predicate<LayerManager> extendCheck;
+ private final Supplier<BloomFilter> filterSupplier;
+
+ /**
+ * Creates a new Builder with defaults of NEVER_ADVANCE and NO_CLEANUP
+ *
+ * @return A builder.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param filterSupplier the supplier of new Bloom filters to add the the list
+ * when necessary.
+ * @param extendCheck The predicate that checks if a new filter should be
+ * added to the list.
+ * @param filterCleanup the consumer the removes any old filters from the list.
+ */
+ private LayerManager(Supplier<BloomFilter> filterSupplier, Predicate<LayerManager> extendCheck,
+ Consumer<LinkedList<BloomFilter>> filterCleanup) {
+ this.filterSupplier = filterSupplier;
+ this.extendCheck = extendCheck;
+ this.filterCleanup = filterCleanup;
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Creates a deep copy of this LayerManager.
+ *
+ * @return a copy of this layer Manager.
+ */
+ public LayerManager copy() {
+ LayerManager newMgr = new LayerManager(filterSupplier, extendCheck, filterCleanup);
+ newMgr.filters.clear();
+ for (BloomFilter bf : filters) {
+ newMgr.filters.add(bf.copy());
+ }
+ return newMgr;
+ }
+
+ /**
+ * Forces an advance to the next depth for subsequent merges. Executes the same
+ * logic as when {@code ExtendCheck} returns {@code true}
+ */
+ public void next() {
+ if (!filters.isEmpty() && filters.getLast().cardinality() == 0) {
+ filters.removeLast();
+ }
+ this.filterCleanup.accept(filters);
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Returns the number of filters in the LayerManager.
+ *
+ * @return the current depth.
+ */
+ public final int getDepth() {
+ return filters.size();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth. The filter at depth 0 is the
+ * oldest filter.
+ *
+ * @param depth the depth at which the desired filter is to be found.
+ * @return the filter.
+ * @throws NoSuchElementException if depth is not in the range
+ * [0,filters.size())
+ */
+ public final BloomFilter get(int depth) {
+ if (depth < 0 || depth >= filters.size()) {
+ throw new NoSuchElementException(String.format("Depth must be in the range [0,%s)", filters.size()));
+ }
+ return filters.get(depth);
+ }
+
+ /**
+ * Returns the current target filter. If the a new filter should be created
+ * based on {@code extendCheck} it will be created before this method returns.
+ *
+ * @return the current target filter.
+ */
+ public final BloomFilter target() {
+ if (extendCheck.test(this)) {
+ next();
+ }
+ return filters.peekLast();
+ }
+
+ /**
+ * Clear all the filters in the layer manager, and set up a new one as the
+ * target.
+ */
+ public final void clear() {
+ filters.clear();
+ next();
+ }
+
+ /**
+ * Executes a Bloom filter Predicate on each Bloom filter in the manager in
+ * depth order. Oldest filter first.
+ *
+ * @param bloomFilterPredicate the predicate to evaluate each Bloom filter with.
+ * @return {@code false} when the first filter fails the predicate test. Returns
+ * {@code true} if all filters pass the test.
+ */
+ @Override
+ public boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate) {
+ for (BloomFilter bf : filters) {
+ if (!bloomFilterPredicate.test(bf)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Builder to create Layer Manager
+ */
+ public static class Builder {
+ private Predicate<LayerManager> extendCheck;
+ private Supplier<BloomFilter> supplier;
+ private Consumer<LinkedList<BloomFilter>> cleanup;
+
+ private Builder() {
+ extendCheck = ExtendCheck.NEVER_ADVANCE;
+ cleanup = Cleanup.NO_CLEANUP;
+ }
+
+ public LayerManager build() {
+ if (supplier == null) {
Review Comment:
I think that NPE may be a more appropriate exception here.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
Review Comment:
Simplify to `Perform tests on whether to reduce a collection of Bloom filters.`
Should this be typed to LayerManager instead of dealing with a Collection? It seems strange to have the ExtendCheck work with a LayerManager and the Cleanup work on a collection.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
+ };
+
+ /**
+ * Removes the earliest filters in the list when the the number of filters
+ * exceeds maxSize.
+ *
+ * @param maxSize the maximum number of filters for the list.
+ * @return A Consumer for the LayerManager filterCleanup constructor argument.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> onMaxSize(int maxSize) {
+ return (ll) -> {
+ while (ll.size() > maxSize) {
+ ll.removeFirst();
+ }
+ };
+ }
+ }
+
+ private final LinkedList<BloomFilter> filters = new LinkedList<>();
+ private final Consumer<LinkedList<BloomFilter>> filterCleanup;
+ private final Predicate<LayerManager> extendCheck;
+ private final Supplier<BloomFilter> filterSupplier;
+
+ /**
+ * Creates a new Builder with defaults of NEVER_ADVANCE and NO_CLEANUP
+ *
+ * @return A builder.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param filterSupplier the supplier of new Bloom filters to add the the list
+ * when necessary.
+ * @param extendCheck The predicate that checks if a new filter should be
+ * added to the list.
+ * @param filterCleanup the consumer the removes any old filters from the list.
+ */
+ private LayerManager(Supplier<BloomFilter> filterSupplier, Predicate<LayerManager> extendCheck,
+ Consumer<LinkedList<BloomFilter>> filterCleanup) {
+ this.filterSupplier = filterSupplier;
+ this.extendCheck = extendCheck;
+ this.filterCleanup = filterCleanup;
+ filters.add(this.filterSupplier.get());
Review Comment:
This method is a potential source of a programming error since the supplier could return null. I suggest we help out here by refactoring `filters.add(this.filterSupplier.get());` (which is used twice) to a new method that checks the supplied filter is not null.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
Review Comment:
Simplify to `Perform tests on whether to extend the depth of a LayerManager.`
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
+ * items than the Shape would indicate and yet still return a false positive
+ * rate in line with the Shape and not the over population.
+ * <p>
+ * This implementation uses a LayerManager to handle the manipulation of the
+ * layers.
+ * </p>
+ * <ul>
+ * <li>Level 0 is the oldest layer and the highest level is the newest.</li>
+ * <li>There is always at least one enclosed filter.</li>
+ * <li>The newest filter is the {@code target} into which merges are performed.
+ * <li>Whenever the target is retrieved, or a {@code merge} operation is
+ * performed the code checks if any older layers should be removed, and if so
+ * removes them. It also checks it a new layer should be added, and if so adds
+ * it and sets the {@code target} before the operation.</li>
+ * </ul>
+ */
+public class LayeredBloomFilter implements BloomFilter, BloomFilterProducer {
+ private final Shape shape;
+ private LayerManager layerManager;
+
+ /**
+ * Creates a fixed size layered bloom filter that adds new filters to the list,
+ * but never merges them. List will never exceed maxDepth. As additional filters
+ * are added earlier filters are removed.
+ *
+ * @param shape The shape for the enclosed Bloom filters
Review Comment:
Missing a period at the end
##########
src/main/java/org/apache/commons/collections4/bloomfilter/BloomFilterProducer.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.function.BiPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Produces Bloom filters that are copies of Bloom filters in a collection (e.g.
+ * LayerBloomFilter).
+ *
+ * @since 4.5
+ */
+public interface BloomFilterProducer {
+ /**
+ * Executes a Bloom filter Predicate on each Bloom filter in the manager in
+ * depth order. Oldest filter first.
+ *
+ * @param bloomFilterPredicate the predicate to evaluate each Bloom filter with.
+ * @return {@code false} when the first filter fails the predicate test. Returns
+ * {@code true} if all filters pass the test.
+ */
+ boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate);
+
+ /**
+ * Return a deep copy of the BloomFilterProducer data as a Bloom filter array.
+ * <p>
+ * The default implementation of this method is slow. It is recommended that
+ * implementing classes reimplement this method.
+ * </p>
+ *
+ * @return An array of Bloom filters.
+ */
+ default BloomFilter[] asBloomFilterArray() {
+ class Filters {
Review Comment:
I think this is not necessary. It has been borrowed from special handling of primitive arrays in BitMapProducer. I think we can use ArrayList here since we are dealing with objects:
```java
final ArrayList<BloomFilter> filters = new ArrayList<>();
forEachBloomFilter(f -> filters.add(f.copy()));
return filters.toArray(new BloomFilter[0]);
```
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
+ };
+
+ /**
+ * Removes the earliest filters in the list when the the number of filters
+ * exceeds maxSize.
+ *
+ * @param maxSize the maximum number of filters for the list.
+ * @return A Consumer for the LayerManager filterCleanup constructor argument.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> onMaxSize(int maxSize) {
+ return (ll) -> {
+ while (ll.size() > maxSize) {
+ ll.removeFirst();
+ }
+ };
+ }
+ }
+
+ private final LinkedList<BloomFilter> filters = new LinkedList<>();
+ private final Consumer<LinkedList<BloomFilter>> filterCleanup;
+ private final Predicate<LayerManager> extendCheck;
+ private final Supplier<BloomFilter> filterSupplier;
+
+ /**
+ * Creates a new Builder with defaults of NEVER_ADVANCE and NO_CLEANUP
+ *
+ * @return A builder.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param filterSupplier the supplier of new Bloom filters to add the the list
+ * when necessary.
+ * @param extendCheck The predicate that checks if a new filter should be
+ * added to the list.
+ * @param filterCleanup the consumer the removes any old filters from the list.
+ */
+ private LayerManager(Supplier<BloomFilter> filterSupplier, Predicate<LayerManager> extendCheck,
+ Consumer<LinkedList<BloomFilter>> filterCleanup) {
+ this.filterSupplier = filterSupplier;
+ this.extendCheck = extendCheck;
+ this.filterCleanup = filterCleanup;
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Creates a deep copy of this LayerManager.
+ *
+ * @return a copy of this layer Manager.
+ */
+ public LayerManager copy() {
+ LayerManager newMgr = new LayerManager(filterSupplier, extendCheck, filterCleanup);
+ newMgr.filters.clear();
+ for (BloomFilter bf : filters) {
+ newMgr.filters.add(bf.copy());
+ }
+ return newMgr;
+ }
+
+ /**
+ * Forces an advance to the next depth for subsequent merges. Executes the same
+ * logic as when {@code ExtendCheck} returns {@code true}
+ */
+ public void next() {
+ if (!filters.isEmpty() && filters.getLast().cardinality() == 0) {
+ filters.removeLast();
+ }
+ this.filterCleanup.accept(filters);
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Returns the number of filters in the LayerManager.
+ *
+ * @return the current depth.
+ */
+ public final int getDepth() {
+ return filters.size();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth. The filter at depth 0 is the
+ * oldest filter.
+ *
+ * @param depth the depth at which the desired filter is to be found.
+ * @return the filter.
+ * @throws NoSuchElementException if depth is not in the range
+ * [0,filters.size())
+ */
+ public final BloomFilter get(int depth) {
+ if (depth < 0 || depth >= filters.size()) {
+ throw new NoSuchElementException(String.format("Depth must be in the range [0,%s)", filters.size()));
+ }
+ return filters.get(depth);
+ }
+
+ /**
+ * Returns the current target filter. If the a new filter should be created
+ * based on {@code extendCheck} it will be created before this method returns.
+ *
+ * @return the current target filter.
+ */
+ public final BloomFilter target() {
Review Comment:
`getTarget`
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
+ };
+
+ /**
+ * Removes the earliest filters in the list when the the number of filters
+ * exceeds maxSize.
+ *
+ * @param maxSize the maximum number of filters for the list.
+ * @return A Consumer for the LayerManager filterCleanup constructor argument.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> onMaxSize(int maxSize) {
+ return (ll) -> {
+ while (ll.size() > maxSize) {
+ ll.removeFirst();
+ }
+ };
+ }
+ }
+
+ private final LinkedList<BloomFilter> filters = new LinkedList<>();
+ private final Consumer<LinkedList<BloomFilter>> filterCleanup;
+ private final Predicate<LayerManager> extendCheck;
+ private final Supplier<BloomFilter> filterSupplier;
+
+ /**
+ * Creates a new Builder with defaults of NEVER_ADVANCE and NO_CLEANUP
+ *
+ * @return A builder.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param filterSupplier the supplier of new Bloom filters to add the the list
+ * when necessary.
+ * @param extendCheck The predicate that checks if a new filter should be
+ * added to the list.
+ * @param filterCleanup the consumer the removes any old filters from the list.
+ */
+ private LayerManager(Supplier<BloomFilter> filterSupplier, Predicate<LayerManager> extendCheck,
+ Consumer<LinkedList<BloomFilter>> filterCleanup) {
+ this.filterSupplier = filterSupplier;
+ this.extendCheck = extendCheck;
+ this.filterCleanup = filterCleanup;
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Creates a deep copy of this LayerManager.
+ *
+ * @return a copy of this layer Manager.
+ */
+ public LayerManager copy() {
+ LayerManager newMgr = new LayerManager(filterSupplier, extendCheck, filterCleanup);
Review Comment:
Since you create and add a new filter then immediately clear it this should be optimised. The cost of creating a filter could be quite high. I would suggest a flag to the LayerManager constructor to indicate that filters should be initialised.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
Review Comment:
In my IDE I get unrecognised characters for your hyphens: V1â586âV1â591. In a standard text editor it works fine as a UTF-8 encoded file. It is also fine on GH.
However the V1 is redundant so I would change this to pp. 586-591 using a minus sign for the hyphen character.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
+ };
+
+ /**
+ * Removes the earliest filters in the list when the the number of filters
+ * exceeds maxSize.
+ *
+ * @param maxSize the maximum number of filters for the list.
+ * @return A Consumer for the LayerManager filterCleanup constructor argument.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> onMaxSize(int maxSize) {
+ return (ll) -> {
+ while (ll.size() > maxSize) {
+ ll.removeFirst();
+ }
+ };
+ }
+ }
+
+ private final LinkedList<BloomFilter> filters = new LinkedList<>();
+ private final Consumer<LinkedList<BloomFilter>> filterCleanup;
+ private final Predicate<LayerManager> extendCheck;
+ private final Supplier<BloomFilter> filterSupplier;
+
+ /**
+ * Creates a new Builder with defaults of NEVER_ADVANCE and NO_CLEANUP
+ *
+ * @return A builder.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param filterSupplier the supplier of new Bloom filters to add the the list
+ * when necessary.
+ * @param extendCheck The predicate that checks if a new filter should be
+ * added to the list.
+ * @param filterCleanup the consumer the removes any old filters from the list.
+ */
+ private LayerManager(Supplier<BloomFilter> filterSupplier, Predicate<LayerManager> extendCheck,
+ Consumer<LinkedList<BloomFilter>> filterCleanup) {
+ this.filterSupplier = filterSupplier;
+ this.extendCheck = extendCheck;
+ this.filterCleanup = filterCleanup;
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Creates a deep copy of this LayerManager.
+ *
+ * @return a copy of this layer Manager.
+ */
+ public LayerManager copy() {
+ LayerManager newMgr = new LayerManager(filterSupplier, extendCheck, filterCleanup);
+ newMgr.filters.clear();
+ for (BloomFilter bf : filters) {
+ newMgr.filters.add(bf.copy());
+ }
+ return newMgr;
+ }
+
+ /**
+ * Forces an advance to the next depth for subsequent merges. Executes the same
+ * logic as when {@code ExtendCheck} returns {@code true}
+ */
+ public void next() {
+ if (!filters.isEmpty() && filters.getLast().cardinality() == 0) {
+ filters.removeLast();
+ }
+ this.filterCleanup.accept(filters);
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Returns the number of filters in the LayerManager.
+ *
+ * @return the current depth.
+ */
+ public final int getDepth() {
+ return filters.size();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth. The filter at depth 0 is the
+ * oldest filter.
+ *
+ * @param depth the depth at which the desired filter is to be found.
+ * @return the filter.
+ * @throws NoSuchElementException if depth is not in the range
+ * [0,filters.size())
+ */
+ public final BloomFilter get(int depth) {
+ if (depth < 0 || depth >= filters.size()) {
+ throw new NoSuchElementException(String.format("Depth must be in the range [0,%s)", filters.size()));
+ }
+ return filters.get(depth);
+ }
+
+ /**
+ * Returns the current target filter. If the a new filter should be created
Review Comment:
`If a new filter`
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
+ };
+
+ /**
+ * Removes the earliest filters in the list when the the number of filters
+ * exceeds maxSize.
+ *
+ * @param maxSize the maximum number of filters for the list.
+ * @return A Consumer for the LayerManager filterCleanup constructor argument.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> onMaxSize(int maxSize) {
+ return (ll) -> {
+ while (ll.size() > maxSize) {
+ ll.removeFirst();
+ }
+ };
+ }
+ }
+
+ private final LinkedList<BloomFilter> filters = new LinkedList<>();
+ private final Consumer<LinkedList<BloomFilter>> filterCleanup;
+ private final Predicate<LayerManager> extendCheck;
+ private final Supplier<BloomFilter> filterSupplier;
+
+ /**
+ * Creates a new Builder with defaults of NEVER_ADVANCE and NO_CLEANUP
+ *
+ * @return A builder.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param filterSupplier the supplier of new Bloom filters to add the the list
+ * when necessary.
+ * @param extendCheck The predicate that checks if a new filter should be
+ * added to the list.
+ * @param filterCleanup the consumer the removes any old filters from the list.
+ */
+ private LayerManager(Supplier<BloomFilter> filterSupplier, Predicate<LayerManager> extendCheck,
+ Consumer<LinkedList<BloomFilter>> filterCleanup) {
+ this.filterSupplier = filterSupplier;
+ this.extendCheck = extendCheck;
+ this.filterCleanup = filterCleanup;
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Creates a deep copy of this LayerManager.
+ *
+ * @return a copy of this layer Manager.
+ */
+ public LayerManager copy() {
+ LayerManager newMgr = new LayerManager(filterSupplier, extendCheck, filterCleanup);
+ newMgr.filters.clear();
+ for (BloomFilter bf : filters) {
+ newMgr.filters.add(bf.copy());
+ }
+ return newMgr;
+ }
+
+ /**
+ * Forces an advance to the next depth for subsequent merges. Executes the same
+ * logic as when {@code ExtendCheck} returns {@code true}
+ */
+ public void next() {
+ if (!filters.isEmpty() && filters.getLast().cardinality() == 0) {
+ filters.removeLast();
+ }
+ this.filterCleanup.accept(filters);
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Returns the number of filters in the LayerManager.
+ *
+ * @return the current depth.
+ */
+ public final int getDepth() {
+ return filters.size();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth. The filter at depth 0 is the
+ * oldest filter.
+ *
+ * @param depth the depth at which the desired filter is to be found.
+ * @return the filter.
+ * @throws NoSuchElementException if depth is not in the range
+ * [0,filters.size())
+ */
+ public final BloomFilter get(int depth) {
+ if (depth < 0 || depth >= filters.size()) {
+ throw new NoSuchElementException(String.format("Depth must be in the range [0,%s)", filters.size()));
+ }
+ return filters.get(depth);
+ }
+
+ /**
+ * Returns the current target filter. If the a new filter should be created
+ * based on {@code extendCheck} it will be created before this method returns.
+ *
+ * @return the current target filter.
+ */
+ public final BloomFilter target() {
+ if (extendCheck.test(this)) {
+ next();
+ }
+ return filters.peekLast();
+ }
+
+ /**
+ * Clear all the filters in the layer manager, and set up a new one as the
+ * target.
+ */
+ public final void clear() {
+ filters.clear();
+ next();
Review Comment:
No need to call `next`, this can simply do `filters.add(this.filterSupplier.get());`. Note that call perhaps should be guarded against the supplier generating null (see other comments).
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
+ };
+
+ /**
+ * Removes the earliest filters in the list when the the number of filters
+ * exceeds maxSize.
+ *
+ * @param maxSize the maximum number of filters for the list.
+ * @return A Consumer for the LayerManager filterCleanup constructor argument.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> onMaxSize(int maxSize) {
+ return (ll) -> {
+ while (ll.size() > maxSize) {
+ ll.removeFirst();
+ }
+ };
+ }
+ }
+
+ private final LinkedList<BloomFilter> filters = new LinkedList<>();
+ private final Consumer<LinkedList<BloomFilter>> filterCleanup;
+ private final Predicate<LayerManager> extendCheck;
+ private final Supplier<BloomFilter> filterSupplier;
+
+ /**
+ * Creates a new Builder with defaults of NEVER_ADVANCE and NO_CLEANUP
+ *
+ * @return A builder.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param filterSupplier the supplier of new Bloom filters to add the the list
+ * when necessary.
+ * @param extendCheck The predicate that checks if a new filter should be
+ * added to the list.
+ * @param filterCleanup the consumer the removes any old filters from the list.
Review Comment:
`the consumer that removes`
##########
src/main/java/org/apache/commons/collections4/bloomfilter/BloomFilterProducer.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.function.BiPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Produces Bloom filters that are copies of Bloom filters in a collection (e.g.
+ * LayerBloomFilter).
+ *
+ * @since 4.5
+ */
+public interface BloomFilterProducer {
+ /**
+ * Executes a Bloom filter Predicate on each Bloom filter in the manager in
+ * depth order. Oldest filter first.
+ *
+ * @param bloomFilterPredicate the predicate to evaluate each Bloom filter with.
+ * @return {@code false} when the first filter fails the predicate test. Returns
+ * {@code true} if all filters pass the test.
+ */
+ boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate);
+
+ /**
+ * Return a deep copy of the BloomFilterProducer data as a Bloom filter array.
+ * <p>
+ * The default implementation of this method is slow. It is recommended that
Review Comment:
Is this comment applicable? It was copied from BitMapProducer. However even there it may not be applicable. The comment probably originates from IndexProducer.asIndexArray which does have a slow default implementation.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
+ };
+
+ /**
+ * Removes the earliest filters in the list when the the number of filters
+ * exceeds maxSize.
+ *
+ * @param maxSize the maximum number of filters for the list.
+ * @return A Consumer for the LayerManager filterCleanup constructor argument.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> onMaxSize(int maxSize) {
+ return (ll) -> {
+ while (ll.size() > maxSize) {
+ ll.removeFirst();
+ }
+ };
+ }
+ }
+
+ private final LinkedList<BloomFilter> filters = new LinkedList<>();
+ private final Consumer<LinkedList<BloomFilter>> filterCleanup;
+ private final Predicate<LayerManager> extendCheck;
+ private final Supplier<BloomFilter> filterSupplier;
+
+ /**
+ * Creates a new Builder with defaults of NEVER_ADVANCE and NO_CLEANUP
+ *
+ * @return A builder.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param filterSupplier the supplier of new Bloom filters to add the the list
+ * when necessary.
+ * @param extendCheck The predicate that checks if a new filter should be
+ * added to the list.
+ * @param filterCleanup the consumer the removes any old filters from the list.
+ */
+ private LayerManager(Supplier<BloomFilter> filterSupplier, Predicate<LayerManager> extendCheck,
+ Consumer<LinkedList<BloomFilter>> filterCleanup) {
+ this.filterSupplier = filterSupplier;
+ this.extendCheck = extendCheck;
+ this.filterCleanup = filterCleanup;
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Creates a deep copy of this LayerManager.
+ *
+ * @return a copy of this layer Manager.
+ */
+ public LayerManager copy() {
+ LayerManager newMgr = new LayerManager(filterSupplier, extendCheck, filterCleanup);
+ newMgr.filters.clear();
+ for (BloomFilter bf : filters) {
+ newMgr.filters.add(bf.copy());
+ }
+ return newMgr;
+ }
+
+ /**
+ * Forces an advance to the next depth for subsequent merges. Executes the same
+ * logic as when {@code ExtendCheck} returns {@code true}
+ */
+ public void next() {
+ if (!filters.isEmpty() && filters.getLast().cardinality() == 0) {
Review Comment:
It is not clear to me why the filter must be empty (cardinality == 0) here.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
+ };
+
+ /**
+ * Removes the earliest filters in the list when the the number of filters
+ * exceeds maxSize.
+ *
+ * @param maxSize the maximum number of filters for the list.
+ * @return A Consumer for the LayerManager filterCleanup constructor argument.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> onMaxSize(int maxSize) {
+ return (ll) -> {
+ while (ll.size() > maxSize) {
+ ll.removeFirst();
+ }
+ };
+ }
+ }
+
+ private final LinkedList<BloomFilter> filters = new LinkedList<>();
+ private final Consumer<LinkedList<BloomFilter>> filterCleanup;
+ private final Predicate<LayerManager> extendCheck;
+ private final Supplier<BloomFilter> filterSupplier;
+
+ /**
+ * Creates a new Builder with defaults of NEVER_ADVANCE and NO_CLEANUP
+ *
+ * @return A builder.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param filterSupplier the supplier of new Bloom filters to add the the list
+ * when necessary.
+ * @param extendCheck The predicate that checks if a new filter should be
+ * added to the list.
+ * @param filterCleanup the consumer the removes any old filters from the list.
+ */
+ private LayerManager(Supplier<BloomFilter> filterSupplier, Predicate<LayerManager> extendCheck,
+ Consumer<LinkedList<BloomFilter>> filterCleanup) {
+ this.filterSupplier = filterSupplier;
+ this.extendCheck = extendCheck;
+ this.filterCleanup = filterCleanup;
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Creates a deep copy of this LayerManager.
+ *
+ * @return a copy of this layer Manager.
+ */
+ public LayerManager copy() {
+ LayerManager newMgr = new LayerManager(filterSupplier, extendCheck, filterCleanup);
+ newMgr.filters.clear();
+ for (BloomFilter bf : filters) {
+ newMgr.filters.add(bf.copy());
+ }
+ return newMgr;
+ }
+
+ /**
+ * Forces an advance to the next depth for subsequent merges. Executes the same
Review Comment:
This comment is jumbled (as it was copied from the LayeredBloomFilter). Perhaps it should read:
```
* Forces an advance to the next depth. This method will clean-up the current layers
* and generate a new filter layer.
*
* <p>This method is used within {@link #target()} when the configured
* {@code ExtendCheck} returns {@code true}.
```
Q. Does this need to be public? In common usage should a layer manager only act using behaviour defined by the constructor arguments to determine when to extend and when to clean-up. Exposing this in the public API allows the layers to be forced through the depth.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
+ };
+
+ /**
+ * Removes the earliest filters in the list when the the number of filters
+ * exceeds maxSize.
+ *
+ * @param maxSize the maximum number of filters for the list.
+ * @return A Consumer for the LayerManager filterCleanup constructor argument.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> onMaxSize(int maxSize) {
+ return (ll) -> {
+ while (ll.size() > maxSize) {
+ ll.removeFirst();
+ }
+ };
+ }
+ }
+
+ private final LinkedList<BloomFilter> filters = new LinkedList<>();
+ private final Consumer<LinkedList<BloomFilter>> filterCleanup;
+ private final Predicate<LayerManager> extendCheck;
+ private final Supplier<BloomFilter> filterSupplier;
+
+ /**
+ * Creates a new Builder with defaults of NEVER_ADVANCE and NO_CLEANUP
+ *
+ * @return A builder.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param filterSupplier the supplier of new Bloom filters to add the the list
+ * when necessary.
+ * @param extendCheck The predicate that checks if a new filter should be
+ * added to the list.
+ * @param filterCleanup the consumer the removes any old filters from the list.
+ */
+ private LayerManager(Supplier<BloomFilter> filterSupplier, Predicate<LayerManager> extendCheck,
+ Consumer<LinkedList<BloomFilter>> filterCleanup) {
Review Comment:
I do not think we should type to `LinkedList`. IIUC the filters are used as layers and modified at the top and bottom so it is the `Deque` interface that is applicable. A `List` would only be used if insertion/deletion at intermediate layers is required.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
+ };
+
+ /**
+ * Removes the earliest filters in the list when the the number of filters
+ * exceeds maxSize.
+ *
+ * @param maxSize the maximum number of filters for the list.
+ * @return A Consumer for the LayerManager filterCleanup constructor argument.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> onMaxSize(int maxSize) {
+ return (ll) -> {
+ while (ll.size() > maxSize) {
+ ll.removeFirst();
+ }
+ };
+ }
+ }
+
+ private final LinkedList<BloomFilter> filters = new LinkedList<>();
+ private final Consumer<LinkedList<BloomFilter>> filterCleanup;
+ private final Predicate<LayerManager> extendCheck;
+ private final Supplier<BloomFilter> filterSupplier;
+
+ /**
+ * Creates a new Builder with defaults of NEVER_ADVANCE and NO_CLEANUP
+ *
+ * @return A builder.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param filterSupplier the supplier of new Bloom filters to add the the list
+ * when necessary.
+ * @param extendCheck The predicate that checks if a new filter should be
+ * added to the list.
+ * @param filterCleanup the consumer the removes any old filters from the list.
+ */
+ private LayerManager(Supplier<BloomFilter> filterSupplier, Predicate<LayerManager> extendCheck,
+ Consumer<LinkedList<BloomFilter>> filterCleanup) {
+ this.filterSupplier = filterSupplier;
+ this.extendCheck = extendCheck;
+ this.filterCleanup = filterCleanup;
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Creates a deep copy of this LayerManager.
+ *
+ * @return a copy of this layer Manager.
+ */
+ public LayerManager copy() {
+ LayerManager newMgr = new LayerManager(filterSupplier, extendCheck, filterCleanup);
+ newMgr.filters.clear();
+ for (BloomFilter bf : filters) {
+ newMgr.filters.add(bf.copy());
+ }
+ return newMgr;
+ }
+
+ /**
+ * Forces an advance to the next depth for subsequent merges. Executes the same
+ * logic as when {@code ExtendCheck} returns {@code true}
+ */
+ public void next() {
+ if (!filters.isEmpty() && filters.getLast().cardinality() == 0) {
+ filters.removeLast();
+ }
+ this.filterCleanup.accept(filters);
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Returns the number of filters in the LayerManager.
Review Comment:
Perhaps this should document the fact that the value should always be positive, i.e. there is always at least 1 layer.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
+ * items than the Shape would indicate and yet still return a false positive
+ * rate in line with the Shape and not the over population.
+ * <p>
+ * This implementation uses a LayerManager to handle the manipulation of the
+ * layers.
+ * </p>
+ * <ul>
+ * <li>Level 0 is the oldest layer and the highest level is the newest.</li>
+ * <li>There is always at least one enclosed filter.</li>
+ * <li>The newest filter is the {@code target} into which merges are performed.
+ * <li>Whenever the target is retrieved, or a {@code merge} operation is
+ * performed the code checks if any older layers should be removed, and if so
+ * removes them. It also checks it a new layer should be added, and if so adds
+ * it and sets the {@code target} before the operation.</li>
+ * </ul>
+ */
+public class LayeredBloomFilter implements BloomFilter, BloomFilterProducer {
+ private final Shape shape;
+ private LayerManager layerManager;
+
+ /**
+ * Creates a fixed size layered bloom filter that adds new filters to the list,
+ * but never merges them. List will never exceed maxDepth. As additional filters
+ * are added earlier filters are removed.
+ *
+ * @param shape The shape for the enclosed Bloom filters
+ * @param maxDepth The maximum depth of layers.
+ * @return An empty layered Bloom filter of the specified shape and depth.
+ */
+ public static LayeredBloomFilter fixed(final Shape shape, int maxDepth) {
+ LayerManager manager = LayerManager.builder().withExtendCheck(LayerManager.ExtendCheck.ADVANCE_ON_POPULATED)
+ .withCleanup(LayerManager.Cleanup.onMaxSize(maxDepth)).withSuplier(() -> new SimpleBloomFilter(shape))
+ .build();
+ return new LayeredBloomFilter(shape, manager);
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param shape the Shape of the enclosed Bloom filters
+ * @param layerManager the LayerManager to manage the layers.
+ */
+ public LayeredBloomFilter(Shape shape, LayerManager layerManager) {
+ this.shape = shape;
+ this.layerManager = layerManager;
+ }
+
+ @Override
+ public LayeredBloomFilter copy() {
+ return new LayeredBloomFilter(shape, layerManager.copy());
+ }
+
+ /**
+ * Gets the depth of the deepest layer.
+ *
+ * @return the depth of the deepest layer.
+ */
+ public final int getDepth() {
+ return layerManager.getDepth();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth
+ *
+ * @param depth the depth of the filter to return.
+ * @return the Bloom filter at the specified depth.
+ * @throws NoSuchElementException if depth is not in the range [0,getDepth())
+ */
+ public BloomFilter get(int depth) {
Review Comment:
Why provide access to layers?
See also the same comments in LayerManager.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/CountingPredicate.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.function.BiPredicate;
+import java.util.function.Predicate;
+
+/**
+ * A predicate that applies the test func to each member of the @{code ary} in
+ * sequence for each call to @{code test()}. if the @{code ary} is exhausted,
+ * the subsequent calls to @{code test} are executed with a {@code null} value.
+ * If the calls to @{code test} do not exhaust the @{code ary} the @{code
+ * forEachRemaining} method can be called to execute the @code{text} with a
+ * {@code null} value for each remaining @{code idx} value.
+ *
+ * @param <T> the type of object being compared.
+ *
Review Comment:
Remove empty line
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
+ * items than the Shape would indicate and yet still return a false positive
+ * rate in line with the Shape and not the over population.
+ * <p>
+ * This implementation uses a LayerManager to handle the manipulation of the
+ * layers.
+ * </p>
+ * <ul>
+ * <li>Level 0 is the oldest layer and the highest level is the newest.</li>
+ * <li>There is always at least one enclosed filter.</li>
+ * <li>The newest filter is the {@code target} into which merges are performed.
+ * <li>Whenever the target is retrieved, or a {@code merge} operation is
+ * performed the code checks if any older layers should be removed, and if so
+ * removes them. It also checks it a new layer should be added, and if so adds
+ * it and sets the {@code target} before the operation.</li>
+ * </ul>
+ */
+public class LayeredBloomFilter implements BloomFilter, BloomFilterProducer {
+ private final Shape shape;
+ private LayerManager layerManager;
+
+ /**
+ * Creates a fixed size layered bloom filter that adds new filters to the list,
+ * but never merges them. List will never exceed maxDepth. As additional filters
+ * are added earlier filters are removed.
+ *
+ * @param shape The shape for the enclosed Bloom filters
+ * @param maxDepth The maximum depth of layers.
+ * @return An empty layered Bloom filter of the specified shape and depth.
+ */
+ public static LayeredBloomFilter fixed(final Shape shape, int maxDepth) {
+ LayerManager manager = LayerManager.builder().withExtendCheck(LayerManager.ExtendCheck.ADVANCE_ON_POPULATED)
+ .withCleanup(LayerManager.Cleanup.onMaxSize(maxDepth)).withSuplier(() -> new SimpleBloomFilter(shape))
+ .build();
+ return new LayeredBloomFilter(shape, manager);
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param shape the Shape of the enclosed Bloom filters
+ * @param layerManager the LayerManager to manage the layers.
+ */
+ public LayeredBloomFilter(Shape shape, LayerManager layerManager) {
+ this.shape = shape;
+ this.layerManager = layerManager;
+ }
+
+ @Override
+ public LayeredBloomFilter copy() {
+ return new LayeredBloomFilter(shape, layerManager.copy());
+ }
+
+ /**
+ * Gets the depth of the deepest layer.
+ *
+ * @return the depth of the deepest layer.
+ */
+ public final int getDepth() {
+ return layerManager.getDepth();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth
+ *
+ * @param depth the depth of the filter to return.
+ * @return the Bloom filter at the specified depth.
+ * @throws NoSuchElementException if depth is not in the range [0,getDepth())
+ */
+ public BloomFilter get(int depth) {
+ return layerManager.get(depth);
+ }
+
+ @Override
+ public int cardinality() {
+ return SetOperations.cardinality(this);
+ }
+
+ @Override
+ public final void clear() {
+ layerManager.clear();
+ }
+
+ /**
+ * Clears the Bloom filter (removes all set bits) at the specified level.
+ *
+ * @param level the level to clear.
+ */
+ public final void clear(int level) {
Review Comment:
Why provide access to clear layers? Is this some requirement of the original paper. IMO it is dangerous to allow access to internals of a managed object that functions collectively.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/WrappedBloomFilter.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+
+/**
+ * An abstract class to assist in implementing Bloom filter decorators.
+ *
+ * @since 4.5
+ */
+public abstract class WrappedBloomFilter implements BloomFilter {
+ final BloomFilter wrapped;
+
+ public WrappedBloomFilter(BloomFilter bf) {
Review Comment:
Javadoc the constructor. You can state that `bf` is maintained as a reference and not copied.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/BloomFilterProducer.java:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.function.BiPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Produces Bloom filters that are copies of Bloom filters in a collection (e.g.
+ * LayerBloomFilter).
+ *
+ * @since 4.5
+ */
+public interface BloomFilterProducer {
+ /**
+ * Executes a Bloom filter Predicate on each Bloom filter in the manager in
+ * depth order. Oldest filter first.
+ *
+ * @param bloomFilterPredicate the predicate to evaluate each Bloom filter with.
+ * @return {@code false} when the first filter fails the predicate test. Returns
+ * {@code true} if all filters pass the test.
+ */
+ boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate);
+
+ /**
+ * Return a deep copy of the BloomFilterProducer data as a Bloom filter array.
+ * <p>
+ * The default implementation of this method is slow. It is recommended that
+ * implementing classes reimplement this method.
+ * </p>
+ *
+ * @return An array of Bloom filters.
+ */
+ default BloomFilter[] asBloomFilterArray() {
+ class Filters {
+ private BloomFilter[] data = new BloomFilter[16];
+ private int size;
+
+ boolean add(final BloomFilter filter) {
+ if (size == data.length) {
+ // This will throw an out-of-memory error if there are too many Bloom filters.
+ data = Arrays.copyOf(data, size * 2);
+ }
+ data[size++] = filter.copy();
+ return true;
+ }
+
+ BloomFilter[] toArray() {
+ // Edge case to avoid a large array copy
+ return size == data.length ? data : Arrays.copyOf(data, size);
+ }
+ }
+ final Filters filters = new Filters();
+ forEachBloomFilter(filters::add);
+ return filters.toArray();
+ }
+
+ /**
+ * Applies the {@code func} to each Bloom filter pair in order. Will apply all
+ * of the Bloom filters from the other BloomFilterProducer to this producer. If
+ * this producer does not have as many BloomFilters it will provide
+ * {@code null} for all excess calls to the BiPredicate.
+ *
+ * @param other The other BloomFilterProducer that provides the y values in the
+ * (x,y) pair.
+ * @param func The function to apply.
+ * @return A LongPredicate that tests this BitMapProducers bitmap values in
+ * order.
+ */
+ default boolean forEachBloomFilterPair(final BloomFilterProducer other,
+ final BiPredicate<BloomFilter, BloomFilter> func) {
+ final CountingPredicate<BloomFilter> p = new CountingPredicate<>(asBloomFilterArray(), func);
+ return other.forEachBloomFilter(p) && p.forEachRemaining();
+ }
+
Review Comment:
Remove empty line
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
+ * items than the Shape would indicate and yet still return a false positive
+ * rate in line with the Shape and not the over population.
+ * <p>
+ * This implementation uses a LayerManager to handle the manipulation of the
+ * layers.
+ * </p>
+ * <ul>
+ * <li>Level 0 is the oldest layer and the highest level is the newest.</li>
+ * <li>There is always at least one enclosed filter.</li>
+ * <li>The newest filter is the {@code target} into which merges are performed.
+ * <li>Whenever the target is retrieved, or a {@code merge} operation is
+ * performed the code checks if any older layers should be removed, and if so
+ * removes them. It also checks it a new layer should be added, and if so adds
+ * it and sets the {@code target} before the operation.</li>
+ * </ul>
+ */
+public class LayeredBloomFilter implements BloomFilter, BloomFilterProducer {
+ private final Shape shape;
+ private LayerManager layerManager;
+
+ /**
+ * Creates a fixed size layered bloom filter that adds new filters to the list,
+ * but never merges them. List will never exceed maxDepth. As additional filters
+ * are added earlier filters are removed.
+ *
+ * @param shape The shape for the enclosed Bloom filters
+ * @param maxDepth The maximum depth of layers.
+ * @return An empty layered Bloom filter of the specified shape and depth.
+ */
+ public static LayeredBloomFilter fixed(final Shape shape, int maxDepth) {
+ LayerManager manager = LayerManager.builder().withExtendCheck(LayerManager.ExtendCheck.ADVANCE_ON_POPULATED)
+ .withCleanup(LayerManager.Cleanup.onMaxSize(maxDepth)).withSuplier(() -> new SimpleBloomFilter(shape))
+ .build();
+ return new LayeredBloomFilter(shape, manager);
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param shape the Shape of the enclosed Bloom filters
+ * @param layerManager the LayerManager to manage the layers.
+ */
+ public LayeredBloomFilter(Shape shape, LayerManager layerManager) {
+ this.shape = shape;
+ this.layerManager = layerManager;
+ }
+
+ @Override
+ public LayeredBloomFilter copy() {
+ return new LayeredBloomFilter(shape, layerManager.copy());
+ }
+
+ /**
+ * Gets the depth of the deepest layer.
+ *
+ * @return the depth of the deepest layer.
+ */
+ public final int getDepth() {
+ return layerManager.getDepth();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth
+ *
+ * @param depth the depth of the filter to return.
+ * @return the Bloom filter at the specified depth.
+ * @throws NoSuchElementException if depth is not in the range [0,getDepth())
+ */
+ public BloomFilter get(int depth) {
+ return layerManager.get(depth);
+ }
+
+ @Override
+ public int cardinality() {
+ return SetOperations.cardinality(this);
+ }
+
+ @Override
+ public final void clear() {
+ layerManager.clear();
+ }
+
+ /**
+ * Clears the Bloom filter (removes all set bits) at the specified level.
+ *
+ * @param level the level to clear.
+ */
+ public final void clear(int level) {
+ layerManager.get(level).clear();
+ }
+
+ /**
+ * Get the Bloom filter that is currently being merged into. This method ensures
+ * that the {@code target} filter meets the criteria specified within the
+ * {@code LayerManager}. if the {@code LayerManager} determines that a new
+ * target filter is required it is constructed and used.
+ *
+ * @return the current Bloom filter.
+ * @see LayerManager
+ */
+ public final BloomFilter target() {
+ return layerManager.target();
+ }
+
+ /**
+ * Processes the Bloom filters in depth order with the most recent filters
+ * first. Each filter is passed to the predicate in turn. The function exits on
+ * the first {@code false} returned by the predicate.
+ *
+ * @param bloomFilterPredicate the predicate to execute.
+ * @return {@code true} if all filters passed the predicate, {@code false}
+ * otherwise.
+ */
+ @Override
+ public final boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate) {
+ return layerManager.forEachBloomFilter(bloomFilterPredicate);
+ }
+
+ /**
+ * Create a standard (non-layered) Bloom filter by merging all of the layers.
+ *
+ * @return the merged bloom filter.
+ */
+ public BloomFilter flatten() {
+ BloomFilter bf = new SimpleBloomFilter(shape);
+ forEachBloomFilter(bf::merge);
+ return bf;
+ }
+
+ /**
+ * Finds the layers in which the Hasher is found.
+ *
+ * @param hasher the Hasher to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the IndexProducer is found.
+ *
+ * @param indexProducer the Index producer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the BitMapProducer is found.
+ *
+ * @param bitMapProducer the BitMapProducer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the Bloom filter is found.
+ *
+ * @param bf the Bloom filter to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(BloomFilter bf) {
+ Finder finder = new Finder(bf);
+ forEachBloomFilter(finder);
+ return finder.getResult();
+ }
+
+ /**
+ * Returns {@code true} if this any layer contained by this filter contains the
+ * specified filter.
+ * <p>
+ * If the {@code other} is another Layered Bloom filter each filter within the
+ * {@code other} is checked to see if it exits within this filter.
+ * </p>
+ *
+ * @param other the other Bloom filter
+ * @return {@code true} if this filter contains the other filter.
+ */
+ @Override
+ public boolean contains(final BloomFilter other) {
+ if (other instanceof LayeredBloomFilter) {
+ boolean[] result = { true };
+ // return false when we have found a match.
+ ((LayeredBloomFilter) other).forEachBloomFilter(x -> {
+ result[0] &= contains(x);
+ return result[0];
+ });
+ return result[0];
+ }
+ return !forEachBloomFilter(x -> !x.contains(other));
+ }
+
+ /**
+ * Creates a Bloom filter from a Hasher.
+ *
+ * @param hasher the hasher to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return bf;
+ }
+
+ /**
+ * Creates a Bloom filter from an IndexProducer.
+ *
+ * @param indexProducer the IndexProducer to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return bf;
+ }
+
+ /**
+ * Creates a Bloom filter from a BitMapProducer.
+ *
+ * @param bitMapProducer the BitMapProducer to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return bf;
+ }
+
+ @Override
+ public int characteristics() {
+ return 0;
Review Comment:
`return target().characteristics();` ???
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
Review Comment:
Needs a check that `maxN` is strictly positive.
Can be a lambda:
```
return manager -> {
};
```
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
Review Comment:
This needs a check that `breakAt` is strictly positive (and a test for breakAt={0, -1})
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
+ * items than the Shape would indicate and yet still return a false positive
+ * rate in line with the Shape and not the over population.
+ * <p>
+ * This implementation uses a LayerManager to handle the manipulation of the
+ * layers.
+ * </p>
+ * <ul>
+ * <li>Level 0 is the oldest layer and the highest level is the newest.</li>
+ * <li>There is always at least one enclosed filter.</li>
+ * <li>The newest filter is the {@code target} into which merges are performed.
+ * <li>Whenever the target is retrieved, or a {@code merge} operation is
+ * performed the code checks if any older layers should be removed, and if so
+ * removes them. It also checks it a new layer should be added, and if so adds
+ * it and sets the {@code target} before the operation.</li>
+ * </ul>
+ */
+public class LayeredBloomFilter implements BloomFilter, BloomFilterProducer {
+ private final Shape shape;
+ private LayerManager layerManager;
+
+ /**
+ * Creates a fixed size layered bloom filter that adds new filters to the list,
+ * but never merges them. List will never exceed maxDepth. As additional filters
+ * are added earlier filters are removed.
+ *
+ * @param shape The shape for the enclosed Bloom filters
+ * @param maxDepth The maximum depth of layers.
+ * @return An empty layered Bloom filter of the specified shape and depth.
+ */
+ public static LayeredBloomFilter fixed(final Shape shape, int maxDepth) {
+ LayerManager manager = LayerManager.builder().withExtendCheck(LayerManager.ExtendCheck.ADVANCE_ON_POPULATED)
+ .withCleanup(LayerManager.Cleanup.onMaxSize(maxDepth)).withSuplier(() -> new SimpleBloomFilter(shape))
+ .build();
+ return new LayeredBloomFilter(shape, manager);
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param shape the Shape of the enclosed Bloom filters
+ * @param layerManager the LayerManager to manage the layers.
+ */
+ public LayeredBloomFilter(Shape shape, LayerManager layerManager) {
+ this.shape = shape;
+ this.layerManager = layerManager;
+ }
+
+ @Override
+ public LayeredBloomFilter copy() {
+ return new LayeredBloomFilter(shape, layerManager.copy());
+ }
+
+ /**
+ * Gets the depth of the deepest layer.
+ *
+ * @return the depth of the deepest layer.
+ */
+ public final int getDepth() {
+ return layerManager.getDepth();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth
+ *
+ * @param depth the depth of the filter to return.
+ * @return the Bloom filter at the specified depth.
+ * @throws NoSuchElementException if depth is not in the range [0,getDepth())
+ */
+ public BloomFilter get(int depth) {
+ return layerManager.get(depth);
+ }
+
+ @Override
+ public int cardinality() {
+ return SetOperations.cardinality(this);
+ }
+
+ @Override
+ public final void clear() {
+ layerManager.clear();
+ }
+
+ /**
+ * Clears the Bloom filter (removes all set bits) at the specified level.
+ *
+ * @param level the level to clear.
+ */
+ public final void clear(int level) {
+ layerManager.get(level).clear();
+ }
+
+ /**
+ * Get the Bloom filter that is currently being merged into. This method ensures
+ * that the {@code target} filter meets the criteria specified within the
+ * {@code LayerManager}. if the {@code LayerManager} determines that a new
+ * target filter is required it is constructed and used.
+ *
+ * @return the current Bloom filter.
+ * @see LayerManager
+ */
+ public final BloomFilter target() {
+ return layerManager.target();
+ }
+
+ /**
+ * Processes the Bloom filters in depth order with the most recent filters
+ * first. Each filter is passed to the predicate in turn. The function exits on
+ * the first {@code false} returned by the predicate.
+ *
+ * @param bloomFilterPredicate the predicate to execute.
+ * @return {@code true} if all filters passed the predicate, {@code false}
+ * otherwise.
+ */
+ @Override
+ public final boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate) {
+ return layerManager.forEachBloomFilter(bloomFilterPredicate);
+ }
+
+ /**
+ * Create a standard (non-layered) Bloom filter by merging all of the layers.
+ *
+ * @return the merged bloom filter.
+ */
+ public BloomFilter flatten() {
Review Comment:
Should this method be in the `BloomFilterProducer` interface? It requires a Shape. For the interface the Shape could be taken from the first filter and the rest then assumed to match. IDK what to return when the producer outputs no filters, e.g. null, or throw a NoSuchElementException.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
+ * items than the Shape would indicate and yet still return a false positive
+ * rate in line with the Shape and not the over population.
+ * <p>
+ * This implementation uses a LayerManager to handle the manipulation of the
+ * layers.
+ * </p>
+ * <ul>
+ * <li>Level 0 is the oldest layer and the highest level is the newest.</li>
+ * <li>There is always at least one enclosed filter.</li>
+ * <li>The newest filter is the {@code target} into which merges are performed.
+ * <li>Whenever the target is retrieved, or a {@code merge} operation is
+ * performed the code checks if any older layers should be removed, and if so
+ * removes them. It also checks it a new layer should be added, and if so adds
+ * it and sets the {@code target} before the operation.</li>
+ * </ul>
+ */
+public class LayeredBloomFilter implements BloomFilter, BloomFilterProducer {
+ private final Shape shape;
+ private LayerManager layerManager;
+
+ /**
+ * Creates a fixed size layered bloom filter that adds new filters to the list,
+ * but never merges them. List will never exceed maxDepth. As additional filters
+ * are added earlier filters are removed.
+ *
+ * @param shape The shape for the enclosed Bloom filters
+ * @param maxDepth The maximum depth of layers.
+ * @return An empty layered Bloom filter of the specified shape and depth.
+ */
+ public static LayeredBloomFilter fixed(final Shape shape, int maxDepth) {
+ LayerManager manager = LayerManager.builder().withExtendCheck(LayerManager.ExtendCheck.ADVANCE_ON_POPULATED)
+ .withCleanup(LayerManager.Cleanup.onMaxSize(maxDepth)).withSuplier(() -> new SimpleBloomFilter(shape))
+ .build();
+ return new LayeredBloomFilter(shape, manager);
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param shape the Shape of the enclosed Bloom filters
+ * @param layerManager the LayerManager to manage the layers.
+ */
+ public LayeredBloomFilter(Shape shape, LayerManager layerManager) {
+ this.shape = shape;
+ this.layerManager = layerManager;
+ }
+
+ @Override
+ public LayeredBloomFilter copy() {
+ return new LayeredBloomFilter(shape, layerManager.copy());
+ }
+
+ /**
+ * Gets the depth of the deepest layer.
+ *
+ * @return the depth of the deepest layer.
+ */
+ public final int getDepth() {
+ return layerManager.getDepth();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth
+ *
+ * @param depth the depth of the filter to return.
+ * @return the Bloom filter at the specified depth.
+ * @throws NoSuchElementException if depth is not in the range [0,getDepth())
+ */
+ public BloomFilter get(int depth) {
+ return layerManager.get(depth);
+ }
+
+ @Override
+ public int cardinality() {
+ return SetOperations.cardinality(this);
+ }
+
+ @Override
+ public final void clear() {
+ layerManager.clear();
+ }
+
+ /**
+ * Clears the Bloom filter (removes all set bits) at the specified level.
+ *
+ * @param level the level to clear.
+ */
+ public final void clear(int level) {
+ layerManager.get(level).clear();
+ }
+
+ /**
+ * Get the Bloom filter that is currently being merged into. This method ensures
+ * that the {@code target} filter meets the criteria specified within the
+ * {@code LayerManager}. if the {@code LayerManager} determines that a new
+ * target filter is required it is constructed and used.
+ *
+ * @return the current Bloom filter.
+ * @see LayerManager
+ */
+ public final BloomFilter target() {
+ return layerManager.target();
+ }
+
+ /**
+ * Processes the Bloom filters in depth order with the most recent filters
+ * first. Each filter is passed to the predicate in turn. The function exits on
+ * the first {@code false} returned by the predicate.
+ *
+ * @param bloomFilterPredicate the predicate to execute.
+ * @return {@code true} if all filters passed the predicate, {@code false}
+ * otherwise.
+ */
+ @Override
+ public final boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate) {
+ return layerManager.forEachBloomFilter(bloomFilterPredicate);
+ }
+
+ /**
+ * Create a standard (non-layered) Bloom filter by merging all of the layers.
+ *
+ * @return the merged bloom filter.
+ */
+ public BloomFilter flatten() {
+ BloomFilter bf = new SimpleBloomFilter(shape);
+ forEachBloomFilter(bf::merge);
+ return bf;
+ }
+
+ /**
+ * Finds the layers in which the Hasher is found.
+ *
+ * @param hasher the Hasher to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the IndexProducer is found.
+ *
+ * @param indexProducer the Index producer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the BitMapProducer is found.
+ *
+ * @param bitMapProducer the BitMapProducer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the Bloom filter is found.
+ *
+ * @param bf the Bloom filter to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(BloomFilter bf) {
+ Finder finder = new Finder(bf);
+ forEachBloomFilter(finder);
+ return finder.getResult();
+ }
+
+ /**
+ * Returns {@code true} if this any layer contained by this filter contains the
+ * specified filter.
+ * <p>
+ * If the {@code other} is another Layered Bloom filter each filter within the
+ * {@code other} is checked to see if it exits within this filter.
+ * </p>
+ *
+ * @param other the other Bloom filter
+ * @return {@code true} if this filter contains the other filter.
+ */
+ @Override
+ public boolean contains(final BloomFilter other) {
+ if (other instanceof LayeredBloomFilter) {
Review Comment:
Could this use BloomFilterProducer here? This allows you to perform contains using other collections of Bloom filters.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
+ * items than the Shape would indicate and yet still return a false positive
+ * rate in line with the Shape and not the over population.
+ * <p>
+ * This implementation uses a LayerManager to handle the manipulation of the
+ * layers.
+ * </p>
+ * <ul>
+ * <li>Level 0 is the oldest layer and the highest level is the newest.</li>
+ * <li>There is always at least one enclosed filter.</li>
+ * <li>The newest filter is the {@code target} into which merges are performed.
+ * <li>Whenever the target is retrieved, or a {@code merge} operation is
+ * performed the code checks if any older layers should be removed, and if so
+ * removes them. It also checks it a new layer should be added, and if so adds
+ * it and sets the {@code target} before the operation.</li>
+ * </ul>
+ */
+public class LayeredBloomFilter implements BloomFilter, BloomFilterProducer {
+ private final Shape shape;
+ private LayerManager layerManager;
+
+ /**
+ * Creates a fixed size layered bloom filter that adds new filters to the list,
+ * but never merges them. List will never exceed maxDepth. As additional filters
+ * are added earlier filters are removed.
+ *
+ * @param shape The shape for the enclosed Bloom filters
+ * @param maxDepth The maximum depth of layers.
+ * @return An empty layered Bloom filter of the specified shape and depth.
+ */
+ public static LayeredBloomFilter fixed(final Shape shape, int maxDepth) {
+ LayerManager manager = LayerManager.builder().withExtendCheck(LayerManager.ExtendCheck.ADVANCE_ON_POPULATED)
+ .withCleanup(LayerManager.Cleanup.onMaxSize(maxDepth)).withSuplier(() -> new SimpleBloomFilter(shape))
+ .build();
+ return new LayeredBloomFilter(shape, manager);
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param shape the Shape of the enclosed Bloom filters
+ * @param layerManager the LayerManager to manage the layers.
+ */
+ public LayeredBloomFilter(Shape shape, LayerManager layerManager) {
+ this.shape = shape;
+ this.layerManager = layerManager;
+ }
+
+ @Override
+ public LayeredBloomFilter copy() {
+ return new LayeredBloomFilter(shape, layerManager.copy());
+ }
+
+ /**
+ * Gets the depth of the deepest layer.
+ *
+ * @return the depth of the deepest layer.
+ */
+ public final int getDepth() {
+ return layerManager.getDepth();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth
+ *
+ * @param depth the depth of the filter to return.
+ * @return the Bloom filter at the specified depth.
+ * @throws NoSuchElementException if depth is not in the range [0,getDepth())
+ */
+ public BloomFilter get(int depth) {
+ return layerManager.get(depth);
+ }
+
+ @Override
+ public int cardinality() {
+ return SetOperations.cardinality(this);
+ }
+
+ @Override
+ public final void clear() {
+ layerManager.clear();
+ }
+
+ /**
+ * Clears the Bloom filter (removes all set bits) at the specified level.
+ *
+ * @param level the level to clear.
+ */
+ public final void clear(int level) {
+ layerManager.get(level).clear();
+ }
+
+ /**
+ * Get the Bloom filter that is currently being merged into. This method ensures
+ * that the {@code target} filter meets the criteria specified within the
+ * {@code LayerManager}. if the {@code LayerManager} determines that a new
+ * target filter is required it is constructed and used.
+ *
+ * @return the current Bloom filter.
+ * @see LayerManager
+ */
+ public final BloomFilter target() {
+ return layerManager.target();
+ }
+
+ /**
+ * Processes the Bloom filters in depth order with the most recent filters
+ * first. Each filter is passed to the predicate in turn. The function exits on
+ * the first {@code false} returned by the predicate.
+ *
+ * @param bloomFilterPredicate the predicate to execute.
+ * @return {@code true} if all filters passed the predicate, {@code false}
+ * otherwise.
+ */
+ @Override
+ public final boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate) {
+ return layerManager.forEachBloomFilter(bloomFilterPredicate);
+ }
+
+ /**
+ * Create a standard (non-layered) Bloom filter by merging all of the layers.
+ *
+ * @return the merged bloom filter.
+ */
+ public BloomFilter flatten() {
+ BloomFilter bf = new SimpleBloomFilter(shape);
+ forEachBloomFilter(bf::merge);
+ return bf;
+ }
+
+ /**
+ * Finds the layers in which the Hasher is found.
+ *
+ * @param hasher the Hasher to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the IndexProducer is found.
+ *
+ * @param indexProducer the Index producer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the BitMapProducer is found.
+ *
+ * @param bitMapProducer the BitMapProducer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the Bloom filter is found.
+ *
+ * @param bf the Bloom filter to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(BloomFilter bf) {
+ Finder finder = new Finder(bf);
+ forEachBloomFilter(finder);
+ return finder.getResult();
+ }
+
+ /**
+ * Returns {@code true} if this any layer contained by this filter contains the
+ * specified filter.
+ * <p>
+ * If the {@code other} is another Layered Bloom filter each filter within the
+ * {@code other} is checked to see if it exits within this filter.
+ * </p>
+ *
+ * @param other the other Bloom filter
+ * @return {@code true} if this filter contains the other filter.
+ */
+ @Override
+ public boolean contains(final BloomFilter other) {
+ if (other instanceof LayeredBloomFilter) {
+ boolean[] result = { true };
+ // return false when we have found a match.
+ ((LayeredBloomFilter) other).forEachBloomFilter(x -> {
+ result[0] &= contains(x);
+ return result[0];
+ });
+ return result[0];
+ }
+ return !forEachBloomFilter(x -> !x.contains(other));
+ }
+
+ /**
+ * Creates a Bloom filter from a Hasher.
+ *
+ * @param hasher the hasher to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return bf;
+ }
+
+ /**
+ * Creates a Bloom filter from an IndexProducer.
+ *
+ * @param indexProducer the IndexProducer to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return bf;
+ }
+
+ /**
+ * Creates a Bloom filter from a BitMapProducer.
+ *
+ * @param bitMapProducer the BitMapProducer to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return bf;
+ }
+
+ @Override
+ public int characteristics() {
+ return 0;
+ }
+
+ @Override
+ public final Shape getShape() {
+ return shape;
+ }
+
+ @Override
+ public boolean contains(final Hasher hasher) {
+ return contains(createFilter(hasher));
+ }
+
+ @Override
+ public boolean contains(final BitMapProducer bitMapProducer) {
+ return contains(createFilter(bitMapProducer));
+ }
+
+ @Override
+ public boolean contains(IndexProducer indexProducer) {
+ return contains(createFilter(indexProducer));
+ }
+
+ @Override
+ public boolean merge(BloomFilter bf) {
+ return target().merge(bf);
+ }
+
+ @Override
+ public boolean merge(IndexProducer indexProducer) {
+ return target().merge(indexProducer);
+ }
+
+ @Override
+ public boolean merge(BitMapProducer bitMapProducer) {
+ return target().merge(bitMapProducer);
+ }
+
+ @Override
+ public boolean forEachIndex(IntPredicate predicate) {
+ return forEachBloomFilter(bf -> bf.forEachIndex(predicate));
+ }
+
+ @Override
+ public boolean forEachBitMap(LongPredicate predicate) {
+ BloomFilter merged = new SimpleBloomFilter(shape);
+ if (forEachBloomFilter(merged::merge) && !merged.forEachBitMap(predicate)) {
+ return false;
+ }
+ return true;
+ }
+
+ @Override
+ public int estimateN() {
+ BloomFilter result = new SimpleBloomFilter(shape);
+ forEachBloomFilter(result::merge);
+ return result.estimateN();
Review Comment:
`flatten().estimateN()`
Is this the best estimate for a layered filter?
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
+ * items than the Shape would indicate and yet still return a false positive
+ * rate in line with the Shape and not the over population.
+ * <p>
+ * This implementation uses a LayerManager to handle the manipulation of the
+ * layers.
+ * </p>
+ * <ul>
+ * <li>Level 0 is the oldest layer and the highest level is the newest.</li>
+ * <li>There is always at least one enclosed filter.</li>
+ * <li>The newest filter is the {@code target} into which merges are performed.
+ * <li>Whenever the target is retrieved, or a {@code merge} operation is
+ * performed the code checks if any older layers should be removed, and if so
+ * removes them. It also checks it a new layer should be added, and if so adds
+ * it and sets the {@code target} before the operation.</li>
+ * </ul>
+ */
+public class LayeredBloomFilter implements BloomFilter, BloomFilterProducer {
+ private final Shape shape;
+ private LayerManager layerManager;
+
+ /**
+ * Creates a fixed size layered bloom filter that adds new filters to the list,
+ * but never merges them. List will never exceed maxDepth. As additional filters
+ * are added earlier filters are removed.
+ *
+ * @param shape The shape for the enclosed Bloom filters
+ * @param maxDepth The maximum depth of layers.
+ * @return An empty layered Bloom filter of the specified shape and depth.
+ */
+ public static LayeredBloomFilter fixed(final Shape shape, int maxDepth) {
+ LayerManager manager = LayerManager.builder().withExtendCheck(LayerManager.ExtendCheck.ADVANCE_ON_POPULATED)
+ .withCleanup(LayerManager.Cleanup.onMaxSize(maxDepth)).withSuplier(() -> new SimpleBloomFilter(shape))
+ .build();
+ return new LayeredBloomFilter(shape, manager);
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param shape the Shape of the enclosed Bloom filters
+ * @param layerManager the LayerManager to manage the layers.
+ */
+ public LayeredBloomFilter(Shape shape, LayerManager layerManager) {
+ this.shape = shape;
+ this.layerManager = layerManager;
+ }
+
+ @Override
+ public LayeredBloomFilter copy() {
+ return new LayeredBloomFilter(shape, layerManager.copy());
+ }
+
+ /**
+ * Gets the depth of the deepest layer.
+ *
+ * @return the depth of the deepest layer.
+ */
+ public final int getDepth() {
+ return layerManager.getDepth();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth
+ *
+ * @param depth the depth of the filter to return.
+ * @return the Bloom filter at the specified depth.
+ * @throws NoSuchElementException if depth is not in the range [0,getDepth())
+ */
+ public BloomFilter get(int depth) {
+ return layerManager.get(depth);
+ }
+
+ @Override
+ public int cardinality() {
+ return SetOperations.cardinality(this);
+ }
+
+ @Override
+ public final void clear() {
+ layerManager.clear();
+ }
+
+ /**
+ * Clears the Bloom filter (removes all set bits) at the specified level.
+ *
+ * @param level the level to clear.
+ */
+ public final void clear(int level) {
+ layerManager.get(level).clear();
+ }
+
+ /**
+ * Get the Bloom filter that is currently being merged into. This method ensures
+ * that the {@code target} filter meets the criteria specified within the
+ * {@code LayerManager}. if the {@code LayerManager} determines that a new
+ * target filter is required it is constructed and used.
+ *
+ * @return the current Bloom filter.
+ * @see LayerManager
+ */
+ public final BloomFilter target() {
+ return layerManager.target();
+ }
+
+ /**
+ * Processes the Bloom filters in depth order with the most recent filters
+ * first. Each filter is passed to the predicate in turn. The function exits on
+ * the first {@code false} returned by the predicate.
+ *
+ * @param bloomFilterPredicate the predicate to execute.
+ * @return {@code true} if all filters passed the predicate, {@code false}
+ * otherwise.
+ */
+ @Override
+ public final boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate) {
+ return layerManager.forEachBloomFilter(bloomFilterPredicate);
+ }
+
+ /**
+ * Create a standard (non-layered) Bloom filter by merging all of the layers.
+ *
+ * @return the merged bloom filter.
+ */
+ public BloomFilter flatten() {
+ BloomFilter bf = new SimpleBloomFilter(shape);
+ forEachBloomFilter(bf::merge);
+ return bf;
+ }
+
+ /**
+ * Finds the layers in which the Hasher is found.
+ *
+ * @param hasher the Hasher to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the IndexProducer is found.
+ *
+ * @param indexProducer the Index producer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the BitMapProducer is found.
+ *
+ * @param bitMapProducer the BitMapProducer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the Bloom filter is found.
+ *
+ * @param bf the Bloom filter to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(BloomFilter bf) {
+ Finder finder = new Finder(bf);
+ forEachBloomFilter(finder);
+ return finder.getResult();
+ }
+
+ /**
+ * Returns {@code true} if this any layer contained by this filter contains the
+ * specified filter.
+ * <p>
+ * If the {@code other} is another Layered Bloom filter each filter within the
Review Comment:
`is another BloomFilterProducer` ???
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
+ };
+
+ /**
+ * Removes the earliest filters in the list when the the number of filters
+ * exceeds maxSize.
+ *
+ * @param maxSize the maximum number of filters for the list.
+ * @return A Consumer for the LayerManager filterCleanup constructor argument.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> onMaxSize(int maxSize) {
+ return (ll) -> {
+ while (ll.size() > maxSize) {
+ ll.removeFirst();
+ }
+ };
+ }
+ }
+
+ private final LinkedList<BloomFilter> filters = new LinkedList<>();
+ private final Consumer<LinkedList<BloomFilter>> filterCleanup;
+ private final Predicate<LayerManager> extendCheck;
+ private final Supplier<BloomFilter> filterSupplier;
+
+ /**
+ * Creates a new Builder with defaults of NEVER_ADVANCE and NO_CLEANUP
+ *
+ * @return A builder.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param filterSupplier the supplier of new Bloom filters to add the the list
+ * when necessary.
+ * @param extendCheck The predicate that checks if a new filter should be
+ * added to the list.
+ * @param filterCleanup the consumer the removes any old filters from the list.
+ */
+ private LayerManager(Supplier<BloomFilter> filterSupplier, Predicate<LayerManager> extendCheck,
+ Consumer<LinkedList<BloomFilter>> filterCleanup) {
+ this.filterSupplier = filterSupplier;
+ this.extendCheck = extendCheck;
+ this.filterCleanup = filterCleanup;
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Creates a deep copy of this LayerManager.
+ *
+ * @return a copy of this layer Manager.
+ */
+ public LayerManager copy() {
+ LayerManager newMgr = new LayerManager(filterSupplier, extendCheck, filterCleanup);
+ newMgr.filters.clear();
+ for (BloomFilter bf : filters) {
+ newMgr.filters.add(bf.copy());
+ }
+ return newMgr;
+ }
+
+ /**
+ * Forces an advance to the next depth for subsequent merges. Executes the same
+ * logic as when {@code ExtendCheck} returns {@code true}
+ */
+ public void next() {
+ if (!filters.isEmpty() && filters.getLast().cardinality() == 0) {
+ filters.removeLast();
+ }
+ this.filterCleanup.accept(filters);
+ filters.add(this.filterSupplier.get());
+ }
+
+ /**
+ * Returns the number of filters in the LayerManager.
+ *
+ * @return the current depth.
+ */
+ public final int getDepth() {
+ return filters.size();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth. The filter at depth 0 is the
+ * oldest filter.
+ *
+ * @param depth the depth at which the desired filter is to be found.
+ * @return the filter.
+ * @throws NoSuchElementException if depth is not in the range
+ * [0,filters.size())
+ */
+ public final BloomFilter get(int depth) {
Review Comment:
Does this need to be public? What is the use of being able to get a certain layer? If the layers are to function collectively why are you going to use a filter from a given depth?
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
+ * items than the Shape would indicate and yet still return a false positive
+ * rate in line with the Shape and not the over population.
+ * <p>
+ * This implementation uses a LayerManager to handle the manipulation of the
+ * layers.
+ * </p>
+ * <ul>
+ * <li>Level 0 is the oldest layer and the highest level is the newest.</li>
+ * <li>There is always at least one enclosed filter.</li>
+ * <li>The newest filter is the {@code target} into which merges are performed.
+ * <li>Whenever the target is retrieved, or a {@code merge} operation is
+ * performed the code checks if any older layers should be removed, and if so
+ * removes them. It also checks it a new layer should be added, and if so adds
+ * it and sets the {@code target} before the operation.</li>
+ * </ul>
+ */
+public class LayeredBloomFilter implements BloomFilter, BloomFilterProducer {
+ private final Shape shape;
+ private LayerManager layerManager;
+
+ /**
+ * Creates a fixed size layered bloom filter that adds new filters to the list,
+ * but never merges them. List will never exceed maxDepth. As additional filters
+ * are added earlier filters are removed.
+ *
+ * @param shape The shape for the enclosed Bloom filters
+ * @param maxDepth The maximum depth of layers.
+ * @return An empty layered Bloom filter of the specified shape and depth.
+ */
+ public static LayeredBloomFilter fixed(final Shape shape, int maxDepth) {
+ LayerManager manager = LayerManager.builder().withExtendCheck(LayerManager.ExtendCheck.ADVANCE_ON_POPULATED)
+ .withCleanup(LayerManager.Cleanup.onMaxSize(maxDepth)).withSuplier(() -> new SimpleBloomFilter(shape))
+ .build();
+ return new LayeredBloomFilter(shape, manager);
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param shape the Shape of the enclosed Bloom filters
+ * @param layerManager the LayerManager to manage the layers.
+ */
+ public LayeredBloomFilter(Shape shape, LayerManager layerManager) {
+ this.shape = shape;
+ this.layerManager = layerManager;
+ }
+
+ @Override
+ public LayeredBloomFilter copy() {
+ return new LayeredBloomFilter(shape, layerManager.copy());
+ }
+
+ /**
+ * Gets the depth of the deepest layer.
+ *
+ * @return the depth of the deepest layer.
+ */
+ public final int getDepth() {
+ return layerManager.getDepth();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth
+ *
+ * @param depth the depth of the filter to return.
+ * @return the Bloom filter at the specified depth.
+ * @throws NoSuchElementException if depth is not in the range [0,getDepth())
+ */
+ public BloomFilter get(int depth) {
+ return layerManager.get(depth);
+ }
+
+ @Override
+ public int cardinality() {
+ return SetOperations.cardinality(this);
+ }
+
+ @Override
+ public final void clear() {
+ layerManager.clear();
+ }
+
+ /**
+ * Clears the Bloom filter (removes all set bits) at the specified level.
+ *
+ * @param level the level to clear.
+ */
+ public final void clear(int level) {
+ layerManager.get(level).clear();
+ }
+
+ /**
+ * Get the Bloom filter that is currently being merged into. This method ensures
+ * that the {@code target} filter meets the criteria specified within the
+ * {@code LayerManager}. if the {@code LayerManager} determines that a new
+ * target filter is required it is constructed and used.
+ *
+ * @return the current Bloom filter.
+ * @see LayerManager
+ */
+ public final BloomFilter target() {
+ return layerManager.target();
+ }
+
+ /**
+ * Processes the Bloom filters in depth order with the most recent filters
+ * first. Each filter is passed to the predicate in turn. The function exits on
+ * the first {@code false} returned by the predicate.
+ *
+ * @param bloomFilterPredicate the predicate to execute.
+ * @return {@code true} if all filters passed the predicate, {@code false}
+ * otherwise.
+ */
+ @Override
+ public final boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate) {
+ return layerManager.forEachBloomFilter(bloomFilterPredicate);
+ }
+
+ /**
+ * Create a standard (non-layered) Bloom filter by merging all of the layers.
+ *
+ * @return the merged bloom filter.
+ */
+ public BloomFilter flatten() {
+ BloomFilter bf = new SimpleBloomFilter(shape);
+ forEachBloomFilter(bf::merge);
+ return bf;
+ }
+
+ /**
+ * Finds the layers in which the Hasher is found.
+ *
+ * @param hasher the Hasher to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the IndexProducer is found.
+ *
+ * @param indexProducer the Index producer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the BitMapProducer is found.
+ *
+ * @param bitMapProducer the BitMapProducer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the Bloom filter is found.
+ *
+ * @param bf the Bloom filter to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(BloomFilter bf) {
+ Finder finder = new Finder(bf);
+ forEachBloomFilter(finder);
+ return finder.getResult();
+ }
+
+ /**
+ * Returns {@code true} if this any layer contained by this filter contains the
+ * specified filter.
+ * <p>
+ * If the {@code other} is another Layered Bloom filter each filter within the
+ * {@code other} is checked to see if it exits within this filter.
+ * </p>
+ *
+ * @param other the other Bloom filter
+ * @return {@code true} if this filter contains the other filter.
+ */
+ @Override
+ public boolean contains(final BloomFilter other) {
+ if (other instanceof LayeredBloomFilter) {
+ boolean[] result = { true };
+ // return false when we have found a match.
+ ((LayeredBloomFilter) other).forEachBloomFilter(x -> {
+ result[0] &= contains(x);
+ return result[0];
+ });
+ return result[0];
+ }
+ return !forEachBloomFilter(x -> !x.contains(other));
+ }
+
+ /**
+ * Creates a Bloom filter from a Hasher.
+ *
+ * @param hasher the hasher to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return bf;
+ }
+
+ /**
+ * Creates a Bloom filter from an IndexProducer.
+ *
+ * @param indexProducer the IndexProducer to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return bf;
+ }
+
+ /**
+ * Creates a Bloom filter from a BitMapProducer.
+ *
+ * @param bitMapProducer the BitMapProducer to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return bf;
+ }
+
+ @Override
+ public int characteristics() {
+ return 0;
+ }
+
+ @Override
+ public final Shape getShape() {
+ return shape;
+ }
+
+ @Override
+ public boolean contains(final Hasher hasher) {
+ return contains(createFilter(hasher));
+ }
+
+ @Override
+ public boolean contains(final BitMapProducer bitMapProducer) {
+ return contains(createFilter(bitMapProducer));
+ }
+
+ @Override
+ public boolean contains(IndexProducer indexProducer) {
+ return contains(createFilter(indexProducer));
+ }
+
+ @Override
+ public boolean merge(BloomFilter bf) {
+ return target().merge(bf);
+ }
+
+ @Override
+ public boolean merge(IndexProducer indexProducer) {
+ return target().merge(indexProducer);
+ }
+
+ @Override
+ public boolean merge(BitMapProducer bitMapProducer) {
+ return target().merge(bitMapProducer);
+ }
+
+ @Override
+ public boolean forEachIndex(IntPredicate predicate) {
+ return forEachBloomFilter(bf -> bf.forEachIndex(predicate));
+ }
+
+ @Override
+ public boolean forEachBitMap(LongPredicate predicate) {
+ BloomFilter merged = new SimpleBloomFilter(shape);
+ if (forEachBloomFilter(merged::merge) && !merged.forEachBitMap(predicate)) {
+ return false;
+ }
+ return true;
+ }
+
+ @Override
+ public int estimateN() {
+ BloomFilter result = new SimpleBloomFilter(shape);
+ forEachBloomFilter(result::merge);
+ return result.estimateN();
+ }
+
+ @Override
+ public int estimateUnion(final BloomFilter other) {
+ Objects.requireNonNull(other, "other");
+ final BloomFilter cpy = this.flatten();
+ cpy.merge(other);
+ return cpy.estimateN();
+ }
+
+ @Override
+ public int estimateIntersection(final BloomFilter other) {
+ Objects.requireNonNull(other, "other");
+ long eThis = estimateN();
+ long eOther = other.estimateN();
+ if (eThis == Integer.MAX_VALUE && eOther == Integer.MAX_VALUE) {
+ // if both are infinite the union is infinite and we return Integer.MAX_VALUE
+ return Integer.MAX_VALUE;
+ }
+ long estimate;
+ // if one is infinite the intersection is the other.
+ if (eThis == Integer.MAX_VALUE) {
+ estimate = eOther;
+ } else if (eOther == Integer.MAX_VALUE) {
+ estimate = eThis;
+ } else {
+ long eUnion = estimateUnion(other);
+ if (eUnion == Integer.MAX_VALUE) {
+ throw new IllegalArgumentException("The estimated N for the union of the filters is infinite");
+ }
+ // maximum estimate value using integer values is: 46144189292 thus
+ // eThis + eOther can not overflow the long value.
+ estimate = eThis + eOther - eUnion;
+ estimate = estimate < 0 ? 0 : estimate;
+ }
+ return estimate > Integer.MAX_VALUE ? Integer.MAX_VALUE : (int) estimate;
+ }
+
+ /**
+ * Forces and advance to the next layer. Executes the same logic as when
+ * LayerManager.extendCheck returns {@code true}
+ *
+ * @see LayerManager
+ */
+ public void next() {
Review Comment:
What is the use case for this?
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
Review Comment:
Needs a check that `maxN` is strictly positive.
Can be a lambda:
```
return manager -> {
};
```
##########
src/test/java/org/apache/commons/collections4/bloomfilter/LayerManagerTest.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+
+import org.junit.jupiter.api.Test;
+
+public class LayerManagerTest {
+
+ private Shape shape = Shape.fromKM(17, 72);
+
+ private LayerManager.Builder testBuilder() {
+ return LayerManager.builder().withSuplier(() -> new SimpleBloomFilter(shape));
+ }
+
+ @Test
+ public void testAdvanceOnPopulated() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.ADVANCE_ON_POPULATED;
+ LayerManager layerManager = testBuilder().build();
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.FROM1);
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testNeverAdvance() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.NEVER_ADVANCE;
+ LayerManager layerManager = testBuilder().build();
+ assertFalse(underTest.test(layerManager));
Review Comment:
This could be put into a loop with e.g. 10 iterations to show it does not advance on repeat calls.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/WrappedBloomFilter.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+
+/**
+ * An abstract class to assist in implementing Bloom filter decorators.
+ *
+ * @since 4.5
+ */
+public abstract class WrappedBloomFilter implements BloomFilter {
Review Comment:
This class is (so far) only used in tests. Please move it to the test directory tree.
##########
src/test/java/org/apache/commons/collections4/bloomfilter/LayerManagerTest.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+
+import org.junit.jupiter.api.Test;
+
+public class LayerManagerTest {
+
+ private Shape shape = Shape.fromKM(17, 72);
+
+ private LayerManager.Builder testBuilder() {
+ return LayerManager.builder().withSuplier(() -> new SimpleBloomFilter(shape));
+ }
+
+ @Test
+ public void testAdvanceOnPopulated() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.ADVANCE_ON_POPULATED;
+ LayerManager layerManager = testBuilder().build();
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.FROM1);
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testNeverAdvance() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.NEVER_ADVANCE;
+ LayerManager layerManager = testBuilder().build();
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.FROM1);
+ assertFalse(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnCount() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnCount(4);
Review Comment:
4 should be a variable (breakAt) and used inside the loop as `int i = 1; i < breakAt; i++`
You could then parameterize the test using different breakAt values.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
Review Comment:
I think public static fields may be cleaner if replaced by methods that return the equivalent private singletons. This means you can obtain any ExtendCheck via a method, not a mix of methods or constants.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
+ * items than the Shape would indicate and yet still return a false positive
+ * rate in line with the Shape and not the over population.
+ * <p>
+ * This implementation uses a LayerManager to handle the manipulation of the
+ * layers.
+ * </p>
+ * <ul>
+ * <li>Level 0 is the oldest layer and the highest level is the newest.</li>
+ * <li>There is always at least one enclosed filter.</li>
+ * <li>The newest filter is the {@code target} into which merges are performed.
+ * <li>Whenever the target is retrieved, or a {@code merge} operation is
+ * performed the code checks if any older layers should be removed, and if so
+ * removes them. It also checks it a new layer should be added, and if so adds
+ * it and sets the {@code target} before the operation.</li>
+ * </ul>
+ */
+public class LayeredBloomFilter implements BloomFilter, BloomFilterProducer {
+ private final Shape shape;
+ private LayerManager layerManager;
+
+ /**
+ * Creates a fixed size layered bloom filter that adds new filters to the list,
+ * but never merges them. List will never exceed maxDepth. As additional filters
+ * are added earlier filters are removed.
+ *
+ * @param shape The shape for the enclosed Bloom filters
+ * @param maxDepth The maximum depth of layers.
+ * @return An empty layered Bloom filter of the specified shape and depth.
+ */
+ public static LayeredBloomFilter fixed(final Shape shape, int maxDepth) {
+ LayerManager manager = LayerManager.builder().withExtendCheck(LayerManager.ExtendCheck.ADVANCE_ON_POPULATED)
+ .withCleanup(LayerManager.Cleanup.onMaxSize(maxDepth)).withSuplier(() -> new SimpleBloomFilter(shape))
+ .build();
+ return new LayeredBloomFilter(shape, manager);
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param shape the Shape of the enclosed Bloom filters
+ * @param layerManager the LayerManager to manage the layers.
+ */
+ public LayeredBloomFilter(Shape shape, LayerManager layerManager) {
+ this.shape = shape;
+ this.layerManager = layerManager;
+ }
+
+ @Override
+ public LayeredBloomFilter copy() {
+ return new LayeredBloomFilter(shape, layerManager.copy());
+ }
+
+ /**
+ * Gets the depth of the deepest layer.
+ *
+ * @return the depth of the deepest layer.
+ */
+ public final int getDepth() {
+ return layerManager.getDepth();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth
+ *
+ * @param depth the depth of the filter to return.
+ * @return the Bloom filter at the specified depth.
+ * @throws NoSuchElementException if depth is not in the range [0,getDepth())
+ */
+ public BloomFilter get(int depth) {
+ return layerManager.get(depth);
+ }
+
+ @Override
+ public int cardinality() {
+ return SetOperations.cardinality(this);
+ }
+
+ @Override
+ public final void clear() {
+ layerManager.clear();
+ }
+
+ /**
+ * Clears the Bloom filter (removes all set bits) at the specified level.
+ *
+ * @param level the level to clear.
+ */
+ public final void clear(int level) {
+ layerManager.get(level).clear();
+ }
+
+ /**
+ * Get the Bloom filter that is currently being merged into. This method ensures
+ * that the {@code target} filter meets the criteria specified within the
+ * {@code LayerManager}. if the {@code LayerManager} determines that a new
+ * target filter is required it is constructed and used.
+ *
+ * @return the current Bloom filter.
+ * @see LayerManager
+ */
+ public final BloomFilter target() {
+ return layerManager.target();
+ }
+
+ /**
+ * Processes the Bloom filters in depth order with the most recent filters
+ * first. Each filter is passed to the predicate in turn. The function exits on
+ * the first {@code false} returned by the predicate.
+ *
+ * @param bloomFilterPredicate the predicate to execute.
+ * @return {@code true} if all filters passed the predicate, {@code false}
+ * otherwise.
+ */
+ @Override
+ public final boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate) {
+ return layerManager.forEachBloomFilter(bloomFilterPredicate);
+ }
+
+ /**
+ * Create a standard (non-layered) Bloom filter by merging all of the layers.
+ *
+ * @return the merged bloom filter.
+ */
+ public BloomFilter flatten() {
+ BloomFilter bf = new SimpleBloomFilter(shape);
+ forEachBloomFilter(bf::merge);
+ return bf;
+ }
+
+ /**
+ * Finds the layers in which the Hasher is found.
+ *
+ * @param hasher the Hasher to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the IndexProducer is found.
+ *
+ * @param indexProducer the Index producer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the BitMapProducer is found.
+ *
+ * @param bitMapProducer the BitMapProducer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the Bloom filter is found.
+ *
+ * @param bf the Bloom filter to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(BloomFilter bf) {
+ Finder finder = new Finder(bf);
+ forEachBloomFilter(finder);
+ return finder.getResult();
+ }
+
+ /**
+ * Returns {@code true} if this any layer contained by this filter contains the
+ * specified filter.
+ * <p>
+ * If the {@code other} is another Layered Bloom filter each filter within the
+ * {@code other} is checked to see if it exits within this filter.
+ * </p>
+ *
+ * @param other the other Bloom filter
+ * @return {@code true} if this filter contains the other filter.
+ */
+ @Override
+ public boolean contains(final BloomFilter other) {
+ if (other instanceof LayeredBloomFilter) {
+ boolean[] result = { true };
+ // return false when we have found a match.
+ ((LayeredBloomFilter) other).forEachBloomFilter(x -> {
+ result[0] &= contains(x);
+ return result[0];
+ });
+ return result[0];
+ }
+ return !forEachBloomFilter(x -> !x.contains(other));
+ }
+
+ /**
+ * Creates a Bloom filter from a Hasher.
+ *
+ * @param hasher the hasher to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return bf;
+ }
+
+ /**
+ * Creates a Bloom filter from an IndexProducer.
+ *
+ * @param indexProducer the IndexProducer to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return bf;
+ }
+
+ /**
+ * Creates a Bloom filter from a BitMapProducer.
+ *
+ * @param bitMapProducer the BitMapProducer to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return bf;
+ }
+
+ @Override
+ public int characteristics() {
+ return 0;
+ }
+
+ @Override
+ public final Shape getShape() {
+ return shape;
+ }
+
+ @Override
+ public boolean contains(final Hasher hasher) {
+ return contains(createFilter(hasher));
+ }
+
+ @Override
+ public boolean contains(final BitMapProducer bitMapProducer) {
+ return contains(createFilter(bitMapProducer));
+ }
+
+ @Override
+ public boolean contains(IndexProducer indexProducer) {
+ return contains(createFilter(indexProducer));
+ }
+
+ @Override
+ public boolean merge(BloomFilter bf) {
+ return target().merge(bf);
+ }
+
+ @Override
+ public boolean merge(IndexProducer indexProducer) {
+ return target().merge(indexProducer);
+ }
+
+ @Override
+ public boolean merge(BitMapProducer bitMapProducer) {
+ return target().merge(bitMapProducer);
+ }
+
+ @Override
+ public boolean forEachIndex(IntPredicate predicate) {
+ return forEachBloomFilter(bf -> bf.forEachIndex(predicate));
+ }
+
+ @Override
+ public boolean forEachBitMap(LongPredicate predicate) {
+ BloomFilter merged = new SimpleBloomFilter(shape);
+ if (forEachBloomFilter(merged::merge) && !merged.forEachBitMap(predicate)) {
Review Comment:
The logic here seems wrong. If `forEachBloomFilter(merged::merge)` returns false then the method will return true despite not ever using the predicate.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(double maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ BloomFilter bf = manager.filters.peekLast();
+ return maxN <= bf.getShape().estimateN(bf.cardinality());
+ }
+ };
+ }
+ }
+
+ /**
+ * Static methods to create a Consumer of a LinkedList of BloomFilter to manage
+ * the size of the list.
+ *
+ */
+ public static class Cleanup {
+ private Cleanup() {
+ }
+
+ /**
+ * A Cleanup that never removes anything.
+ */
+ public static final Consumer<LinkedList<BloomFilter>> NO_CLEANUP = x -> {
Review Comment:
Access singleton via a static method.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
+ * items than the Shape would indicate and yet still return a false positive
+ * rate in line with the Shape and not the over population.
+ * <p>
+ * This implementation uses a LayerManager to handle the manipulation of the
+ * layers.
+ * </p>
+ * <ul>
+ * <li>Level 0 is the oldest layer and the highest level is the newest.</li>
+ * <li>There is always at least one enclosed filter.</li>
+ * <li>The newest filter is the {@code target} into which merges are performed.
+ * <li>Whenever the target is retrieved, or a {@code merge} operation is
+ * performed the code checks if any older layers should be removed, and if so
+ * removes them. It also checks it a new layer should be added, and if so adds
+ * it and sets the {@code target} before the operation.</li>
+ * </ul>
+ */
+public class LayeredBloomFilter implements BloomFilter, BloomFilterProducer {
+ private final Shape shape;
+ private LayerManager layerManager;
+
+ /**
+ * Creates a fixed size layered bloom filter that adds new filters to the list,
+ * but never merges them. List will never exceed maxDepth. As additional filters
+ * are added earlier filters are removed.
+ *
+ * @param shape The shape for the enclosed Bloom filters
+ * @param maxDepth The maximum depth of layers.
+ * @return An empty layered Bloom filter of the specified shape and depth.
+ */
+ public static LayeredBloomFilter fixed(final Shape shape, int maxDepth) {
+ LayerManager manager = LayerManager.builder().withExtendCheck(LayerManager.ExtendCheck.ADVANCE_ON_POPULATED)
+ .withCleanup(LayerManager.Cleanup.onMaxSize(maxDepth)).withSuplier(() -> new SimpleBloomFilter(shape))
+ .build();
+ return new LayeredBloomFilter(shape, manager);
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param shape the Shape of the enclosed Bloom filters
+ * @param layerManager the LayerManager to manage the layers.
+ */
+ public LayeredBloomFilter(Shape shape, LayerManager layerManager) {
+ this.shape = shape;
+ this.layerManager = layerManager;
+ }
+
+ @Override
+ public LayeredBloomFilter copy() {
+ return new LayeredBloomFilter(shape, layerManager.copy());
+ }
+
+ /**
+ * Gets the depth of the deepest layer.
+ *
+ * @return the depth of the deepest layer.
+ */
+ public final int getDepth() {
+ return layerManager.getDepth();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth
+ *
+ * @param depth the depth of the filter to return.
+ * @return the Bloom filter at the specified depth.
+ * @throws NoSuchElementException if depth is not in the range [0,getDepth())
+ */
+ public BloomFilter get(int depth) {
+ return layerManager.get(depth);
+ }
+
+ @Override
+ public int cardinality() {
+ return SetOperations.cardinality(this);
+ }
+
+ @Override
+ public final void clear() {
+ layerManager.clear();
+ }
+
+ /**
+ * Clears the Bloom filter (removes all set bits) at the specified level.
+ *
+ * @param level the level to clear.
+ */
+ public final void clear(int level) {
+ layerManager.get(level).clear();
+ }
+
+ /**
+ * Get the Bloom filter that is currently being merged into. This method ensures
+ * that the {@code target} filter meets the criteria specified within the
+ * {@code LayerManager}. if the {@code LayerManager} determines that a new
+ * target filter is required it is constructed and used.
+ *
+ * @return the current Bloom filter.
+ * @see LayerManager
+ */
+ public final BloomFilter target() {
Review Comment:
`getTarget`
Why provide this method? Why does the top layer of the filter matter to the end user. The target is used correctly internally by merge operations, and layers are used correctly by contains.
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
+ * items than the Shape would indicate and yet still return a false positive
+ * rate in line with the Shape and not the over population.
+ * <p>
+ * This implementation uses a LayerManager to handle the manipulation of the
+ * layers.
+ * </p>
+ * <ul>
+ * <li>Level 0 is the oldest layer and the highest level is the newest.</li>
+ * <li>There is always at least one enclosed filter.</li>
+ * <li>The newest filter is the {@code target} into which merges are performed.
+ * <li>Whenever the target is retrieved, or a {@code merge} operation is
+ * performed the code checks if any older layers should be removed, and if so
+ * removes them. It also checks it a new layer should be added, and if so adds
+ * it and sets the {@code target} before the operation.</li>
+ * </ul>
+ */
+public class LayeredBloomFilter implements BloomFilter, BloomFilterProducer {
+ private final Shape shape;
+ private LayerManager layerManager;
+
+ /**
+ * Creates a fixed size layered bloom filter that adds new filters to the list,
+ * but never merges them. List will never exceed maxDepth. As additional filters
+ * are added earlier filters are removed.
+ *
+ * @param shape The shape for the enclosed Bloom filters
+ * @param maxDepth The maximum depth of layers.
+ * @return An empty layered Bloom filter of the specified shape and depth.
+ */
+ public static LayeredBloomFilter fixed(final Shape shape, int maxDepth) {
+ LayerManager manager = LayerManager.builder().withExtendCheck(LayerManager.ExtendCheck.ADVANCE_ON_POPULATED)
+ .withCleanup(LayerManager.Cleanup.onMaxSize(maxDepth)).withSuplier(() -> new SimpleBloomFilter(shape))
+ .build();
+ return new LayeredBloomFilter(shape, manager);
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param shape the Shape of the enclosed Bloom filters
+ * @param layerManager the LayerManager to manage the layers.
+ */
+ public LayeredBloomFilter(Shape shape, LayerManager layerManager) {
+ this.shape = shape;
+ this.layerManager = layerManager;
+ }
+
+ @Override
+ public LayeredBloomFilter copy() {
+ return new LayeredBloomFilter(shape, layerManager.copy());
+ }
+
+ /**
+ * Gets the depth of the deepest layer.
+ *
+ * @return the depth of the deepest layer.
+ */
+ public final int getDepth() {
+ return layerManager.getDepth();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth
+ *
+ * @param depth the depth of the filter to return.
+ * @return the Bloom filter at the specified depth.
+ * @throws NoSuchElementException if depth is not in the range [0,getDepth())
+ */
+ public BloomFilter get(int depth) {
+ return layerManager.get(depth);
+ }
+
+ @Override
+ public int cardinality() {
+ return SetOperations.cardinality(this);
+ }
+
+ @Override
+ public final void clear() {
+ layerManager.clear();
+ }
+
+ /**
+ * Clears the Bloom filter (removes all set bits) at the specified level.
+ *
+ * @param level the level to clear.
+ */
+ public final void clear(int level) {
+ layerManager.get(level).clear();
+ }
+
+ /**
+ * Get the Bloom filter that is currently being merged into. This method ensures
+ * that the {@code target} filter meets the criteria specified within the
+ * {@code LayerManager}. if the {@code LayerManager} determines that a new
+ * target filter is required it is constructed and used.
+ *
+ * @return the current Bloom filter.
+ * @see LayerManager
+ */
+ public final BloomFilter target() {
+ return layerManager.target();
+ }
+
+ /**
+ * Processes the Bloom filters in depth order with the most recent filters
+ * first. Each filter is passed to the predicate in turn. The function exits on
+ * the first {@code false} returned by the predicate.
+ *
+ * @param bloomFilterPredicate the predicate to execute.
+ * @return {@code true} if all filters passed the predicate, {@code false}
+ * otherwise.
+ */
+ @Override
+ public final boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate) {
+ return layerManager.forEachBloomFilter(bloomFilterPredicate);
+ }
+
+ /**
+ * Create a standard (non-layered) Bloom filter by merging all of the layers.
+ *
+ * @return the merged bloom filter.
+ */
+ public BloomFilter flatten() {
+ BloomFilter bf = new SimpleBloomFilter(shape);
+ forEachBloomFilter(bf::merge);
+ return bf;
+ }
+
+ /**
+ * Finds the layers in which the Hasher is found.
+ *
+ * @param hasher the Hasher to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the IndexProducer is found.
+ *
+ * @param indexProducer the Index producer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the BitMapProducer is found.
+ *
+ * @param bitMapProducer the BitMapProducer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the Bloom filter is found.
+ *
+ * @param bf the Bloom filter to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(BloomFilter bf) {
+ Finder finder = new Finder(bf);
+ forEachBloomFilter(finder);
+ return finder.getResult();
+ }
+
+ /**
+ * Returns {@code true} if this any layer contained by this filter contains the
+ * specified filter.
+ * <p>
+ * If the {@code other} is another Layered Bloom filter each filter within the
+ * {@code other} is checked to see if it exits within this filter.
+ * </p>
+ *
+ * @param other the other Bloom filter
+ * @return {@code true} if this filter contains the other filter.
+ */
+ @Override
+ public boolean contains(final BloomFilter other) {
+ if (other instanceof LayeredBloomFilter) {
+ boolean[] result = { true };
+ // return false when we have found a match.
+ ((LayeredBloomFilter) other).forEachBloomFilter(x -> {
+ result[0] &= contains(x);
+ return result[0];
+ });
+ return result[0];
+ }
+ return !forEachBloomFilter(x -> !x.contains(other));
+ }
+
+ /**
+ * Creates a Bloom filter from a Hasher.
+ *
+ * @param hasher the hasher to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return bf;
+ }
+
+ /**
+ * Creates a Bloom filter from an IndexProducer.
+ *
+ * @param indexProducer the IndexProducer to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return bf;
+ }
+
+ /**
+ * Creates a Bloom filter from a BitMapProducer.
+ *
+ * @param bitMapProducer the BitMapProducer to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return bf;
+ }
+
+ @Override
+ public int characteristics() {
+ return 0;
+ }
+
+ @Override
+ public final Shape getShape() {
+ return shape;
+ }
+
+ @Override
+ public boolean contains(final Hasher hasher) {
+ return contains(createFilter(hasher));
+ }
+
+ @Override
+ public boolean contains(final BitMapProducer bitMapProducer) {
+ return contains(createFilter(bitMapProducer));
+ }
+
+ @Override
+ public boolean contains(IndexProducer indexProducer) {
+ return contains(createFilter(indexProducer));
+ }
+
+ @Override
+ public boolean merge(BloomFilter bf) {
+ return target().merge(bf);
+ }
+
+ @Override
+ public boolean merge(IndexProducer indexProducer) {
+ return target().merge(indexProducer);
+ }
+
+ @Override
+ public boolean merge(BitMapProducer bitMapProducer) {
+ return target().merge(bitMapProducer);
+ }
+
+ @Override
+ public boolean forEachIndex(IntPredicate predicate) {
+ return forEachBloomFilter(bf -> bf.forEachIndex(predicate));
+ }
+
+ @Override
+ public boolean forEachBitMap(LongPredicate predicate) {
+ BloomFilter merged = new SimpleBloomFilter(shape);
+ if (forEachBloomFilter(merged::merge) && !merged.forEachBitMap(predicate)) {
+ return false;
+ }
+ return true;
+ }
+
+ @Override
+ public int estimateN() {
+ BloomFilter result = new SimpleBloomFilter(shape);
+ forEachBloomFilter(result::merge);
+ return result.estimateN();
+ }
+
+ @Override
+ public int estimateUnion(final BloomFilter other) {
+ Objects.requireNonNull(other, "other");
+ final BloomFilter cpy = this.flatten();
+ cpy.merge(other);
+ return cpy.estimateN();
+ }
+
+ @Override
+ public int estimateIntersection(final BloomFilter other) {
Review Comment:
Why override to use the int variation of the double computation in the BloomFilter interface? IIUC since cardinality() and estimateN() both use a flattened version of the layers then they should be the same with allowance for rounding. Am I missing something?
I note that the default implementation in BloomFilter uses the Shape of the current filter for both estimates. With this variation the estimate will use the shape of the filter that is estimating N. Is this a bug? Is the estimateN method only valid if the shape matches?
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayerManager.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.LinkedList;
+import java.util.NoSuchElementException;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+/**
+ * Implementation of the methods to manage the Layers in a Layered Bloom filter.
+ * <p>
+ * The manager comprises a list of Bloom filters that are managed based on
+ * various rules. The last filter in the list is known as the {@code target} and
+ * is the filter into which merges are performed. The Layered manager utilizes
+ * three methods to manage the list.
+ * </p>
+ * <ul>
+ * <li>ExtendCheck - A Predicate that if true causes a new Bloom filter to be
+ * created as the new target.</li>
+ * <li>FilterSupplier - A Supplier that produces empty Bloom filters to be used
+ * as a new target.</li>
+ * <li>Cleanup - A Consumer of a LinkedList of BloomFilter that removes any
+ * expired or out dated filters from the list.</li>
+ * </ul>
+ * <p>
+ * When extendCheck returns {@code true} the following steps are taken:
+ * </p>
+ * <ol>
+ * <li>If the current target is empty it is removed.</li>
+ * <li>{@code Cleanup} is called</li>
+ * <li>{@code FilterSuplier} is executed and the new filter added to the list as
+ * the {@code target} filter.</li>
+ * </ol>
+ *
+ * @since 4.5
+ */
+public class LayerManager implements BloomFilterProducer {
+
+ /**
+ * Static methods an variable for standard extend checks.
+ *
+ */
+ public static class ExtendCheck {
+ private ExtendCheck() {
+ }
+
+ /**
+ * Advances the target once a merge has been performed.
+ */
+ public static final Predicate<LayerManager> ADVANCE_ON_POPULATED = lm -> {
+ return !lm.filters.isEmpty() && !lm.filters.peekLast().forEachBitMap(y -> y == 0);
+ };
+
+ /**
+ * Does not automatically advance the target. next() must be called directly to
+ * perform the advance.
+ */
+ public static final Predicate<LayerManager> NEVER_ADVANCE = x -> false;
+
+ /**
+ * Calculates the estimated number of Bloom filters (n) that have been merged
+ * into the target and compares that with the estimated maximum expected n based
+ * on the shape. If the target is full then a new target is created.
+ *
+ * @param shape The shape of the filters in the LayerManager.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnCalculatedFull(Shape shape) {
+ return advanceOnSaturation(shape.estimateMaxN());
+ }
+
+ /**
+ * Creates a new target after a specific number of filters have been added to
+ * the current target.
+ *
+ * @param breakAt the number of filters to merge into each filter in the list.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static Predicate<LayerManager> advanceOnCount(int breakAt) {
+ return new Predicate<LayerManager>() {
+ int count = 0;
+
+ @Override
+ public boolean test(LayerManager filter) {
+ return ++count % breakAt == 0;
+ }
+ };
+ }
+
+ /**
+ * Creates a new target after the current target is saturated. Saturation is
+ * defined as the estimated N of the target Bloom filter being greater than the
+ * maxN specified.
+ * <p>
+ * This method uses the integer estimation found in the Bloom filter. To use the
+ * estimation from the Shape use the double version of this function.
+ *
+ * @param maxN the maximum number of estimated items in the filter.
+ * @return A Predicate suitable for the LayerManager extendCheck parameter.
+ */
+ public static final Predicate<LayerManager> advanceOnSaturation(int maxN) {
+ return new Predicate<LayerManager>() {
+ @Override
+ public boolean test(LayerManager manager) {
+ if (manager.filters.isEmpty()) {
+ return false;
+ }
+ return maxN <= manager.filters.peekLast().estimateN();
+ }
+
Review Comment:
Remove empty line
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
+ * items than the Shape would indicate and yet still return a false positive
+ * rate in line with the Shape and not the over population.
+ * <p>
+ * This implementation uses a LayerManager to handle the manipulation of the
+ * layers.
+ * </p>
+ * <ul>
+ * <li>Level 0 is the oldest layer and the highest level is the newest.</li>
+ * <li>There is always at least one enclosed filter.</li>
+ * <li>The newest filter is the {@code target} into which merges are performed.
+ * <li>Whenever the target is retrieved, or a {@code merge} operation is
+ * performed the code checks if any older layers should be removed, and if so
+ * removes them. It also checks it a new layer should be added, and if so adds
+ * it and sets the {@code target} before the operation.</li>
+ * </ul>
+ */
+public class LayeredBloomFilter implements BloomFilter, BloomFilterProducer {
+ private final Shape shape;
+ private LayerManager layerManager;
+
+ /**
+ * Creates a fixed size layered bloom filter that adds new filters to the list,
+ * but never merges them. List will never exceed maxDepth. As additional filters
+ * are added earlier filters are removed.
+ *
+ * @param shape The shape for the enclosed Bloom filters
+ * @param maxDepth The maximum depth of layers.
+ * @return An empty layered Bloom filter of the specified shape and depth.
+ */
+ public static LayeredBloomFilter fixed(final Shape shape, int maxDepth) {
+ LayerManager manager = LayerManager.builder().withExtendCheck(LayerManager.ExtendCheck.ADVANCE_ON_POPULATED)
+ .withCleanup(LayerManager.Cleanup.onMaxSize(maxDepth)).withSuplier(() -> new SimpleBloomFilter(shape))
+ .build();
+ return new LayeredBloomFilter(shape, manager);
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param shape the Shape of the enclosed Bloom filters
+ * @param layerManager the LayerManager to manage the layers.
+ */
+ public LayeredBloomFilter(Shape shape, LayerManager layerManager) {
+ this.shape = shape;
+ this.layerManager = layerManager;
+ }
+
+ @Override
+ public LayeredBloomFilter copy() {
+ return new LayeredBloomFilter(shape, layerManager.copy());
+ }
+
+ /**
+ * Gets the depth of the deepest layer.
+ *
+ * @return the depth of the deepest layer.
+ */
+ public final int getDepth() {
+ return layerManager.getDepth();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth
+ *
+ * @param depth the depth of the filter to return.
+ * @return the Bloom filter at the specified depth.
+ * @throws NoSuchElementException if depth is not in the range [0,getDepth())
+ */
+ public BloomFilter get(int depth) {
+ return layerManager.get(depth);
+ }
+
+ @Override
+ public int cardinality() {
+ return SetOperations.cardinality(this);
+ }
+
+ @Override
+ public final void clear() {
+ layerManager.clear();
+ }
+
+ /**
+ * Clears the Bloom filter (removes all set bits) at the specified level.
+ *
+ * @param level the level to clear.
+ */
+ public final void clear(int level) {
+ layerManager.get(level).clear();
+ }
+
+ /**
+ * Get the Bloom filter that is currently being merged into. This method ensures
+ * that the {@code target} filter meets the criteria specified within the
+ * {@code LayerManager}. if the {@code LayerManager} determines that a new
+ * target filter is required it is constructed and used.
+ *
+ * @return the current Bloom filter.
+ * @see LayerManager
+ */
+ public final BloomFilter target() {
+ return layerManager.target();
+ }
+
+ /**
+ * Processes the Bloom filters in depth order with the most recent filters
+ * first. Each filter is passed to the predicate in turn. The function exits on
+ * the first {@code false} returned by the predicate.
+ *
+ * @param bloomFilterPredicate the predicate to execute.
+ * @return {@code true} if all filters passed the predicate, {@code false}
+ * otherwise.
+ */
+ @Override
+ public final boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate) {
+ return layerManager.forEachBloomFilter(bloomFilterPredicate);
+ }
+
+ /**
+ * Create a standard (non-layered) Bloom filter by merging all of the layers.
+ *
+ * @return the merged bloom filter.
+ */
+ public BloomFilter flatten() {
+ BloomFilter bf = new SimpleBloomFilter(shape);
+ forEachBloomFilter(bf::merge);
+ return bf;
+ }
+
+ /**
+ * Finds the layers in which the Hasher is found.
+ *
+ * @param hasher the Hasher to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the IndexProducer is found.
+ *
+ * @param indexProducer the Index producer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the BitMapProducer is found.
+ *
+ * @param bitMapProducer the BitMapProducer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the Bloom filter is found.
+ *
+ * @param bf the Bloom filter to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(BloomFilter bf) {
+ Finder finder = new Finder(bf);
+ forEachBloomFilter(finder);
+ return finder.getResult();
+ }
+
+ /**
+ * Returns {@code true} if this any layer contained by this filter contains the
+ * specified filter.
+ * <p>
+ * If the {@code other} is another Layered Bloom filter each filter within the
+ * {@code other} is checked to see if it exits within this filter.
+ * </p>
+ *
+ * @param other the other Bloom filter
+ * @return {@code true} if this filter contains the other filter.
+ */
+ @Override
+ public boolean contains(final BloomFilter other) {
+ if (other instanceof LayeredBloomFilter) {
+ boolean[] result = { true };
+ // return false when we have found a match.
+ ((LayeredBloomFilter) other).forEachBloomFilter(x -> {
+ result[0] &= contains(x);
+ return result[0];
+ });
+ return result[0];
+ }
+ return !forEachBloomFilter(x -> !x.contains(other));
+ }
+
+ /**
+ * Creates a Bloom filter from a Hasher.
+ *
+ * @param hasher the hasher to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return bf;
+ }
+
+ /**
+ * Creates a Bloom filter from an IndexProducer.
+ *
+ * @param indexProducer the IndexProducer to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return bf;
+ }
+
+ /**
+ * Creates a Bloom filter from a BitMapProducer.
+ *
+ * @param bitMapProducer the BitMapProducer to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return bf;
+ }
+
+ @Override
+ public int characteristics() {
+ return 0;
+ }
+
+ @Override
+ public final Shape getShape() {
+ return shape;
+ }
+
+ @Override
+ public boolean contains(final Hasher hasher) {
+ return contains(createFilter(hasher));
+ }
+
+ @Override
+ public boolean contains(final BitMapProducer bitMapProducer) {
+ return contains(createFilter(bitMapProducer));
+ }
+
+ @Override
+ public boolean contains(IndexProducer indexProducer) {
+ return contains(createFilter(indexProducer));
+ }
+
+ @Override
+ public boolean merge(BloomFilter bf) {
+ return target().merge(bf);
+ }
+
+ @Override
+ public boolean merge(IndexProducer indexProducer) {
+ return target().merge(indexProducer);
+ }
+
+ @Override
+ public boolean merge(BitMapProducer bitMapProducer) {
+ return target().merge(bitMapProducer);
+ }
+
+ @Override
+ public boolean forEachIndex(IntPredicate predicate) {
+ return forEachBloomFilter(bf -> bf.forEachIndex(predicate));
+ }
+
+ @Override
+ public boolean forEachBitMap(LongPredicate predicate) {
Review Comment:
Should this have a comment that the flattened bit map from all layers is used, ie. is like `flatten().forEachBitMap(predicate)`
##########
src/main/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilter.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import java.util.Arrays;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.function.IntPredicate;
+import java.util.function.LongPredicate;
+import java.util.function.Predicate;
+
+/**
+ * Layered Bloom filters are described in Zhiwang, Cen; Jungang, Xu; Jian, Sun
+ * (2010), "A multi-layer Bloom filter for duplicated URL detection", Proc. 3rd
+ * International Conference on Advanced Computer Theory and Engineering (ICACTE
+ * 2010), vol. 1, pp. V1–586–V1–591, doi:10.1109/ICACTE.2010.5578947, ISBN
+ * 978-1-4244-6539-2, S2CID 3108985
+ * <p>
+ * In short, Layered Bloom filter contains several bloom filters arranged in
+ * layers.
+ * </p>
+ * <ul>
+ * <li>When membership in the filter is checked each layer in turn is checked
+ * and if a match is found {@code true} is returned.</li>
+ * <li>When merging each bloom filter is merged into the last newest filter in
+ * the list of layers.</li>
+ * <li>When questions of cardinality are asked the cardinality of the union of
+ * the enclosed Bloom filters is used.</li>
+ * </ul>
+ * The net result is that the layered Bloom filter can be populated with more
+ * items than the Shape would indicate and yet still return a false positive
+ * rate in line with the Shape and not the over population.
+ * <p>
+ * This implementation uses a LayerManager to handle the manipulation of the
+ * layers.
+ * </p>
+ * <ul>
+ * <li>Level 0 is the oldest layer and the highest level is the newest.</li>
+ * <li>There is always at least one enclosed filter.</li>
+ * <li>The newest filter is the {@code target} into which merges are performed.
+ * <li>Whenever the target is retrieved, or a {@code merge} operation is
+ * performed the code checks if any older layers should be removed, and if so
+ * removes them. It also checks it a new layer should be added, and if so adds
+ * it and sets the {@code target} before the operation.</li>
+ * </ul>
+ */
+public class LayeredBloomFilter implements BloomFilter, BloomFilterProducer {
+ private final Shape shape;
+ private LayerManager layerManager;
+
+ /**
+ * Creates a fixed size layered bloom filter that adds new filters to the list,
+ * but never merges them. List will never exceed maxDepth. As additional filters
+ * are added earlier filters are removed.
+ *
+ * @param shape The shape for the enclosed Bloom filters
+ * @param maxDepth The maximum depth of layers.
+ * @return An empty layered Bloom filter of the specified shape and depth.
+ */
+ public static LayeredBloomFilter fixed(final Shape shape, int maxDepth) {
+ LayerManager manager = LayerManager.builder().withExtendCheck(LayerManager.ExtendCheck.ADVANCE_ON_POPULATED)
+ .withCleanup(LayerManager.Cleanup.onMaxSize(maxDepth)).withSuplier(() -> new SimpleBloomFilter(shape))
+ .build();
+ return new LayeredBloomFilter(shape, manager);
+ }
+
+ /**
+ * Constructor.
+ *
+ * @param shape the Shape of the enclosed Bloom filters
+ * @param layerManager the LayerManager to manage the layers.
+ */
+ public LayeredBloomFilter(Shape shape, LayerManager layerManager) {
+ this.shape = shape;
+ this.layerManager = layerManager;
+ }
+
+ @Override
+ public LayeredBloomFilter copy() {
+ return new LayeredBloomFilter(shape, layerManager.copy());
+ }
+
+ /**
+ * Gets the depth of the deepest layer.
+ *
+ * @return the depth of the deepest layer.
+ */
+ public final int getDepth() {
+ return layerManager.getDepth();
+ }
+
+ /**
+ * Gets the Bloom filter at the specified depth
+ *
+ * @param depth the depth of the filter to return.
+ * @return the Bloom filter at the specified depth.
+ * @throws NoSuchElementException if depth is not in the range [0,getDepth())
+ */
+ public BloomFilter get(int depth) {
+ return layerManager.get(depth);
+ }
+
+ @Override
+ public int cardinality() {
+ return SetOperations.cardinality(this);
+ }
+
+ @Override
+ public final void clear() {
+ layerManager.clear();
+ }
+
+ /**
+ * Clears the Bloom filter (removes all set bits) at the specified level.
+ *
+ * @param level the level to clear.
+ */
+ public final void clear(int level) {
+ layerManager.get(level).clear();
+ }
+
+ /**
+ * Get the Bloom filter that is currently being merged into. This method ensures
+ * that the {@code target} filter meets the criteria specified within the
+ * {@code LayerManager}. if the {@code LayerManager} determines that a new
+ * target filter is required it is constructed and used.
+ *
+ * @return the current Bloom filter.
+ * @see LayerManager
+ */
+ public final BloomFilter target() {
+ return layerManager.target();
+ }
+
+ /**
+ * Processes the Bloom filters in depth order with the most recent filters
+ * first. Each filter is passed to the predicate in turn. The function exits on
+ * the first {@code false} returned by the predicate.
+ *
+ * @param bloomFilterPredicate the predicate to execute.
+ * @return {@code true} if all filters passed the predicate, {@code false}
+ * otherwise.
+ */
+ @Override
+ public final boolean forEachBloomFilter(Predicate<BloomFilter> bloomFilterPredicate) {
+ return layerManager.forEachBloomFilter(bloomFilterPredicate);
+ }
+
+ /**
+ * Create a standard (non-layered) Bloom filter by merging all of the layers.
+ *
+ * @return the merged bloom filter.
+ */
+ public BloomFilter flatten() {
+ BloomFilter bf = new SimpleBloomFilter(shape);
+ forEachBloomFilter(bf::merge);
+ return bf;
+ }
+
+ /**
+ * Finds the layers in which the Hasher is found.
+ *
+ * @param hasher the Hasher to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the IndexProducer is found.
+ *
+ * @param indexProducer the Index producer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the BitMapProducer is found.
+ *
+ * @param bitMapProducer the BitMapProducer to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return find(bf);
+ }
+
+ /**
+ * Finds the layers in which the Bloom filter is found.
+ *
+ * @param bf the Bloom filter to search for.
+ * @return an array of layer indices in which the Bloom filter is found.
+ */
+ public int[] find(BloomFilter bf) {
+ Finder finder = new Finder(bf);
+ forEachBloomFilter(finder);
+ return finder.getResult();
+ }
+
+ /**
+ * Returns {@code true} if this any layer contained by this filter contains the
+ * specified filter.
+ * <p>
+ * If the {@code other} is another Layered Bloom filter each filter within the
+ * {@code other} is checked to see if it exits within this filter.
+ * </p>
+ *
+ * @param other the other Bloom filter
+ * @return {@code true} if this filter contains the other filter.
+ */
+ @Override
+ public boolean contains(final BloomFilter other) {
+ if (other instanceof LayeredBloomFilter) {
+ boolean[] result = { true };
+ // return false when we have found a match.
+ ((LayeredBloomFilter) other).forEachBloomFilter(x -> {
+ result[0] &= contains(x);
+ return result[0];
+ });
+ return result[0];
+ }
+ return !forEachBloomFilter(x -> !x.contains(other));
+ }
+
+ /**
+ * Creates a Bloom filter from a Hasher.
+ *
+ * @param hasher the hasher to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final Hasher hasher) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(hasher);
+ return bf;
+ }
+
+ /**
+ * Creates a Bloom filter from an IndexProducer.
+ *
+ * @param indexProducer the IndexProducer to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final IndexProducer indexProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(indexProducer);
+ return bf;
+ }
+
+ /**
+ * Creates a Bloom filter from a BitMapProducer.
+ *
+ * @param bitMapProducer the BitMapProducer to create the filter from.
+ * @return the BloomFilter.
+ */
+ private BloomFilter createFilter(final BitMapProducer bitMapProducer) {
+ SimpleBloomFilter bf = new SimpleBloomFilter(shape);
+ bf.merge(bitMapProducer);
+ return bf;
+ }
+
+ @Override
+ public int characteristics() {
+ return 0;
+ }
+
+ @Override
+ public final Shape getShape() {
+ return shape;
+ }
+
+ @Override
+ public boolean contains(final Hasher hasher) {
+ return contains(createFilter(hasher));
+ }
+
+ @Override
+ public boolean contains(final BitMapProducer bitMapProducer) {
+ return contains(createFilter(bitMapProducer));
+ }
+
+ @Override
+ public boolean contains(IndexProducer indexProducer) {
+ return contains(createFilter(indexProducer));
+ }
+
+ @Override
+ public boolean merge(BloomFilter bf) {
+ return target().merge(bf);
+ }
+
+ @Override
+ public boolean merge(IndexProducer indexProducer) {
+ return target().merge(indexProducer);
+ }
+
+ @Override
+ public boolean merge(BitMapProducer bitMapProducer) {
+ return target().merge(bitMapProducer);
+ }
+
+ @Override
+ public boolean forEachIndex(IntPredicate predicate) {
Review Comment:
Should this have a comment that the indices from each layer are returned (i.e. are not unique).
##########
src/test/java/org/apache/commons/collections4/bloomfilter/LayerManagerTest.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+
+import org.junit.jupiter.api.Test;
+
+public class LayerManagerTest {
+
+ private Shape shape = Shape.fromKM(17, 72);
+
+ private LayerManager.Builder testBuilder() {
+ return LayerManager.builder().withSuplier(() -> new SimpleBloomFilter(shape));
+ }
+
+ @Test
+ public void testAdvanceOnPopulated() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.ADVANCE_ON_POPULATED;
+ LayerManager layerManager = testBuilder().build();
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.FROM1);
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testNeverAdvance() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.NEVER_ADVANCE;
+ LayerManager layerManager = testBuilder().build();
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.FROM1);
+ assertFalse(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnCount() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnCount(4);
+ LayerManager layerManager = testBuilder().build();
+ for (int i = 0; i < 3; i++) {
+ assertFalse(underTest.test(layerManager), "at " + i);
+ layerManager.target().merge(TestingHashers.FROM1);
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnCalculatedFull() {
+ Double maxN = shape.estimateMaxN();
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnCalculatedFull(shape);
+ LayerManager layerManager = testBuilder().build();
+ while (layerManager.target().getShape().estimateN(layerManager.target().cardinality()) < maxN) {
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.randomHasher());
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnSaturationWithDouble() {
+ Double maxN = shape.estimateMaxN();
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnSaturation(maxN);
+ LayerManager layerManager = testBuilder().build();
+ while (layerManager.target().getShape().estimateN(layerManager.target().cardinality()) < maxN) {
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.randomHasher());
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnSaturationWithInt() {
Review Comment:
Given the similarity, why do we want to support the `int` version too?
##########
src/test/java/org/apache/commons/collections4/bloomfilter/LayerManagerTest.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+
+import org.junit.jupiter.api.Test;
+
+public class LayerManagerTest {
+
+ private Shape shape = Shape.fromKM(17, 72);
+
+ private LayerManager.Builder testBuilder() {
+ return LayerManager.builder().withSuplier(() -> new SimpleBloomFilter(shape));
+ }
+
+ @Test
+ public void testAdvanceOnPopulated() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.ADVANCE_ON_POPULATED;
+ LayerManager layerManager = testBuilder().build();
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.FROM1);
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testNeverAdvance() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.NEVER_ADVANCE;
+ LayerManager layerManager = testBuilder().build();
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.FROM1);
+ assertFalse(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnCount() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnCount(4);
+ LayerManager layerManager = testBuilder().build();
+ for (int i = 0; i < 3; i++) {
+ assertFalse(underTest.test(layerManager), "at " + i);
+ layerManager.target().merge(TestingHashers.FROM1);
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnCalculatedFull() {
+ Double maxN = shape.estimateMaxN();
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnCalculatedFull(shape);
+ LayerManager layerManager = testBuilder().build();
+ while (layerManager.target().getShape().estimateN(layerManager.target().cardinality()) < maxN) {
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.randomHasher());
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnSaturationWithDouble() {
+ Double maxN = shape.estimateMaxN();
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnSaturation(maxN);
+ LayerManager layerManager = testBuilder().build();
+ while (layerManager.target().getShape().estimateN(layerManager.target().cardinality()) < maxN) {
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.randomHasher());
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnSaturationWithInt() {
+ int maxN = (int) Math.floor(shape.estimateMaxN());
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnSaturation(maxN);
+ LayerManager layerManager = testBuilder().build();
+ while (layerManager.target().estimateN() < maxN) {
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.randomHasher());
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testOnMaxSize() {
+ Consumer<LinkedList<BloomFilter>> underTest = LayerManager.Cleanup.onMaxSize(5);
Review Comment:
Max size should be a variable and you could change to a ParameterizedTest
```java
@ParameterizedTest
@ValueSource(ints = {5, 10})
public void testOnMaxSize(int maxSize) {
Consumer<LinkedList<BloomFilter>> underTest = LayerManager.Cleanup.onMaxSize(maxSize);
LinkedList<BloomFilter> list = new LinkedList<>();
for (int i = 0; i < maxSize; i++) {
```
##########
src/test/java/org/apache/commons/collections4/bloomfilter/LayerManagerTest.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+
+import org.junit.jupiter.api.Test;
+
+public class LayerManagerTest {
+
+ private Shape shape = Shape.fromKM(17, 72);
+
+ private LayerManager.Builder testBuilder() {
+ return LayerManager.builder().withSuplier(() -> new SimpleBloomFilter(shape));
+ }
+
+ @Test
+ public void testAdvanceOnPopulated() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.ADVANCE_ON_POPULATED;
+ LayerManager layerManager = testBuilder().build();
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.FROM1);
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testNeverAdvance() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.NEVER_ADVANCE;
+ LayerManager layerManager = testBuilder().build();
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.FROM1);
+ assertFalse(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnCount() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnCount(4);
+ LayerManager layerManager = testBuilder().build();
+ for (int i = 0; i < 3; i++) {
+ assertFalse(underTest.test(layerManager), "at " + i);
+ layerManager.target().merge(TestingHashers.FROM1);
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnCalculatedFull() {
+ Double maxN = shape.estimateMaxN();
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnCalculatedFull(shape);
+ LayerManager layerManager = testBuilder().build();
+ while (layerManager.target().getShape().estimateN(layerManager.target().cardinality()) < maxN) {
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.randomHasher());
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnSaturationWithDouble() {
+ Double maxN = shape.estimateMaxN();
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnSaturation(maxN);
+ LayerManager layerManager = testBuilder().build();
+ while (layerManager.target().getShape().estimateN(layerManager.target().cardinality()) < maxN) {
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.randomHasher());
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnSaturationWithInt() {
+ int maxN = (int) Math.floor(shape.estimateMaxN());
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnSaturation(maxN);
+ LayerManager layerManager = testBuilder().build();
+ while (layerManager.target().estimateN() < maxN) {
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.randomHasher());
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testOnMaxSize() {
+ Consumer<LinkedList<BloomFilter>> underTest = LayerManager.Cleanup.onMaxSize(5);
+ LinkedList<BloomFilter> list = new LinkedList<>();
+ for (int i = 0; i < 5; i++) {
+ assertEquals(i, list.size());
+ list.add(new SimpleBloomFilter(shape));
+ underTest.accept(list);
+ }
+ assertEquals(5, list.size());
+ list.add(new SimpleBloomFilter(shape));
+ underTest.accept(list);
+
+ assertEquals(5, list.size());
+ list.add(new SimpleBloomFilter(shape));
+ underTest.accept(list);
+
+ assertEquals(5, list.size());
+ list.add(new SimpleBloomFilter(shape));
+ underTest.accept(list);
+ }
+
+ @Test
+ public void testCopy() {
+ LayerManager underTest = LayerManager.builder().withSuplier(() -> new SimpleBloomFilter(shape)).build();
+ underTest.target().merge(TestingHashers.randomHasher());
+ underTest.next();
+ underTest.target().merge(TestingHashers.randomHasher());
+ underTest.next();
+ underTest.target().merge(TestingHashers.randomHasher());
+ assertEquals(3, underTest.getDepth());
+
+ LayerManager copy = underTest.copy();
+ assertFalse(underTest == copy);
+ assertFalse(underTest.equals(copy));
Review Comment:
Assertions.assertNotSame
Assertions.assertNotEquals
I would comment this one: `, "Object.equals is not implemented"
##########
src/test/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilterTest.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+
+import org.junit.jupiter.api.Test;
+
+public class LayeredBloomFilterTest extends AbstractBloomFilterTest<LayeredBloomFilter> {
+
+ @Override
+ protected LayeredBloomFilter createEmptyFilter(Shape shape) {
+ return LayeredBloomFilter.fixed(shape, 10);
+ }
+
+ protected BloomFilter makeFilter(int... values) {
+ return makeFilter(IndexProducer.fromIndexArray(values));
+ }
+
+ protected BloomFilter makeFilter(IndexProducer p) {
+ BloomFilter bf = new SparseBloomFilter(getTestShape());
+ bf.merge(p);
+ return bf;
+ }
+
+ protected BloomFilter makeFilter(Hasher h) {
+ BloomFilter bf = new SparseBloomFilter(getTestShape());
+ bf.merge(h);
+ return bf;
+ }
+
+ @Test
+ public void testMultipleFilters() {
+ LayeredBloomFilter filter = LayeredBloomFilter.fixed(getTestShape(), 10);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ assertEquals(2, filter.getDepth());
+ assertTrue(filter.contains(makeFilter(TestingHashers.FROM1)));
+ assertTrue(filter.contains(makeFilter(TestingHashers.FROM11)));
+ BloomFilter t1 = makeFilter(6, 7, 17, 18, 19);
+ assertFalse(filter.contains(t1));
+ assertFalse(filter.copy().contains(t1));
+ assertTrue(filter.flatten().contains(t1));
+ }
+
+ private LayeredBloomFilter setupFindTest() {
+ LayeredBloomFilter filter = LayeredBloomFilter.fixed(getTestShape(), 10);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ filter.merge(new IncrementingHasher(11, 2));
+ filter.merge(TestingHashers.populateFromHashersFrom1AndFrom11(new SimpleBloomFilter(getTestShape())));
+ return filter;
+ }
+
+ @Test
+ public void testFindBloomFilter() {
+ LayeredBloomFilter filter = setupFindTest();
+ int[] result = filter.find(TestingHashers.FROM1);
+ assertEquals(2, result.length);
+ assertEquals(0, result[0]);
+ assertEquals(3, result[1]);
+ result = filter.find(TestingHashers.FROM11);
+ assertEquals(2, result.length);
+ assertEquals(1, result[0]);
+ assertEquals(3, result[1]);
+ }
+
+ @Test
+ public void testFindBitMapProducer() {
+ LayeredBloomFilter filter = setupFindTest();
+
+ IndexProducer idxProducer = TestingHashers.FROM1.indices(getTestShape());
+ BitMapProducer producer = BitMapProducer.fromIndexProducer(idxProducer, getTestShape().getNumberOfBits());
+
+ int[] result = filter.find(producer);
+ assertEquals(2, result.length);
+ assertEquals(0, result[0]);
+ assertEquals(3, result[1]);
+
+ idxProducer = TestingHashers.FROM11.indices(getTestShape());
+ producer = BitMapProducer.fromIndexProducer(idxProducer, getTestShape().getNumberOfBits());
+ result = filter.find(producer);
+ assertEquals(2, result.length);
+ assertEquals(1, result[0]);
+ assertEquals(3, result[1]);
+ }
+
+ @Test
+ public void testFindIndexProducer() {
+ IndexProducer producer = TestingHashers.FROM1.indices(getTestShape());
+ LayeredBloomFilter filter = setupFindTest();
+
+ int[] result = filter.find(producer);
+ assertEquals(2, result.length);
+ assertEquals(0, result[0]);
+ assertEquals(3, result[1]);
+
+ producer = TestingHashers.FROM11.indices(getTestShape());
+ result = filter.find(producer);
+ assertEquals(2, result.length);
+ assertEquals(1, result[0]);
+ assertEquals(3, result[1]);
+ }
+
+ /**
+ * Tests that the estimated union calculations are correct.
+ */
+ @Test
+ public final void testEstimateUnionCrossTypes() {
+ final BloomFilter bf = createFilter(getTestShape(), TestingHashers.FROM1);
+ final BloomFilter bf2 = new DefaultBloomFilterTest.SparseDefaultBloomFilter(getTestShape());
+ bf2.merge(TestingHashers.FROM11);
+
+ assertEquals(2, bf.estimateUnion(bf2));
+ assertEquals(2, bf2.estimateUnion(bf));
+ }
+
+ @Test
+ public final void testGetLayer() {
+ BloomFilter bf = new SimpleBloomFilter(getTestShape());
+ bf.merge(TestingHashers.FROM11);
+ LayeredBloomFilter filter = LayeredBloomFilter.fixed(getTestShape(), 10);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ filter.merge(new IncrementingHasher(11, 2));
+ filter.merge(TestingHashers.populateFromHashersFrom1AndFrom11(new SimpleBloomFilter(getTestShape())));
+ assertArrayEquals(bf.asBitMapArray(), filter.get(1).asBitMapArray());
+ }
+
+ @Test
+ public final void testClearLayer() {
+ LayeredBloomFilter filter = LayeredBloomFilter.fixed(getTestShape(), 10);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ filter.merge(new IncrementingHasher(11, 2));
+ filter.merge(TestingHashers.populateFromHashersFrom1AndFrom11(new SimpleBloomFilter(getTestShape())));
+ filter.clear(1);
+ assertEquals(0, filter.get(1).cardinality());
+ }
+
+ @Test
+ public final void testNext() {
+ LayerManager layerManager = LayerManager.builder().withSuplier(() -> new SimpleBloomFilter(getTestShape()))
+ .build();
+
+ LayeredBloomFilter filter = new LayeredBloomFilter(getTestShape(), layerManager);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ assertEquals(1, filter.getDepth());
+ filter.next();
+ filter.merge(new IncrementingHasher(11, 2));
+ assertEquals(2, filter.getDepth());
+ assertTrue(filter.get(0).contains(TestingHashers.FROM1));
+ assertTrue(filter.get(0).contains(TestingHashers.FROM11));
+ assertFalse(filter.get(0).contains(new IncrementingHasher(11, 2)));
+ assertFalse(filter.get(1).contains(TestingHashers.FROM1));
+ assertFalse(filter.get(1).contains(TestingHashers.FROM11));
+ assertTrue(filter.get(1).contains(new IncrementingHasher(11, 2)));
+ }
+
+ // ***** TESTS THAT CHECK LAYERED PROCESSING ******
+
+ // ***example of instrumentation ***
+ private static List<String> dbgInstrument = new ArrayList<>();
+ // instrumentation to record timestamps in dbgInstrument list
+ private Predicate<BloomFilter> dbg = (bf) -> {
+ TimestampedBloomFilter tbf = (TimestampedBloomFilter) bf;
+ long ts = System.currentTimeMillis();
+ dbgInstrument.add(String.format("T:%s (Elapsed:%s)- EstN:%s (Card:%s)\n", tbf.timestamp, ts - tbf.timestamp,
+ tbf.estimateN(), tbf.cardinality()));
+ return true;
+ };
+ // *** end of instrumentation ***
+
+ /**
+ * Creates a LayeredBloomFilter that retains enclosed filters for
+ * {@code duration} and limits the contents of each enclosed filter to a time
+ * {@code quanta}. This filter uses the timestamped Bloom filter internally.
+ *
+ * @param shape The shape of the Bloom filters.
+ * @param duration The length of time to keep filters in the list.
+ * @param dUnit The unit of time to apply to duration.
+ * @param quanta The quantization factor for each filter. Individual filters
+ * will span at most this much time.
+ * @param qUnit the unit of time to apply to quanta.
+ * @return LayeredBloomFilter with the above properties.
+ */
+ static LayeredBloomFilter createTimedLayeredFilter(Shape shape, long duration, TimeUnit dUnit, long quanta,
+ TimeUnit qUnit) {
+ LayerManager layerManager = LayerManager.builder()
+ .withSuplier(() -> new TimestampedBloomFilter(new SimpleBloomFilter(shape)))
+ .withCleanup(new CleanByTime(duration, dUnit))
+ .withExtendCheck(new AdvanceOnTimeQuanta(shape, quanta, qUnit)
+ .or(LayerManager.ExtendCheck.advanceOnCalculatedFull(shape)))
+ .build();
+ return new LayeredBloomFilter(shape, layerManager);
+ }
+
+ /**
+ * A Predicate that advances after a quantum of time.
+ */
+ static class AdvanceOnTimeQuanta implements Predicate<LayerManager> {
+ long quanta;
+
+ AdvanceOnTimeQuanta(Shape shape, long quanta, TimeUnit unit) {
+ this.quanta = unit.toMillis(quanta);
+ }
+
+ @Override
+ public boolean test(LayerManager lm) {
+ int depth = lm.getDepth();
+ if (depth == 0) {
Review Comment:
Can this be zero? Even a clear operation then adds back a single filter layer.
##########
src/test/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilterTest.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+
+import org.junit.jupiter.api.Test;
+
+public class LayeredBloomFilterTest extends AbstractBloomFilterTest<LayeredBloomFilter> {
+
+ @Override
+ protected LayeredBloomFilter createEmptyFilter(Shape shape) {
+ return LayeredBloomFilter.fixed(shape, 10);
+ }
+
+ protected BloomFilter makeFilter(int... values) {
+ return makeFilter(IndexProducer.fromIndexArray(values));
+ }
+
+ protected BloomFilter makeFilter(IndexProducer p) {
+ BloomFilter bf = new SparseBloomFilter(getTestShape());
+ bf.merge(p);
+ return bf;
+ }
+
+ protected BloomFilter makeFilter(Hasher h) {
+ BloomFilter bf = new SparseBloomFilter(getTestShape());
+ bf.merge(h);
+ return bf;
+ }
+
+ @Test
+ public void testMultipleFilters() {
+ LayeredBloomFilter filter = LayeredBloomFilter.fixed(getTestShape(), 10);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ assertEquals(2, filter.getDepth());
+ assertTrue(filter.contains(makeFilter(TestingHashers.FROM1)));
+ assertTrue(filter.contains(makeFilter(TestingHashers.FROM11)));
+ BloomFilter t1 = makeFilter(6, 7, 17, 18, 19);
+ assertFalse(filter.contains(t1));
+ assertFalse(filter.copy().contains(t1));
+ assertTrue(filter.flatten().contains(t1));
+ }
+
+ private LayeredBloomFilter setupFindTest() {
+ LayeredBloomFilter filter = LayeredBloomFilter.fixed(getTestShape(), 10);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ filter.merge(new IncrementingHasher(11, 2));
+ filter.merge(TestingHashers.populateFromHashersFrom1AndFrom11(new SimpleBloomFilter(getTestShape())));
+ return filter;
+ }
+
+ @Test
+ public void testFindBloomFilter() {
+ LayeredBloomFilter filter = setupFindTest();
+ int[] result = filter.find(TestingHashers.FROM1);
+ assertEquals(2, result.length);
+ assertEquals(0, result[0]);
+ assertEquals(3, result[1]);
+ result = filter.find(TestingHashers.FROM11);
+ assertEquals(2, result.length);
+ assertEquals(1, result[0]);
+ assertEquals(3, result[1]);
+ }
+
+ @Test
+ public void testFindBitMapProducer() {
+ LayeredBloomFilter filter = setupFindTest();
+
+ IndexProducer idxProducer = TestingHashers.FROM1.indices(getTestShape());
+ BitMapProducer producer = BitMapProducer.fromIndexProducer(idxProducer, getTestShape().getNumberOfBits());
+
+ int[] result = filter.find(producer);
+ assertEquals(2, result.length);
+ assertEquals(0, result[0]);
+ assertEquals(3, result[1]);
+
+ idxProducer = TestingHashers.FROM11.indices(getTestShape());
+ producer = BitMapProducer.fromIndexProducer(idxProducer, getTestShape().getNumberOfBits());
+ result = filter.find(producer);
+ assertEquals(2, result.length);
+ assertEquals(1, result[0]);
+ assertEquals(3, result[1]);
+ }
+
+ @Test
+ public void testFindIndexProducer() {
+ IndexProducer producer = TestingHashers.FROM1.indices(getTestShape());
+ LayeredBloomFilter filter = setupFindTest();
+
+ int[] result = filter.find(producer);
+ assertEquals(2, result.length);
+ assertEquals(0, result[0]);
+ assertEquals(3, result[1]);
+
+ producer = TestingHashers.FROM11.indices(getTestShape());
+ result = filter.find(producer);
+ assertEquals(2, result.length);
+ assertEquals(1, result[0]);
+ assertEquals(3, result[1]);
+ }
+
+ /**
+ * Tests that the estimated union calculations are correct.
+ */
+ @Test
+ public final void testEstimateUnionCrossTypes() {
+ final BloomFilter bf = createFilter(getTestShape(), TestingHashers.FROM1);
+ final BloomFilter bf2 = new DefaultBloomFilterTest.SparseDefaultBloomFilter(getTestShape());
+ bf2.merge(TestingHashers.FROM11);
+
+ assertEquals(2, bf.estimateUnion(bf2));
+ assertEquals(2, bf2.estimateUnion(bf));
+ }
+
+ @Test
+ public final void testGetLayer() {
+ BloomFilter bf = new SimpleBloomFilter(getTestShape());
+ bf.merge(TestingHashers.FROM11);
+ LayeredBloomFilter filter = LayeredBloomFilter.fixed(getTestShape(), 10);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ filter.merge(new IncrementingHasher(11, 2));
+ filter.merge(TestingHashers.populateFromHashersFrom1AndFrom11(new SimpleBloomFilter(getTestShape())));
+ assertArrayEquals(bf.asBitMapArray(), filter.get(1).asBitMapArray());
+ }
+
+ @Test
+ public final void testClearLayer() {
+ LayeredBloomFilter filter = LayeredBloomFilter.fixed(getTestShape(), 10);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ filter.merge(new IncrementingHasher(11, 2));
+ filter.merge(TestingHashers.populateFromHashersFrom1AndFrom11(new SimpleBloomFilter(getTestShape())));
+ filter.clear(1);
+ assertEquals(0, filter.get(1).cardinality());
+ }
+
+ @Test
+ public final void testNext() {
+ LayerManager layerManager = LayerManager.builder().withSuplier(() -> new SimpleBloomFilter(getTestShape()))
+ .build();
+
+ LayeredBloomFilter filter = new LayeredBloomFilter(getTestShape(), layerManager);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ assertEquals(1, filter.getDepth());
+ filter.next();
+ filter.merge(new IncrementingHasher(11, 2));
+ assertEquals(2, filter.getDepth());
+ assertTrue(filter.get(0).contains(TestingHashers.FROM1));
+ assertTrue(filter.get(0).contains(TestingHashers.FROM11));
+ assertFalse(filter.get(0).contains(new IncrementingHasher(11, 2)));
+ assertFalse(filter.get(1).contains(TestingHashers.FROM1));
+ assertFalse(filter.get(1).contains(TestingHashers.FROM11));
+ assertTrue(filter.get(1).contains(new IncrementingHasher(11, 2)));
+ }
+
+ // ***** TESTS THAT CHECK LAYERED PROCESSING ******
+
+ // ***example of instrumentation ***
+ private static List<String> dbgInstrument = new ArrayList<>();
+ // instrumentation to record timestamps in dbgInstrument list
+ private Predicate<BloomFilter> dbg = (bf) -> {
+ TimestampedBloomFilter tbf = (TimestampedBloomFilter) bf;
+ long ts = System.currentTimeMillis();
+ dbgInstrument.add(String.format("T:%s (Elapsed:%s)- EstN:%s (Card:%s)\n", tbf.timestamp, ts - tbf.timestamp,
+ tbf.estimateN(), tbf.cardinality()));
+ return true;
+ };
+ // *** end of instrumentation ***
+
+ /**
+ * Creates a LayeredBloomFilter that retains enclosed filters for
+ * {@code duration} and limits the contents of each enclosed filter to a time
+ * {@code quanta}. This filter uses the timestamped Bloom filter internally.
+ *
+ * @param shape The shape of the Bloom filters.
+ * @param duration The length of time to keep filters in the list.
+ * @param dUnit The unit of time to apply to duration.
+ * @param quanta The quantization factor for each filter. Individual filters
+ * will span at most this much time.
+ * @param qUnit the unit of time to apply to quanta.
+ * @return LayeredBloomFilter with the above properties.
+ */
+ static LayeredBloomFilter createTimedLayeredFilter(Shape shape, long duration, TimeUnit dUnit, long quanta,
+ TimeUnit qUnit) {
+ LayerManager layerManager = LayerManager.builder()
+ .withSuplier(() -> new TimestampedBloomFilter(new SimpleBloomFilter(shape)))
+ .withCleanup(new CleanByTime(duration, dUnit))
+ .withExtendCheck(new AdvanceOnTimeQuanta(shape, quanta, qUnit)
+ .or(LayerManager.ExtendCheck.advanceOnCalculatedFull(shape)))
+ .build();
+ return new LayeredBloomFilter(shape, layerManager);
+ }
+
+ /**
+ * A Predicate that advances after a quantum of time.
+ */
+ static class AdvanceOnTimeQuanta implements Predicate<LayerManager> {
+ long quanta;
+
+ AdvanceOnTimeQuanta(Shape shape, long quanta, TimeUnit unit) {
+ this.quanta = unit.toMillis(quanta);
+ }
+
+ @Override
+ public boolean test(LayerManager lm) {
+ int depth = lm.getDepth();
+ if (depth == 0) {
+ return false;
+ }
+ // can not use getTarget() as it causes recursion.
+ TimestampedBloomFilter bf = (TimestampedBloomFilter) lm.get(depth - 1);
+ return bf.timestamp + quanta < System.currentTimeMillis();
+ }
+ }
+
+ /**
+ * A Consumer that cleans the list based on how long each filters has been in
+ * the list.
+ *
+ */
+ static class CleanByTime implements Consumer<LinkedList<BloomFilter>> {
+ long elapsedTime;
+
+ CleanByTime(long duration, TimeUnit unit) {
+ elapsedTime = unit.toMillis(duration);
+ }
+
+ @Override
+ public void accept(LinkedList<BloomFilter> t) {
+ long min = System.currentTimeMillis() - elapsedTime;
+ while (!t.isEmpty() && ((TimestampedBloomFilter) t.getFirst()).getTimestamp() < min) {
+ TimestampedBloomFilter bf = (TimestampedBloomFilter) t.getFirst();
+ dbgInstrument.add(String.format("Removing old entry: T:%s (Aged: %s) \n", bf.getTimestamp(),
+ (min - bf.getTimestamp())));
+ t.removeFirst();
+ }
+ }
+ }
+
+ /**
+ * A Bloomfilter implementation that tracks the creation time.
+ */
+ static class TimestampedBloomFilter extends WrappedBloomFilter {
+ final long timestamp;
+
+ TimestampedBloomFilter(BloomFilter bf) {
+ super(bf);
+ this.timestamp = System.currentTimeMillis();
+ }
+
+ public long getTimestamp() {
+ return timestamp;
+ }
+ }
+
+ @Test
+ public void testExpiration() throws InterruptedException {
Review Comment:
Can you experiment with shorter time periods here. The test takes 7.5 seconds in sleep. If you reduce the quanta of the time from 1 second to 150ms then it should still be stable across platform implementations of sleep but reduces the run time of the test ~ 6-fold.
##########
src/test/java/org/apache/commons/collections4/bloomfilter/LayerManagerTest.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+
+import org.junit.jupiter.api.Test;
+
+public class LayerManagerTest {
+
+ private Shape shape = Shape.fromKM(17, 72);
+
+ private LayerManager.Builder testBuilder() {
+ return LayerManager.builder().withSuplier(() -> new SimpleBloomFilter(shape));
+ }
+
+ @Test
+ public void testAdvanceOnPopulated() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.ADVANCE_ON_POPULATED;
+ LayerManager layerManager = testBuilder().build();
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.FROM1);
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testNeverAdvance() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.NEVER_ADVANCE;
+ LayerManager layerManager = testBuilder().build();
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.FROM1);
+ assertFalse(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnCount() {
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnCount(4);
+ LayerManager layerManager = testBuilder().build();
+ for (int i = 0; i < 3; i++) {
+ assertFalse(underTest.test(layerManager), "at " + i);
+ layerManager.target().merge(TestingHashers.FROM1);
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnCalculatedFull() {
+ Double maxN = shape.estimateMaxN();
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnCalculatedFull(shape);
+ LayerManager layerManager = testBuilder().build();
+ while (layerManager.target().getShape().estimateN(layerManager.target().cardinality()) < maxN) {
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.randomHasher());
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnSaturationWithDouble() {
+ Double maxN = shape.estimateMaxN();
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnSaturation(maxN);
+ LayerManager layerManager = testBuilder().build();
+ while (layerManager.target().getShape().estimateN(layerManager.target().cardinality()) < maxN) {
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.randomHasher());
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testAdvanceOnSaturationWithInt() {
+ int maxN = (int) Math.floor(shape.estimateMaxN());
+ Predicate<LayerManager> underTest = LayerManager.ExtendCheck.advanceOnSaturation(maxN);
+ LayerManager layerManager = testBuilder().build();
+ while (layerManager.target().estimateN() < maxN) {
+ assertFalse(underTest.test(layerManager));
+ layerManager.target().merge(TestingHashers.randomHasher());
+ }
+ assertTrue(underTest.test(layerManager));
+ }
+
+ @Test
+ public void testOnMaxSize() {
+ Consumer<LinkedList<BloomFilter>> underTest = LayerManager.Cleanup.onMaxSize(5);
+ LinkedList<BloomFilter> list = new LinkedList<>();
+ for (int i = 0; i < 5; i++) {
+ assertEquals(i, list.size());
+ list.add(new SimpleBloomFilter(shape));
+ underTest.accept(list);
+ }
+ assertEquals(5, list.size());
+ list.add(new SimpleBloomFilter(shape));
+ underTest.accept(list);
+
+ assertEquals(5, list.size());
+ list.add(new SimpleBloomFilter(shape));
+ underTest.accept(list);
+
+ assertEquals(5, list.size());
+ list.add(new SimpleBloomFilter(shape));
+ underTest.accept(list);
+ }
+
+ @Test
+ public void testCopy() {
+ LayerManager underTest = LayerManager.builder().withSuplier(() -> new SimpleBloomFilter(shape)).build();
+ underTest.target().merge(TestingHashers.randomHasher());
+ underTest.next();
+ underTest.target().merge(TestingHashers.randomHasher());
+ underTest.next();
+ underTest.target().merge(TestingHashers.randomHasher());
+ assertEquals(3, underTest.getDepth());
+
+ LayerManager copy = underTest.copy();
+ assertFalse(underTest == copy);
+ assertFalse(underTest.equals(copy));
+
+ assertEquals(underTest.getDepth(), copy.getDepth());
+ assertTrue(
+ underTest.forEachBloomFilterPair(copy, (x, y) -> Arrays.equals(x.asBitMapArray(), y.asBitMapArray())));
+ }
+
+ @Test
+ public void testClear() {
+ LayerManager underTest = LayerManager.builder().withSuplier(() -> new SimpleBloomFilter(shape)).build();
+ underTest.target().merge(TestingHashers.randomHasher());
+ underTest.next();
+ underTest.target().merge(TestingHashers.randomHasher());
+ underTest.next();
+ underTest.target().merge(TestingHashers.randomHasher());
+ assertEquals(3, underTest.getDepth());
+ underTest.clear();
+ assertEquals(1, underTest.getDepth());
+ assertEquals(0, underTest.target().cardinality());
+ }
+
+ @Test
+ public void testNext() {
+ LayerManager underTest = LayerManager.builder().withSuplier(() -> new SimpleBloomFilter(shape)).build();
+ assertEquals(1, underTest.getDepth());
+ underTest.target().merge(TestingHashers.randomHasher());
+ assertEquals(1, underTest.getDepth());
+ underTest.next();
+ assertEquals(2, underTest.getDepth());
+ }
+
+ @Test
+ public void testGetDepth() {
Review Comment:
This is a duplicate of `testNext`. Unless you are doing something different I would merge the two tests as textNextAndGetDepth.
##########
src/test/java/org/apache/commons/collections4/bloomfilter/LayeredBloomFilterTest.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.collections4.bloomfilter;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Predicate;
+
+import org.junit.jupiter.api.Test;
+
+public class LayeredBloomFilterTest extends AbstractBloomFilterTest<LayeredBloomFilter> {
+
+ @Override
+ protected LayeredBloomFilter createEmptyFilter(Shape shape) {
+ return LayeredBloomFilter.fixed(shape, 10);
+ }
+
+ protected BloomFilter makeFilter(int... values) {
+ return makeFilter(IndexProducer.fromIndexArray(values));
+ }
+
+ protected BloomFilter makeFilter(IndexProducer p) {
+ BloomFilter bf = new SparseBloomFilter(getTestShape());
+ bf.merge(p);
+ return bf;
+ }
+
+ protected BloomFilter makeFilter(Hasher h) {
+ BloomFilter bf = new SparseBloomFilter(getTestShape());
+ bf.merge(h);
+ return bf;
+ }
+
+ @Test
+ public void testMultipleFilters() {
+ LayeredBloomFilter filter = LayeredBloomFilter.fixed(getTestShape(), 10);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ assertEquals(2, filter.getDepth());
+ assertTrue(filter.contains(makeFilter(TestingHashers.FROM1)));
+ assertTrue(filter.contains(makeFilter(TestingHashers.FROM11)));
+ BloomFilter t1 = makeFilter(6, 7, 17, 18, 19);
+ assertFalse(filter.contains(t1));
+ assertFalse(filter.copy().contains(t1));
+ assertTrue(filter.flatten().contains(t1));
+ }
+
+ private LayeredBloomFilter setupFindTest() {
+ LayeredBloomFilter filter = LayeredBloomFilter.fixed(getTestShape(), 10);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ filter.merge(new IncrementingHasher(11, 2));
+ filter.merge(TestingHashers.populateFromHashersFrom1AndFrom11(new SimpleBloomFilter(getTestShape())));
+ return filter;
+ }
+
+ @Test
+ public void testFindBloomFilter() {
+ LayeredBloomFilter filter = setupFindTest();
+ int[] result = filter.find(TestingHashers.FROM1);
+ assertEquals(2, result.length);
+ assertEquals(0, result[0]);
+ assertEquals(3, result[1]);
+ result = filter.find(TestingHashers.FROM11);
+ assertEquals(2, result.length);
+ assertEquals(1, result[0]);
+ assertEquals(3, result[1]);
+ }
+
+ @Test
+ public void testFindBitMapProducer() {
+ LayeredBloomFilter filter = setupFindTest();
+
+ IndexProducer idxProducer = TestingHashers.FROM1.indices(getTestShape());
+ BitMapProducer producer = BitMapProducer.fromIndexProducer(idxProducer, getTestShape().getNumberOfBits());
+
+ int[] result = filter.find(producer);
+ assertEquals(2, result.length);
+ assertEquals(0, result[0]);
+ assertEquals(3, result[1]);
+
+ idxProducer = TestingHashers.FROM11.indices(getTestShape());
+ producer = BitMapProducer.fromIndexProducer(idxProducer, getTestShape().getNumberOfBits());
+ result = filter.find(producer);
+ assertEquals(2, result.length);
+ assertEquals(1, result[0]);
+ assertEquals(3, result[1]);
+ }
+
+ @Test
+ public void testFindIndexProducer() {
+ IndexProducer producer = TestingHashers.FROM1.indices(getTestShape());
+ LayeredBloomFilter filter = setupFindTest();
+
+ int[] result = filter.find(producer);
+ assertEquals(2, result.length);
+ assertEquals(0, result[0]);
+ assertEquals(3, result[1]);
+
+ producer = TestingHashers.FROM11.indices(getTestShape());
+ result = filter.find(producer);
+ assertEquals(2, result.length);
+ assertEquals(1, result[0]);
+ assertEquals(3, result[1]);
+ }
+
+ /**
+ * Tests that the estimated union calculations are correct.
+ */
+ @Test
+ public final void testEstimateUnionCrossTypes() {
+ final BloomFilter bf = createFilter(getTestShape(), TestingHashers.FROM1);
+ final BloomFilter bf2 = new DefaultBloomFilterTest.SparseDefaultBloomFilter(getTestShape());
+ bf2.merge(TestingHashers.FROM11);
+
+ assertEquals(2, bf.estimateUnion(bf2));
+ assertEquals(2, bf2.estimateUnion(bf));
+ }
+
+ @Test
+ public final void testGetLayer() {
+ BloomFilter bf = new SimpleBloomFilter(getTestShape());
+ bf.merge(TestingHashers.FROM11);
+ LayeredBloomFilter filter = LayeredBloomFilter.fixed(getTestShape(), 10);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ filter.merge(new IncrementingHasher(11, 2));
+ filter.merge(TestingHashers.populateFromHashersFrom1AndFrom11(new SimpleBloomFilter(getTestShape())));
+ assertArrayEquals(bf.asBitMapArray(), filter.get(1).asBitMapArray());
+ }
+
+ @Test
+ public final void testClearLayer() {
+ LayeredBloomFilter filter = LayeredBloomFilter.fixed(getTestShape(), 10);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ filter.merge(new IncrementingHasher(11, 2));
+ filter.merge(TestingHashers.populateFromHashersFrom1AndFrom11(new SimpleBloomFilter(getTestShape())));
+ filter.clear(1);
+ assertEquals(0, filter.get(1).cardinality());
+ }
+
+ @Test
+ public final void testNext() {
+ LayerManager layerManager = LayerManager.builder().withSuplier(() -> new SimpleBloomFilter(getTestShape()))
+ .build();
+
+ LayeredBloomFilter filter = new LayeredBloomFilter(getTestShape(), layerManager);
+ filter.merge(TestingHashers.FROM1);
+ filter.merge(TestingHashers.FROM11);
+ assertEquals(1, filter.getDepth());
+ filter.next();
+ filter.merge(new IncrementingHasher(11, 2));
+ assertEquals(2, filter.getDepth());
+ assertTrue(filter.get(0).contains(TestingHashers.FROM1));
+ assertTrue(filter.get(0).contains(TestingHashers.FROM11));
+ assertFalse(filter.get(0).contains(new IncrementingHasher(11, 2)));
+ assertFalse(filter.get(1).contains(TestingHashers.FROM1));
+ assertFalse(filter.get(1).contains(TestingHashers.FROM11));
+ assertTrue(filter.get(1).contains(new IncrementingHasher(11, 2)));
+ }
+
+ // ***** TESTS THAT CHECK LAYERED PROCESSING ******
+
+ // ***example of instrumentation ***
+ private static List<String> dbgInstrument = new ArrayList<>();
+ // instrumentation to record timestamps in dbgInstrument list
+ private Predicate<BloomFilter> dbg = (bf) -> {
+ TimestampedBloomFilter tbf = (TimestampedBloomFilter) bf;
+ long ts = System.currentTimeMillis();
+ dbgInstrument.add(String.format("T:%s (Elapsed:%s)- EstN:%s (Card:%s)\n", tbf.timestamp, ts - tbf.timestamp,
+ tbf.estimateN(), tbf.cardinality()));
+ return true;
+ };
+ // *** end of instrumentation ***
+
+ /**
+ * Creates a LayeredBloomFilter that retains enclosed filters for
+ * {@code duration} and limits the contents of each enclosed filter to a time
+ * {@code quanta}. This filter uses the timestamped Bloom filter internally.
+ *
+ * @param shape The shape of the Bloom filters.
+ * @param duration The length of time to keep filters in the list.
+ * @param dUnit The unit of time to apply to duration.
+ * @param quanta The quantization factor for each filter. Individual filters
+ * will span at most this much time.
+ * @param qUnit the unit of time to apply to quanta.
+ * @return LayeredBloomFilter with the above properties.
+ */
+ static LayeredBloomFilter createTimedLayeredFilter(Shape shape, long duration, TimeUnit dUnit, long quanta,
+ TimeUnit qUnit) {
+ LayerManager layerManager = LayerManager.builder()
+ .withSuplier(() -> new TimestampedBloomFilter(new SimpleBloomFilter(shape)))
+ .withCleanup(new CleanByTime(duration, dUnit))
+ .withExtendCheck(new AdvanceOnTimeQuanta(shape, quanta, qUnit)
+ .or(LayerManager.ExtendCheck.advanceOnCalculatedFull(shape)))
+ .build();
+ return new LayeredBloomFilter(shape, layerManager);
+ }
+
+ /**
+ * A Predicate that advances after a quantum of time.
+ */
+ static class AdvanceOnTimeQuanta implements Predicate<LayerManager> {
+ long quanta;
+
+ AdvanceOnTimeQuanta(Shape shape, long quanta, TimeUnit unit) {
+ this.quanta = unit.toMillis(quanta);
+ }
+
+ @Override
+ public boolean test(LayerManager lm) {
+ int depth = lm.getDepth();
+ if (depth == 0) {
+ return false;
+ }
+ // can not use getTarget() as it causes recursion.
+ TimestampedBloomFilter bf = (TimestampedBloomFilter) lm.get(depth - 1);
+ return bf.timestamp + quanta < System.currentTimeMillis();
+ }
+ }
+
+ /**
+ * A Consumer that cleans the list based on how long each filters has been in
+ * the list.
+ *
+ */
+ static class CleanByTime implements Consumer<LinkedList<BloomFilter>> {
+ long elapsedTime;
+
+ CleanByTime(long duration, TimeUnit unit) {
+ elapsedTime = unit.toMillis(duration);
+ }
+
+ @Override
+ public void accept(LinkedList<BloomFilter> t) {
+ long min = System.currentTimeMillis() - elapsedTime;
+ while (!t.isEmpty() && ((TimestampedBloomFilter) t.getFirst()).getTimestamp() < min) {
+ TimestampedBloomFilter bf = (TimestampedBloomFilter) t.getFirst();
+ dbgInstrument.add(String.format("Removing old entry: T:%s (Aged: %s) \n", bf.getTimestamp(),
+ (min - bf.getTimestamp())));
+ t.removeFirst();
+ }
+ }
+ }
+
+ /**
+ * A Bloomfilter implementation that tracks the creation time.
+ */
+ static class TimestampedBloomFilter extends WrappedBloomFilter {
+ final long timestamp;
+
+ TimestampedBloomFilter(BloomFilter bf) {
+ super(bf);
+ this.timestamp = System.currentTimeMillis();
+ }
+
+ public long getTimestamp() {
+ return timestamp;
+ }
+ }
+
+ @Test
+ public void testExpiration() throws InterruptedException {
+ // this test uses the instrumentation noted above to track changes for debugging
+ // purposes.
+
+ // list of timestamps that are expected to be expired.
+ List<Long> lst = new ArrayList<>();
+ Shape shape = Shape.fromNM(4, 64);
+
+ // create a filter that removes filters that are 4 seconds old
+ // and quantises time to 1 second intervals.
+ LayeredBloomFilter underTest = createTimedLayeredFilter(shape, 4, TimeUnit.SECONDS, 1, TimeUnit.SECONDS);
+
+ for (int i = 0; i < 10; i++) {
+ underTest.merge(TestingHashers.randomHasher());
+ }
+ underTest.forEachBloomFilter(dbg.and(x -> lst.add(((TimestampedBloomFilter) x).timestamp)));
+ assertTrue(underTest.getDepth() > 1);
+
+ Thread.sleep(TimeUnit.SECONDS.toMillis(2));
+ for (int i = 0; i < 10; i++) {
+ underTest.merge(TestingHashers.randomHasher());
+ }
+ dbgInstrument.add("=== AFTER 2 seconds ====\n");
+ underTest.forEachBloomFilter(dbg);
+
+ Thread.sleep(TimeUnit.SECONDS.toMillis(1));
+ for (int i = 0; i < 10; i++) {
+ underTest.merge(TestingHashers.randomHasher());
+ }
+ dbgInstrument.add("=== AFTER 3 seconds ====\n");
+ underTest.forEachBloomFilter(dbg);
+
+ // sleep 1.5 seconds to ensure we cross the 4 second boundary
+ Thread.sleep(TimeUnit.SECONDS.toMillis(4) + 500);
Review Comment:
To match the comments this can be TimeUnit.SECONDS.toMillis(1) and the test still works. That saves 3 seconds of time. But it could be improved by using a shorter sleep time across the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@commons.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org