You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2021/11/09 13:45:33 UTC
[GitHub] [iotdb] Cpaulyz opened a new pull request #4350: [IOTDB-1950] Add Bloom Filter cache for query
Cpaulyz opened a new pull request #4350:
URL: https://github.com/apache/iotdb/pull/4350
### Contribution
* Add `BloomFilterCache` to cache bloom filter instead of read from TsFile directly
* Add `BloomFilterCacheTest`
* Update `iotdb-engine.properties` about allocated memory for BloomFilterCache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] JackieTien97 commented on a change in pull request #4350: [IOTDB-1950] Add Bloom Filter cache for query
Posted by GitBox <gi...@apache.org>.
JackieTien97 commented on a change in pull request #4350:
URL: https://github.com/apache/iotdb/pull/4350#discussion_r747169501
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/cache/BloomFilterCache.java
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.engine.cache;
+
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.query.control.FileReaderManager;
+import org.apache.iotdb.db.utils.TestOnly;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.utils.BloomFilter;
+import org.apache.iotdb.tsfile.utils.FilePathUtils;
+import org.apache.iotdb.tsfile.utils.Pair;
+import org.apache.iotdb.tsfile.utils.RamUsageEstimator;
+
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.LoadingCache;
+import com.github.benmanes.caffeine.cache.Weigher;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/** This class is used to cache <code>BloomFilter</code> in IoTDB. The caching strategy is LRU. */
+public class BloomFilterCache {
+
+ private static final Logger logger = LoggerFactory.getLogger(BloomFilterCache.class);
+ private static final Logger DEBUG_LOGGER = LoggerFactory.getLogger("QUERY_DEBUG");
+ private static final IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+ private static final long MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE =
+ config.getAllocateMemoryForBloomFilterCache();
+ private static final boolean CACHE_ENABLE = config.isMetaDataCacheEnable();
+
+ private final LoadingCache<BloomFilterCacheKey, BloomFilter> lruCache;
+
+ private BloomFilterCache() {
+ if (CACHE_ENABLE) {
+ logger.info("BloomFilterCache size = " + MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE);
+ }
+ lruCache =
+ Caffeine.newBuilder()
+ .maximumWeight(MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE)
+ .weigher(
+ (Weigher<BloomFilterCacheKey, BloomFilter>)
+ (key, bloomFilter) ->
+ (int)
+ (RamUsageEstimator.shallowSizeOf(key)
+ + RamUsageEstimator.sizeOf(key.tsFilePrefixPath)
+ + RamUsageEstimator.sizeOf(bloomFilter)))
+ .recordStats()
+ .build(
+ key -> {
+ try {
+ TsFileSequenceReader reader =
+ FileReaderManager.getInstance().get(key.filePath, true);
+ return reader.readBloomFilter();
+ } catch (IOException e) {
+ logger.error(
+ "Something wrong happened in reading bloom filter in tsfile {}",
+ key.filePath,
+ e);
+ throw e;
+ }
+ });
+ }
+
+ public static BloomFilterCache getInstance() {
+ return BloomFilterCacheHolder.INSTANCE;
+ }
+
+ public BloomFilter get(BloomFilterCacheKey key) throws IOException {
+ return get(key, false);
+ }
+
+ public BloomFilter get(BloomFilterCacheKey key, boolean debug) throws IOException {
+ if (!CACHE_ENABLE) {
+ TsFileSequenceReader reader = FileReaderManager.getInstance().get(key.filePath, true);
+ return reader.readBloomFilter();
+ }
+
+ BloomFilter bloomFilter = lruCache.get(key);
+
+ if (debug) {
+ DEBUG_LOGGER.info("get bloomFilter from cache where filePath is: " + key.filePath);
+ }
+
+ return bloomFilter;
+ }
+
+ public double calculateChunkHitRatio() {
+ return lruCache.stats().hitRate();
+ }
+
+ public long getEvictionCount() {
+ return lruCache.stats().evictionCount();
+ }
+
+ public long getMaxMemory() {
+ return MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE;
+ }
+
+ public double getAverageLoadPenalty() {
+ return lruCache.stats().averageLoadPenalty();
+ }
+
+ /** clear LRUCache. */
+ public void clear() {
+ lruCache.invalidateAll();
+ lruCache.cleanUp();
+ }
+
+ public void remove(BloomFilterCacheKey key) {
Review comment:
```suggestion
@TestOnly
public void remove(BloomFilterCacheKey key) {
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/cache/TimeSeriesMetadataCache.java
##########
@@ -118,7 +118,9 @@ public TimeseriesMetadata get(
if (!CACHE_ENABLE) {
// bloom filter part
TsFileSequenceReader reader = FileReaderManager.getInstance().get(key.filePath, true);
- BloomFilter bloomFilter = reader.readBloomFilter();
+ BloomFilter bloomFilter =
+ BloomFilterCache.getInstance()
+ .get(new BloomFilterCache.BloomFilterCacheKey(key.filePath));
Review comment:
If the `CACHE_ENABLE` is false, there is no need to read from `BloomFilterCache`, just use `reader.readBloomFilter()`.
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/cache/TimeSeriesMetadataCache.java
##########
@@ -210,7 +214,9 @@ public TimeseriesMetadata get(
if (!CACHE_ENABLE) {
// bloom filter part
TsFileSequenceReader reader = FileReaderManager.getInstance().get(key.filePath, true);
- BloomFilter bloomFilter = reader.readBloomFilter();
+ BloomFilter bloomFilter =
+ BloomFilterCache.getInstance()
+ .get(new BloomFilterCache.BloomFilterCacheKey(key.filePath));
Review comment:
```suggestion
.get(new BloomFilterCache.BloomFilterCacheKey(key.filePath), debug);
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/cache/TimeSeriesMetadataCache.java
##########
@@ -239,7 +245,9 @@ public TimeseriesMetadata get(
Path path = new Path(key.device, key.measurement);
// bloom filter part
TsFileSequenceReader reader = FileReaderManager.getInstance().get(key.filePath, true);
- BloomFilter bloomFilter = reader.readBloomFilter();
+ BloomFilter bloomFilter =
+ BloomFilterCache.getInstance()
+ .get(new BloomFilterCache.BloomFilterCacheKey(key.filePath));
Review comment:
```suggestion
.get(new BloomFilterCache.BloomFilterCacheKey(key.filePath)), debug;
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/cache/BloomFilterCache.java
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.engine.cache;
+
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.query.control.FileReaderManager;
+import org.apache.iotdb.db.utils.TestOnly;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.utils.BloomFilter;
+import org.apache.iotdb.tsfile.utils.FilePathUtils;
+import org.apache.iotdb.tsfile.utils.Pair;
+import org.apache.iotdb.tsfile.utils.RamUsageEstimator;
+
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.LoadingCache;
+import com.github.benmanes.caffeine.cache.Weigher;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/** This class is used to cache <code>BloomFilter</code> in IoTDB. The caching strategy is LRU. */
+public class BloomFilterCache {
+
+ private static final Logger logger = LoggerFactory.getLogger(BloomFilterCache.class);
+ private static final Logger DEBUG_LOGGER = LoggerFactory.getLogger("QUERY_DEBUG");
+ private static final IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+ private static final long MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE =
+ config.getAllocateMemoryForBloomFilterCache();
+ private static final boolean CACHE_ENABLE = config.isMetaDataCacheEnable();
+
+ private final LoadingCache<BloomFilterCacheKey, BloomFilter> lruCache;
+
+ private BloomFilterCache() {
+ if (CACHE_ENABLE) {
+ logger.info("BloomFilterCache size = " + MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE);
+ }
+ lruCache =
+ Caffeine.newBuilder()
+ .maximumWeight(MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE)
+ .weigher(
+ (Weigher<BloomFilterCacheKey, BloomFilter>)
+ (key, bloomFilter) ->
+ (int)
+ (RamUsageEstimator.shallowSizeOf(key)
+ + RamUsageEstimator.sizeOf(key.tsFilePrefixPath)
+ + RamUsageEstimator.sizeOf(bloomFilter)))
+ .recordStats()
+ .build(
+ key -> {
+ try {
+ TsFileSequenceReader reader =
+ FileReaderManager.getInstance().get(key.filePath, true);
+ return reader.readBloomFilter();
+ } catch (IOException e) {
+ logger.error(
+ "Something wrong happened in reading bloom filter in tsfile {}",
+ key.filePath,
+ e);
+ throw e;
+ }
+ });
+ }
+
+ public static BloomFilterCache getInstance() {
+ return BloomFilterCacheHolder.INSTANCE;
+ }
+
+ public BloomFilter get(BloomFilterCacheKey key) throws IOException {
+ return get(key, false);
+ }
+
+ public BloomFilter get(BloomFilterCacheKey key, boolean debug) throws IOException {
+ if (!CACHE_ENABLE) {
+ TsFileSequenceReader reader = FileReaderManager.getInstance().get(key.filePath, true);
+ return reader.readBloomFilter();
+ }
+
+ BloomFilter bloomFilter = lruCache.get(key);
+
+ if (debug) {
+ DEBUG_LOGGER.info("get bloomFilter from cache where filePath is: " + key.filePath);
+ }
+
+ return bloomFilter;
+ }
+
+ public double calculateChunkHitRatio() {
+ return lruCache.stats().hitRate();
+ }
+
+ public long getEvictionCount() {
+ return lruCache.stats().evictionCount();
+ }
+
+ public long getMaxMemory() {
+ return MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE;
+ }
+
+ public double getAverageLoadPenalty() {
+ return lruCache.stats().averageLoadPenalty();
+ }
Review comment:
Add all these stats into `CacheHitRatioMonitor` class, referring to ChunkCache.
##########
File path: server/src/test/java/org/apache/iotdb/db/engine/cache/BloomFilterCacheTest.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.engine.cache;
+
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.utils.BloomFilter;
+import org.apache.iotdb.tsfile.write.TsFileWriter;
+import org.apache.iotdb.tsfile.write.record.Tablet;
+import org.apache.iotdb.tsfile.write.schema.IMeasurementSchema;
+import org.apache.iotdb.tsfile.write.schema.Schema;
+import org.apache.iotdb.tsfile.write.schema.UnaryMeasurementSchema;
+
+import org.apache.commons.io.FileUtils;
+import org.junit.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class BloomFilterCacheTest {
+ private List<String> pathList;
+ private int pathSize = 3;
+ private BloomFilterCache bloomFilterCache;
+
+ @Before
+ public void setUp() throws Exception {
+ // set up and create tsFile
+ pathList = new ArrayList<>();
+ for (int i = 0; i < pathSize; i++) {
+ String path =
+ "target"
+ .concat(File.separator)
+ .concat("data")
+ .concat(File.separator)
+ .concat("data")
+ .concat(File.separator)
+ .concat("sequence")
+ .concat(File.separator)
+ .concat("root.sg" + (i + 1))
+ .concat(File.separator)
+ .concat("0")
+ .concat(File.separator)
+ .concat("0")
+ .concat(File.separator)
+ .concat("1-0-0-0.tsfile");
+ String device = "d" + (i + 1);
+ pathList.add(path);
+ createTsFile(path, device);
+ }
+ bloomFilterCache = BloomFilterCache.getInstance();
+ }
+
+ @After
+ public void tearDown() {
+ try {
+ for (String filePath : pathList) {
+ FileUtils.forceDelete(new File(filePath));
+ }
+ } catch (IOException e) {
+ Assert.fail(e.getMessage());
+ }
+ }
+
+ @Test
+ public void testGet() {
+ try {
+ for (String filePath : pathList) {
+ BloomFilter bloomFilter =
+ bloomFilterCache.get(new BloomFilterCache.BloomFilterCacheKey(filePath));
+ Assert.assertNotEquals(bloomFilter, null);
Review comment:
I think it's better to use `FileReaderManager.getInstance().get()` to get a `TsFileSequenceReader`, and use that to read a `BloomFilter`, then compare the content of these two bloomfilters. Otherwise, we only know it is not null, but we don't know whether the content of it is correct.
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/cache/BloomFilterCache.java
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.engine.cache;
+
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.query.control.FileReaderManager;
+import org.apache.iotdb.db.utils.TestOnly;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.utils.BloomFilter;
+import org.apache.iotdb.tsfile.utils.FilePathUtils;
+import org.apache.iotdb.tsfile.utils.Pair;
+import org.apache.iotdb.tsfile.utils.RamUsageEstimator;
+
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.LoadingCache;
+import com.github.benmanes.caffeine.cache.Weigher;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/** This class is used to cache <code>BloomFilter</code> in IoTDB. The caching strategy is LRU. */
+public class BloomFilterCache {
+
+ private static final Logger logger = LoggerFactory.getLogger(BloomFilterCache.class);
+ private static final Logger DEBUG_LOGGER = LoggerFactory.getLogger("QUERY_DEBUG");
+ private static final IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+ private static final long MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE =
+ config.getAllocateMemoryForBloomFilterCache();
+ private static final boolean CACHE_ENABLE = config.isMetaDataCacheEnable();
+
+ private final LoadingCache<BloomFilterCacheKey, BloomFilter> lruCache;
+
+ private BloomFilterCache() {
+ if (CACHE_ENABLE) {
+ logger.info("BloomFilterCache size = " + MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE);
+ }
+ lruCache =
+ Caffeine.newBuilder()
+ .maximumWeight(MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE)
+ .weigher(
+ (Weigher<BloomFilterCacheKey, BloomFilter>)
+ (key, bloomFilter) ->
+ (int)
+ (RamUsageEstimator.shallowSizeOf(key)
+ + RamUsageEstimator.sizeOf(key.tsFilePrefixPath)
+ + RamUsageEstimator.sizeOf(bloomFilter)))
+ .recordStats()
+ .build(
+ key -> {
+ try {
+ TsFileSequenceReader reader =
+ FileReaderManager.getInstance().get(key.filePath, true);
+ return reader.readBloomFilter();
+ } catch (IOException e) {
+ logger.error(
+ "Something wrong happened in reading bloom filter in tsfile {}",
+ key.filePath,
+ e);
+ throw e;
+ }
+ });
+ }
+
+ public static BloomFilterCache getInstance() {
+ return BloomFilterCacheHolder.INSTANCE;
+ }
+
+ public BloomFilter get(BloomFilterCacheKey key) throws IOException {
+ return get(key, false);
+ }
+
+ public BloomFilter get(BloomFilterCacheKey key, boolean debug) throws IOException {
+ if (!CACHE_ENABLE) {
+ TsFileSequenceReader reader = FileReaderManager.getInstance().get(key.filePath, true);
+ return reader.readBloomFilter();
+ }
+
+ BloomFilter bloomFilter = lruCache.get(key);
+
+ if (debug) {
+ DEBUG_LOGGER.info("get bloomFilter from cache where filePath is: " + key.filePath);
+ }
+
+ return bloomFilter;
+ }
+
+ public double calculateChunkHitRatio() {
+ return lruCache.stats().hitRate();
+ }
+
+ public long getEvictionCount() {
+ return lruCache.stats().evictionCount();
+ }
+
+ public long getMaxMemory() {
+ return MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE;
+ }
+
+ public double getAverageLoadPenalty() {
+ return lruCache.stats().averageLoadPenalty();
+ }
+
+ /** clear LRUCache. */
+ public void clear() {
+ lruCache.invalidateAll();
+ lruCache.cleanUp();
+ }
+
+ public void remove(BloomFilterCacheKey key) {
+ lruCache.invalidate(key);
+ }
+
+ @TestOnly
+ public BloomFilter getIfPresent(BloomFilterCacheKey key) {
+ return lruCache.getIfPresent(key);
+ }
+
+ @TestOnly
+ public void clearUp() {
+ lruCache.cleanUp();
+ }
Review comment:
```suggestion
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/cache/TimeSeriesMetadataCache.java
##########
@@ -143,7 +145,9 @@ public TimeseriesMetadata get(
Path path = new Path(key.device, key.measurement);
// bloom filter part
TsFileSequenceReader reader = FileReaderManager.getInstance().get(key.filePath, true);
- BloomFilter bloomFilter = reader.readBloomFilter();
+ BloomFilter bloomFilter =
+ BloomFilterCache.getInstance()
+ .get(new BloomFilterCache.BloomFilterCacheKey(key.filePath));
Review comment:
```suggestion
BloomFilter bloomFilter =
BloomFilterCache.getInstance()
.get(new BloomFilterCache.BloomFilterCacheKey(key.filePath), debug);
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/cache/BloomFilterCache.java
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.engine.cache;
+
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.query.control.FileReaderManager;
+import org.apache.iotdb.db.utils.TestOnly;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.utils.BloomFilter;
+import org.apache.iotdb.tsfile.utils.FilePathUtils;
+import org.apache.iotdb.tsfile.utils.Pair;
+import org.apache.iotdb.tsfile.utils.RamUsageEstimator;
+
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.LoadingCache;
+import com.github.benmanes.caffeine.cache.Weigher;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/** This class is used to cache <code>BloomFilter</code> in IoTDB. The caching strategy is LRU. */
+public class BloomFilterCache {
+
+ private static final Logger logger = LoggerFactory.getLogger(BloomFilterCache.class);
+ private static final Logger DEBUG_LOGGER = LoggerFactory.getLogger("QUERY_DEBUG");
+ private static final IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+ private static final long MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE =
+ config.getAllocateMemoryForBloomFilterCache();
+ private static final boolean CACHE_ENABLE = config.isMetaDataCacheEnable();
+
+ private final LoadingCache<BloomFilterCacheKey, BloomFilter> lruCache;
+
+ private BloomFilterCache() {
+ if (CACHE_ENABLE) {
+ logger.info("BloomFilterCache size = " + MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE);
+ }
+ lruCache =
+ Caffeine.newBuilder()
+ .maximumWeight(MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE)
+ .weigher(
+ (Weigher<BloomFilterCacheKey, BloomFilter>)
+ (key, bloomFilter) ->
+ (int)
+ (RamUsageEstimator.shallowSizeOf(key)
+ + RamUsageEstimator.sizeOf(key.tsFilePrefixPath)
+ + RamUsageEstimator.sizeOf(bloomFilter)))
+ .recordStats()
+ .build(
+ key -> {
+ try {
+ TsFileSequenceReader reader =
+ FileReaderManager.getInstance().get(key.filePath, true);
+ return reader.readBloomFilter();
+ } catch (IOException e) {
+ logger.error(
+ "Something wrong happened in reading bloom filter in tsfile {}",
+ key.filePath,
+ e);
+ throw e;
+ }
+ });
+ }
+
+ public static BloomFilterCache getInstance() {
+ return BloomFilterCacheHolder.INSTANCE;
+ }
+
+ public BloomFilter get(BloomFilterCacheKey key) throws IOException {
+ return get(key, false);
+ }
+
+ public BloomFilter get(BloomFilterCacheKey key, boolean debug) throws IOException {
+ if (!CACHE_ENABLE) {
+ TsFileSequenceReader reader = FileReaderManager.getInstance().get(key.filePath, true);
+ return reader.readBloomFilter();
+ }
+
+ BloomFilter bloomFilter = lruCache.get(key);
+
+ if (debug) {
+ DEBUG_LOGGER.info("get bloomFilter from cache where filePath is: " + key.filePath);
+ }
+
+ return bloomFilter;
+ }
+
+ public double calculateChunkHitRatio() {
+ return lruCache.stats().hitRate();
+ }
+
+ public long getEvictionCount() {
+ return lruCache.stats().evictionCount();
+ }
+
+ public long getMaxMemory() {
+ return MEMORY_THRESHOLD_IN_BLOOM_FILTER_CACHE;
+ }
+
+ public double getAverageLoadPenalty() {
+ return lruCache.stats().averageLoadPenalty();
+ }
+
+ /** clear LRUCache. */
+ public void clear() {
+ lruCache.invalidateAll();
+ lruCache.cleanUp();
+ }
+
+ public void remove(BloomFilterCacheKey key) {
+ lruCache.invalidate(key);
+ }
+
+ @TestOnly
+ public BloomFilter getIfPresent(BloomFilterCacheKey key) {
+ return lruCache.getIfPresent(key);
+ }
+
+ @TestOnly
+ public void clearUp() {
+ lruCache.cleanUp();
+ }
+
+ public static class BloomFilterCacheKey {
+
+ private final String filePath;
Review comment:
```suggestion
// There is no need to add this field size while calculating the size of BloomFilterCacheKey, because filePath is get from TsFileResource, different BloomFilterCacheKey of the same file share this String.
private final String filePath;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #4350: [IOTDB-1950] Add Bloom Filter cache for query
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #4350:
URL: https://github.com/apache/iotdb/pull/4350#issuecomment-964205566
[](https://coveralls.io/builds/44183881)
Coverage decreased (-0.05%) to 66.676% when pulling **fdd8b44b45af2f12ebafe5bc0aeeebcad82eeafc on Cpaulyz:feat/bloomFilterCache** into **206412a7ea01ad44a93abbb496904ae139ea37e7 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #4350: [IOTDB-1950] Add Bloom Filter cache for query
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #4350:
URL: https://github.com/apache/iotdb/pull/4350#issuecomment-964205566
[](https://coveralls.io/builds/44192287)
Coverage decreased (-0.02%) to 66.703% when pulling **e481dd666dafb6b7edc560971b45bac46c24336e on Cpaulyz:feat/bloomFilterCache** into **206412a7ea01ad44a93abbb496904ae139ea37e7 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] JackieTien97 merged pull request #4350: [IOTDB-1950] Add Bloom Filter cache for query
Posted by GitBox <gi...@apache.org>.
JackieTien97 merged pull request #4350:
URL: https://github.com/apache/iotdb/pull/4350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #4350: [IOTDB-1950] Add Bloom Filter cache for query
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #4350:
URL: https://github.com/apache/iotdb/pull/4350#issuecomment-964205566
[](https://coveralls.io/builds/44192181)
Coverage increased (+0.0004%) to 66.726% when pulling **e481dd666dafb6b7edc560971b45bac46c24336e on Cpaulyz:feat/bloomFilterCache** into **206412a7ea01ad44a93abbb496904ae139ea37e7 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #4350: [IOTDB-1950] Add Bloom Filter cache for query
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #4350:
URL: https://github.com/apache/iotdb/pull/4350#issuecomment-964205566
[](https://coveralls.io/builds/44155216)
Coverage increased (+0.01%) to 66.737% when pulling **f6bcc4765822a43c3db6bdd8c9b54337ec3d82c1 on Cpaulyz:feat/bloomFilterCache** into **206412a7ea01ad44a93abbb496904ae139ea37e7 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #4350: [IOTDB-1950] Add Bloom Filter cache for query
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #4350:
URL: https://github.com/apache/iotdb/pull/4350#issuecomment-964205566
[](https://coveralls.io/builds/44133740)
Coverage increased (+0.03%) to 66.753% when pulling **9d22b577933ada22955790b36c67bad80e28a65a on Cpaulyz:feat/bloomFilterCache** into **206412a7ea01ad44a93abbb496904ae139ea37e7 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls commented on pull request #4350: [IOTDB-1950] Add Bloom Filter cache for query
Posted by GitBox <gi...@apache.org>.
coveralls commented on pull request #4350:
URL: https://github.com/apache/iotdb/pull/4350#issuecomment-964205566
[](https://coveralls.io/builds/44128773)
Coverage remained the same at 66.725% when pulling **c660ed5bb87fc8ed28ab82ff590c1685e664fb0b on Cpaulyz:feat/bloomFilterCache** into **206412a7ea01ad44a93abbb496904ae139ea37e7 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #4350: [IOTDB-1950] Add Bloom Filter cache for query
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #4350:
URL: https://github.com/apache/iotdb/pull/4350#issuecomment-964205566
[](https://coveralls.io/builds/44192200)
Coverage decreased (-0.03%) to 66.692% when pulling **e481dd666dafb6b7edc560971b45bac46c24336e on Cpaulyz:feat/bloomFilterCache** into **206412a7ea01ad44a93abbb496904ae139ea37e7 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org