You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2021/05/14 02:39:50 UTC
[GitHub] [iotdb] neuyilan commented on a change in pull request #2950: [IOTDB-1263] optimize matched paths
neuyilan commented on a change in pull request #2950:
URL: https://github.com/apache/iotdb/pull/2950#discussion_r632234790
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/metadata/CMManager.java
##########
@@ -551,6 +564,135 @@ public synchronized boolean containsKey(PartialPath key) {
}
}
+ private class RemotePathAliasCache extends LRUCache<String, List<PartialPath>> {
+ private long valueSizeThreshold;
+ private AtomicLong valueSize = new AtomicLong(0);
+
+ // period executor thread
+ private ScheduledExecutorService periodExecutorService;
+
+ // executor every partition group update cache
+ private ExecutorService executorService = Executors.newFixedThreadPool(4);
+
+ private int cacheUpdateInteval;
+
+ RemotePathAliasCache(int cacheSize) {
+ super(cacheSize);
+ valueSizeThreshold = cacheSize * 100;
+ periodExecutorService = Executors.newSingleThreadScheduledExecutor();
+
+ cacheUpdateInteval = clusterConfig.getQueryPathsCacheUpdateInterval();
+ periodExecutorService.scheduleWithFixedDelay(
+ this::executeCacheUpdate, cacheUpdateInteval, cacheUpdateInteval, TimeUnit.MILLISECONDS);
+ }
+
+ protected Map<String, List<PartialPath>> initCache(int cacheSize) {
+ return new LinkedHashMap<String, List<PartialPath>>(cacheSize, 0.75f, true) {
+ @Override
+ protected boolean removeEldestEntry(Map.Entry eldest) {
+ boolean isRemoveEldest = size() > cacheSize || valueSize.get() > valueSizeThreshold;
+ if (isRemoveEldest) {
+ // get eldest Entry value
+ long size = cache.entrySet().iterator().next().getValue().size();
+ valueSize.addAndGet(size * -1);
+ }
+ return isRemoveEldest;
+ }
+ };
+ }
+
+ public synchronized void put(String key, List<PartialPath> value) {
+ // Statistic value size
+ valueSize.addAndGet(value.size());
+ cache.put(key, value);
+ }
+
+ @Override
+ protected List<PartialPath> loadObjectByKey(String key) {
+ return cache.get(key);
+ }
+
+ public void executeCacheUpdate() {
+ // split the paths by the data groups that should hold them
+ Map<PartitionGroup, List<String>> partitionGroupListMap = Maps.newHashMap();
+ for (Map.Entry<String, List<PartialPath>> entry : cache.entrySet()) {
+ String fullPath = entry.getKey();
+ PartialPath storageGroup = null;
+ try {
+ storageGroup = IoTDB.metaManager.getStorageGroupPath(new PartialPath(fullPath));
+ } catch (StorageGroupNotSetException | IllegalPathException e) {
+ logger.warn("Failed to get storage group path, fullPath is {}", fullPath, e);
+ continue;
+ }
+
+ PartitionGroup partitionGroup =
+ metaGroupMember.getPartitionTable().route(storageGroup.getFullPath(), 0);
+ partitionGroupListMap.computeIfAbsent(partitionGroup, p -> new ArrayList<>()).add(fullPath);
+ }
+ logger.debug("Start to execute cache updating");
+
+ for (Map.Entry<PartitionGroup, List<String>> entry : partitionGroupListMap.entrySet()) {
+ this.executorService.submit(
+ new PeriodUpdateCacheTask(entry.getKey(), entry.getValue(), getRemotePathAliasCache()));
Review comment:
There are maybe some concurrent problems when using the cache? As it will be modified in `thePeriodUpdateCacheTask` and for one storage group it not update atomic, while also be queried in this class, sometimes when we called the `loadObjectByKey()` method, we may get some path results which contain both are old while the others are new?
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/metadata/CMManager.java
##########
@@ -551,6 +564,135 @@ public synchronized boolean containsKey(PartialPath key) {
}
}
+ private class RemotePathAliasCache extends LRUCache<String, List<PartialPath>> {
+ private long valueSizeThreshold;
+ private AtomicLong valueSize = new AtomicLong(0);
+
+ // period executor thread
+ private ScheduledExecutorService periodExecutorService;
+
+ // executor every partition group update cache
+ private ExecutorService executorService = Executors.newFixedThreadPool(4);
Review comment:
Why the core size is 4? Any special considerations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org