You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "linxt20 (via GitHub)" <gi...@apache.org> on 2024/02/19 11:59:55 UTC
[PR] P btree container divide to two buffer [iotdb]
linxt20 opened a new pull request, #12048:
URL: https://github.com/apache/iotdb/pull/12048
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PBTree: Implement dual-buffer container for MNode management [iotdb]
Posted by "linxt20 (via GitHub)" <gi...@apache.org>.
linxt20 commented on code in PR #12048:
URL: https://github.com/apache/iotdb/pull/12048#discussion_r1498758080
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/IMNodeChildBuffer.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.container;
+
+import org.apache.iotdb.commons.schema.node.utils.IMNodeContainer;
+import org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.ICachedMNode;
+
+import java.util.Iterator;
+import java.util.Map;
+
+public interface IMNodeChildBuffer extends IMNodeContainer<ICachedMNode> {
+
+ Iterator<ICachedMNode> getMNodeChildBufferIterator();
+
+ Map<String, ICachedMNode> getFlushingBuffer();
+
+ Map<String, ICachedMNode> getReceivingBuffer();
+
+ void transferReceivingBufferToFlushingBuffer();
+
+ ICachedMNode removeFromFlushingBuffer(Object key);
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PBTree: Implement dual-buffer container for MNode management [iotdb]
Posted by "MarcosZyk (via GitHub)" <gi...@apache.org>.
MarcosZyk commented on code in PR #12048:
URL: https://github.com/apache/iotdb/pull/12048#discussion_r1500148151
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/MNodeChildBuffer.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.container;
+
+import org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.ICachedMNode;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static java.util.Collections.emptySet;
+
+public abstract class MNodeChildBuffer implements IMNodeChildBuffer {
+
+ protected Map<String, ICachedMNode> flushingBuffer; // Store old data nodes for disk flushing
+ protected Map<String, ICachedMNode> receivingBuffer; // Store newly created or modified data nodes
+
+ protected int totalSize =
+ 0; // The total size is the union size of flushingBuffer and receivingBuffer.
+
+ private static final IMNodeChildBuffer EMPTY_BUFFER = new MNodeChildBuffer.EmptyBuffer();
+
+ public static IMNodeChildBuffer emptyMNodeChildBuffer() {
+ return EMPTY_BUFFER;
+ }
+
+ @Override
+ public Iterator<ICachedMNode> getMNodeChildBufferIterator() {
+ return new MNodeChildBufferIterator();
+ }
+
+ @Override
+ public Map<String, ICachedMNode> getFlushingBuffer() {
+ return flushingBuffer == null ? Collections.emptyMap() : flushingBuffer;
+ }
+
+ @Override
+ public Map<String, ICachedMNode> getReceivingBuffer() {
+ return receivingBuffer == null ? Collections.emptyMap() : receivingBuffer;
+ }
+
+ @Override
+ public void transferReceivingBufferToFlushingBuffer() {
+ if (flushingBuffer == null) {
+ flushingBuffer = new ConcurrentHashMap<>();
+ }
+ if (receivingBuffer != null) {
+ flushingBuffer.putAll(receivingBuffer);
+ receivingBuffer.clear();
+ }
+ }
+
+ @Override
+ public int size() {
+ return totalSize;
+ }
+
+ @Override
+ public boolean isEmpty() {
+ return totalSize == 0;
+ }
+
+ private boolean containKey(Map<String, ICachedMNode> map, Object key) {
+ return map != null && map.containsKey(key);
+ }
+
+ @Override
+ public boolean containsKey(Object key) {
+ return containKey(flushingBuffer, key) || containKey(receivingBuffer, key);
+ }
+
+ private boolean containValue(Map<String, ICachedMNode> map, Object value) {
+ return map != null && map.containsValue(value);
+ }
+
+ @Override
+ public boolean containsValue(Object value) {
+ return containValue(flushingBuffer, value) || containValue(receivingBuffer, value);
+ }
+
+ protected ICachedMNode get(Map<String, ICachedMNode> map, Object key) {
+ return map != null ? map.get(key) : null;
+ }
+
+ @Override
+ public synchronized ICachedMNode get(Object key) {
+ ICachedMNode result = get(receivingBuffer, key);
+ if (result != null) return result;
+ else {
+ return get(flushingBuffer, key);
+ }
+ }
+
+ private ICachedMNode remove(Map<String, ICachedMNode> map, Object key) {
+ return map == null ? null : map.remove(key);
+ }
+
+ @Override
+ public synchronized ICachedMNode remove(Object key) {
+ // There are some duplicate keys in recevingBuffer and flushingBuffer.
+ ICachedMNode result1 = remove(flushingBuffer, key);
+ ICachedMNode result2 = remove(receivingBuffer, key);
+ // If the first one is empty, then look at the second result; if the first one is not empty,
+ // then the second one is either empty or the same as the first one
+ ICachedMNode result = result1 != null ? result1 : result2;
+ if (result != null) {
+ totalSize--;
+ }
+ return result;
+ }
+
+ @Override
+ public void clear() {
+ if (receivingBuffer != null) {
+ receivingBuffer.clear();
+ }
+ if (flushingBuffer != null) {
+ flushingBuffer.clear();
+ }
+ totalSize = 0;
+ }
+
+ private Set<String> keySet(Map<String, ICachedMNode> map) {
+ return map == null ? Collections.emptySet() : map.keySet();
+ }
+
+ @Nonnull
+ @Override
+ public Set<String> keySet() {
+ Set<String> result = new TreeSet<>();
+ // This is a set structure. If there are duplicates, set will automatically remove them.
+ result.addAll(keySet(receivingBuffer));
+ result.addAll(keySet(flushingBuffer));
+ return result;
+ }
+
+ private Collection<ICachedMNode> values(Map<String, ICachedMNode> map) {
+ return map == null ? Collections.emptyList() : map.values();
+ }
+
+ @Nonnull
+ @Override
+ public Collection<ICachedMNode> values() {
+ Collection<ICachedMNode> result = new HashSet<>();
+ result.addAll(values(flushingBuffer));
+ result.addAll(values(receivingBuffer));
+ return result;
+ }
+
+ private Set<Entry<String, ICachedMNode>> entrySet(Map<String, ICachedMNode> map) {
+ return map == null ? Collections.emptySet() : map.entrySet();
+ }
+
+ @Nonnull
+ @Override
+ public Set<Entry<String, ICachedMNode>> entrySet() {
+ Set<Entry<String, ICachedMNode>> result = new HashSet<>();
+ // HashSet will automatically remove duplicates
+ result.addAll(entrySet(receivingBuffer));
+ result.addAll(entrySet(flushingBuffer));
+ return result;
+ }
+
+ @Nullable
+ @Override
+ public synchronized ICachedMNode replace(String key, ICachedMNode value) {
+ throw new UnsupportedOperationException();
+ }
+
+ private class MNodeChildBufferIterator implements Iterator<ICachedMNode> {
+
+ int flushingIndex = 0;
+ int receivingIndex = 0;
+ List<ICachedMNode> flushingBufferList;
+ List<ICachedMNode> receivingBufferList;
+
+ MNodeChildBufferIterator() {
+ // use merge sort to merge them and remove duplicates.
+ List<ICachedMNode> list = new ArrayList<>();
+ receivingBufferList = new ArrayList<>(getReceivingBuffer().values());
+ flushingBufferList = new ArrayList<>(getFlushingBuffer().values());
+ receivingBufferList.sort(Comparator.comparing(ICachedMNode::getName));
+ flushingBufferList.sort(Comparator.comparing(ICachedMNode::getName));
+ }
+
+ private ICachedMNode tryGetNext() {
+ // There are only three situations here, namely, both are left, and each of the two is left,
+ // and then gradually merge and remove duplicates.
+ if (receivingIndex < receivingBufferList.size()
+ && flushingIndex < flushingBufferList.size()) {
+ ICachedMNode node1 = receivingBufferList.get(receivingIndex);
+ ICachedMNode node2 = flushingBufferList.get(flushingIndex);
+ if (node1.getName().compareTo(node2.getName()) < 0) {
+ receivingIndex++;
+ return node1;
+ } else if (node1.getName().compareTo(node2.getName()) > 0) {
+ flushingIndex++;
+ return node2;
+ } else {
+ receivingIndex++;
+ flushingIndex++;
+ return node1;
+ }
+ }
+ if (receivingIndex < receivingBufferList.size()) {
+ return receivingBufferList.get(receivingIndex++);
+ }
+ if (flushingIndex < flushingBufferList.size()) {
+ return flushingBufferList.get(flushingIndex++);
+ }
+ throw new NoSuchElementException();
+ }
+
+ @Override
+ public boolean hasNext() {
+ return flushingIndex < flushingBufferList.size()
+ || receivingIndex < receivingBufferList.size();
+ }
+
+ @Override
+ public ICachedMNode next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ return tryGetNext();
+ }
+ }
+
+ private static class EmptyBuffer extends AbstractMap<String, ICachedMNode>
Review Comment:
Separate this static class from current file. Make it an individual class file.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/MNodeChildBuffer.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.container;
+
+import org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.ICachedMNode;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static java.util.Collections.emptySet;
+
+public abstract class MNodeChildBuffer implements IMNodeChildBuffer {
+
+ protected Map<String, ICachedMNode> flushingBuffer; // Store old data nodes for disk flushing
+ protected Map<String, ICachedMNode> receivingBuffer; // Store newly created or modified data nodes
+
+ protected int totalSize =
+ 0; // The total size is the union size of flushingBuffer and receivingBuffer.
+
+ private static final IMNodeChildBuffer EMPTY_BUFFER = new MNodeChildBuffer.EmptyBuffer();
+
+ public static IMNodeChildBuffer emptyMNodeChildBuffer() {
+ return EMPTY_BUFFER;
+ }
+
+ @Override
+ public Iterator<ICachedMNode> getMNodeChildBufferIterator() {
+ return new MNodeChildBufferIterator();
+ }
+
+ @Override
+ public Map<String, ICachedMNode> getFlushingBuffer() {
+ return flushingBuffer == null ? Collections.emptyMap() : flushingBuffer;
+ }
+
+ @Override
+ public Map<String, ICachedMNode> getReceivingBuffer() {
+ return receivingBuffer == null ? Collections.emptyMap() : receivingBuffer;
+ }
+
+ @Override
+ public void transferReceivingBufferToFlushingBuffer() {
+ if (flushingBuffer == null) {
+ flushingBuffer = new ConcurrentHashMap<>();
+ }
+ if (receivingBuffer != null) {
+ flushingBuffer.putAll(receivingBuffer);
+ receivingBuffer.clear();
+ }
+ }
+
+ @Override
+ public int size() {
+ return totalSize;
+ }
+
+ @Override
+ public boolean isEmpty() {
+ return totalSize == 0;
+ }
+
+ private boolean containKey(Map<String, ICachedMNode> map, Object key) {
+ return map != null && map.containsKey(key);
+ }
+
+ @Override
+ public boolean containsKey(Object key) {
+ return containKey(flushingBuffer, key) || containKey(receivingBuffer, key);
+ }
+
+ private boolean containValue(Map<String, ICachedMNode> map, Object value) {
+ return map != null && map.containsValue(value);
+ }
+
+ @Override
+ public boolean containsValue(Object value) {
+ return containValue(flushingBuffer, value) || containValue(receivingBuffer, value);
+ }
+
+ protected ICachedMNode get(Map<String, ICachedMNode> map, Object key) {
+ return map != null ? map.get(key) : null;
+ }
+
+ @Override
+ public synchronized ICachedMNode get(Object key) {
+ ICachedMNode result = get(receivingBuffer, key);
+ if (result != null) return result;
+ else {
+ return get(flushingBuffer, key);
+ }
+ }
+
+ private ICachedMNode remove(Map<String, ICachedMNode> map, Object key) {
+ return map == null ? null : map.remove(key);
+ }
+
+ @Override
+ public synchronized ICachedMNode remove(Object key) {
+ // There are some duplicate keys in recevingBuffer and flushingBuffer.
+ ICachedMNode result1 = remove(flushingBuffer, key);
+ ICachedMNode result2 = remove(receivingBuffer, key);
+ // If the first one is empty, then look at the second result; if the first one is not empty,
+ // then the second one is either empty or the same as the first one
+ ICachedMNode result = result1 != null ? result1 : result2;
+ if (result != null) {
+ totalSize--;
+ }
+ return result;
+ }
+
+ @Override
+ public void clear() {
+ if (receivingBuffer != null) {
+ receivingBuffer.clear();
+ }
+ if (flushingBuffer != null) {
+ flushingBuffer.clear();
+ }
+ totalSize = 0;
+ }
+
+ private Set<String> keySet(Map<String, ICachedMNode> map) {
+ return map == null ? Collections.emptySet() : map.keySet();
+ }
+
+ @Nonnull
+ @Override
+ public Set<String> keySet() {
+ Set<String> result = new TreeSet<>();
+ // This is a set structure. If there are duplicates, set will automatically remove them.
+ result.addAll(keySet(receivingBuffer));
+ result.addAll(keySet(flushingBuffer));
+ return result;
+ }
+
+ private Collection<ICachedMNode> values(Map<String, ICachedMNode> map) {
+ return map == null ? Collections.emptyList() : map.values();
+ }
+
+ @Nonnull
+ @Override
+ public Collection<ICachedMNode> values() {
+ Collection<ICachedMNode> result = new HashSet<>();
+ result.addAll(values(flushingBuffer));
+ result.addAll(values(receivingBuffer));
+ return result;
+ }
+
+ private Set<Entry<String, ICachedMNode>> entrySet(Map<String, ICachedMNode> map) {
+ return map == null ? Collections.emptySet() : map.entrySet();
+ }
+
+ @Nonnull
+ @Override
+ public Set<Entry<String, ICachedMNode>> entrySet() {
+ Set<Entry<String, ICachedMNode>> result = new HashSet<>();
+ // HashSet will automatically remove duplicates
+ result.addAll(entrySet(receivingBuffer));
+ result.addAll(entrySet(flushingBuffer));
+ return result;
+ }
+
+ @Nullable
+ @Override
+ public synchronized ICachedMNode replace(String key, ICachedMNode value) {
+ throw new UnsupportedOperationException();
+ }
+
+ private class MNodeChildBufferIterator implements Iterator<ICachedMNode> {
+
+ int flushingIndex = 0;
+ int receivingIndex = 0;
+ List<ICachedMNode> flushingBufferList;
+ List<ICachedMNode> receivingBufferList;
+
+ MNodeChildBufferIterator() {
+ // use merge sort to merge them and remove duplicates.
+ List<ICachedMNode> list = new ArrayList<>();
+ receivingBufferList = new ArrayList<>(getReceivingBuffer().values());
+ flushingBufferList = new ArrayList<>(getFlushingBuffer().values());
+ receivingBufferList.sort(Comparator.comparing(ICachedMNode::getName));
+ flushingBufferList.sort(Comparator.comparing(ICachedMNode::getName));
+ }
+
+ private ICachedMNode tryGetNext() {
+ // There are only three situations here, namely, both are left, and each of the two is left,
+ // and then gradually merge and remove duplicates.
+ if (receivingIndex < receivingBufferList.size()
+ && flushingIndex < flushingBufferList.size()) {
+ ICachedMNode node1 = receivingBufferList.get(receivingIndex);
+ ICachedMNode node2 = flushingBufferList.get(flushingIndex);
+ if (node1.getName().compareTo(node2.getName()) < 0) {
+ receivingIndex++;
+ return node1;
+ } else if (node1.getName().compareTo(node2.getName()) > 0) {
+ flushingIndex++;
+ return node2;
+ } else {
+ receivingIndex++;
+ flushingIndex++;
+ return node1;
+ }
+ }
+ if (receivingIndex < receivingBufferList.size()) {
+ return receivingBufferList.get(receivingIndex++);
+ }
+ if (flushingIndex < flushingBufferList.size()) {
+ return flushingBufferList.get(flushingIndex++);
+ }
+ throw new NoSuchElementException();
+ }
+
+ @Override
+ public boolean hasNext() {
+ return flushingIndex < flushingBufferList.size()
+ || receivingIndex < receivingBufferList.size();
+ }
+
+ @Override
+ public ICachedMNode next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ return tryGetNext();
+ }
+ }
+
+ private static class EmptyBuffer extends AbstractMap<String, ICachedMNode>
Review Comment:
Separate this static class from current file. Make it an individual class file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PBTree: Implement dual-buffer container for MNode management [iotdb]
Posted by "MarcosZyk (via GitHub)" <gi...@apache.org>.
MarcosZyk merged PR #12048:
URL: https://github.com/apache/iotdb/pull/12048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PBTree: Implement dual-buffer container for MNode management [iotdb]
Posted by "linxt20 (via GitHub)" <gi...@apache.org>.
linxt20 commented on code in PR #12048:
URL: https://github.com/apache/iotdb/pull/12048#discussion_r1500208902
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/CachedMNodeContainer.java:
##########
@@ -383,10 +374,11 @@ public synchronized String toString() {
private class CachedMNodeContainerIterator implements Iterator<ICachedMNode> {
Iterator<ICachedMNode> iterator;
- byte status = 0;
+ byte status;
- CachedMNodeContainerIterator() {
- iterator = getChildCache().values().iterator();
+ CachedMNodeContainerIterator(byte _status) {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PBTree: Implement dual-buffer container for MNode management [iotdb]
Posted by "MarcosZyk (via GitHub)" <gi...@apache.org>.
MarcosZyk commented on code in PR #12048:
URL: https://github.com/apache/iotdb/pull/12048#discussion_r1500154231
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/MNodeChildBuffer.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.container;
+
+import org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.ICachedMNode;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static java.util.Collections.emptySet;
+
+public abstract class MNodeChildBuffer implements IMNodeChildBuffer {
+
+ protected Map<String, ICachedMNode> flushingBuffer; // Store old data nodes for disk flushing
+ protected Map<String, ICachedMNode> receivingBuffer; // Store newly created or modified data nodes
+
+ protected int totalSize =
+ 0; // The total size is the union size of flushingBuffer and receivingBuffer.
+
+ private static final IMNodeChildBuffer EMPTY_BUFFER = new MNodeChildBuffer.EmptyBuffer();
+
+ public static IMNodeChildBuffer emptyMNodeChildBuffer() {
+ return EMPTY_BUFFER;
+ }
+
+ @Override
+ public Iterator<ICachedMNode> getMNodeChildBufferIterator() {
+ return new MNodeChildBufferIterator();
+ }
+
+ @Override
+ public Map<String, ICachedMNode> getFlushingBuffer() {
+ return flushingBuffer == null ? Collections.emptyMap() : flushingBuffer;
+ }
+
+ @Override
+ public Map<String, ICachedMNode> getReceivingBuffer() {
+ return receivingBuffer == null ? Collections.emptyMap() : receivingBuffer;
+ }
+
+ @Override
+ public void transferReceivingBufferToFlushingBuffer() {
+ if (flushingBuffer == null) {
+ flushingBuffer = new ConcurrentHashMap<>();
+ }
+ if (receivingBuffer != null) {
+ flushingBuffer.putAll(receivingBuffer);
+ receivingBuffer.clear();
+ }
+ }
+
+ @Override
+ public int size() {
+ return totalSize;
+ }
+
+ @Override
+ public boolean isEmpty() {
+ return totalSize == 0;
+ }
+
+ private boolean containKey(Map<String, ICachedMNode> map, Object key) {
+ return map != null && map.containsKey(key);
+ }
+
+ @Override
+ public boolean containsKey(Object key) {
+ return containKey(flushingBuffer, key) || containKey(receivingBuffer, key);
+ }
+
+ private boolean containValue(Map<String, ICachedMNode> map, Object value) {
+ return map != null && map.containsValue(value);
+ }
+
+ @Override
+ public boolean containsValue(Object value) {
+ return containValue(flushingBuffer, value) || containValue(receivingBuffer, value);
+ }
+
+ protected ICachedMNode get(Map<String, ICachedMNode> map, Object key) {
+ return map != null ? map.get(key) : null;
+ }
+
+ @Override
+ public synchronized ICachedMNode get(Object key) {
+ ICachedMNode result = get(receivingBuffer, key);
+ if (result != null) return result;
Review Comment:
block this line of code with {}
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/IMNodeChildBuffer.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.container;
+
+import org.apache.iotdb.commons.schema.node.utils.IMNodeContainer;
+import org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.ICachedMNode;
+
+import java.util.Iterator;
+import java.util.Map;
+
+public interface IMNodeChildBuffer extends IMNodeContainer<ICachedMNode> {
+
+ // get the iterator of the whole Buffer which is make a merge sort of the ReceivingBuffer and the
+ // FlushingBuffer
+ Iterator<ICachedMNode> getMNodeChildBufferIterator();
+
+ // only get the FlushingBuffer
+ Map<String, ICachedMNode> getFlushingBuffer();
+
+ // only get the ReceivingBuffer
+ Map<String, ICachedMNode> getReceivingBuffer();
Review Comment:
Add attention notes: there shall not be write operation on the returned map instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PBTree: Implement dual-buffer container for MNode management [iotdb]
Posted by "linxt20 (via GitHub)" <gi...@apache.org>.
linxt20 commented on code in PR #12048:
URL: https://github.com/apache/iotdb/pull/12048#discussion_r1500212238
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/MNodeChildBuffer.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.container;
+
+import org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.ICachedMNode;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static java.util.Collections.emptySet;
+
+public abstract class MNodeChildBuffer implements IMNodeChildBuffer {
+
+ protected Map<String, ICachedMNode> flushingBuffer; // Store old data nodes for disk flushing
+ protected Map<String, ICachedMNode> receivingBuffer; // Store newly created or modified data nodes
+
+ protected int totalSize =
+ 0; // The total size is the union size of flushingBuffer and receivingBuffer.
+
+ private static final IMNodeChildBuffer EMPTY_BUFFER = new MNodeChildBuffer.EmptyBuffer();
+
+ public static IMNodeChildBuffer emptyMNodeChildBuffer() {
+ return EMPTY_BUFFER;
+ }
+
+ @Override
+ public Iterator<ICachedMNode> getMNodeChildBufferIterator() {
+ return new MNodeChildBufferIterator();
+ }
+
+ @Override
+ public Map<String, ICachedMNode> getFlushingBuffer() {
+ return flushingBuffer == null ? Collections.emptyMap() : flushingBuffer;
+ }
+
+ @Override
+ public Map<String, ICachedMNode> getReceivingBuffer() {
+ return receivingBuffer == null ? Collections.emptyMap() : receivingBuffer;
+ }
+
+ @Override
+ public void transferReceivingBufferToFlushingBuffer() {
+ if (flushingBuffer == null) {
+ flushingBuffer = new ConcurrentHashMap<>();
+ }
+ if (receivingBuffer != null) {
+ flushingBuffer.putAll(receivingBuffer);
+ receivingBuffer.clear();
+ }
+ }
+
+ @Override
+ public int size() {
+ return totalSize;
+ }
+
+ @Override
+ public boolean isEmpty() {
+ return totalSize == 0;
+ }
+
+ private boolean containKey(Map<String, ICachedMNode> map, Object key) {
+ return map != null && map.containsKey(key);
+ }
+
+ @Override
+ public boolean containsKey(Object key) {
+ return containKey(flushingBuffer, key) || containKey(receivingBuffer, key);
+ }
+
+ private boolean containValue(Map<String, ICachedMNode> map, Object value) {
+ return map != null && map.containsValue(value);
+ }
+
+ @Override
+ public boolean containsValue(Object value) {
+ return containValue(flushingBuffer, value) || containValue(receivingBuffer, value);
+ }
+
+ protected ICachedMNode get(Map<String, ICachedMNode> map, Object key) {
+ return map != null ? map.get(key) : null;
+ }
+
+ @Override
+ public synchronized ICachedMNode get(Object key) {
+ ICachedMNode result = get(receivingBuffer, key);
+ if (result != null) return result;
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PBTree: Implement dual-buffer container for MNode management [iotdb]
Posted by "linxt20 (via GitHub)" <gi...@apache.org>.
linxt20 commented on code in PR #12048:
URL: https://github.com/apache/iotdb/pull/12048#discussion_r1498743075
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/memory/MemoryManager.java:
##########
@@ -322,6 +323,11 @@ public ICachedMNode next() {
private void tryGetNext() {
ICachedMNode node;
CacheEntry cacheEntry;
+ if (!bufferedNodeIterator.hasNext() && status == 0) {
+ // 此时说明NewChildBuffer的flushingBuffer已经遍历完毕，需要遍历UpdateChildBuffer的flushingBuffer
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PBTree: Implement dual-buffer container for MNode management [iotdb]
Posted by "linxt20 (via GitHub)" <gi...@apache.org>.
linxt20 commented on code in PR #12048:
URL: https://github.com/apache/iotdb/pull/12048#discussion_r1498752788
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/CachedMNodeContainer.java:
##########
@@ -425,14 +416,10 @@ private boolean changeStatus() {
private class BufferIterator implements Iterator<ICachedMNode> {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PBTree: Implement dual-buffer container for MNode management [iotdb]
Posted by "MarcosZyk (via GitHub)" <gi...@apache.org>.
MarcosZyk commented on code in PR #12048:
URL: https://github.com/apache/iotdb/pull/12048#discussion_r1500149994
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/CachedMNodeContainer.java:
##########
@@ -383,10 +374,11 @@ public synchronized String toString() {
private class CachedMNodeContainerIterator implements Iterator<ICachedMNode> {
Iterator<ICachedMNode> iterator;
- byte status = 0;
+ byte status;
- CachedMNodeContainerIterator() {
- iterator = getChildCache().values().iterator();
+ CachedMNodeContainerIterator(byte _status) {
Review Comment:
Rename the _status to status.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/MNodeChildBuffer.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.container;
+
+import org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.ICachedMNode;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static java.util.Collections.emptySet;
+
+public abstract class MNodeChildBuffer implements IMNodeChildBuffer {
+
+ protected Map<String, ICachedMNode> flushingBuffer; // Store old data nodes for disk flushing
+ protected Map<String, ICachedMNode> receivingBuffer; // Store newly created or modified data nodes
+
+ protected int totalSize =
+ 0; // The total size is the union size of flushingBuffer and receivingBuffer.
+
+ private static final IMNodeChildBuffer EMPTY_BUFFER = new MNodeChildBuffer.EmptyBuffer();
+
+ public static IMNodeChildBuffer emptyMNodeChildBuffer() {
+ return EMPTY_BUFFER;
+ }
+
+ @Override
+ public Iterator<ICachedMNode> getMNodeChildBufferIterator() {
+ return new MNodeChildBufferIterator();
+ }
+
+ @Override
+ public Map<String, ICachedMNode> getFlushingBuffer() {
+ return flushingBuffer == null ? Collections.emptyMap() : flushingBuffer;
+ }
+
+ @Override
+ public Map<String, ICachedMNode> getReceivingBuffer() {
+ return receivingBuffer == null ? Collections.emptyMap() : receivingBuffer;
+ }
+
+ @Override
+ public void transferReceivingBufferToFlushingBuffer() {
+ if (flushingBuffer == null) {
+ flushingBuffer = new ConcurrentHashMap<>();
+ }
+ if (receivingBuffer != null) {
+ flushingBuffer.putAll(receivingBuffer);
+ receivingBuffer.clear();
+ }
+ }
+
+ @Override
+ public int size() {
+ return totalSize;
+ }
+
+ @Override
+ public boolean isEmpty() {
+ return totalSize == 0;
+ }
+
+ private boolean containKey(Map<String, ICachedMNode> map, Object key) {
+ return map != null && map.containsKey(key);
+ }
+
+ @Override
+ public boolean containsKey(Object key) {
+ return containKey(flushingBuffer, key) || containKey(receivingBuffer, key);
+ }
+
+ private boolean containValue(Map<String, ICachedMNode> map, Object value) {
+ return map != null && map.containsValue(value);
+ }
+
+ @Override
+ public boolean containsValue(Object value) {
+ return containValue(flushingBuffer, value) || containValue(receivingBuffer, value);
+ }
+
+ protected ICachedMNode get(Map<String, ICachedMNode> map, Object key) {
+ return map != null ? map.get(key) : null;
+ }
+
+ @Override
+ public synchronized ICachedMNode get(Object key) {
+ ICachedMNode result = get(receivingBuffer, key);
+ if (result != null) return result;
+ else {
+ return get(flushingBuffer, key);
+ }
+ }
+
+ private ICachedMNode remove(Map<String, ICachedMNode> map, Object key) {
+ return map == null ? null : map.remove(key);
+ }
+
+ @Override
+ public synchronized ICachedMNode remove(Object key) {
+ // There are some duplicate keys in recevingBuffer and flushingBuffer.
+ ICachedMNode result1 = remove(flushingBuffer, key);
+ ICachedMNode result2 = remove(receivingBuffer, key);
+ // If the first one is empty, then look at the second result; if the first one is not empty,
+ // then the second one is either empty or the same as the first one
+ ICachedMNode result = result1 != null ? result1 : result2;
+ if (result != null) {
+ totalSize--;
+ }
+ return result;
+ }
+
+ @Override
+ public void clear() {
+ if (receivingBuffer != null) {
+ receivingBuffer.clear();
+ }
+ if (flushingBuffer != null) {
+ flushingBuffer.clear();
+ }
+ totalSize = 0;
+ }
+
+ private Set<String> keySet(Map<String, ICachedMNode> map) {
+ return map == null ? Collections.emptySet() : map.keySet();
+ }
+
+ @Nonnull
+ @Override
+ public Set<String> keySet() {
+ Set<String> result = new TreeSet<>();
+ // This is a set structure. If there are duplicates, set will automatically remove them.
+ result.addAll(keySet(receivingBuffer));
+ result.addAll(keySet(flushingBuffer));
+ return result;
+ }
+
+ private Collection<ICachedMNode> values(Map<String, ICachedMNode> map) {
+ return map == null ? Collections.emptyList() : map.values();
+ }
+
+ @Nonnull
+ @Override
+ public Collection<ICachedMNode> values() {
+ Collection<ICachedMNode> result = new HashSet<>();
+ result.addAll(values(flushingBuffer));
+ result.addAll(values(receivingBuffer));
+ return result;
+ }
+
+ private Set<Entry<String, ICachedMNode>> entrySet(Map<String, ICachedMNode> map) {
+ return map == null ? Collections.emptySet() : map.entrySet();
+ }
+
+ @Nonnull
+ @Override
+ public Set<Entry<String, ICachedMNode>> entrySet() {
+ Set<Entry<String, ICachedMNode>> result = new HashSet<>();
+ // HashSet will automatically remove duplicates
+ result.addAll(entrySet(receivingBuffer));
+ result.addAll(entrySet(flushingBuffer));
+ return result;
+ }
+
+ @Nullable
+ @Override
+ public synchronized ICachedMNode replace(String key, ICachedMNode value) {
+ throw new UnsupportedOperationException();
+ }
+
+ private class MNodeChildBufferIterator implements Iterator<ICachedMNode> {
+
+ int flushingIndex = 0;
+ int receivingIndex = 0;
+ List<ICachedMNode> flushingBufferList;
+ List<ICachedMNode> receivingBufferList;
+
+ MNodeChildBufferIterator() {
+ // use merge sort to merge them and remove duplicates.
+ List<ICachedMNode> list = new ArrayList<>();
+ receivingBufferList = new ArrayList<>(getReceivingBuffer().values());
+ flushingBufferList = new ArrayList<>(getFlushingBuffer().values());
+ receivingBufferList.sort(Comparator.comparing(ICachedMNode::getName));
+ flushingBufferList.sort(Comparator.comparing(ICachedMNode::getName));
+ }
+
+ private ICachedMNode tryGetNext() {
+ // There are only three situations here, namely, both are left, and each of the two is left,
+ // and then gradually merge and remove duplicates.
+ if (receivingIndex < receivingBufferList.size()
+ && flushingIndex < flushingBufferList.size()) {
+ ICachedMNode node1 = receivingBufferList.get(receivingIndex);
+ ICachedMNode node2 = flushingBufferList.get(flushingIndex);
+ if (node1.getName().compareTo(node2.getName()) < 0) {
+ receivingIndex++;
+ return node1;
+ } else if (node1.getName().compareTo(node2.getName()) > 0) {
+ flushingIndex++;
+ return node2;
+ } else {
+ receivingIndex++;
+ flushingIndex++;
+ return node1;
+ }
+ }
+ if (receivingIndex < receivingBufferList.size()) {
+ return receivingBufferList.get(receivingIndex++);
+ }
+ if (flushingIndex < flushingBufferList.size()) {
+ return flushingBufferList.get(flushingIndex++);
+ }
+ throw new NoSuchElementException();
+ }
+
+ @Override
+ public boolean hasNext() {
+ return flushingIndex < flushingBufferList.size()
+ || receivingIndex < receivingBufferList.size();
+ }
+
+ @Override
+ public ICachedMNode next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ return tryGetNext();
+ }
+ }
+
+ private static class EmptyBuffer extends AbstractMap<String, ICachedMNode>
Review Comment:
Separate this static class from current file. Make it an individual class file.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/CachedMNodeContainer.java:
##########
@@ -335,17 +326,17 @@ public synchronized void updateMNode(String name) {
ICachedMNode node = removeFromMap(childCache, name);
if (node != null) {
if (updatedChildBuffer == null) {
- updatedChildBuffer = new ConcurrentHashMap<>();
+ updatedChildBuffer = new MNodeUpdateChildBuffer();
}
updatedChildBuffer.put(name, node);
}
}
@Override
public synchronized void moveMNodeToCache(String name) {
- ICachedMNode node = removeFromMap(newChildBuffer, name);
+ ICachedMNode node = getNewChildBuffer().removeFromFlushingBuffer(name);
if (node == null) {
- node = removeFromMap(updatedChildBuffer, name);
+ node = getUpdatedChildBuffer().removeFromFlushingBuffer(name);
Review Comment:
These two operation shall not be executed continuously. It should be split to two interfaces-- moveMNodeFromNewChildBufferToCache and modeMNodeFromUpdateChildBufferToCache. Each of the interface should be invoked based on the status in MemoryManager#VolatileSubtreeIterator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PBTree: Implement dual-buffer container for MNode management [iotdb]
Posted by "linxt20 (via GitHub)" <gi...@apache.org>.
linxt20 commented on code in PR #12048:
URL: https://github.com/apache/iotdb/pull/12048#discussion_r1500208313
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/memory/MemoryManager.java:
##########
@@ -338,8 +344,13 @@ private void tryGetNext() {
cacheEntry = getCacheEntry(node);
synchronized (cacheEntry) {
- cacheEntry.setVolatile(false);
- memoryStatistics.removeVolatileNode();
+ if (status == 1
+ && container.getUpdatedChildReceivingBuffer().containsKey(node.getName())) {
+ // 此时是处于updatebuffer当中，此时需要判断cacheEntry在updatebuffer的receivingbuffer当中是否有重复点，如果没有再设置为false
+ } else {
+ cacheEntry.setVolatile(false);
+ memoryStatistics.removeVolatileNode();
+ }
container.moveMNodeToCache(node.getName());
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PBTree: Implement dual-buffer container for MNode management [iotdb]
Posted by "MarcosZyk (via GitHub)" <gi...@apache.org>.
MarcosZyk commented on code in PR #12048:
URL: https://github.com/apache/iotdb/pull/12048#discussion_r1498699673
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/memory/MemoryManager.java:
##########
@@ -322,6 +323,11 @@ public ICachedMNode next() {
private void tryGetNext() {
ICachedMNode node;
CacheEntry cacheEntry;
+ if (!bufferedNodeIterator.hasNext() && status == 0) {
+ // 此时说明NewChildBuffer的flushingBuffer已经遍历完毕，需要遍历UpdateChildBuffer的flushingBuffer
Review Comment:
Comments shall be in English.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/IMNodeChildBuffer.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.container;
+
+import org.apache.iotdb.commons.schema.node.utils.IMNodeContainer;
+import org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.pbtree.mnode.ICachedMNode;
+
+import java.util.Iterator;
+import java.util.Map;
+
+public interface IMNodeChildBuffer extends IMNodeContainer<ICachedMNode> {
+
+ Iterator<ICachedMNode> getMNodeChildBufferIterator();
+
+ Map<String, ICachedMNode> getFlushingBuffer();
+
+ Map<String, ICachedMNode> getReceivingBuffer();
+
+ void transferReceivingBufferToFlushingBuffer();
+
+ ICachedMNode removeFromFlushingBuffer(Object key);
Review Comment:
Add javadoc to describe the use case of each method.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/CachedMNodeContainer.java:
##########
@@ -425,14 +416,10 @@ private boolean changeStatus() {
private class BufferIterator implements Iterator<ICachedMNode> {
Review Comment:
It seems this class can be removed since newChildBuffer.iterator can be used directly.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/memory/MemoryManager.java:
##########
@@ -338,8 +344,13 @@ private void tryGetNext() {
cacheEntry = getCacheEntry(node);
synchronized (cacheEntry) {
- cacheEntry.setVolatile(false);
- memoryStatistics.removeVolatileNode();
+ if (status == 1
+ && container.getUpdatedChildReceivingBuffer().containsKey(node.getName())) {
+ // 此时是处于updatebuffer当中，此时需要判断cacheEntry在updatebuffer的receivingbuffer当中是否有重复点，如果没有再设置为false
+ } else {
+ cacheEntry.setVolatile(false);
+ memoryStatistics.removeVolatileNode();
+ }
container.moveMNodeToCache(node.getName());
Review Comment:
These three operations belong to one case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PBTree: Implement dual-buffer container for MNode management [iotdb]
Posted by "linxt20 (via GitHub)" <gi...@apache.org>.
linxt20 commented on code in PR #12048:
URL: https://github.com/apache/iotdb/pull/12048#discussion_r1500217694
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/CachedMNodeContainer.java:
##########
@@ -335,17 +326,17 @@ public synchronized void updateMNode(String name) {
ICachedMNode node = removeFromMap(childCache, name);
if (node != null) {
if (updatedChildBuffer == null) {
- updatedChildBuffer = new ConcurrentHashMap<>();
+ updatedChildBuffer = new MNodeUpdateChildBuffer();
}
updatedChildBuffer.put(name, node);
}
}
@Override
public synchronized void moveMNodeToCache(String name) {
- ICachedMNode node = removeFromMap(newChildBuffer, name);
+ ICachedMNode node = getNewChildBuffer().removeFromFlushingBuffer(name);
if (node == null) {
- node = removeFromMap(updatedChildBuffer, name);
+ node = getUpdatedChildBuffer().removeFromFlushingBuffer(name);
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org