You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2021/02/23 11:37:10 UTC
[GitHub] [iotdb] haimeiguo opened a new pull request #2724: filter slots for snapshots
haimeiguo opened a new pull request #2724:
URL: https://github.com/apache/iotdb/pull/2724
currently we generate snapshot for slots in all storage groups.
this pr fixes it and generates timeseries schema snapshot for slots in current data group member only
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] neuyilan merged pull request #2724: filter slots for snapshots
Posted by GitBox <gi...@apache.org>.
neuyilan merged pull request #2724:
URL: https://github.com/apache/iotdb/pull/2724
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] mychaow commented on a change in pull request #2724: filter slots for snapshots
Posted by GitBox <gi...@apache.org>.
mychaow commented on a change in pull request #2724:
URL: https://github.com/apache/iotdb/pull/2724#discussion_r581511225
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/log/manage/FilePartitionedSnapshotLogManager.java
##########
@@ -119,9 +119,20 @@ private void collectTsFilesAndFillTimeseriesSchemas() throws IOException {
// 2.register the measurement
for (Map.Entry<Integer, Collection<TimeseriesSchema>> entry : slotTimeseries.entrySet()) {
int slotNum = entry.getKey();
- FileSnapshot snapshot = slotSnapshots.computeIfAbsent(slotNum, s -> new FileSnapshot());
- if (snapshot.getTimeseriesSchemas().isEmpty()) {
- snapshot.setTimeseriesSchemas(entry.getValue());
+
+ boolean slotExistsInPartition = true;
+ if (dataGroupMember.getMetaGroupMember() != null) {
Review comment:
same as following.
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/log/manage/PartitionedSnapshotLogManager.java
##########
@@ -99,10 +99,20 @@ void collectTimeseriesSchemas() {
.calculateSlotByTime(
storageGroupName, 0, ((SlotPartitionTable) partitionTable).getTotalSlotNumbers());
- Collection<TimeseriesSchema> schemas =
- slotTimeseries.computeIfAbsent(slot, s -> new HashSet<>());
- IoTDB.metaManager.collectTimeseriesSchema(sgNode, schemas);
- logger.debug("{}: {} timeseries are snapshot in slot {}", getName(), schemas.size(), slot);
+ boolean slotExistsInPartition = true;
+ if (dataGroupMember.getMetaGroupMember() != null) {
Review comment:
could move it to the line 94.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] neuyilan commented on a change in pull request #2724: filter slots for snapshots
Posted by GitBox <gi...@apache.org>.
neuyilan commented on a change in pull request #2724:
URL: https://github.com/apache/iotdb/pull/2724#discussion_r581532127
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/log/manage/FilePartitionedSnapshotLogManager.java
##########
@@ -117,11 +117,22 @@ private void collectTsFilesAndFillTimeseriesSchemas() throws IOException {
collectTsFiles();
// 2.register the measurement
+ boolean slotExistsInPartition = true;
for (Map.Entry<Integer, Collection<TimeseriesSchema>> entry : slotTimeseries.entrySet()) {
int slotNum = entry.getKey();
- FileSnapshot snapshot = slotSnapshots.computeIfAbsent(slotNum, s -> new FileSnapshot());
- if (snapshot.getTimeseriesSchemas().isEmpty()) {
- snapshot.setTimeseriesSchemas(entry.getValue());
+
+ if (dataGroupMember.getMetaGroupMember() != null) {
+ List<Integer> slots =
+ ((SlotPartitionTable) dataGroupMember.getMetaGroupMember().getPartitionTable())
+ .getNodeSlots(dataGroupMember.getHeader());
Review comment:
Move those codes before the for loop, we do not need to get the slots for this raft group per loop, it's not efficient.
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/log/manage/PartitionedSnapshotLogManager.java
##########
@@ -92,17 +92,27 @@ public Snapshot getSnapshot(long minIndex) {
void collectTimeseriesSchemas() {
slotTimeseries.clear();
List<StorageGroupMNode> allSgNodes = IoTDB.metaManager.getAllStorageGroupNodes();
+ boolean slotExistsInPartition = true;
for (MNode sgNode : allSgNodes) {
String storageGroupName = sgNode.getFullPath();
int slot =
SlotPartitionTable.getSlotStrategy()
.calculateSlotByTime(
storageGroupName, 0, ((SlotPartitionTable) partitionTable).getTotalSlotNumbers());
- Collection<TimeseriesSchema> schemas =
- slotTimeseries.computeIfAbsent(slot, s -> new HashSet<>());
- IoTDB.metaManager.collectTimeseriesSchema(sgNode, schemas);
- logger.debug("{}: {} timeseries are snapshot in slot {}", getName(), schemas.size(), slot);
+ if (dataGroupMember.getMetaGroupMember() != null) {
+ List<Integer> slots =
+ ((SlotPartitionTable) dataGroupMember.getMetaGroupMember().getPartitionTable())
+ .getNodeSlots(dataGroupMember.getHeader());
Review comment:
the same as above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org