You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2021/11/30 13:28:15 UTC
[GitHub] [iotdb] jun0315 opened a new pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
jun0315 opened a new pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502
see https://issues.apache.org/jira/browse/IOTDB-2075
test result https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=195728781&draftShareId=fe6ddf54-ae5e-4ab6-a206-f5d27bde6535&
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r770174169
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -384,4 +388,59 @@ public void unsetIsExecuted(int i) {
results.remove(i);
}
}
+
+ public int getDifferentStorageGroupsCount() {
+ if (differentStorageGroupsCount == null) {
+ Set<String> insertPlanSGSet = new HashSet<>();
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ IStorageGroupMNode storageGroupMNode = null;
+ try {
+ storageGroupMNode =
Review comment:
```suggestion
try {
IStorageGroupMNode storageGroupMNode =
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jun0315 commented on pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
jun0315 commented on pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#issuecomment-988792035
# Test Result
| | Serialization | 2 corePoolSize | 4 corePoolSize | 6 corePoolSize | 8 corePoolSize | 10 corePoolSize | 12 corePoolSize |
| --------------------------- | ------------- | -------------- | -------------- | -------------- | -------------- | --------------- | --------------- |
| 1000 Plans 48 rows 16 columns | 1397 | 1292 | 1160 | 1040 | 1085 | 1089 | 1102 |
We can find that the efficiency is the highest when we take Runtime.getRuntime().availableProcessors()/2.
Next, we default to 1000 Plans 48 rows and change the different columns.
| | Serialization | 6 corePoolSize |
| ------------------ | ------------- | -------------- |
| 4 columns | 901 | 1023 |
| 8 columns | 958 | 1068 |
| 12 columns | 1196 | 1259 |
| 14 columns | 1278 | 1239 |
| **15 columns** | **1215** | **1001** |
| **16 columns** | **1364** | **1014** |
We can find that when there are 15 columns and 16 columns, the thread pool speed will exceed the serial speed, so we set the threshold to 15.
To sum up, we set the number of core threads in the thread pool to min(the number of different sg, availableProcessors()/2), and need columns >= 15
## Test Device
MacBook Pro (16-inch, 2019)
2.6 GHz 6-core processor Core i7
16 GB 2667 MHz DDR4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r770173082
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -1524,8 +1540,56 @@ public void insertTablet(InsertMultiTabletPlan insertMultiTabletPlan)
}
}
+ private void insertTabletParallel(InsertMultiTabletPlan insertMultiTabletPlan)
+ throws BatchProcessException {
+ updateInsertTabletsPool(insertMultiTabletPlan.getDifferentStorageGroupsCount());
+
+ List<InsertTabletPlan> planList = insertMultiTabletPlan.getInsertTabletPlanList();
+ List<Future<?>> futureList = new ArrayList<>();
+
+ Map<Integer, TSStatus> results = insertMultiTabletPlan.getResults();
+ for (InsertTabletPlan plan : planList) {
+ Future<?> f =
+ insertionPool.submit(
+ () -> {
+ insertTablet(plan);
Review comment:
```suggestion
try {
insertTablet(plan);
} catch (Exception e) {
return e;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls commented on pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
coveralls commented on pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#issuecomment-982712528
[](https://coveralls.io/builds/44638629)
Coverage increased (+0.005%) to 67.327% when pulling **866170bd05583017442b4fbd9292192d5c0cef42 on jun0315:accelerate_insert_tablets** into **cbb09d0b99fa9317de06a277c4ee393a577029e9 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r770176511
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -384,4 +388,59 @@ public void unsetIsExecuted(int i) {
results.remove(i);
}
}
+
+ public int getDifferentStorageGroupsCount() {
+ if (differentStorageGroupsCount == null) {
+ Set<String> insertPlanSGSet = new HashSet<>();
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ IStorageGroupMNode storageGroupMNode = null;
+ try {
+ storageGroupMNode =
+ IoTDB.metaManager.getStorageGroupNodeByPath(insertTabletPlan.getDeviceId());
+ insertPlanSGSet.add(storageGroupMNode.getFullPath());
+ } catch (MetadataException ignored) {
+ }
+ }
+ differentStorageGroupsCount = insertPlanSGSet.size();
+ }
+ return differentStorageGroupsCount;
+ }
+
+ public boolean isEnableMultiThreading() {
+ // After testing, we can find that when there are 10 columns , the thread pool speed will exceed
+ // the serial speed, so we set the threshold to 10.
Review comment:
```suggestion
```
Explain why need the InsertMultiTabletEnableMultithreadingColumnThreshold configuration parameter here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jun0315 commented on pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
jun0315 commented on pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#issuecomment-993441623
Since the default maximum row is 10000, we redo the following experiment with row = 10000
# Test Result
| | Serialization | 2 corePoolSize | 4 corePoolSize | 6 corePoolSize | 8 corePoolSize | 10 corePoolSize | 12 corePoolSize |
| ------------------------------ | ------------- | -------------- | -------------- | -------------- | -------------- | --------------- | --------------- |
| 1000 Plans 10000 row 16 column | 1952 | 1906 | 1742 | 1608 | 1736 | 1670 | 1641 |
We can also find Runtime.getRuntime().availableProcessors() = 12 on known test machine
We can find that the efficiency is the highest when we take Runtime.getRuntime().availableProcessors()/2.
| | Serialization | 6 corePoolSize |
| ---------------------------- | ------------- | -------------- |
| 1000Plans 10000row 8 column | 710 | 809 |
| 1000Plans 10000row 10 column | 1018 | 960 |
| 1000Plans 10000row 12 column | 1912 | 1717 |
We can find that when there are 10 columns , the thread pool speed will exceed the serial speed, so we set the threshold to 10.
Therefore, we set the number of core threads in the thread pool to min(the number of different sg, availableProcessors()/2), and need columns >= 10
It should be noted that in the latest test, we found that if the number of sg is large and exceeds twice the recommended number of CPU threads, it **may** lead to failure to allocate out of heap memory and NPE. Therefore, we will also turn off multithreading in this case.
## Test Device
MacBook Pro (16-inch, 2019)
2.6 GHz 6-core processor Core i7
16 GB 2667 MHz DDR4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r769379700
##########
File path: server/src/assembly/resources/conf/iotdb-engine.properties
##########
@@ -884,6 +884,15 @@ timestamp_precision=ms
# Datatype: int
# select_into_insert_tablet_plan_row_limit=10000
+
+####################
+### Insert-Tablets Configuration
+####################
+
+# When the insert plan rowcount reaches a certain quantity and exceeds a certain ratio, turn on Multithreading acceleration
+# Datatype: int
+# insert_multi_tablet_enable_thread_pool_row_count_threshold=10
Review comment:
```suggestion
# insert_multi_tablet_enable_multithreading_column_threshold=10
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r770173949
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -384,4 +388,59 @@ public void unsetIsExecuted(int i) {
results.remove(i);
}
}
+
+ public int getDifferentStorageGroupsCount() {
+ if (differentStorageGroupsCount == null) {
+ Set<String> insertPlanSGSet = new HashSet<>();
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ IStorageGroupMNode storageGroupMNode = null;
+ try {
+ storageGroupMNode =
+ IoTDB.metaManager.getStorageGroupNodeByPath(insertTabletPlan.getDeviceId());
+ insertPlanSGSet.add(storageGroupMNode.getFullPath());
+ } catch (MetadataException ignored) {
Review comment:
This can not be ignored. Throwing a RuntimeException here is OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r769400761
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -384,4 +388,52 @@ public void unsetIsExecuted(int i) {
results.remove(i);
}
}
+
+ public int getInsertPlanSGSize() {
+ if (insertPlanSGSize == null) {
+ Set<String> insertPlanSGSet = new HashSet<>();
+ for (InsertTabletPlan insertTabletPlan : insertTabletPlanList) {
+ IStorageGroupMNode storageGroupMNode = null;
+ try {
+ storageGroupMNode =
+ IoTDB.metaManager.getStorageGroupNodeByPath(insertTabletPlan.getDeviceId());
+ insertPlanSGSet.add(storageGroupMNode.getFullPath());
+ } catch (MetadataException ignored) {
+ }
+ }
+ insertPlanSGSize = insertPlanSGSet.size();
+ }
+ return insertPlanSGSize;
+ }
+
+ public boolean needMultiThread() {
+ // After testing, we can find that when there are 10 columns , the thread pool speed will exceed
Review comment:
```suggestion
if (IoTDB.isClusterMode()) {
isEnableMultithreading = false;
return isEnableMultithreading;
}
// After testing, we can find that when there are 10 columns , the thread pool speed will exceed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#issuecomment-982712528
[](https://coveralls.io/builds/44709645)
Coverage decreased (-0.03%) to 67.294% when pulling **3bdaf8b6554ab4c4d93fc4a56346c30e8034c87f on jun0315:accelerate_insert_tablets** into **cbb09d0b99fa9317de06a277c4ee393a577029e9 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jun0315 edited a comment on pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
jun0315 edited a comment on pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#issuecomment-988792035
# Test Result
| | Serialization | 2 corePoolSize | 4 corePoolSize | 6 corePoolSize | 8 corePoolSize | 10 corePoolSize | 12 corePoolSize |
| --------------------------- | ------------- | -------------- | -------------- | -------------- | -------------- | --------------- | --------------- |
| 1000 Plans 48 rows 16 columns | 1397 | 1292 | 1160 | 1040 | 1085 | 1089 | 1102 |
Runtime.getRuntime().availableProcessors() = 12 on known test machine
We can find that the efficiency is the highest when we take Runtime.getRuntime().availableProcessors()/2.
Next, we default to 1000 Plans 48 rows and change the different columns.
| | Serialization | 6 corePoolSize |
| ------------------ | ------------- | -------------- |
| 4 columns | 901 | 1023 |
| 8 columns | 958 | 1068 |
| 12 columns | 1196 | 1259 |
| 14 columns | 1278 | 1239 |
| **15 columns** | **1215** | **1001** |
| **16 columns** | **1364** | **1014** |
We can find that when there are 15 columns and 16 columns, the thread pool speed will exceed the serial speed, so we set the threshold to 15.
To sum up, we set the number of core threads in the thread pool to min(the number of different sg, availableProcessors()/2), and need columns >= 15
## Test Device
MacBook Pro (16-inch, 2019)
2.6 GHz 6-core processor Core i7
16 GB 2667 MHz DDR4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jun0315 closed pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
jun0315 closed pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r769380802
##########
File path: server/src/main/java/org/apache/iotdb/db/concurrent/ThreadName.java
##########
@@ -44,6 +44,7 @@
LOAD_TSFILE("Load-TsFile"),
TIME_COST_STATISTIC("TIME_COST_STATISTIC"),
QUERY_SERVICE("Query"),
+ INSERT_SERVICE("InsertTablets"),
Review comment:
```suggestion
INSERTION_SERVICE("MultithreadingInsertionPool"),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#issuecomment-982712528
[](https://coveralls.io/builds/44704074)
Coverage decreased (-0.03%) to 67.291% when pulling **311783804fddd58d53cfb37d49efc3599116d089 on jun0315:accelerate_insert_tablets** into **cbb09d0b99fa9317de06a277c4ee393a577029e9 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#issuecomment-982712528
[](https://coveralls.io/builds/44730875)
Coverage decreased (-0.02%) to 67.303% when pulling **954c25386f717dbccf4caba18df006bd335103ab on jun0315:accelerate_insert_tablets** into **cbb09d0b99fa9317de06a277c4ee393a577029e9 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r769399096
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -96,6 +96,10 @@
boolean[] isExecuted;
+ Boolean needMulti;
+
+ Integer insertPlanSGSize;
Review comment:
```suggestion
Boolean isEnableMultithreading;
Integer differentStorageGroupsCount;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r769398804
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertMultiTabletPlan.java
##########
@@ -384,4 +388,52 @@ public void unsetIsExecuted(int i) {
results.remove(i);
}
}
+
+ public int getInsertPlanSGSize() {
Review comment:
```suggestion
public int getDifferentStorageGroupsCount() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] dake0805 commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
dake0805 commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r770759614
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -1524,8 +1540,72 @@ public void insertTablet(InsertMultiTabletPlan insertMultiTabletPlan)
}
}
+ private void insertTabletParallel(InsertMultiTabletPlan insertMultiTabletPlan)
+ throws BatchProcessException {
+ updateInsertTabletsPool(insertMultiTabletPlan.getDifferentStorageGroupsCount());
+
+ List<InsertTabletPlan> planList = insertMultiTabletPlan.getInsertTabletPlanList();
+ List<Future<?>> futureList = new ArrayList<>();
+
+ Map<Integer, TSStatus> results = insertMultiTabletPlan.getResults();
+
+ List<InsertTabletPlan> runPlanList = new ArrayList<>();
+ Map<Integer, Integer> runIndexToRealIndex = new HashMap<>();
+ for (int i = 0; i < planList.size(); i++) {
+ if (!(results.containsKey(i) || insertMultiTabletPlan.isExecuted(i))) {
+ runPlanList.add(planList.get(i));
+ runIndexToRealIndex.put(runPlanList.size() - 1, i);
+ }
+ }
+ for (InsertTabletPlan plan : runPlanList) {
+ Future<?> f =
+ insertionPool.submit(
+ () -> {
+ insertTablet(plan);
+ return null;
+ });
+ futureList.add(f);
+ }
+ for (int i = 0; i < futureList.size(); i++) {
+ try {
+ futureList.get(i).get();
Review comment:
Future.get() should set timeout param , or else it may cause blocking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#issuecomment-982712528
[](https://coveralls.io/builds/44730279)
Coverage decreased (-0.04%) to 67.283% when pulling **954c25386f717dbccf4caba18df006bd335103ab on jun0315:accelerate_insert_tablets** into **cbb09d0b99fa9317de06a277c4ee393a577029e9 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r770173487
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -1524,8 +1540,56 @@ public void insertTablet(InsertMultiTabletPlan insertMultiTabletPlan)
}
}
+ private void insertTabletParallel(InsertMultiTabletPlan insertMultiTabletPlan)
+ throws BatchProcessException {
+ updateInsertTabletsPool(insertMultiTabletPlan.getDifferentStorageGroupsCount());
+
+ List<InsertTabletPlan> planList = insertMultiTabletPlan.getInsertTabletPlanList();
+ List<Future<?>> futureList = new ArrayList<>();
+
+ Map<Integer, TSStatus> results = insertMultiTabletPlan.getResults();
+ for (InsertTabletPlan plan : planList) {
+ Future<?> f =
+ insertionPool.submit(
+ () -> {
+ insertTablet(plan);
+ return null;
+ });
+ futureList.add(f);
+ }
+ for (int i = 0; i < futureList.size(); i++) {
+ try {
+ futureList.get(i).get();
+ } catch (Exception e) {
+ if (e.getCause() instanceof QueryProcessException) {
+ QueryProcessException qe = (QueryProcessException) e.getCause();
+ results.put(i, RpcUtils.getStatus(qe.getErrorCode(), qe.getMessage()));
+ } else {
+ results.put(i, RpcUtils.getStatus(TIME_OUT, e.getMessage()));
Review comment:
The exception may not only be a TIME_OUT exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r770173247
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -1524,8 +1540,56 @@ public void insertTablet(InsertMultiTabletPlan insertMultiTabletPlan)
}
}
+ private void insertTabletParallel(InsertMultiTabletPlan insertMultiTabletPlan)
+ throws BatchProcessException {
+ updateInsertTabletsPool(insertMultiTabletPlan.getDifferentStorageGroupsCount());
+
+ List<InsertTabletPlan> planList = insertMultiTabletPlan.getInsertTabletPlanList();
+ List<Future<?>> futureList = new ArrayList<>();
+
+ Map<Integer, TSStatus> results = insertMultiTabletPlan.getResults();
+ for (InsertTabletPlan plan : planList) {
+ Future<?> f =
+ insertionPool.submit(
+ () -> {
+ insertTablet(plan);
+ return null;
+ });
+ futureList.add(f);
+ }
+ for (int i = 0; i < futureList.size(); i++) {
+ try {
+ futureList.get(i).get();
Review comment:
```suggestion
futureList.get(i).get();
```
Handle the exception here :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#issuecomment-982712528
[](https://coveralls.io/builds/44730159)
Coverage decreased (-0.05%) to 67.275% when pulling **954c25386f717dbccf4caba18df006bd335103ab on jun0315:accelerate_insert_tablets** into **cbb09d0b99fa9317de06a277c4ee393a577029e9 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jun0315 closed pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
jun0315 closed pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r770172216
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -1524,8 +1540,56 @@ public void insertTablet(InsertMultiTabletPlan insertMultiTabletPlan)
}
}
+ private void insertTabletParallel(InsertMultiTabletPlan insertMultiTabletPlan)
+ throws BatchProcessException {
+ updateInsertTabletsPool(insertMultiTabletPlan.getDifferentStorageGroupsCount());
+
+ List<InsertTabletPlan> planList = insertMultiTabletPlan.getInsertTabletPlanList();
+ List<Future<?>> futureList = new ArrayList<>();
+
+ Map<Integer, TSStatus> results = insertMultiTabletPlan.getResults();
+ for (InsertTabletPlan plan : planList) {
+ Future<?> f =
+ insertionPool.submit(
+ () -> {
+ insertTablet(plan);
+ return null;
+ });
+ futureList.add(f);
+ }
+ for (int i = 0; i < futureList.size(); i++) {
+ try {
+ futureList.get(i).get();
+ } catch (Exception e) {
+ if (e.getCause() instanceof QueryProcessException) {
+ QueryProcessException qe = (QueryProcessException) e.getCause();
+ results.put(i, RpcUtils.getStatus(qe.getErrorCode(), qe.getMessage()));
+ } else {
+ results.put(i, RpcUtils.getStatus(TIME_OUT, e.getMessage()));
+ }
+ }
+ }
+ if (!results.isEmpty()) {
+ throw new BatchProcessException(insertMultiTabletPlan.getFailingStatus());
+ }
+ }
+
+ private void updateInsertTabletsPool(int sgSize) {
+ int updateCoreSize = Math.min(sgSize, Runtime.getRuntime().availableProcessors() / 2);
+ if (insertionPool == null || insertionPool.isTerminated()) {
+ insertionPool =
+ (ThreadPoolExecutor)
+ IoTDBThreadPoolFactory.newFixedThreadPool(
+ Runtime.getRuntime().availableProcessors(), ThreadName.INSERT_SERVICE.getName());
Review comment:
```suggestion
updateCoreSize, ThreadName.INSERT_SERVICE.getName());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jun0315 commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
jun0315 commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r771259286
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -1524,8 +1540,72 @@ public void insertTablet(InsertMultiTabletPlan insertMultiTabletPlan)
}
}
+ private void insertTabletParallel(InsertMultiTabletPlan insertMultiTabletPlan)
+ throws BatchProcessException {
+ updateInsertTabletsPool(insertMultiTabletPlan.getDifferentStorageGroupsCount());
+
+ List<InsertTabletPlan> planList = insertMultiTabletPlan.getInsertTabletPlanList();
+ List<Future<?>> futureList = new ArrayList<>();
+
+ Map<Integer, TSStatus> results = insertMultiTabletPlan.getResults();
+
+ List<InsertTabletPlan> runPlanList = new ArrayList<>();
+ Map<Integer, Integer> runIndexToRealIndex = new HashMap<>();
+ for (int i = 0; i < planList.size(); i++) {
+ if (!(results.containsKey(i) || insertMultiTabletPlan.isExecuted(i))) {
+ runPlanList.add(planList.get(i));
+ runIndexToRealIndex.put(runPlanList.size() - 1, i);
+ }
+ }
+ for (InsertTabletPlan plan : runPlanList) {
+ Future<?> f =
+ insertionPool.submit(
+ () -> {
+ insertTablet(plan);
+ return null;
+ });
+ futureList.add(f);
+ }
+ for (int i = 0; i < futureList.size(); i++) {
+ try {
+ futureList.get(i).get();
Review comment:
Thank you for your reiew, because this is a database system. If there is a blockage, there is usually a bug in the system. At this time, we hope to throw a problem to check the bug. Maybe we don't need to set timeout param, which may be different from the business system in the company.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jun0315 commented on pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
jun0315 commented on pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#issuecomment-982756528
test result:
tablet count use serial use thread pool
100 169 166
1000 378 308
5000 1044 918
10000 1375 1497
50000 7648 7172
Because the speed of insertTable is very fast, about 0.15ms, there is no acceleration effect. In order to verify this conjecture,
insertTable sleep 10 ms
tablet count use serial use thread pool
100 1206 216
1000 12310 1167
5000 59562 5177
test device:
MacBook Pro (16-inch, 2019)
2.6 GHz 6-core processor Core i7
16 GB 2667 MHz DDR4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#issuecomment-982712528
[](https://coveralls.io/builds/44708627)
Coverage decreased (-0.03%) to 67.296% when pulling **3bdaf8b6554ab4c4d93fc4a56346c30e8034c87f on jun0315:accelerate_insert_tablets** into **cbb09d0b99fa9317de06a277c4ee393a577029e9 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu merged pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu merged pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] jun0315 removed a comment on pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
jun0315 removed a comment on pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#issuecomment-982756528
test result:
tablet count use serial use thread pool
100 169 166
1000 378 308
5000 1044 918
10000 1375 1497
50000 7648 7172
Because the speed of insertTable is very fast, about 0.15ms, there is no acceleration effect. In order to verify this conjecture,
insertTable sleep 10 ms
tablet count use serial use thread pool
100 1206 216
1000 12310 1167
5000 59562 5177
test device:
MacBook Pro (16-inch, 2019)
2.6 GHz 6-core processor Core i7
16 GB 2667 MHz DDR4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r769409839
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -1506,26 +1513,77 @@ public void insert(InsertRowPlan insertRowPlan) throws QueryProcessException {
@Override
public void insertTablet(InsertMultiTabletPlan insertMultiTabletPlan)
throws QueryProcessException {
- for (int i = 0; i < insertMultiTabletPlan.getInsertTabletPlanList().size(); i++) {
- if (insertMultiTabletPlan.getResults().containsKey(i)
- || insertMultiTabletPlan.isExecuted(i)) {
- continue;
+ if (insertMultiTabletPlan.needMultiThread()) {
+ updateInsertTabletsPool(insertMultiTabletPlan.getInsertPlanSGSize());
+ }
+
+ List<InsertTabletPlan> planList = insertMultiTabletPlan.getInsertTabletPlanList();
+ List<Future<?>> futureList = new ArrayList<>();
+
+ Map<Integer, TSStatus> results = insertMultiTabletPlan.getResults();
+
+ List<InsertTabletPlan> runPlanList = new ArrayList<>();
+ Map<Integer, Integer> runIndexToRealIndex = new HashMap<>();
+ for (int i = 0; i < planList.size(); i++) {
+ if (!(results.containsKey(i) || insertMultiTabletPlan.isExecuted(i))) {
+ runPlanList.add(planList.get(i));
+ runIndexToRealIndex.put(runPlanList.size() - 1, i);
}
- try {
- insertTablet(insertMultiTabletPlan.getInsertTabletPlanList().get(i));
- } catch (QueryProcessException e) {
- insertMultiTabletPlan
- .getResults()
- .put(i, RpcUtils.getStatus(e.getErrorCode(), e.getMessage()));
+ }
+ for (int i = 0; i < runPlanList.size(); i++) {
+ InsertTabletPlan plan = runPlanList.get(i);
+ if (insertMultiTabletPlan.needMultiThread()) {
+ Future<?> f = insertTabletsPool.submit(() -> asyncInsertTablet(plan));
+ futureList.add(f);
+ } else {
+ try {
+ insertTablet(plan);
+ } catch (QueryProcessException e) {
+ results.put(i, RpcUtils.getStatus(e.getErrorCode(), e.getMessage()));
+ }
}
}
- if (!insertMultiTabletPlan.getResults().isEmpty()) {
+ if (CollectionUtils.isNotEmpty(futureList)) {
+ for (int i = 0; i < futureList.size(); i++) {
+ try {
+ futureList.get(i).get();
+ } catch (InterruptedException ignored) {
+ } catch (ExecutionException e) {
Review comment:
```suggestion
} catch (Exception e) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#discussion_r770172722
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -1524,8 +1540,56 @@ public void insertTablet(InsertMultiTabletPlan insertMultiTabletPlan)
}
}
+ private void insertTabletParallel(InsertMultiTabletPlan insertMultiTabletPlan)
+ throws BatchProcessException {
+ updateInsertTabletsPool(insertMultiTabletPlan.getDifferentStorageGroupsCount());
+
+ List<InsertTabletPlan> planList = insertMultiTabletPlan.getInsertTabletPlanList();
+ List<Future<?>> futureList = new ArrayList<>();
+
+ Map<Integer, TSStatus> results = insertMultiTabletPlan.getResults();
+ for (InsertTabletPlan plan : planList) {
+ Future<?> f =
Review comment:
```suggestion
Future<Exception> f =
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #4502: [IOTDB-2075] Accelerate the process of insertTablets by using thread pool
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #4502:
URL: https://github.com/apache/iotdb/pull/4502#issuecomment-982712528
[](https://coveralls.io/builds/44730234)
Coverage decreased (-0.03%) to 67.29% when pulling **954c25386f717dbccf4caba18df006bd335103ab on jun0315:accelerate_insert_tablets** into **cbb09d0b99fa9317de06a277c4ee393a577029e9 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org