You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2020/01/08 03:22:03 UTC
[GitHub] [incubator-iotdb] SilverNarcissus opened a new pull request #712:
[IOTDB-408] Improve batch insert in partitioned storage group
SilverNarcissus opened a new pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712
Improve batch insert in partitioned storage group.
This pr is build from [IOTDB-333] and is for testing performance.
DO NOT merge without contact with me. Thanks~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r365630913
##########
File path: server/src/assembly/resources/conf/iotdb-engine.properties
##########
@@ -415,4 +415,17 @@ value_encoder=PLAIN
# Compression configuration
# Data compression method, TsFile supports UNCOMPRESSED or SNAPPY. Default value is UNCOMPRESSED which means no compression
-compressor=SNAPPY
\ No newline at end of file
+compressor=SNAPPY
+
+# Time range for divide storage group
+# Time series data will divide into groups by this time range
+# Unit is second
+# default value is one day, which is 86400 seconds
+partition_interval = 86400
+
+# the num of memtables in each storage group
+# this config decide how many time series in different time partition can be insert concurrently
Review comment:
see above
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull
request #712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
SilverNarcissus commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r366113716
##########
File path: server/src/assembly/resources/conf/iotdb-engine.properties
##########
@@ -415,4 +415,17 @@ value_encoder=PLAIN
# Compression configuration
# Data compression method, TsFile supports UNCOMPRESSED or SNAPPY. Default value is UNCOMPRESSED which means no compression
-compressor=SNAPPY
\ No newline at end of file
+compressor=SNAPPY
+
+# Time range for divide storage group
+# Time series data will divide into groups by this time range
+# Unit is second
+# default value is one day, which is 86400 seconds
+partition_interval = 86400
+
+# the num of memtables in each storage group
+# this config decide how many time series in different time partition can be insert concurrently
Review comment:
fixed~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull
request #712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
SilverNarcissus commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r366113638
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/version/SimpleFileVersionController.java
##########
@@ -31,27 +31,49 @@
* SimpleFileVersionController uses a local file and its file name to store the version.
*/
public class SimpleFileVersionController implements VersionController {
- private static final Logger logger = LoggerFactory.getLogger(SimpleFileVersionController.class);
+ private static final Logger logger = LoggerFactory.getLogger(SimpleFileVersionController.class);
+ private static final String FILE_PREFIX = "Version-";
/**
* Every time currVersion - prevVersion >= saveInterval, currVersion is persisted and prevVersion
* is set to currVersion. When recovering from file, the version number is automatically increased
* by saveInterval to avoid conflicts.
*/
private static long saveInterval = 100;
- private static final String FILE_PREFIX = "Version-";
+ /**
+ * time partition id to dividing time series into different storage group
Review comment:
fixed~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r365631640
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -131,38 +134,59 @@
* avoid some tsfileResource is changed (e.g., from unsealed to sealed) when a query is executed.
*/
private final ReadWriteLock closeQueryLock = new ReentrantReadWriteLock();
+ /**
+ * time partition id in the storage group -> tsFileProcessor for this time partition
+ */
+ private final TreeMap<Long, TsFileProcessor> workSequenceTsFileProcessors = new TreeMap<>();
+ /**
+ * time partition id in the storage group -> tsFileProcessor for this time partition
+ */
+ private final TreeMap<Long, TsFileProcessor> workUnsequenceTsFileProcessors = new TreeMap<>();
+ /**
+ * Time range for dividing storage group, unit is second
+ */
+ private long partitionIntervalForStorageGroup;
/**
* the schema of time series that belong this storage group
*/
private Schema schema;
// includes sealed and unsealed sequence TsFiles
- private List<TsFileResource> sequenceFileList = new ArrayList<>();
- private TsFileProcessor workSequenceTsFileProcessor = null;
+ private TreeSet<TsFileResource> sequenceFileList = new TreeSet<>(
Review comment:
Rename the field since it is no more a List.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull
request #712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
SilverNarcissus commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r366113647
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -889,27 +1126,42 @@ public void delete(String deviceId, String measurementId, long timestamp) throws
List<ModificationFile> updatedModFiles = new ArrayList<>();
try {
- Long lastUpdateTime = latestTimeForEachDevice.get(deviceId);
- // no tsfile data, the delete operation is invalid
+ Long lastUpdateTime = null;
+ for (Map<String, Long> latestTimeMap : latestTimeForEachDevice.values()) {
+ Long curTime = latestTimeMap.get(deviceId);
+ if (curTime != null && (lastUpdateTime == null || lastUpdateTime < curTime)) {
+ lastUpdateTime = curTime;
+ }
+ }
+
+ // There is no tsfile data, the delete operation is invalid
if (lastUpdateTime == null) {
logger.debug("No device {} in SG {}, deletion invalid", deviceId, storageGroupName);
return;
}
+ // time partition to divide storage group
+ long timePartitionId = fromTimeToTimePartition(timestamp);
// write log
if (IoTDBDescriptor.getInstance().getConfig().isEnableWal()) {
- if (workSequenceTsFileProcessor != null) {
- workSequenceTsFileProcessor.getLogNode()
- .write(new DeletePlan(timestamp, new Path(deviceId, measurementId)));
+ for (Map.Entry<Long, TsFileProcessor> entry : workSequenceTsFileProcessors.entrySet()) {
+ if (entry.getKey() <= timePartitionId) {
+ entry.getValue().getLogNode()
+ .write(new DeletePlan(timestamp, new Path(deviceId, measurementId)));
Review comment:
fixed~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] SilverNarcissus commented on issue #712:
[IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
SilverNarcissus commented on issue #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#issuecomment-573963992
This pr has been merged by https://github.com/apache/incubator-iotdb/pull/664. So this pr will be closed now~ If you want to review the code, please check PR664 as above
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull
request #712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
SilverNarcissus commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r366113667
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -131,38 +134,59 @@
* avoid some tsfileResource is changed (e.g., from unsealed to sealed) when a query is executed.
*/
private final ReadWriteLock closeQueryLock = new ReentrantReadWriteLock();
+ /**
+ * time partition id in the storage group -> tsFileProcessor for this time partition
+ */
+ private final TreeMap<Long, TsFileProcessor> workSequenceTsFileProcessors = new TreeMap<>();
+ /**
+ * time partition id in the storage group -> tsFileProcessor for this time partition
+ */
+ private final TreeMap<Long, TsFileProcessor> workUnsequenceTsFileProcessors = new TreeMap<>();
+ /**
+ * Time range for dividing storage group, unit is second
+ */
+ private long partitionIntervalForStorageGroup;
/**
* the schema of time series that belong this storage group
*/
private Schema schema;
// includes sealed and unsealed sequence TsFiles
- private List<TsFileResource> sequenceFileList = new ArrayList<>();
- private TsFileProcessor workSequenceTsFileProcessor = null;
+ private TreeSet<TsFileResource> sequenceFileList = new TreeSet<>(
Review comment:
fixed~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r365637259
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/version/SimpleFileVersionController.java
##########
@@ -31,27 +31,49 @@
* SimpleFileVersionController uses a local file and its file name to store the version.
*/
public class SimpleFileVersionController implements VersionController {
- private static final Logger logger = LoggerFactory.getLogger(SimpleFileVersionController.class);
+ private static final Logger logger = LoggerFactory.getLogger(SimpleFileVersionController.class);
+ private static final String FILE_PREFIX = "Version-";
/**
* Every time currVersion - prevVersion >= saveInterval, currVersion is persisted and prevVersion
* is set to currVersion. When recovering from file, the version number is automatically increased
* by saveInterval to avoid conflicts.
*/
private static long saveInterval = 100;
- private static final String FILE_PREFIX = "Version-";
+ /**
+ * time partition id to dividing time series into different storage group
Review comment:
-> time partition id within a storage group, each partition in a storage group has its own VersionController
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] SilverNarcissus closed pull request #712:
[IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
SilverNarcissus closed pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull
request #712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
SilverNarcissus commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r366113729
##########
File path: docs/Documentation/UserGuide/3-Server/4-Config Manual.md
##########
@@ -236,6 +236,25 @@ The detail of each variables are as follows:
|Default| true |
|Effective|Trigger|
+* partition\_interval
+
+|Name| partition\_interval |
+|:---:|:---|
+|Description| Time range for dividing storage group, time series data will be divided into groups by this time range |
+|Type|Int64|
+|Default| 86400 |
+|Effective|Should not be changed after first start up|
+
+
+* memtable\_num\_in\_each\_storage\_group
+
+|Name| memtable\_num\_in\_each\_storage\_group |
+|:---:|:---|
+|Description| This config decide how many time series in different time partition can be insert concurrently.</br> For example, your partitionInterval is 86400 and you want to insert data in 3 different days, you should set this param >= 6 (for sequence and unsequence) |
Review comment:
fixed~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r365631200
##########
File path: server/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java
##########
@@ -299,6 +299,13 @@ private void loadProps() {
conf.setDefaultTTL(Long.parseLong(properties.getProperty("default_ttl",
String.valueOf(conf.getDefaultTTL()))));
+ // Time range for divide storage group
Review comment:
divide -> dividing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r365630892
##########
File path: docs/Documentation/UserGuide/3-Server/4-Config Manual.md
##########
@@ -236,6 +236,25 @@ The detail of each variables are as follows:
|Default| true |
|Effective|Trigger|
+* partition\_interval
+
+|Name| partition\_interval |
+|:---:|:---|
+|Description| Time range for dividing storage group, time series data will be divided into groups by this time range |
+|Type|Int64|
+|Default| 86400 |
+|Effective|Should not be changed after first start up|
+
+
+* memtable\_num\_in\_each\_storage\_group
+
+|Name| memtable\_num\_in\_each\_storage\_group |
+|:---:|:---|
+|Description| This config decide how many time series in different time partition can be insert concurrently.</br> For example, your partitionInterval is 86400 and you want to insert data in 3 different days, you should set this param >= 6 (for sequence and unsequence) |
Review comment:
-> "This config decides how many time partitions in a storage group can be inserted concurrently"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r365634852
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -534,7 +678,58 @@ private TsFileProcessor getOrCreateTsFileProcessor(boolean sequence) {
return tsFileProcessor;
}
- private TsFileProcessor createTsFileProcessor(boolean sequence)
+ /**
+ * get processor from hashmap, flush oldest processor is necessary
Review comment:
is necessary -> if necessary
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull
request #712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
SilverNarcissus commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r366113685
##########
File path: server/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java
##########
@@ -299,6 +299,13 @@ private void loadProps() {
conf.setDefaultTTL(Long.parseLong(properties.getProperty("default_ttl",
String.valueOf(conf.getDefaultTTL()))));
+ // Time range for divide storage group
Review comment:
fixed~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull
request #712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
SilverNarcissus commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r366113655
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -534,7 +678,58 @@ private TsFileProcessor getOrCreateTsFileProcessor(boolean sequence) {
return tsFileProcessor;
}
- private TsFileProcessor createTsFileProcessor(boolean sequence)
+ /**
+ * get processor from hashmap, flush oldest processor is necessary
Review comment:
fixed~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull
request #712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
SilverNarcissus commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r366113615
##########
File path: session/src/main/java/org/apache/iotdb/session/Session.java
##########
@@ -201,6 +206,98 @@ public TSExecuteBatchStatementResp insertBatch(RowBatch rowBatch)
}
}
+ /**
+ * use batch interface to insert data
+ *
+ * @param rowBatch data batch
+ */
+ public TSExecuteBatchStatementResp insertBatch(RowBatch rowBatch)
+ throws IoTDBSessionException {
+ sortRowBatch(rowBatch);
+
+ return insertSortedBatch(rowBatch);
+ }
+
+ private void sortRowBatch(RowBatch rowBatch){
+ /*
+ * following part of code sort the batch data by time,
+ * so we can insert continuous data in value list to get a better performance
+ */
+ // sort to get index, and use index to sort value list
Review comment:
fixed~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r365635985
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -889,27 +1126,42 @@ public void delete(String deviceId, String measurementId, long timestamp) throws
List<ModificationFile> updatedModFiles = new ArrayList<>();
try {
- Long lastUpdateTime = latestTimeForEachDevice.get(deviceId);
- // no tsfile data, the delete operation is invalid
+ Long lastUpdateTime = null;
+ for (Map<String, Long> latestTimeMap : latestTimeForEachDevice.values()) {
+ Long curTime = latestTimeMap.get(deviceId);
+ if (curTime != null && (lastUpdateTime == null || lastUpdateTime < curTime)) {
+ lastUpdateTime = curTime;
+ }
+ }
+
+ // There is no tsfile data, the delete operation is invalid
if (lastUpdateTime == null) {
logger.debug("No device {} in SG {}, deletion invalid", deviceId, storageGroupName);
return;
}
+ // time partition to divide storage group
+ long timePartitionId = fromTimeToTimePartition(timestamp);
// write log
if (IoTDBDescriptor.getInstance().getConfig().isEnableWal()) {
- if (workSequenceTsFileProcessor != null) {
- workSequenceTsFileProcessor.getLogNode()
- .write(new DeletePlan(timestamp, new Path(deviceId, measurementId)));
+ for (Map.Entry<Long, TsFileProcessor> entry : workSequenceTsFileProcessors.entrySet()) {
+ if (entry.getKey() <= timePartitionId) {
+ entry.getValue().getLogNode()
+ .write(new DeletePlan(timestamp, new Path(deviceId, measurementId)));
Review comment:
Maybe you can create a DeletePlan outside of the loop and reuse it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r365642004
##########
File path: session/src/main/java/org/apache/iotdb/session/Session.java
##########
@@ -201,6 +206,98 @@ public TSExecuteBatchStatementResp insertBatch(RowBatch rowBatch)
}
}
+ /**
+ * use batch interface to insert data
+ *
+ * @param rowBatch data batch
+ */
+ public TSExecuteBatchStatementResp insertBatch(RowBatch rowBatch)
+ throws IoTDBSessionException {
+ sortRowBatch(rowBatch);
+
+ return insertSortedBatch(rowBatch);
+ }
+
+ private void sortRowBatch(RowBatch rowBatch){
+ /*
+ * following part of code sort the batch data by time,
+ * so we can insert continuous data in value list to get a better performance
+ */
+ // sort to get index, and use index to sort value list
Review comment:
I would recommend that you check if the batch is already sorted before doing the actual sort.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull
request #712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
SilverNarcissus commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r366113676
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java
##########
@@ -439,14 +451,29 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
getProcessor(storageGroupName).loadNewTsFile(newTsFileResource);
}
- public boolean deleteTsfile(File deletedTsfile) throws StorageEngineException {
+ public boolean deleteTsfileForSync(File deletedTsfile)
+ throws StorageEngineException {
return getProcessor(deletedTsfile.getParentFile().getName()).deleteTsfile(deletedTsfile);
}
+ public boolean deleteTsfile(File deletedTsfile) throws StorageEngineException {
+ return getProcessor(getSgByEngineFile(deletedTsfile)).deleteTsfile(deletedTsfile);
+ }
+
public boolean moveTsfile(File tsfileToBeMoved, File targetDir)
throws StorageEngineException, IOException {
- return getProcessor(tsfileToBeMoved.getParentFile().getName())
- .moveTsfile(tsfileToBeMoved, targetDir);
+ return getProcessor(getSgByEngineFile(tsfileToBeMoved)).moveTsfile(tsfileToBeMoved, targetDir);
+ }
+
+ /**
+ * The engine file means that the file is in the engine, which is different from those external
+ * files which are not loaded.
Review comment:
fixed~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#712: [IOTDB-408] Improve batch insert in partitioned storage group
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #712: [IOTDB-408] Improve batch insert in partitioned storage group
URL: https://github.com/apache/incubator-iotdb/pull/712#discussion_r365631378
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java
##########
@@ -439,14 +451,29 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
getProcessor(storageGroupName).loadNewTsFile(newTsFileResource);
}
- public boolean deleteTsfile(File deletedTsfile) throws StorageEngineException {
+ public boolean deleteTsfileForSync(File deletedTsfile)
+ throws StorageEngineException {
return getProcessor(deletedTsfile.getParentFile().getName()).deleteTsfile(deletedTsfile);
}
+ public boolean deleteTsfile(File deletedTsfile) throws StorageEngineException {
+ return getProcessor(getSgByEngineFile(deletedTsfile)).deleteTsfile(deletedTsfile);
+ }
+
public boolean moveTsfile(File tsfileToBeMoved, File targetDir)
throws StorageEngineException, IOException {
- return getProcessor(tsfileToBeMoved.getParentFile().getName())
- .moveTsfile(tsfileToBeMoved, targetDir);
+ return getProcessor(getSgByEngineFile(tsfileToBeMoved)).moveTsfile(tsfileToBeMoved, targetDir);
+ }
+
+ /**
+ * The engine file means that the file is in the engine, which is different from those external
+ * files which are not loaded.
Review comment:
Then maybe it would be more clear to call it InternalFile.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services