[GitHub] [incubator-iotdb] haimeiguo opened a new pull request #1693: 863: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

haimeiguo opened a new pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693


   Users may not care about out of order data.
   So I add a configuration parameter (enable_discard_out_of_order_data) to allow iotdb to discard the out-of-order data
   
   jira: https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-863?filter=allopenissues


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

SilverNarcissus commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r492513870



##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -26,6 +26,7 @@
 import java.util.Collections;

Review comment:
       use this line to change the time partition and add some test over different time partition. ```IoTDBDescriptor.getInstance().getConfig().setPartitionInterval(100);```
   
   do not forget to set it back to original value. You can refer to ```IoTDBMultiSeriesIT.java``` for this. Thank you very much XD




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r486802944



##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);

Review comment:
       remember to reset this to its default value in the end of the test
   
   We usually do this as follows:
   
   boolean defaultValue = config.isEnableDiscard...;
   config.setEnableDiscard(true);
   do some test
   config.setEnableDiscard(defaultValue);

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);

Review comment:
       remember to reset this to its default value in the end of the test
   
   We usually do this as follows:
   
   boolean defaultValue = config.isEnableDiscard...;
   config.setEnableDiscard(true);
   do some test
   config.setEnableDiscard(defaultValue);

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);

Review comment:
       remember to reset this to its default value in the end of the test
   
   We usually do this as follows:
   
   boolean defaultValue = config.isEnableDiscard...;
   config.setEnableDiscard(true);
   do some test
   config.setEnableDiscard(defaultValue);

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);

Review comment:
       remember to reset this to its default value in the end of the test
   
   We usually do this as follows:
   
   boolean defaultValue = config.isEnableDiscard...;
   config.setEnableDiscard(true);
   do some test
   config.setEnableDiscard(defaultValue);

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);

Review comment:
       remember to reset this to its default value in the end of the test
   
   We usually do this as follows:
   
   boolean defaultValue = config.isEnableDiscard...;
   config.setEnableDiscard(true);
   do some test
   config.setEnableDiscard(defaultValue);




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] haimeiguo commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

haimeiguo commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r484307510



##########
File path: server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
##########
@@ -144,6 +144,8 @@
 
   private volatile boolean readOnly = false;
 
+  private boolean enable_discard_out_of_order_data = false;

Review comment:
       edited! thank you 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

SilverNarcissus commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r486811407



##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -706,8 +714,11 @@ public void insertTablet(InsertTabletPlan insertTabletPlan) throws BatchInsertio
           // judge if we should insert sequence
           if (!isSequence && time > lastFlushTime) {
             // insert into unsequence and then start sequence
-            noFailure = insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
-                beforeTimePartition) && noFailure;
+            if (!IoTDBDescriptor.getInstance().getConfig().isEnableDiscardOutOfOrderData()) {
+              noFailure =
+                  insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
+                      beforeTimePartition) && noFailure;
+            }

Review comment:
       I think you forget to control last batch of data and update cache part. These part also need to filter unsequence data

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());

Review comment:
       You should also add some flush operation to test your code

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();

Review comment:
       You may change the time partition range to test some conner case of your code




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

SilverNarcissus commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r494795361



##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -267,6 +269,279 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet1()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(100);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementMNode[] measurementMNodes = new MeasurementMNode[2];
+    measurementMNodes[0] = new MeasurementMNode(null, "s0",
+        new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN), null);
+    measurementMNodes[1] = new MeasurementMNode(null, "s1",
+        new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN), null);
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+    config.setPartitionInterval(defaultTimePartition);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet2()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(500);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementMNode[] measurementMNodes = new MeasurementMNode[2];
+    measurementMNodes[0] = new MeasurementMNode(null, "s0",
+        new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN), null);
+    measurementMNodes[1] = new MeasurementMNode(null, "s1",
+        new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN), null);
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+    config.setPartitionInterval(defaultTimePartition);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet3()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(1000);

Review comment:
       Sure, you set the time partition. But your insert time range only from 0 -150 which never cross the partition interval. By the way, you should also enable the partition like this:
   ```
   config.setEnablePartition(true);
   ```
   and do not forget:
   ```
   config.setEnablePartition(false);
   ```
   
   Thank you very much~




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] haimeiguo commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

haimeiguo commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r484384697



##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -733,10 +733,15 @@ public void insert(InsertRowPlan insertRowPlan) throws WriteProcessException {
       partitionLatestFlushedTimeForEachDevice
           .computeIfAbsent(timePartitionId, id -> new HashMap<>());
 
+      boolean isSequence = insertRowPlan.getTime() > partitionLatestFlushedTimeForEachDevice.get(timePartitionId)

Review comment:
       edited! thank you




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

SilverNarcissus commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r486811407



##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -706,8 +714,11 @@ public void insertTablet(InsertTabletPlan insertTabletPlan) throws BatchInsertio
           // judge if we should insert sequence
           if (!isSequence && time > lastFlushTime) {
             // insert into unsequence and then start sequence
-            noFailure = insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
-                beforeTimePartition) && noFailure;
+            if (!IoTDBDescriptor.getInstance().getConfig().isEnableDiscardOutOfOrderData()) {
+              noFailure =
+                  insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
+                      beforeTimePartition) && noFailure;
+            }

Review comment:
       I think you forget to control last batch of data and update cache part. These part also need to filter unsequence data




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

SilverNarcissus commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r486814622



##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());

Review comment:
       You should also add some flush operation to test your code

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();

Review comment:
       You may change the time partition range to test some conner case of your code




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

SilverNarcissus commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r494795361



##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -267,6 +269,279 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet1()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(100);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementMNode[] measurementMNodes = new MeasurementMNode[2];
+    measurementMNodes[0] = new MeasurementMNode(null, "s0",
+        new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN), null);
+    measurementMNodes[1] = new MeasurementMNode(null, "s1",
+        new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN), null);
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+    config.setPartitionInterval(defaultTimePartition);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet2()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(500);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementMNode[] measurementMNodes = new MeasurementMNode[2];
+    measurementMNodes[0] = new MeasurementMNode(null, "s0",
+        new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN), null);
+    measurementMNodes[1] = new MeasurementMNode(null, "s1",
+        new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN), null);
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+    config.setPartitionInterval(defaultTimePartition);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet3()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(1000);

Review comment:
       Sure, you set time partition. But you insert time range only from 0 -150 which never cross the partition interval. By the way, you should also enable the partition like this:
   ```
   config.setEnablePartition(true);
   ```
   and do not forget:
   ```
   config.setEnablePartition(false);
   ```
   
   Thank you very much~




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

SilverNarcissus commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r486811407



##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -706,8 +714,11 @@ public void insertTablet(InsertTabletPlan insertTabletPlan) throws BatchInsertio
           // judge if we should insert sequence
           if (!isSequence && time > lastFlushTime) {
             // insert into unsequence and then start sequence
-            noFailure = insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
-                beforeTimePartition) && noFailure;
+            if (!IoTDBDescriptor.getInstance().getConfig().isEnableDiscardOutOfOrderData()) {
+              noFailure =
+                  insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
+                      beforeTimePartition) && noFailure;
+            }

Review comment:
       I think you forget to control last batch of data and update cache part. These part also need to filter unsequence data

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());

Review comment:
       You should also add some flush operation to test your code

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();

Review comment:
       You may change the time partition range to test some conner case of your code

##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -706,8 +714,11 @@ public void insertTablet(InsertTabletPlan insertTabletPlan) throws BatchInsertio
           // judge if we should insert sequence
           if (!isSequence && time > lastFlushTime) {
             // insert into unsequence and then start sequence
-            noFailure = insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
-                beforeTimePartition) && noFailure;
+            if (!IoTDBDescriptor.getInstance().getConfig().isEnableDiscardOutOfOrderData()) {
+              noFailure =
+                  insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
+                      beforeTimePartition) && noFailure;
+            }

Review comment:
       I think you forget to control last batch of data and update cache part. These part also need to filter unsequence data

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());

Review comment:
       You should also add some flush operation to test your code

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();

Review comment:
       You may change the time partition range to test some conner case of your code

##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -706,8 +714,11 @@ public void insertTablet(InsertTabletPlan insertTabletPlan) throws BatchInsertio
           // judge if we should insert sequence
           if (!isSequence && time > lastFlushTime) {
             // insert into unsequence and then start sequence
-            noFailure = insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
-                beforeTimePartition) && noFailure;
+            if (!IoTDBDescriptor.getInstance().getConfig().isEnableDiscardOutOfOrderData()) {
+              noFailure =
+                  insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
+                      beforeTimePartition) && noFailure;
+            }

Review comment:
       I think you forget to control last batch of data and update cache part. These part also need to filter unsequence data

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());

Review comment:
       You should also add some flush operation to test your code

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();

Review comment:
       You may change the time partition range to test some conner case of your code

##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -706,8 +714,11 @@ public void insertTablet(InsertTabletPlan insertTabletPlan) throws BatchInsertio
           // judge if we should insert sequence
           if (!isSequence && time > lastFlushTime) {
             // insert into unsequence and then start sequence
-            noFailure = insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
-                beforeTimePartition) && noFailure;
+            if (!IoTDBDescriptor.getInstance().getConfig().isEnableDiscardOutOfOrderData()) {
+              noFailure =
+                  insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
+                      beforeTimePartition) && noFailure;
+            }

Review comment:
       I think you forget to control last batch of data and update cache part. These part also need to filter unsequence data

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());

Review comment:
       You should also add some flush operation to test your code

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();

Review comment:
       You may change the time partition range to test some conner case of your code

##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -706,8 +714,11 @@ public void insertTablet(InsertTabletPlan insertTabletPlan) throws BatchInsertio
           // judge if we should insert sequence
           if (!isSequence && time > lastFlushTime) {
             // insert into unsequence and then start sequence
-            noFailure = insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
-                beforeTimePartition) && noFailure;
+            if (!IoTDBDescriptor.getInstance().getConfig().isEnableDiscardOutOfOrderData()) {
+              noFailure =
+                  insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
+                      beforeTimePartition) && noFailure;
+            }

Review comment:
       I think you forget to control last batch of data and update cache part. These part also need to filter unsequence data

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());

Review comment:
       You should also add some flush operation to test your code

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();

Review comment:
       You may change the time partition range to test some conner case of your code

##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -706,8 +714,11 @@ public void insertTablet(InsertTabletPlan insertTabletPlan) throws BatchInsertio
           // judge if we should insert sequence
           if (!isSequence && time > lastFlushTime) {
             // insert into unsequence and then start sequence
-            noFailure = insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
-                beforeTimePartition) && noFailure;
+            if (!IoTDBDescriptor.getInstance().getConfig().isEnableDiscardOutOfOrderData()) {
+              noFailure =
+                  insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
+                      beforeTimePartition) && noFailure;
+            }

Review comment:
       I think you forget to control last batch of data and update cache part. These part also need to filter unsequence data

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());

Review comment:
       You should also add some flush operation to test your code

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();

Review comment:
       You may change the time partition range to test some conner case of your code

##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -706,8 +714,11 @@ public void insertTablet(InsertTabletPlan insertTabletPlan) throws BatchInsertio
           // judge if we should insert sequence
           if (!isSequence && time > lastFlushTime) {
             // insert into unsequence and then start sequence
-            noFailure = insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
-                beforeTimePartition) && noFailure;
+            if (!IoTDBDescriptor.getInstance().getConfig().isEnableDiscardOutOfOrderData()) {
+              noFailure =
+                  insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
+                      beforeTimePartition) && noFailure;
+            }

Review comment:
       I think you forget to control last batch of data and update cache part. These part also need to filter unsequence data

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());

Review comment:
       You should also add some flush operation to test your code

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();

Review comment:
       You may change the time partition range to test some conner case of your code

##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -706,8 +714,11 @@ public void insertTablet(InsertTabletPlan insertTabletPlan) throws BatchInsertio
           // judge if we should insert sequence
           if (!isSequence && time > lastFlushTime) {
             // insert into unsequence and then start sequence
-            noFailure = insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
-                beforeTimePartition) && noFailure;
+            if (!IoTDBDescriptor.getInstance().getConfig().isEnableDiscardOutOfOrderData()) {
+              noFailure =
+                  insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
+                      beforeTimePartition) && noFailure;
+            }

Review comment:
       I think you forget to control last batch of data and update cache part. These part also need to filter unsequence data

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());

Review comment:
       You should also add some flush operation to test your code

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();

Review comment:
       You may change the time partition range to test some conner case of your code

##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -706,8 +714,11 @@ public void insertTablet(InsertTabletPlan insertTabletPlan) throws BatchInsertio
           // judge if we should insert sequence
           if (!isSequence && time > lastFlushTime) {
             // insert into unsequence and then start sequence
-            noFailure = insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
-                beforeTimePartition) && noFailure;
+            if (!IoTDBDescriptor.getInstance().getConfig().isEnableDiscardOutOfOrderData()) {
+              noFailure =
+                  insertTabletToTsFileProcessor(insertTabletPlan, before, loc, false, results,
+                      beforeTimePartition) && noFailure;
+            }

Review comment:
       I think you forget to control last batch of data and update cache part. These part also need to filter unsequence data

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());

Review comment:
       You should also add some flush operation to test your code

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet()
+      throws QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementSchema[] schemas = new MeasurementSchema[2];
+    schemas[0] = new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN);
+    schemas[1] = new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN);
+
+    MNode deviceMNode = new MNode(null, deviceId);
+    deviceMNode.addChild("s0", new MeasurementMNode(null, null, null, null));
+    deviceMNode.addChild("s1", new MeasurementMNode(null, null, null, null));
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan1.setSchemas(schemas);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+    insertTabletPlan2.setSchemas(schemas);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setDeviceMNode(deviceMNode);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();

Review comment:
       You may change the time partition range to test some conner case of your code




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] haimeiguo commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

haimeiguo commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r493237697



##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -26,6 +26,7 @@
 import java.util.Collections;

Review comment:
       Thank you for your review! I added test cases for time partition = 100, 500 and 1000. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

SilverNarcissus commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r492513870



##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -26,6 +26,7 @@
 import java.util.Collections;

Review comment:
       use this line to change the time partition and add some test over different time partition. ```IoTDBDescriptor.getInstance().getConfig().setPartitionInterval(100);```
   
   do not forget to set it back to original value. You can refer to ```IoTDBMultiSeriesIT.java``` for this. Thank you very much XD




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] mychaow commented on a change in pull request #1693: 863: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

mychaow commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r483937333



##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -808,8 +815,11 @@ public void insertTablet(InsertTabletPlan insertTabletPlan) throws BatchInsertio
           // judge if we should insert sequence
           if (!isSequence && time > lastFlushTime) {

Review comment:
       so, the first row of one timepartition always insert into unsequence memtable?

##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -733,10 +733,15 @@ public void insert(InsertRowPlan insertRowPlan) throws WriteProcessException {
       partitionLatestFlushedTimeForEachDevice
           .computeIfAbsent(timePartitionId, id -> new HashMap<>());
 
+      boolean isSequence = insertRowPlan.getTime() > partitionLatestFlushedTimeForEachDevice.get(timePartitionId)

Review comment:
       you should move the code to the position before previous two line, because the two line  has no meaning if the data will be discarded.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r484208829



##########
File path: server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
##########
@@ -144,6 +144,8 @@
 
   private volatile boolean readOnly = false;
 
+  private boolean enable_discard_out_of_order_data = false;

Review comment:
       ```suggestion
     private boolean enableDiscardOutOfOrderData = false;
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r486802944



##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);

Review comment:
       remember to reset this to its default value in the end of the test
   
   We usually do this as follows:
   
   boolean defaultValue = config.isEnableDiscard...;
   config.setEnableDiscard(true);
   do some test
   config.setEnableDiscard(defaultValue);




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r495521180



##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -267,6 +269,279 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet1()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(100);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementMNode[] measurementMNodes = new MeasurementMNode[2];
+    measurementMNodes[0] = new MeasurementMNode(null, "s0",
+        new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN), null);
+    measurementMNodes[1] = new MeasurementMNode(null, "s1",
+        new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN), null);
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+    config.setPartitionInterval(defaultTimePartition);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet2()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(500);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementMNode[] measurementMNodes = new MeasurementMNode[2];
+    measurementMNodes[0] = new MeasurementMNode(null, "s0",
+        new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN), null);
+    measurementMNodes[1] = new MeasurementMNode(null, "s1",
+        new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN), null);
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+    config.setPartitionInterval(defaultTimePartition);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet3()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(1000);

Review comment:
       boolean defaultEnablePartition = config.getEnablePartition...




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

qiaojialin commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r486802944



##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);

Review comment:
       remember to reset this to its default value in the end of the test
   
   We usually do this as follows:
   
   boolean defaultValue = config.isEnableDiscard...;
   config.setEnableDiscard(true);
   do some test
   config.setEnableDiscard(defaultValue);

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);

Review comment:
       remember to reset this to its default value in the end of the test
   
   We usually do this as follows:
   
   boolean defaultValue = config.isEnableDiscard...;
   config.setEnableDiscard(true);
   do some test
   config.setEnableDiscard(defaultValue);

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);

Review comment:
       remember to reset this to its default value in the end of the test
   
   We usually do this as follows:
   
   boolean defaultValue = config.isEnableDiscard...;
   config.setEnableDiscard(true);
   do some test
   config.setEnableDiscard(defaultValue);

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);

Review comment:
       remember to reset this to its default value in the end of the test
   
   We usually do this as follows:
   
   boolean defaultValue = config.isEnableDiscard...;
   config.setEnableDiscard(true);
   do some test
   config.setEnableDiscard(defaultValue);

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -273,6 +274,112 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    config.setEnableDiscardOutOfOrderData(true);

Review comment:
       remember to reset this to its default value in the end of the test
   
   We usually do this as follows:
   
   boolean defaultValue = config.isEnableDiscard...;
   config.setEnableDiscard(true);
   do some test
   config.setEnableDiscard(defaultValue);




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] SilverNarcissus commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

SilverNarcissus commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r494795361



##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -267,6 +269,279 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet1()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(100);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementMNode[] measurementMNodes = new MeasurementMNode[2];
+    measurementMNodes[0] = new MeasurementMNode(null, "s0",
+        new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN), null);
+    measurementMNodes[1] = new MeasurementMNode(null, "s1",
+        new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN), null);
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+    config.setPartitionInterval(defaultTimePartition);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet2()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(500);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementMNode[] measurementMNodes = new MeasurementMNode[2];
+    measurementMNodes[0] = new MeasurementMNode(null, "s0",
+        new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN), null);
+    measurementMNodes[1] = new MeasurementMNode(null, "s1",
+        new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN), null);
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+    config.setPartitionInterval(defaultTimePartition);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet3()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(1000);

Review comment:
       Sure, you set time partition. But you insert time range only from 0 -150 which never cross the partition interval. By the way, you should also enable the partition like this:
   ```
   config.setEnablePartition(true);
   ```
   and do not forget:
   ```
   config.setEnablePartition(false);
   ```
   
   Thank you very much~

##########
File path: server/src/test/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessorTest.java
##########
@@ -267,6 +269,279 @@ public void testSeqAndUnSeqSyncClose()
     }
   }
 
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertRowPlan()
+      throws WriteProcessException, QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+
+    for (int j = 21; j <= 30; j++) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (int j = 10; j >= 1; j--) {
+      TSRecord record = new TSRecord(j, deviceId);
+      record.addTuple(DataPoint.getDataPoint(TSDataType.INT32, measurementId, String.valueOf(j)));
+      insertToStorageGroupProcessor(record);
+      processor.asyncCloseAllWorkingTsFileProcessors();
+    }
+
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+    Assert.assertEquals(10, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+    for (TsFileResource resource : queryDataSource.getUnseqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet1()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(100);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementMNode[] measurementMNodes = new MeasurementMNode[2];
+    measurementMNodes[0] = new MeasurementMNode(null, "s0",
+        new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN), null);
+    measurementMNodes[1] = new MeasurementMNode(null, "s1",
+        new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN), null);
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+    config.setPartitionInterval(defaultTimePartition);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet2()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(500);
+
+    String[] measurements = new String[2];
+    measurements[0] = "s0";
+    measurements[1] = "s1";
+    List<Integer> dataTypes = new ArrayList<>();
+    dataTypes.add(TSDataType.INT32.ordinal());
+    dataTypes.add(TSDataType.INT64.ordinal());
+
+    MeasurementMNode[] measurementMNodes = new MeasurementMNode[2];
+    measurementMNodes[0] = new MeasurementMNode(null, "s0",
+        new MeasurementSchema("s0", TSDataType.INT32, TSEncoding.PLAIN), null);
+    measurementMNodes[1] = new MeasurementMNode(null, "s1",
+        new MeasurementSchema("s1", TSDataType.INT64, TSEncoding.PLAIN), null);
+
+    InsertTabletPlan insertTabletPlan1 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    long[] times = new long[100];
+    Object[] columns = new Object[2];
+    columns[0] = new int[100];
+    columns[1] = new long[100];
+
+    for (int r = 0; r < 100; r++) {
+      times[r] = r;
+      ((int[]) columns[0])[r] = 1;
+      ((long[]) columns[1])[r] = 1;
+    }
+    insertTabletPlan1.setTimes(times);
+    insertTabletPlan1.setColumns(columns);
+    insertTabletPlan1.setRowCount(times.length);
+    insertTabletPlan1.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan1);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+
+    InsertTabletPlan insertTabletPlan2 = new InsertTabletPlan(new PartialPath("root.vehicle.d0"), measurements,
+        dataTypes);
+
+    for (int r = 50; r < 149; r++) {
+      times[r - 50] = r;
+      ((int[]) columns[0])[r - 50] = 1;
+      ((long[]) columns[1])[r - 50] = 1;
+    }
+    insertTabletPlan2.setTimes(times);
+    insertTabletPlan2.setColumns(columns);
+    insertTabletPlan2.setRowCount(times.length);
+    insertTabletPlan2.setMeasurementMNodes(measurementMNodes);
+
+    processor.insertTablet(insertTabletPlan2);
+    processor.asyncCloseAllWorkingTsFileProcessors();
+    processor.syncCloseAllWorkingTsFileProcessors();
+
+    for (TsFileProcessor tsfileProcessor : processor.getWorkUnsequenceTsFileProcessor()) {
+      tsfileProcessor.syncFlush();
+    }
+
+    QueryDataSource queryDataSource = processor.query(new PartialPath(deviceId), measurementId, context,
+        null, null);
+
+    Assert.assertEquals(2, queryDataSource.getSeqResources().size());
+    Assert.assertEquals(0, queryDataSource.getUnseqResources().size());
+    for (TsFileResource resource : queryDataSource.getSeqResources()) {
+      Assert.assertTrue(resource.isClosed());
+    }
+
+    config.setEnableDiscardOutOfOrderData(defaultValue);
+    config.setPartitionInterval(defaultTimePartition);
+  }
+
+  @Test
+  public void testEnableDiscardOutOfOrderDataForInsertTablet3()
+      throws QueryProcessException, IllegalPathException, IOException {
+    IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+    boolean defaultValue = config.isEnableDiscardOutOfOrderData();
+    config.setEnableDiscardOutOfOrderData(true);
+    long defaultTimePartition = IoTDBDescriptor.getInstance().getConfig().getPartitionInterval();
+    config.setPartitionInterval(1000);

Review comment:
       Sure, you set the time partition. But your insert time range only from 0 -150 which never cross the partition interval. By the way, you should also enable the partition like this:
   ```
   config.setEnablePartition(true);
   ```
   and do not forget:
   ```
   config.setEnablePartition(false);
   ```
   
   Thank you very much~




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

samperson1997 commented on a change in pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693#discussion_r484385614



##########
File path: server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
##########
@@ -144,6 +144,8 @@
 
   private volatile boolean readOnly = false;
 
+  private boolean enable_discard_out_of_order_data = false;

Review comment:
       My pleasure ; )




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-iotdb] SilverNarcissus merged pull request #1693: [IOTDB-863]: add a switch to drop ouf-of-order data that is recognized by IoTDB

[GitBox <gi...@apache.org>]

SilverNarcissus merged pull request #1693:
URL: https://github.com/apache/incubator-iotdb/pull/1693


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org