[GitHub] [incubator-iotdb] LeiRui commented on a change in pull request #455: [IOTDB-251]improve TSQueryDataSet structure in RPC

[GitBox <gi...@apache.org>]

LeiRui commented on a change in pull request #455: [IOTDB-251]improve TSQueryDataSet structure in RPC
URL: https://github.com/apache/incubator-iotdb/pull/455#discussion_r337007825
 
 

 ##########
 File path: server/src/main/java/org/apache/iotdb/db/utils/QueryDataSetUtils.java
 ##########
 @@ -55,66 +55,86 @@ public static TSQueryDataSet convertQueryDataSetByFetchSize(QueryDataSet queryDa
 
   public static TSQueryDataSet convertQueryDataSetByFetchSize(QueryDataSet queryDataSet,
       int fetchSize, WatermarkEncoder watermarkEncoder) throws IOException {
+    List<TSDataType> dataTypes = queryDataSet.getDataTypes();
+    int columnNum = dataTypes.size();
     TSQueryDataSet tsQueryDataSet = new TSQueryDataSet();
-    tsQueryDataSet.setRecords(new ArrayList<>());
+    int columnNumWithTime = columnNum + 1;
+    DataOutputStream[] dataOutputStreams = new DataOutputStream[columnNumWithTime];
+    ByteArrayOutputStream[] byteArrayOutputStreams = new ByteArrayOutputStream[columnNumWithTime];
+    for (int i = 0; i < columnNumWithTime; i++) {
+      byteArrayOutputStreams[i] = new ByteArrayOutputStream();
+      dataOutputStreams[i] = new DataOutputStream(byteArrayOutputStreams[i]);
+    }
+
+    int rowCount = 0;
+    int valueOccupation = 0;
     for (int i = 0; i < fetchSize; i++) {
       if (queryDataSet.hasNext()) {
+        rowCount++;
         RowRecord rowRecord = queryDataSet.next();
         if (watermarkEncoder != null) {
           rowRecord = watermarkEncoder.encodeRecord(rowRecord);
         }
-        tsQueryDataSet.getRecords().add(convertToTSRecord(rowRecord));
+        // use columnOutput to write byte array
+        dataOutputStreams[0].writeLong(rowRecord.getTimestamp());
+        List<Field> fields = rowRecord.getFields();
+        for (int k = 0; k < fields.size(); k++) {
+          Field field = fields.get(k);
+          DataOutputStream dataOutputStream = dataOutputStreams[k + 1]; // DO NOT FORGET +1
+          if (field.getDataType() == null) {
+            dataOutputStream.writeBoolean(true); // is_empty true
+          } else {
+            dataOutputStream.writeBoolean(false); // is_empty false
 
 Review comment:
   Hi, yes, this is scheduled as a possible future improvement as for the inner structure of TSQueryDataSet's value buffer. And more experiments are needed.
   But for now, the core aim of this pr, which is to remove record-wise structure in TSQueryDataSet, is hit.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services