[GitHub] [incubator-iotdb] qiaojialin commented on issue #303: transform old form to new form and use a parameter to control

[GitBox <gi...@apache.org>]

qiaojialin commented on issue #303: transform old form to new form and use a parameter to control
URL: https://github.com/apache/incubator-iotdb/pull/303#issuecomment-515348571
 
 
   > As for the performance consideration. Given that tsfile is read by path, if we want get new table form, we need to split one row of data and build the iterator. I'm not sure whether this step can be parallelized. In current implementation, I just use union function, which means all operations can be in one stage without shuffle.
   
   I mean when users want to use the new table form, there will be no wide row which contains all time series at all. Therefore, we do not need to split a wide row into several narrow rows.
   
   The following is an example: suppose the file schema has "s1, s2, s3"
   
   In DefaultSource.inferSchema(), we return the "timestamp, deviceId, s1, s2, s3" to spark to generate the new table form.
   
   In DefaultSource.buildReader(), the required schema is "timestamp, deviceId, s1, s2, s3", suppose the PartitionedFile has one chunk group (device1, s1, s2, s3). Then, we could build a tsfile reader to query three time series : device1.s1, device1.s2, device1.s3 and directly construct the result to an InternalRow and return to Spark.
   
   This step can be parallelized. Each executor in spark will read a partition of one tsfile. These executors could work independently.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services