[GitHub] [incubator-iotdb] LeiRui commented on a change in pull request #303: transform old form to new form and use a parameter to control

[GitBox <gi...@apache.org>]

LeiRui commented on a change in pull request #303: transform old form to new form and use a parameter to control
URL: https://github.com/apache/incubator-iotdb/pull/303#discussion_r321088354
 
 

 ##########
 File path: spark-tsfile/src/test/scala/org/apache/iotdb/tsfile/TSFileSuit.scala
 ##########
 @@ -214,4 +223,28 @@ class TSFileSuit extends FunSuite with BeforeAndAfterAll {
     Assert.assertEquals(expected, df.schema)
   }
 
+  test("testTransform1") {
+    val df = spark.read.tsfile(tsfile1)
+    df.createOrReplaceTempView("tsfile_table")
+    val newDf = spark.sql("select * from tsfile_table " +
+      "where (`device_1.sensor_1`>0 or `device_1.sensor_2` < 22) and time < 4")
+    Assert.assertEquals(3, newDf.count())
+  }
+
+  test("testTransform2") {
+    val df = spark.read.tsfile(tsfile1, true)
+    df.createOrReplaceTempView("tsfile_table")
+    val newDf = spark.sql("select * from tsfile_table " +
+      "where `device_name` = 'device_1' and (`sensor_1`>0 or `sensor_2` < 22) and time < 4")
+    Assert.assertEquals(3, newDf.count())
+  }
+
+  test("testTransform3") {
 
 Review comment:
   Check the `partitionedFile` in the `DefaultSource.buildReader` and you will find that the partitionedFile's start and length just correspond to the whole tsfile. That actually means the partitioned case is not covered in these tests. 
   
   So, to valid the full function of your code, it would be nice to add more tests to **cover the cases when the tsfile is really partitioned into more than one partitionedFile**.
   
   I currently have no feasible solutions for that. I tried to add the following configurations but the partitionedFile turned out to still be the whole tsfile:
   ```
       val minSplit: Int = 100000
       val maxSplit: Int = 100000
       spark.sparkContext.hadoopConfiguration.setInt("mapred.min.split.size", minSplit)
       spark.sparkContext.hadoopConfiguration.setInt("mapred.max.split.size", maxSplit)
   ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services