[GitHub] [carbondata] niuge01 commented on a change in pull request #3515: [CARBONDATA-3623]: Fixed global sort compaction failure on timestamp column

[GitBox <gi...@apache.org>]

niuge01 commented on a change in pull request #3515: [CARBONDATA-3623]: Fixed global sort compaction failure on timestamp column
URL: https://github.com/apache/carbondata/pull/3515#discussion_r364135260
 
 

 ##########
 File path: integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
 ##########
 @@ -443,23 +438,18 @@ object DataLoadProcessBuilderOnSpark {
       .asScala
       .map(_.getColName)
       .toArray
+
+    /**
+     * [[org.apache.spark.sql.catalyst.expressions.objects.ValidateExternalType]] validates the
+     * datatype of column data and corresponding datatype in schema provided to create dataframe.
+     * Since carbonScanRDD gives Long data for timestamp column and corresponding column datatype in
+     * schema is Timestamp, this validation fails if we use createDataFrame API which takes rdd as
+     * input. Hence, We need to give the List[Row] compatible with the schema datatypes. So using
+     * the createDataFrame API which takes List[Row] and schema as input.
+     */
     val schema = SparkTypeConverter.createSparkSchema(carbonTable, columns)
-    val rdd: RDD[InternalRow] = new CarbonScanRDD[CarbonRow](
-      sparkSession,
-      columnProjection = new CarbonProjection(columns),
-      null,
-      carbonTable.getAbsoluteTableIdentifier,
-      carbonTable.getTableInfo.serialize,
-      carbonTable.getTableInfo,
-      new CarbonInputMetrics,
-      null,
-      classOf[SparkDataTypeConverterImpl],
-      classOf[CarbonRowReadSupport],
-      splits.asJava)
-      .map { row =>
-        new GenericInternalRow(row.getData.asInstanceOf[Array[Any]])
-      }
-    SparkSQLUtil.execute(rdd, schema, sparkSession)
+    val listOfRows = sparkSession.sqlContext.table(carbonTable.getTableName).collect().toList.asJava
 
 Review comment:
   If use List[Row] to instead of RDD, i think it need use more memory to cache all rows, all right?
   May increase OOM risk.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services