[GitHub] [carbondata] QiangCai commented on a change in pull request #3694: [CARBONDATA-3763] Fix wrong insert result during insert stage command

[GitBox <gi...@apache.org>]

QiangCai commented on a change in pull request #3694: [CARBONDATA-3763] Fix wrong insert result during insert stage command
URL: https://github.com/apache/carbondata/pull/3694#discussion_r405359588
 
 

 ##########
 File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CommonLoadUtils.scala
 ##########
 @@ -767,7 +773,16 @@ object CommonLoadUtils {
             internalRow.getInt(index) + DateDirectDictionaryGenerator.cutOffDate)
         }
       }
-      internalRow
+      if (isInsertFromStageCommand) {
+        // Insert stage command, logical plan already consist of LogicalRDD of internalRow.
+        // When it is converted to DataFrame, spark is reusing the same internalRow.
+        // So, need to have a copy before the last transformation.
+        // TODO: Even though copying internalRow is faster, we should avoid it
+        //  by finding a better way
+        internalRow.copy()
+      } else {
+        internalRow
+      }
 
 Review comment:
   How about moving code block to line 750?
   Because this change faced NullPointerException in some scenarios.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services