[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3694: [CARBONDATA-3763] Fix wrong insert result during insert stage command

[GitBox <gi...@apache.org>]

ajantha-bhat commented on a change in pull request #3694: [CARBONDATA-3763] Fix wrong insert result during insert stage command
URL: https://github.com/apache/carbondata/pull/3694#discussion_r405363099
 
 

 ##########
 File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CommonLoadUtils.scala
 ##########
 @@ -767,7 +773,16 @@ object CommonLoadUtils {
             internalRow.getInt(index) + DateDirectDictionaryGenerator.cutOffDate)
         }
       }
-      internalRow
+      if (isInsertFromStageCommand) {
+        // Insert stage command, logical plan already consist of LogicalRDD of internalRow.
+        // When it is converted to DataFrame, spark is reusing the same internalRow.
+        // So, need to have a copy before the last transformation.
+        // TODO: Even though copying internalRow is faster, we should avoid it
+        //  by finding a better way
+        internalRow.copy()
+      } else {
+        internalRow
+      }
 
 Review comment:
   @QiangCai: CI all test case run. when we can get NPE ?
   Shall I just add null check in the same line 782 ?  Becuase moving to line 750, how it can avoid NPE ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services