You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2020/09/28 15:01:35 UTC

[GitHub] [carbondata] akashrn5 opened a new pull request #3961: [wip]Fix cdc merge when there are multiple bucket and join columns and AND or OR join expressions

akashrn5 opened a new pull request #3961:
URL: https://github.com/apache/carbondata/pull/3961


   Fix cdc merge when there are multiple bucket and join columns and AND or OR join expressions
   
    ### Why is this PR needed?
    
    
    ### What changes were proposed in this PR?
   
       
    ### Does this PR introduce any user interface change?
    - No
    - Yes. (please explain the change and update document)
   
    ### Is any new testcase added?
    - No
    - Yes
   
       
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3961: [CARBONDATA-4019]Fix CDC merge failure join expression made of AND/OR expressions.

Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3961:
URL: https://github.com/apache/carbondata/pull/3961#issuecomment-703699501


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2553/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [carbondata] kunal642 commented on a change in pull request #3961: [CARBONDATA-4019]Fix CDC merge failure join expression made of AND/OR expressions.

Posted by GitBox <gi...@apache.org>.
kunal642 commented on a change in pull request #3961:
URL: https://github.com/apache/carbondata/pull/3961#discussion_r500753133



##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/merge/CarbonMergeDataSetCommand.scala
##########
@@ -106,18 +106,22 @@ case class CarbonMergeDataSetCommand(
     // decide join type based on match conditions
     val joinType = decideJoinType
 
-    val joinColumn = mergeMatches.joinExpr.expr.asInstanceOf[EqualTo].left
-      .asInstanceOf[UnresolvedAttribute].nameParts.tail.head
-    // repartition the srsDs, if the target has bucketing and the bucketing column and join column
-    // are same
+    val joinColumns = mergeMatches.joinExpr.expr.collect {
+      case unresolvedAttribute: UnresolvedAttribute if unresolvedAttribute.nameParts.nonEmpty =>
+        unresolvedAttribute.nameParts.tail.head

Review comment:
       please write a comment explaining this logic




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3961: [CARBONDATA-4019]Fix CDC merge failure join expression made of AND/OR expressions.

Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3961:
URL: https://github.com/apache/carbondata/pull/3961#issuecomment-703704746


   Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4301/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [carbondata] kunal642 commented on pull request #3961: [CARBONDATA-4019]Fix CDC merge failure join expression made of AND/OR expressions.

Posted by GitBox <gi...@apache.org>.
kunal642 commented on pull request #3961:
URL: https://github.com/apache/carbondata/pull/3961#issuecomment-705371096


   LGTM


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [carbondata] asfgit closed pull request #3961: [CARBONDATA-4019]Fix CDC merge failure join expression made of AND/OR expressions.

Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3961:
URL: https://github.com/apache/carbondata/pull/3961


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3961: [CARBONDATA-4019]Fix CDC merge failure join expression made of AND/OR expressions.

Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3961:
URL: https://github.com/apache/carbondata/pull/3961#issuecomment-704762415


   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4318/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [carbondata] brijoobopanna commented on pull request #3961: [CARBONDATA-4019]Fix CDC merge failure join expression made of AND/OR expressions.

Posted by GitBox <gi...@apache.org>.
brijoobopanna commented on pull request #3961:
URL: https://github.com/apache/carbondata/pull/3961#issuecomment-703629980


   retest this please
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [carbondata] kunal642 commented on pull request #3961: [CARBONDATA-4019]Fix CDC merge failure join expression made of AND/OR expressions.

Posted by GitBox <gi...@apache.org>.
kunal642 commented on pull request #3961:
URL: https://github.com/apache/carbondata/pull/3961#issuecomment-705371096


   LGTM


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [carbondata] akashrn5 commented on a change in pull request #3961: [CARBONDATA-4019]Fix CDC merge failure join expression made of AND/OR expressions.

Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3961:
URL: https://github.com/apache/carbondata/pull/3961#discussion_r500756989



##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/merge/CarbonMergeDataSetCommand.scala
##########
@@ -106,18 +106,22 @@ case class CarbonMergeDataSetCommand(
     // decide join type based on match conditions
     val joinType = decideJoinType
 
-    val joinColumn = mergeMatches.joinExpr.expr.asInstanceOf[EqualTo].left
-      .asInstanceOf[UnresolvedAttribute].nameParts.tail.head
-    // repartition the srsDs, if the target has bucketing and the bucketing column and join column
-    // are same
+    val joinColumns = mergeMatches.joinExpr.expr.collect {
+      case unresolvedAttribute: UnresolvedAttribute if unresolvedAttribute.nameParts.nonEmpty =>
+        unresolvedAttribute.nameParts.tail.head

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3961: [wip]Fix cdc merge when there are multiple bucket and join columns and AND or OR join expressions

Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3961:
URL: https://github.com/apache/carbondata/pull/3961#issuecomment-700164154






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [carbondata] asfgit closed pull request #3961: [CARBONDATA-4019]Fix CDC merge failure join expression made of AND/OR expressions.

Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3961:
URL: https://github.com/apache/carbondata/pull/3961


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3961: [CARBONDATA-4019]Fix CDC merge failure join expression made of AND/OR expressions.

Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3961:
URL: https://github.com/apache/carbondata/pull/3961#issuecomment-704764695


   Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2568/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org