You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2025/03/07 09:21:24 UTC

[PR] [HUDI-9088] Fix unnecessary scanning of target table in MERGE INTO on Spark [hudi]

yihua opened a new pull request, #12932:
URL: https://github.com/apache/hudi/pull/12932

   ### Change Logs
   
   This PR fixes the unnecessary scanning of the target table in MERGE INTO statement in Spark by only using the source table as the input if the target table has a record key (before this change the the source and target table is left joined, causing all partitions in the target table to be scanned unnecessarily, if the source table only contains data targeting a handful of partitions).  For primary-keyless table the logic is not changed, i.e., the source and target table is left joined to get the meta column values for prepped upsert in MERGE INTO.
   
   ### Impact
   
   Improves performance of MERGE INTO on Spark
   
   ### Risk level
   
   low.  Performance improvement only.  Existing tests have guarded the correctness of MERGE INTO statement.
   
   ### Documentation Update
   
   N/A
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9088][DNM] Fix unnecessary scanning of target table in MERGE INTO on Spark [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #12932:
URL: https://github.com/apache/hudi/pull/12932#issuecomment-2756072513

   Closing this in favor of #12933 for 0.x releases.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9088][DNM] Fix unnecessary scanning of target table in MERGE INTO on Spark [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12932:
URL: https://github.com/apache/hudi/pull/12932#issuecomment-2705942086

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a048ad52f7e90d1a54fd02e896b19c4844bc6220",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a048ad52f7e90d1a54fd02e896b19c4844bc6220",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a048ad52f7e90d1a54fd02e896b19c4844bc6220 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9088][DNM] Fix unnecessary scanning of target table in MERGE INTO on Spark [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12932:
URL: https://github.com/apache/hudi/pull/12932#issuecomment-2708463560

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a048ad52f7e90d1a54fd02e896b19c4844bc6220",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4094",
       "triggerID" : "a048ad52f7e90d1a54fd02e896b19c4844bc6220",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a048ad52f7e90d1a54fd02e896b19c4844bc6220 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4094) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9088][DNM] Fix unnecessary scanning of target table in MERGE INTO on Spark [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua closed pull request #12932: [HUDI-9088][DNM] Fix unnecessary scanning of target table in MERGE INTO on Spark
URL: https://github.com/apache/hudi/pull/12932


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org