You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/28 04:55:08 UTC

[GitHub] [hudi] scxwhite opened a new pull request, #7309: [HUDI-5284] Add new config controls whether input rdds should be first persist before insert.

scxwhite opened a new pull request, #7309:
URL: https://github.com/apache/hudi/pull/7309

   ### Change Logs
   
   When our input data comes from a complex rdd lineage, hudi writing will lead to repeated calculations.
   For example, we will de duplicate according to the key of the input data, and we will obtain all partitions to be written to the data in the tag location. So I think we should cache the data to be written for downstream use.
   
   ### Impact
   
   add new config (hoodie.persist.before.insert) controls whether they should be first persist before insert.
   ### Risk level (write none, low medium or high below)
   
   low
   ### Documentation Update
   
   add new config (hoodie.persist.before.insert) controls whether they should be first persist before insert.
   
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] bvaradar commented on pull request #7309: [HUDI-5284] Add new config controls whether input rdds should be first persist before insert

Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #7309:
URL: https://github.com/apache/hudi/pull/7309#issuecomment-1489752602

   Closing this PR as @codope has confirmed 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] scxwhite commented on a diff in pull request #7309: [HUDI-5284] Add new config controls whether input rdds should be first persist before insert

Posted by GitBox <gi...@apache.org>.
scxwhite commented on code in PR #7309:
URL: https://github.com/apache/hudi/pull/7309#discussion_r1035551565


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -265,6 +265,13 @@ public class HoodieWriteConfig extends HoodieConfig {
       .withDocumentation("When inserted records share same key, controls whether they should be first combined (i.e de-duplicated) before"
           + " writing to storage.");
 
+  public static final ConfigProperty<String> PERSIST_BEFORE_INSERT = ConfigProperty

Review Comment:
   Thank you for your reply.
   You mentioned two points here. 
   - The first is: "we should refrain from adding another config here"
   The reason for adding this configuration is to consider the use of non sorted bulk_insert and other write operations without additional intermediate operations, persistence may not be necessary.
   - The secnod is: "We should optimize the DAG instead"
   The underlying principles of hudi are not clear to data developers. They may not realize that even if they only need to upsert once, they need to cache data to speed up writing.And this will also increase the development cost of data developers.
   Just my own opinion, I look forward to your reply。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] bvaradar commented on pull request #7309: [HUDI-5284] Add new config controls whether input rdds should be first persist before insert

Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #7309:
URL: https://github.com/apache/hudi/pull/7309#issuecomment-1457479669

   @codope @scxwhite  : Since https://github.com/apache/hudi/pull/7914 is landed, Should we close this PR ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7309: [HUDI-5284] Add new config controls whether input rdds should be first persist before insert.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7309:
URL: https://github.com/apache/hudi/pull/7309#issuecomment-1328699516

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0d90c0dab2f0737d9912e5a8b2ec2011b30d381b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13268",
       "triggerID" : "0d90c0dab2f0737d9912e5a8b2ec2011b30d381b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0d90c0dab2f0737d9912e5a8b2ec2011b30d381b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13268) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] codope commented on a diff in pull request #7309: [HUDI-5284] Add new config controls whether input rdds should be first persist before insert.

Posted by GitBox <gi...@apache.org>.
codope commented on code in PR #7309:
URL: https://github.com/apache/hudi/pull/7309#discussion_r1034750722


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -265,6 +265,13 @@ public class HoodieWriteConfig extends HoodieConfig {
       .withDocumentation("When inserted records share same key, controls whether they should be first combined (i.e de-duplicated) before"
           + " writing to storage.");
 
+  public static final ConfigProperty<String> PERSIST_BEFORE_INSERT = ConfigProperty

Review Comment:
   @scxwhite Thanks for raising the PR. This is a very valid issue. However, I think we should refrain from adding another config here. We should optimize the DAG instead.
   The work here seems related to #7039 
   cc @nsivabalan 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7309: [HUDI-5284] Add new config controls whether input rdds should be first persist before insert.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7309:
URL: https://github.com/apache/hudi/pull/7309#issuecomment-1328572185

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0d90c0dab2f0737d9912e5a8b2ec2011b30d381b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13268",
       "triggerID" : "0d90c0dab2f0737d9912e5a8b2ec2011b30d381b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0d90c0dab2f0737d9912e5a8b2ec2011b30d381b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13268) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7309: [HUDI-5284] Add new config controls whether input rdds should be first persist before insert.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7309:
URL: https://github.com/apache/hudi/pull/7309#issuecomment-1328888803

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0d90c0dab2f0737d9912e5a8b2ec2011b30d381b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13268",
       "triggerID" : "0d90c0dab2f0737d9912e5a8b2ec2011b30d381b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d90c0dab2f0737d9912e5a8b2ec2011b30d381b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13280",
       "triggerID" : "1328866614",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 0d90c0dab2f0737d9912e5a8b2ec2011b30d381b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13268) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13280) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] scxwhite commented on a diff in pull request #7309: [HUDI-5284] Add new config controls whether input rdds should be first persist before insert

Posted by GitBox <gi...@apache.org>.
scxwhite commented on code in PR #7309:
URL: https://github.com/apache/hudi/pull/7309#discussion_r1035551565


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -265,6 +265,13 @@ public class HoodieWriteConfig extends HoodieConfig {
       .withDocumentation("When inserted records share same key, controls whether they should be first combined (i.e de-duplicated) before"
           + " writing to storage.");
 
+  public static final ConfigProperty<String> PERSIST_BEFORE_INSERT = ConfigProperty

Review Comment:
   @codope Thank you for your reply.
   You mentioned two points here. 
   - The first is: "we should refrain from adding another config here"
   The reason for adding this configuration is to consider the use of non sorted bulk_insert and other write operations without additional intermediate operations, persistence may not be necessary.
   - The secnod is: "We should optimize the DAG instead"
   The underlying principles of hudi are not clear to data developers. They may not realize that even if they only need to upsert once, they need to cache data to speed up writing.And this will also increase the development cost of data developers.
   Just my own opinion, I look forward to your reply。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] scxwhite commented on pull request #7309: [HUDI-5284] Add new config controls whether input rdds should be first persist before insert.

Posted by GitBox <gi...@apache.org>.
scxwhite commented on PR #7309:
URL: https://github.com/apache/hudi/pull/7309#issuecomment-1328866614

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7309: [HUDI-5284] Add new config controls whether input rdds should be first persist before insert.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7309:
URL: https://github.com/apache/hudi/pull/7309#issuecomment-1329173767

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0d90c0dab2f0737d9912e5a8b2ec2011b30d381b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13268",
       "triggerID" : "0d90c0dab2f0737d9912e5a8b2ec2011b30d381b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d90c0dab2f0737d9912e5a8b2ec2011b30d381b",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13280",
       "triggerID" : "1328866614",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 0d90c0dab2f0737d9912e5a8b2ec2011b30d381b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13268) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13280) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] bvaradar closed pull request #7309: [HUDI-5284] Add new config controls whether input rdds should be first persist before insert

Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar closed pull request #7309: [HUDI-5284] Add new config controls whether input rdds should be first persist before insert
URL: https://github.com/apache/hudi/pull/7309


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #7309: [HUDI-5284] Add new config controls whether input rdds should be first persist before insert.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7309:
URL: https://github.com/apache/hudi/pull/7309#issuecomment-1328569554

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0d90c0dab2f0737d9912e5a8b2ec2011b30d381b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0d90c0dab2f0737d9912e5a8b2ec2011b30d381b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0d90c0dab2f0737d9912e5a8b2ec2011b30d381b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org