You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/17 06:33:40 UTC

[GitHub] [hudi] liuhe0702 opened a new pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

liuhe0702 opened a new pull request #4012:
URL: https://github.com/apache/hudi/pull/4012


   ## What is the purpose of the pull request
   
   Improve performance.
   
   ## Brief change log
   
     - *If there are multiple partitions, isEmpty will causes tasks to be executed in multiple jobs and performance deterioration. I replace isEmpty() with count()=0.*
   
   ## Verify this pull request
   
   *(Please pick either of the following options)*
   
   This pull request is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This pull request is already covered by existing tests, such as *(please describe tests)*.
   
   (or)
   
   This change added tests and can be verified as follows:
   
   *(example:)*
   
     - *Added integration tests for end-to-end.*
     - *Added HoodieClientWriteTest to verify the change.*
     - *Manually verified the change by running a job locally.*
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] liuhe0702 edited a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
liuhe0702 edited a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-979739567


   @xushiyan 
   Data is written to the data file only after the three tasks in the preceding figure are executed.
   In the isEmpty method, tasks in different states are executed in serial mode. In the count method, most tasks are concurrently executed in the same state.
   In the isEmpty method, more tasks are executed, more stages are executed in sequence.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082780444


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7551",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 22c0d5a0e58f7410f65111cb766e19996dc0cff0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7551) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552) 
   * 0000 Unknown: [CANCELED](TBD) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] liuhe0702 edited a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
liuhe0702 edited a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1084422646


   
   
   > 
   
   new pr:https://github.com/apache/hudi/pull/5187


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] liuhe0702 closed pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
liuhe0702 closed pull request #4012:
URL: https://github.com/apache/hudi/pull/4012


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] yanghua commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-971362627


   > @leesf @yanghua Please approve the running workflow.
   
   Done, thanks for your contribution.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-973663414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 638ca6f8424de9523e683e2af55b5c2e42fa024a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-974744735


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 638ca6f8424de9523e683e2af55b5c2e42fa024a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] leesf commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
leesf commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-974744663






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] liuhe0702 commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-976375391


   > @liuhe0702 would you please paste the spark ui jobs/stages to see the difference?
   
   before code modification:
   ![isEmpty](https://user-images.githubusercontent.com/76084319/143007171-c59ce008-3e71-42da-a1db-b2ef927007c6.png)
   
   after code modification
   ![count](https://user-images.githubusercontent.com/76084319/143007380-7778f74a-8ecd-44c9-b9da-2a97b37e5d9c.png)
   
   > 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-971254131


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 638ca6f8424de9523e683e2af55b5c2e42fa024a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] liuhe0702 edited a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
liuhe0702 edited a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-980482343


   > Sorry, still not following . When you say different jobs are started - you are referring to the duplicate jobs you see ? tbh given the input `writeResult.getWriteStatuses.rdd` is cached already, not sure if its a needle mover one way or other.
   
   @vinothchandar 
   
   All transformations in Spark are lazy, in that they do not compute their results right away. Instead, they just remember the transformations applied to some base dataset (e.g. a file). The transformations are only computed when an action requires a result to be returned to the driver program.
   In the scenario we're talking about,the data is written to data files only after isEmpty or count is executed. Data in all partitions needs to be written to data files during data saving. Therefore, the total number of tasks to be executed for the isEmpty mothed are the same with the count mothed.
   Assume that 1000 tasks need to be executed and 1000 computing resources are available. The isEmpty method will divide 1000 tasks into six batches and each batch has 1, 5, 25, 125, 625 and 219 tasks (
   As shown in the preceding figure, three tasks are divided into 2 batches. Each batch contains 1 and 2 tasks). In the isEmpty method, each batch is serially executed and the next batch of tasks is executed only after a batch of tasks is executed。In the count method, 1000 tasks are executed as a batch. When computing resources are sufficient, the time consumed by isEmpty is six times that consumed by count.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082711541


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 22c0d5a0e58f7410f65111cb766e19996dc0cff0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082783807


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7551",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0000",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 0000 Unknown: [CANCELED](TBD) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082711541


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 22c0d5a0e58f7410f65111cb766e19996dc0cff0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-971253002


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 638ca6f8424de9523e683e2af55b5c2e42fa024a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-973663414






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082580093


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7539",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 22c0d5a0e58f7410f65111cb766e19996dc0cff0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7539) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082777028


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 22c0d5a0e58f7410f65111cb766e19996dc0cff0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xushiyan commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
xushiyan commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082785653


   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xushiyan commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
xushiyan commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1080943815


   will catch up here; targeting to land this in 0.11


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xushiyan edited a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
xushiyan edited a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082623653


   @xiarixiaoyao thanks for sharing the analysis. sorry haven't been able to catch up on time. You're right; `collect()` happens right after `isEmpty()` or `count()`. A full compute + a cache hit faster than a full compute + a partial compute. In this code path, we should do count(). It does slow down the other case where error is present, but I think that scenario is not performance-sensitive so we should be ok.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xushiyan edited a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
xushiyan edited a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082623653


   @xiarixiaoyao thanks for sharing the analysis. sorry haven't been able to catch up on time. You're right; `collect()` happens right after `isEmpty()` or `count()`. A full compute + a cache hit faster than a full compute + a partial compute. In this code path, we should do count(). It does slow down the other case where error is present, but that scenario is not performance-sensitive so I think we should be good.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082780444


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7551",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 22c0d5a0e58f7410f65111cb766e19996dc0cff0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7551) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552) 
   * 0000 Unknown: [CANCELED](TBD) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082777028


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 22c0d5a0e58f7410f65111cb766e19996dc0cff0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082676452


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7539",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 22c0d5a0e58f7410f65111cb766e19996dc0cff0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7539) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xushiyan commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
xushiyan commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1084304387


   @liuhe0702 i think the merge commit break the CI parsing. Can you close this one and create a new PR that links back to this? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-973649522


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 638ca6f8424de9523e683e2af55b5c2e42fa024a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-978206676


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 638ca6f8424de9523e683e2af55b5c2e42fa024a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-974744735






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] leesf commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
leesf commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-974777633


   @liuhe0702 would you please paste the spark ui jobs/stages to see the difference?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-973649522


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 638ca6f8424de9523e683e2af55b5c2e42fa024a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xiarixiaoyao commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-973649150


   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-973663414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 638ca6f8424de9523e683e2af55b5c2e42fa024a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] yihua commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-980519014


   @leesf I removed the `release-blockers` label for 0.10.0.  We should still triage the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] liuhe0702 edited a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
liuhe0702 edited a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1084422646


   > 
   @xushiyan new pr:https://github.com/apache/hudi/pull/5187


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082676452


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7539",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 22c0d5a0e58f7410f65111cb766e19996dc0cff0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7539) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-971253002


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 638ca6f8424de9523e683e2af55b5c2e42fa024a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] leesf commented on a change in pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#discussion_r756127631



##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -521,7 +521,7 @@ object HoodieSparkSqlWriter {
                                              jsc: JavaSparkContext,
                                              tableInstantInfo: TableInstantInfo
                                              ): (Boolean, common.util.Option[java.lang.String], common.util.Option[java.lang.String]) = {
-    if(writeResult.getWriteStatuses.rdd.filter(ws => ws.hasErrors).isEmpty()) {
+    if(writeResult.getWriteStatuses.rdd.filter(ws => ws.hasErrors).count() == 0) {

Review comment:
       nit: `if (writeResult.xxx)`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-978208217


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 0000 Unknown: [CANCELED](TBD) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-978208217


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 0000 Unknown: [CANCELED](TBD) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-978206676


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 638ca6f8424de9523e683e2af55b5c2e42fa024a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-974748990


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 638ca6f8424de9523e683e2af55b5c2e42fa024a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] leesf commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
leesf commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-974744663


   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082783807


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7551",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0000",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 0000 Unknown: [CANCELED](TBD) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1065043222


   @xushiyan @nsivabalan can we get to the bottom of this issue? and summarize what we find. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-971254131


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 638ca6f8424de9523e683e2af55b5c2e42fa024a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-978209669


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-979719161


   @leesf @danny0405 would be good to get some closure on this or mark this as a non blocker


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] liuhe0702 commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-979739567


   
   @xushiyan 
   Data is written to the data file only after the three tasks in the preceding figure are executed.
   In the isEmpty method, tasks in different states are executed in serial mode. In the count method, most tasks are concurrently executed in the same state.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xiarixiaoyao commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-980520231


   @leesf @vinothchandar  @xushiyan 
   If there is a large amount of data, the performance gap is very obvious.
   Our environment uses 100g of data, and the performance is significantly improved after using this patch


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on a change in pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#discussion_r756328996



##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
##########
@@ -521,7 +521,7 @@ object HoodieSparkSqlWriter {
                                              jsc: JavaSparkContext,
                                              tableInstantInfo: TableInstantInfo
                                              ): (Boolean, common.util.Option[java.lang.String], common.util.Option[java.lang.String]) = {
-    if(writeResult.getWriteStatuses.rdd.filter(ws => ws.hasErrors).isEmpty()) {
+    if(writeResult.getWriteStatuses.rdd.filter(ws => ws.hasErrors).count() == 0) {

Review comment:
       if the writeStatuses are cached/persisted, would that still trigger recomputations?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-979285232


   the dual invocation is an issue with Spark UI and JavaContext. I would also check to ensure we are not being misled by that. I would also think count() is more expensive.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082716879


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7539",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0000",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7548",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7548",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 0000 Unknown: [CANCELED](TBD) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-980304453


   Sorry, still not following . When you say different jobs are started - you are referring to the duplicate jobs you see ? tbh given the input `writeResult.getWriteStatuses.rdd` is cached already, not sure if its a needle mover one way or other.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082714232


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7539",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7548",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 22c0d5a0e58f7410f65111cb766e19996dc0cff0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7539) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7548) 
   * 0000 Unknown: [CANCELED](TBD) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082790728


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7551",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552",
       "triggerID" : "1082785653",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082787068


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7551",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0000",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1082785653",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 0000 Unknown: [CANCELED](TBD) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082578530


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 22c0d5a0e58f7410f65111cb766e19996dc0cff0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082578530


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 22c0d5a0e58f7410f65111cb766e19996dc0cff0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xushiyan commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
xushiyan commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082623653


   @xiarixiaoyao thanks for sharing the analysis. sorry haven't been able to catch up on time. You're right; `collect()` happens right after `isEmpty()` or `count()`. A full compute + a cache hit faster than a full compute + a partial compute. In this code path, we should do count().


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-978209669


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082580093


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3702",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7539",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 22c0d5a0e58f7410f65111cb766e19996dc0cff0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7539) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] liuhe0702 commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-971319932


   @leesf @yanghua 
   Please approve the running workflow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] liuhe0702 commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1084422646


   
   
   
   > 
   
   https://github.com/apache/hudi/pull/5187


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xushiyan commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
xushiyan commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1084402436


   @liuhe0702 thanks. Would you mind opening a new one?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082787068


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7551",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0000",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7552",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1082785653",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 0000 Unknown: [CANCELED](TBD) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xiarixiaoyao commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082593749


   @vinothchandar @xushiyan 
   in line 121 SparkRDDWriteClient.java,  we call collect for rdd, which will trigger compute again. 
   ```
       List<HoodieWriteStat> writeStats = writeStatuses.map(WriteStatus::getStat).collect();
   ```
   if we use call isEmpty in 595 HoodieSparkSqlWriter, what will happen
   ```
       if (writeResult.getWriteStatuses.rdd.filter(ws => ws.hasErrors).isEmpty()) {
   ```
   1) empty will trigger partial compute
   2) in line 121 SparkRDDWriteClient.java,  compute will trigger again, as cache is not availbale
   
   if we can call count in  595 HoodieSparkSqlWriter, what will happen
   1) count will trigger all compute,  and trigger cache to cache the compute result.
   2) in line 121 SparkRDDWriteClient.java, compute will not trigger again, as we have already cache the compute result.
   
   so i think count is better than isEmpty
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082714232


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7539",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "0000",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7548",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 22c0d5a0e58f7410f65111cb766e19996dc0cff0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7539) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7548) 
   * 0000 Unknown: [CANCELED](TBD) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-1082716879


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7539",
       "triggerID" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0000",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7548",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "22c0d5a0e58f7410f65111cb766e19996dc0cff0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7548",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 0000 Unknown: [CANCELED](TBD) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-974744735


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 638ca6f8424de9523e683e2af55b5c2e42fa024a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] vinothchandar commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-980297058


   https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java#L250 does persist the write statuses


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-974748990


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421",
       "triggerID" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486",
       "triggerID" : "973649150",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "638ca6f8424de9523e683e2af55b5c2e42fa024a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542",
       "triggerID" : "974744663",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 638ca6f8424de9523e683e2af55b5c2e42fa024a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3421) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3486) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3542) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] liuhe0702 commented on pull request #4012: [HUDI-2777] Data import performance deteriorates because multiple Spark jobs are started when data is written to disks.

Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4012:
URL: https://github.com/apache/hudi/pull/4012#issuecomment-980482343


   > Sorry, still not following . When you say different jobs are started - you are referring to the duplicate jobs you see ? tbh given the input `writeResult.getWriteStatuses.rdd` is cached already, not sure if its a needle mover one way or other.
   
   @vinothchandar 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org