You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/12/10 20:14:46 UTC
[GitHub] [hudi] yihua opened a new pull request, #7426: [MINOR] Rebalance Azure CI jobs to save total time
yihua opened a new pull request, #7426:
URL: https://github.com/apache/hudi/pull/7426
### Change Logs
This PR rebalances Azure CI jobs to save total time and avoid timeout. The functional tests of Spark datasource are moved from Job 3 to Job 4 in Azure CI, based on the previous runs shown below: Job 3 takes 2h+ on average, Job 4 takes 1h to 1h20min, and the Spark datasource functional tests take around 20min.
Master
<img width="302" alt="Screen Shot 2022-12-10 at 11 52 29" src="https://user-images.githubusercontent.com/2497195/206873653-f2556f50-363a-4b09-8a64-3847e9cc4916.png">
<img width="304" alt="Screen Shot 2022-12-10 at 11 53 32" src="https://user-images.githubusercontent.com/2497195/206873655-3444a253-388d-49d1-983b-974402cbddd4.png">
<img width="302" alt="Screen Shot 2022-12-10 at 11 54 07" src="https://user-images.githubusercontent.com/2497195/206873656-9dd86170-57d2-4397-9668-1b5eb4c6d16a.png">
<img width="305" alt="Screen Shot 2022-12-10 at 11 54 48" src="https://user-images.githubusercontent.com/2497195/206873657-4947a80e-f346-4e3a-9748-9bd03651a5ed.png">
Branch
<img width="316" alt="Screen Shot 2022-12-10 at 11 59 39" src="https://user-images.githubusercontent.com/2497195/206873661-5a72a66c-397d-4e23-9660-20c657eafd36.png">
<img width="310" alt="Screen Shot 2022-12-10 at 11 59 59" src="https://user-images.githubusercontent.com/2497195/206873662-c296223a-af66-4769-858e-31d431d1a7ad.png">
### Impact
Reducing Azure CI total finishing time.
### Risk level
none
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7426: [MINOR] Rebalance Azure CI jobs to save total time
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7426:
URL: https://github.com/apache/hudi/pull/7426#issuecomment-1345386296
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd031ae3d87a05ccfb973161049fd837ba573d80",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cd031ae3d87a05ccfb973161049fd837ba573d80",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cd031ae3d87a05ccfb973161049fd837ba573d80 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #7426: [MINOR] Rebalance Azure CI jobs to save total time
Posted by GitBox <gi...@apache.org>.
yihua merged PR #7426:
URL: https://github.com/apache/hudi/pull/7426
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7426: [MINOR] Rebalance Azure CI jobs to save total time
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7426:
URL: https://github.com/apache/hudi/pull/7426#issuecomment-1345414104
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd031ae3d87a05ccfb973161049fd837ba573d80",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cd031ae3d87a05ccfb973161049fd837ba573d80",
"triggerType" : "PUSH"
}, {
"hash" : "88db67f71de76a83134c7e6013dbd81f041edc01",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13617",
"triggerID" : "88db67f71de76a83134c7e6013dbd81f041edc01",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cd031ae3d87a05ccfb973161049fd837ba573d80 UNKNOWN
* 88db67f71de76a83134c7e6013dbd81f041edc01 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13617)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7426: [MINOR] Rebalance Azure CI jobs to save total time
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #7426:
URL: https://github.com/apache/hudi/pull/7426#discussion_r1045136950
##########
azure-pipelines.yml:
##########
@@ -138,7 +154,7 @@ stages:
grep "testcase" */target/surefire-reports/*.xml */*/target/surefire-reports/*.xml | awk -F'"' ' { print $6,$4,$2 } ' | sort -nr | head -n 100
displayName: Top 100 long-running testcases
- job: UT_FT_3
- displayName: UT FT spark-datasource
Review Comment:
should we fix the displayname for UT_FT_4 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #7426: [MINOR] Rebalance Azure CI jobs to save total time
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #7426:
URL: https://github.com/apache/hudi/pull/7426#discussion_r1045151370
##########
azure-pipelines.yml:
##########
@@ -138,7 +154,7 @@ stages:
grep "testcase" */target/surefire-reports/*.xml */*/target/surefire-reports/*.xml | awk -F'"' ' { print $6,$4,$2 } ' | sort -nr | head -n 100
displayName: Top 100 long-running testcases
- job: UT_FT_3
- displayName: UT FT spark-datasource
Review Comment:
We don’t need to because there are still functional tests run by other jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7426: [MINOR] Rebalance Azure CI jobs to save total time
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7426:
URL: https://github.com/apache/hudi/pull/7426#issuecomment-1345412660
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd031ae3d87a05ccfb973161049fd837ba573d80",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cd031ae3d87a05ccfb973161049fd837ba573d80",
"triggerType" : "PUSH"
}, {
"hash" : "88db67f71de76a83134c7e6013dbd81f041edc01",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88db67f71de76a83134c7e6013dbd81f041edc01",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cd031ae3d87a05ccfb973161049fd837ba573d80 UNKNOWN
* 88db67f71de76a83134c7e6013dbd81f041edc01 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7426: [MINOR] Rebalance Azure CI jobs to save total time
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7426:
URL: https://github.com/apache/hudi/pull/7426#issuecomment-1345429995
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "cd031ae3d87a05ccfb973161049fd837ba573d80",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cd031ae3d87a05ccfb973161049fd837ba573d80",
"triggerType" : "PUSH"
}, {
"hash" : "88db67f71de76a83134c7e6013dbd81f041edc01",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13617",
"triggerID" : "88db67f71de76a83134c7e6013dbd81f041edc01",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cd031ae3d87a05ccfb973161049fd837ba573d80 UNKNOWN
* 88db67f71de76a83134c7e6013dbd81f041edc01 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13617)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org