You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/07 04:29:31 UTC
[GitHub] [hudi] xicm opened a new pull request, #5526: [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti…
xicm opened a new pull request, #5526:
URL: https://github.com/apache/hudi/pull/5526
## What is the purpose of the pull request
Fix flaky test ITTestHoodieDataSource
## Brief change log
```
TimeUnit.SECONDS.sleep(timeout);
tableResult.getJobClient().ifPresent(JobClient::cancel);
```
When the job finishes before timeout, the mini cluster has been shutdown. We dont't need to call cancel.
While it's not easy to get whether the job is finished, I catch the Exception here.
## Verify this pull request
This pull request is about test
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm closed pull request #5526: [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti…
Posted by GitBox <gi...@apache.org>.
xicm closed pull request #5526: [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti…
URL: https://github.com/apache/hudi/pull/5526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5526: [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5526:
URL: https://github.com/apache/hudi/pull/5526#issuecomment-1120132461
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b9f94381649251917667868f66e1492557c6e13",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7b9f94381649251917667868f66e1492557c6e13",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7b9f94381649251917667868f66e1492557c6e13 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5526: [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5526:
URL: https://github.com/apache/hudi/pull/5526#issuecomment-1120146526
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b9f94381649251917667868f66e1492557c6e13",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8484",
"triggerID" : "7b9f94381649251917667868f66e1492557c6e13",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7b9f94381649251917667868f66e1492557c6e13 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8484)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5526: [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5526:
URL: https://github.com/apache/hudi/pull/5526#issuecomment-1120211526
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b9f94381649251917667868f66e1492557c6e13",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8484",
"triggerID" : "7b9f94381649251917667868f66e1492557c6e13",
"triggerType" : "PUSH"
}, {
"hash" : "d5851284a56659bb22d30f63941d6b2d29a2b4a6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d5851284a56659bb22d30f63941d6b2d29a2b4a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7b9f94381649251917667868f66e1492557c6e13 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8484)
* d5851284a56659bb22d30f63941d6b2d29a2b4a6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #5526: [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti…
Posted by GitBox <gi...@apache.org>.
danny0405 merged PR #5526:
URL: https://github.com/apache/hudi/pull/5526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #5526: [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #5526:
URL: https://github.com/apache/hudi/pull/5526#issuecomment-1120179717
You can update the PR with following commits and there is no need to close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5526: [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5526:
URL: https://github.com/apache/hudi/pull/5526#issuecomment-1120132807
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b9f94381649251917667868f66e1492557c6e13",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8484",
"triggerID" : "7b9f94381649251917667868f66e1492557c6e13",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7b9f94381649251917667868f66e1492557c6e13 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8484)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #5526: [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti…
Posted by GitBox <gi...@apache.org>.
xicm commented on code in PR #5526:
URL: https://github.com/apache/hudi/pull/5526#discussion_r867313899
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestHoodieDataSource.java:
##########
@@ -1358,7 +1358,11 @@ private List<Row> execSelectSql(TableEnvironment tEnv, String select, String sin
TableResult tableResult = tEnv.executeSql("insert into sink " + select);
// wait for the timeout then cancels the job
TimeUnit.SECONDS.sleep(timeout);
- tableResult.getJobClient().ifPresent(JobClient::cancel);
+ try {
+ tableResult.getJobClient().ifPresent(JobClient::cancel);
+ } catch (IllegalStateException e) {
+ log.info("MiniCluster has already been shut down, do nothing.");
+ }
Review Comment:
I found the testWriteAndReadDebeziumJson is flaky a few days ago,it resolved with my changes. But I haven't reproduced it today. It seems I haven't found the root cause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5526: [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5526:
URL: https://github.com/apache/hudi/pull/5526#discussion_r867305446
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestHoodieDataSource.java:
##########
@@ -1358,7 +1358,11 @@ private List<Row> execSelectSql(TableEnvironment tEnv, String select, String sin
TableResult tableResult = tEnv.executeSql("insert into sink " + select);
// wait for the timeout then cancels the job
TimeUnit.SECONDS.sleep(timeout);
- tableResult.getJobClient().ifPresent(JobClient::cancel);
+ try {
+ tableResult.getJobClient().ifPresent(JobClient::cancel);
+ } catch (IllegalStateException e) {
+ log.info("MiniCluster has already been shut down, do nothing.");
+ }
Review Comment:
Thanks for the fix @xicm ! I have applied your patch but it does not solve the problem.
Actually the test is flaky because in recent change:
https://github.com/apache/hudi/blob/52fe1c9faeb83fe51b520e18d0c37b67ad3fcfe4/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/compact/CompactionPlanOperator.java#L112
We have tweaked the compaction scheduling strategy: when there is a pending compaction, we can not schedule another one, the test is flaky based on whether there is pending compaction when scheduling.
My initial idea is we change the test itself:
[4053.patch.zip](https://github.com/apache/hudi/files/8644508/4053.patch.zip)
Hope it helps, or maybe you have better idea :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5526: [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5526:
URL: https://github.com/apache/hudi/pull/5526#issuecomment-1120212058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b9f94381649251917667868f66e1492557c6e13",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8484",
"triggerID" : "7b9f94381649251917667868f66e1492557c6e13",
"triggerType" : "PUSH"
}, {
"hash" : "d5851284a56659bb22d30f63941d6b2d29a2b4a6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8491",
"triggerID" : "d5851284a56659bb22d30f63941d6b2d29a2b4a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7b9f94381649251917667868f66e1492557c6e13 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8484)
* d5851284a56659bb22d30f63941d6b2d29a2b4a6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #5526: [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti…
Posted by GitBox <gi...@apache.org>.
xicm commented on code in PR #5526:
URL: https://github.com/apache/hudi/pull/5526#discussion_r867313229
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestHoodieDataSource.java:
##########
@@ -1358,7 +1358,11 @@ private List<Row> execSelectSql(TableEnvironment tEnv, String select, String sin
TableResult tableResult = tEnv.executeSql("insert into sink " + select);
// wait for the timeout then cancels the job
TimeUnit.SECONDS.sleep(timeout);
- tableResult.getJobClient().ifPresent(JobClient::cancel);
+ try {
+ tableResult.getJobClient().ifPresent(JobClient::cancel);
+ } catch (IllegalStateException e) {
+ log.info("MiniCluster has already been shut down, do nothing.");
+ }
Review Comment:
Sorry, I didn't go into this test. my changes is not the root cause, let me close this pr and you push your patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5526: [HUDI-4053] Flaky ITTestHoodieDataSource.testStreamWriteBatchReadOpti…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5526:
URL: https://github.com/apache/hudi/pull/5526#issuecomment-1120229380
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b9f94381649251917667868f66e1492557c6e13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8484",
"triggerID" : "7b9f94381649251917667868f66e1492557c6e13",
"triggerType" : "PUSH"
}, {
"hash" : "d5851284a56659bb22d30f63941d6b2d29a2b4a6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8491",
"triggerID" : "d5851284a56659bb22d30f63941d6b2d29a2b4a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5851284a56659bb22d30f63941d6b2d29a2b4a6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org