You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/09 02:32:58 UTC
[GitHub] [hudi] yihua opened a new pull request, #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
yihua opened a new pull request, #5269:
URL: https://github.com/apache/hudi/pull/5269
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1261284466
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
}, {
"hash" : "c5709b6c45cda7e0a0253319246b10da11668873",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853",
"triggerID" : "c5709b6c45cda7e0a0253319246b10da11668873",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7966fbbde59b43876542fc39135157835b6c861 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816)
* c5709b6c45cda7e0a0253319246b10da11668873 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1260411915
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7966fbbde59b43876542fc39135157835b6c861 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1125054810
@yihua : can you follow up on this PR when you get a chance. we might need to fix this before 0.11.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaking sink
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #5269:
URL: https://github.com/apache/hudi/pull/5269#discussion_r1000006472
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -47,4 +60,11 @@ public void start(Function<Boolean, Boolean> onShutdownCallback) {
}
super.start(onShutdownCallback);
}
+
+ public void updateWriteConfig(HoodieWriteConfig writeConfig) {
+ synchronized (writeConfigUpdateLock) {
+ this.writeConfig = EmbeddedTimelineServerHelper.updateWriteConfigWithTimelineServer(embeddedTimelineService.get(), writeConfig);
+ isWriteConfigUpdated.set(true);
Review Comment:
let's sync up. I am not sure if I understood your comment fully.
we can't update the writeConfig synchronously. The crux here is that, we need to re-instantiate a new writeClient if writeconfig is updated. but, we can give up on entire locking since all we care is, next time when compaction runs it should re-instate itself w/ updated write config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1260108067
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a360d286f9a9bff3f60cc7231bc0abfe86675a88 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240)
* 460ce44922c50af95bf6c68f111ceb61d07c0289 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaking sink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1278369418
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
}, {
"hash" : "c5709b6c45cda7e0a0253319246b10da11668873",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853",
"triggerID" : "c5709b6c45cda7e0a0253319246b10da11668873",
"triggerType" : "PUSH"
}, {
"hash" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12114",
"triggerID" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"triggerType" : "PUSH"
}, {
"hash" : "5c5dde77caa0524e38ba5def698b490c0a35a6c4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5c5dde77caa0524e38ba5def698b490c0a35a6c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d763b34664c25cdf07d1d2eb818ddbf462a8bd7f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12114)
* 5c5dde77caa0524e38ba5def698b490c0a35a6c4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaking sink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1278460192
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
}, {
"hash" : "c5709b6c45cda7e0a0253319246b10da11668873",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853",
"triggerID" : "c5709b6c45cda7e0a0253319246b10da11668873",
"triggerType" : "PUSH"
}, {
"hash" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12114",
"triggerID" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"triggerType" : "PUSH"
}, {
"hash" : "5c5dde77caa0524e38ba5def698b490c0a35a6c4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12193",
"triggerID" : "5c5dde77caa0524e38ba5def698b490c0a35a6c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c5dde77caa0524e38ba5def698b490c0a35a6c4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12193)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #5269:
URL: https://github.com/apache/hudi/pull/5269#discussion_r993354807
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseCompactor.java:
##########
@@ -39,8 +39,9 @@ public BaseCompactor(BaseHoodieWriteClient<T, I, K, O> compactionClient) {
public abstract void compact(HoodieInstant instant) throws IOException;
- public void updateWriteClient(BaseHoodieWriteClient<T, I, K, O> writeClient) {
- this.compactionClient = writeClient;
+ public void close() {
Review Comment:
ditto
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseClusterer.java:
##########
@@ -40,16 +40,15 @@ public BaseClusterer(BaseHoodieWriteClient<T, I, K, O> clusteringClient) {
/**
* Run clustering for the instant.
+ *
* @param instant
* @throws IOException
*/
public abstract void cluster(HoodieInstant instant) throws IOException;
- /**
- * Update the write client used by async clustering.
- * @param writeClient
- */
- public void updateWriteClient(BaseHoodieWriteClient<T, I, K, O> writeClient) {
- this.clusteringClient = writeClient;
+ public void close() {
+ if (clusteringClient != null) {
+ clusteringClient.close();
+ }
Review Comment:
let's implement AutoCloseable when we have this close() API
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -78,13 +81,26 @@ protected Pair<CompletableFuture, ExecutorService> startService() {
while (!isShutdownRequested()) {
final HoodieInstant instant = fetchNextAsyncServiceInstant();
-
if (null != instant) {
LOG.info("Starting Compaction for instant " + instant);
+ synchronized (writeConfigUpdateLock) {
Review Comment:
from my understanding of previous comments, this still allows async service to receive write config updates? if we think about async service coupling with a table service client and being immutable, we shouldn't need this sort of lock. we just create new async service at deltasync level, and freeze write config for the service internally.
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/DataSourceUtils.java:
##########
@@ -190,7 +192,49 @@ public static HoodieWriteConfig createHoodieConfig(String schemaStr, String base
public static SparkRDDWriteClient createHoodieClient(JavaSparkContext jssc, String schemaStr, String basePath,
String tblName, Map<String, String> parameters) {
- return new SparkRDDWriteClient<>(new HoodieSparkEngineContext(jssc), createHoodieConfig(schemaStr, basePath, tblName, parameters));
+ return createHoodieClient(jssc, schemaStr, basePath, tblName, parameters, Option.empty());
+ }
+
+ public static SparkRDDWriteClient createHoodieClient(JavaSparkContext jssc, String schemaStr, String basePath,
+ String tblName, Map<String, String> parameters, Option<EmbeddedTimelineServiceHandler> embeddedTimelineServiceHandler) {
+ HoodieWriteConfig writeConfig = createHoodieConfig(schemaStr, basePath, tblName, parameters);
+ if (embeddedTimelineServiceHandler.isPresent()) {
+ embeddedTimelineServiceHandler.get().onInstantiation(writeConfig);
+ }
+ return new SparkRDDWriteClient<>(new HoodieSparkEngineContext(jssc), writeConfig, embeddedTimelineServiceHandler.isPresent()
+ ? embeddedTimelineServiceHandler.get().getEmbeddedTimelineService() : Option.empty());
+ }
+
+ /**
+ * For spark structured streaming ingestion, embedded timeline service is instantiated externally and re-used across various write client instantiations (regular writer, table services).
+ * This class helps in coordinating the instantiation of embedded timeline service and exposes the singleton instance.
+ */
+ public static class EmbeddedTimelineServiceHandler {
+
+ private Option<EmbeddedTimelineService> embeddedTimelineService = Option.empty();
+ private final JavaSparkContext jssc;
+
+ public EmbeddedTimelineServiceHandler(JavaSparkContext jssc) {
+ this.jssc = jssc;
+ }
+
+ public void onInstantiation(HoodieWriteConfig writeConfig) {
+ if (writeConfig.isEmbeddedTimelineServerEnabled()) {
+ if (!embeddedTimelineService.isPresent()) {
+ try {
+ embeddedTimelineService = EmbeddedTimelineServerHelper.createEmbeddedTimelineService(new HoodieSparkEngineContext(jssc), writeConfig);
+ } catch (IOException e) {
+ e.printStackTrace();
Review Comment:
avoid e.print
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1093608288
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7937",
"triggerID" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7515d1f98a1299f5a42e242d3e36d84a9a38bee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7937)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1094172575
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6f8d22ccc5efbd87ff993a46ea1977355842602f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1274011725
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
}, {
"hash" : "c5709b6c45cda7e0a0253319246b10da11668873",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853",
"triggerID" : "c5709b6c45cda7e0a0253319246b10da11668873",
"triggerType" : "PUSH"
}, {
"hash" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12114",
"triggerID" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d763b34664c25cdf07d1d2eb818ddbf462a8bd7f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12114)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1260245383
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807)
* b987d84d2c18e5e3cb15268a2689a6294842b2c8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1260167190
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua closed pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaming sink
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua closed pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaming sink
URL: https://github.com/apache/hudi/pull/5269
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1240185610
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6f8d22ccc5efbd87ff993a46ea1977355842602f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944)
* a360d286f9a9bff3f60cc7231bc0abfe86675a88 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1094172828
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6f8d22ccc5efbd87ff993a46ea1977355842602f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1274018688
@yihua : this patch is ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1261275485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
}, {
"hash" : "c5709b6c45cda7e0a0253319246b10da11668873",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c5709b6c45cda7e0a0253319246b10da11668873",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7966fbbde59b43876542fc39135157835b6c861 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816)
* c5709b6c45cda7e0a0253319246b10da11668873 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #5269:
URL: https://github.com/apache/hudi/pull/5269#discussion_r980511991
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -81,8 +82,13 @@ protected Pair<CompletableFuture, ExecutorService> startService() {
if (null != instant) {
LOG.info("Starting Compaction for instant " + instant);
+ synchronized (writeConfigUpdateLock) {
+ compactor = createCompactor();
Review Comment:
+1
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -732,31 +732,31 @@ public void setupWriteClient() throws IOException {
if ((null != schemaProvider)) {
Schema sourceSchema = schemaProvider.getSourceSchema();
Schema targetSchema = schemaProvider.getTargetSchema();
- reInitWriteClient(sourceSchema, targetSchema);
+ createNewWriteClientWithSchema(sourceSchema, targetSchema);
}
}
- private void reInitWriteClient(Schema sourceSchema, Schema targetSchema) throws IOException {
+ private void createNewWriteClientWithSchema(Schema sourceSchema, Schema targetSchema) throws IOException {
LOG.info("Setting up new Hoodie Write Client");
if (isDropPartitionColumns()) {
targetSchema = HoodieAvroUtils.removeFields(targetSchema, getPartitionColumns(keyGenerator, props));
}
registerAvroSchemas(sourceSchema, targetSchema);
- HoodieWriteConfig hoodieCfg = getHoodieClientConfig(targetSchema);
- if (hoodieCfg.isEmbeddedTimelineServerEnabled()) {
+ HoodieWriteConfig writeConfig = getHoodieClientConfig(targetSchema);
+ if (writeConfig.isEmbeddedTimelineServerEnabled()) {
if (!embeddedTimelineService.isPresent()) {
- embeddedTimelineService = EmbeddedTimelineServerHelper.createEmbeddedTimelineService(new HoodieSparkEngineContext(jssc), hoodieCfg);
+ embeddedTimelineService = EmbeddedTimelineServerHelper.createEmbeddedTimelineService(new HoodieSparkEngineContext(jssc), writeConfig);
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1093623854
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7937",
"triggerID" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"triggerType" : "PUSH"
}, {
"hash" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7515d1f98a1299f5a42e242d3e36d84a9a38bee Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7937)
* 784c8cf617b0f4eca028205f87067e08bb55457b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1093629451
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7937",
"triggerID" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"triggerType" : "PUSH"
}, {
"hash" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"triggerType" : "PUSH"
}, {
"hash" : "c411bedee151888a7cb67690be123c0bc0f29797",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c411bedee151888a7cb67690be123c0bc0f29797",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7515d1f98a1299f5a42e242d3e36d84a9a38bee Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7937)
* 784c8cf617b0f4eca028205f87067e08bb55457b UNKNOWN
* c411bedee151888a7cb67690be123c0bc0f29797 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1093665632
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7937",
"triggerID" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"triggerType" : "PUSH"
}, {
"hash" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"triggerType" : "PUSH"
}, {
"hash" : "c411bedee151888a7cb67690be123c0bc0f29797",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7938",
"triggerID" : "c411bedee151888a7cb67690be123c0bc0f29797",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 784c8cf617b0f4eca028205f87067e08bb55457b UNKNOWN
* c411bedee151888a7cb67690be123c0bc0f29797 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7938)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a diff in pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on code in PR #5269:
URL: https://github.com/apache/hudi/pull/5269#discussion_r846787619
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseCompactor.java:
##########
@@ -31,16 +33,30 @@
private static final long serialVersionUID = 1L;
+ protected final transient Object writeClientUpdateLock = new Object();
+ protected final transient List<BaseHoodieWriteClient<T, I, K, O>> oldCompactionClientList = new ArrayList<>();
+
protected transient BaseHoodieWriteClient<T, I, K, O> compactionClient;
+ protected boolean isCompactionRunning = false;
+
public BaseCompactor(BaseHoodieWriteClient<T, I, K, O> compactionClient) {
this.compactionClient = compactionClient;
}
public abstract void compact(HoodieInstant instant) throws IOException;
public void updateWriteClient(BaseHoodieWriteClient<T, I, K, O> writeClient) {
- this.compactionClient = writeClient;
+ synchronized (writeClientUpdateLock) {
+ if (!isCompactionRunning) {
+ this.compactionClient.close();
+ } else {
+ // Store the old compaction client so that they can be closed
Review Comment:
can't we just tie the life cycle of the write client to the life cycle of a single execution of clustering or compaction. i.e from DeltaSync level, when we submit the runnable for a executing a given compaction or clustering, we make a new write client. is that not sufficient? simpler?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseCompactor.java:
##########
@@ -31,16 +33,30 @@
private static final long serialVersionUID = 1L;
+ protected final transient Object writeClientUpdateLock = new Object();
+ protected final transient List<BaseHoodieWriteClient<T, I, K, O>> oldCompactionClientList = new ArrayList<>();
+
protected transient BaseHoodieWriteClient<T, I, K, O> compactionClient;
+ protected boolean isCompactionRunning = false;
+
public BaseCompactor(BaseHoodieWriteClient<T, I, K, O> compactionClient) {
this.compactionClient = compactionClient;
}
public abstract void compact(HoodieInstant instant) throws IOException;
public void updateWriteClient(BaseHoodieWriteClient<T, I, K, O> writeClient) {
- this.compactionClient = writeClient;
+ synchronized (writeClientUpdateLock) {
Review Comment:
reuse code?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/HoodieSparkCompactor.java:
##########
@@ -60,5 +64,12 @@ public void compact(HoodieInstant instant) {
}
// Commit compaction
writeClient.commitCompaction(instant.getTimestamp(), compactionMetadata.getCommitMetadata().get(), Option.empty());
+
+ synchronized (writeClientUpdateLock) {
Review Comment:
Not a fan of all this concurrency handling here. IMO harder to read
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaking sink
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #5269:
URL: https://github.com/apache/hudi/pull/5269#discussion_r1000040906
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -20,24 +20,37 @@
package org.apache.hudi.async;
import org.apache.hudi.client.RunsTableService;
+import org.apache.hudi.client.embedded.EmbeddedTimelineServerHelper;
+import org.apache.hudi.client.embedded.EmbeddedTimelineService;
+import org.apache.hudi.common.util.Option;
import org.apache.hudi.config.HoodieWriteConfig;
+import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.Function;
public abstract class HoodieAsyncTableService extends HoodieAsyncService implements RunsTableService {
+ protected final Object writeConfigUpdateLock = new Object();
protected HoodieWriteConfig writeConfig;
+ protected Option<EmbeddedTimelineService> embeddedTimelineService;
+ protected AtomicBoolean isWriteConfigUpdated = new AtomicBoolean(false);
protected HoodieAsyncTableService() {
}
- protected HoodieAsyncTableService(HoodieWriteConfig writeConfig) {
+ protected HoodieAsyncTableService(HoodieWriteConfig writeConfig, Option<EmbeddedTimelineService> embeddedTimelineService) {
this.writeConfig = writeConfig;
+ this.embeddedTimelineService = embeddedTimelineService;
Review Comment:
@nsivabalan did you update this one? Don't see any changes here
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -47,4 +60,11 @@ public void start(Function<Boolean, Boolean> onShutdownCallback) {
}
super.start(onShutdownCallback);
}
+
+ public void updateWriteConfig(HoodieWriteConfig writeConfig) {
+ synchronized (writeConfigUpdateLock) {
+ this.writeConfig = EmbeddedTimelineServerHelper.updateWriteConfigWithTimelineServer(embeddedTimelineService.get(), writeConfig);
+ isWriteConfigUpdated.set(true);
Review Comment:
@nsivabalan let's do the following instead:
- Let's keep the flag to trigger the refresh
- Let's make the flag more generic (like `shouldResetWriteClient`) so that it's not coupled to writeConfig (there's no need for it to be)
- Let's reuse this logic b/w Clustering/Compaction services (see my comment above)
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -47,4 +60,11 @@ public void start(Function<Boolean, Boolean> onShutdownCallback) {
}
super.start(onShutdownCallback);
}
+
+ public void updateWriteConfig(HoodieWriteConfig writeConfig) {
+ synchronized (writeConfigUpdateLock) {
+ this.writeConfig = EmbeddedTimelineServerHelper.updateWriteConfigWithTimelineServer(embeddedTimelineService.get(), writeConfig);
+ isWriteConfigUpdated.set(true);
Review Comment:
Sync'd up offline: we can reset the write-client (it won't affect already running compaction/clustering) but we won't be able to close existing client synchronously, and will have to do an async follow-up anyway
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -78,13 +81,26 @@ protected Pair<CompletableFuture, ExecutorService> startService() {
while (!isShutdownRequested()) {
Review Comment:
This portion of the code is identical b/w Clustering/Compaction services.
Let's abstract it into a shared base class to avoid duplication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1093621910
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7937",
"triggerID" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"triggerType" : "PUSH"
}, {
"hash" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7515d1f98a1299f5a42e242d3e36d84a9a38bee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7937)
* 784c8cf617b0f4eca028205f87067e08bb55457b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #5269:
URL: https://github.com/apache/hudi/pull/5269#discussion_r846580321
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/HoodieSparkClusteringClient.java:
##########
@@ -53,11 +53,15 @@ public void cluster(HoodieInstant instant) throws IOException {
SparkRDDWriteClient<T> writeClient = (SparkRDDWriteClient<T>) clusteringClient;
Option<HoodieCommitMetadata> commitMetadata = writeClient.cluster(instant.getTimestamp(), true).getCommitMetadata();
Stream<HoodieWriteStat> hoodieWriteStatStream = commitMetadata.get().getPartitionToWriteStats().entrySet().stream().flatMap(e ->
- e.getValue().stream());
+ e.getValue().stream());
long errorsCount = hoodieWriteStatStream.mapToLong(HoodieWriteStat::getTotalWriteErrors).sum();
if (errorsCount > 0) {
// TODO: Should we treat this fatal and throw exception?
LOG.error("Clustering for instant (" + instant + ") failed with write errors");
}
+ if (writeClient != clusteringClient) {
+ // Clustering write client has been updated with new schema, closing the old one
+ writeClient.close();
Review Comment:
Right. I've fixed the logic to consider such cases so that the old write client is properly closed in all cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1093839735
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7937",
"triggerID" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"triggerType" : "PUSH"
}, {
"hash" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"triggerType" : "PUSH"
}, {
"hash" : "c411bedee151888a7cb67690be123c0bc0f29797",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7938",
"triggerID" : "c411bedee151888a7cb67690be123c0bc0f29797",
"triggerType" : "PUSH"
}, {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 784c8cf617b0f4eca028205f87067e08bb55457b UNKNOWN
* 6f8d22ccc5efbd87ff993a46ea1977355842602f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1260241912
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 460ce44922c50af95bf6c68f111ceb61d07c0289 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806)
* f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807)
* b987d84d2c18e5e3cb15268a2689a6294842b2c8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1260290578
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b987d84d2c18e5e3cb15268a2689a6294842b2c8 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814)
* e7966fbbde59b43876542fc39135157835b6c861 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaking sink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1285177056
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
}, {
"hash" : "c5709b6c45cda7e0a0253319246b10da11668873",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853",
"triggerID" : "c5709b6c45cda7e0a0253319246b10da11668873",
"triggerType" : "PUSH"
}, {
"hash" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12114",
"triggerID" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"triggerType" : "PUSH"
}, {
"hash" : "5c5dde77caa0524e38ba5def698b490c0a35a6c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12193",
"triggerID" : "5c5dde77caa0524e38ba5def698b490c0a35a6c4",
"triggerType" : "PUSH"
}, {
"hash" : "96aa0891caaf95715b4877c8f2444bfc3b59386f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12361",
"triggerID" : "96aa0891caaf95715b4877c8f2444bfc3b59386f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 96aa0891caaf95715b4877c8f2444bfc3b59386f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12361)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaming sink
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1611205586
There's still some concern around this PR. Let's keep it out of 0.14.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaking sink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1284831101
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
}, {
"hash" : "c5709b6c45cda7e0a0253319246b10da11668873",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853",
"triggerID" : "c5709b6c45cda7e0a0253319246b10da11668873",
"triggerType" : "PUSH"
}, {
"hash" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12114",
"triggerID" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"triggerType" : "PUSH"
}, {
"hash" : "5c5dde77caa0524e38ba5def698b490c0a35a6c4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12193",
"triggerID" : "5c5dde77caa0524e38ba5def698b490c0a35a6c4",
"triggerType" : "PUSH"
}, {
"hash" : "96aa0891caaf95715b4877c8f2444bfc3b59386f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12361",
"triggerID" : "96aa0891caaf95715b4877c8f2444bfc3b59386f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c5dde77caa0524e38ba5def698b490c0a35a6c4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12193)
* 96aa0891caaf95715b4877c8f2444bfc3b59386f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12361)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1240474858
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a360d286f9a9bff3f60cc7231bc0abfe86675a88 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1260171893
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 460ce44922c50af95bf6c68f111ceb61d07c0289 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806)
* f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1260132585
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 460ce44922c50af95bf6c68f111ceb61d07c0289 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804)
* f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #5269:
URL: https://github.com/apache/hudi/pull/5269#discussion_r866299432
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseCompactor.java:
##########
@@ -31,16 +33,30 @@
private static final long serialVersionUID = 1L;
+ protected final transient Object writeClientUpdateLock = new Object();
+ protected final transient List<BaseHoodieWriteClient<T, I, K, O>> oldCompactionClientList = new ArrayList<>();
+
protected transient BaseHoodieWriteClient<T, I, K, O> compactionClient;
+ protected boolean isCompactionRunning = false;
+
public BaseCompactor(BaseHoodieWriteClient<T, I, K, O> compactionClient) {
this.compactionClient = compactionClient;
}
public abstract void compact(HoodieInstant instant) throws IOException;
public void updateWriteClient(BaseHoodieWriteClient<T, I, K, O> writeClient) {
- this.compactionClient = writeClient;
+ synchronized (writeClientUpdateLock) {
+ if (!isCompactionRunning) {
+ this.compactionClient.close();
+ } else {
+ // Store the old compaction client so that they can be closed
Review Comment:
Agree very strongly with the point above: unless there are very strong argument why we can not re-init Async Service itself, i believe we should follow t/h with an invariant that AS and Write Client lifecycles are coupled and solve this concurrency control issues at the root
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseCompactor.java:
##########
@@ -31,16 +33,30 @@
private static final long serialVersionUID = 1L;
+ protected final transient Object writeClientUpdateLock = new Object();
+ protected final transient List<BaseHoodieWriteClient<T, I, K, O>> oldCompactionClientList = new ArrayList<>();
+
protected transient BaseHoodieWriteClient<T, I, K, O> compactionClient;
+ protected boolean isCompactionRunning = false;
+
public BaseCompactor(BaseHoodieWriteClient<T, I, K, O> compactionClient) {
this.compactionClient = compactionClient;
}
public abstract void compact(HoodieInstant instant) throws IOException;
public void updateWriteClient(BaseHoodieWriteClient<T, I, K, O> writeClient) {
- this.compactionClient = writeClient;
+ synchronized (writeClientUpdateLock) {
+ if (!isCompactionRunning) {
+ this.compactionClient.close();
+ } else {
+ // Store the old compaction client so that they can be closed
Review Comment:
Immutability is hard but very powerful property we should lean in on it as much as possible, only detour from it when there's no other choice (mostly for perf reasons)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #5269:
URL: https://github.com/apache/hudi/pull/5269#discussion_r846563805
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/HoodieSparkClusteringClient.java:
##########
@@ -53,11 +53,15 @@ public void cluster(HoodieInstant instant) throws IOException {
SparkRDDWriteClient<T> writeClient = (SparkRDDWriteClient<T>) clusteringClient;
Option<HoodieCommitMetadata> commitMetadata = writeClient.cluster(instant.getTimestamp(), true).getCommitMetadata();
Stream<HoodieWriteStat> hoodieWriteStatStream = commitMetadata.get().getPartitionToWriteStats().entrySet().stream().flatMap(e ->
- e.getValue().stream());
+ e.getValue().stream());
long errorsCount = hoodieWriteStatStream.mapToLong(HoodieWriteStat::getTotalWriteErrors).sum();
if (errorsCount > 0) {
// TODO: Should we treat this fatal and throw exception?
LOG.error("Clustering for instant (" + instant + ") failed with write errors");
}
+ if (writeClient != clusteringClient) {
+ // Clustering write client has been updated with new schema, closing the old one
+ writeClient.close();
Review Comment:
guess, we could miss to close the old write client if there is no clustering inflight. i.e. when async clustering thread is just waiting and if write client is updated (due to schema change), we might over ride w/ new write client and we may not close the old one. Can we ensure we close it out in all cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1093721102
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7937",
"triggerID" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"triggerType" : "PUSH"
}, {
"hash" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"triggerType" : "PUSH"
}, {
"hash" : "c411bedee151888a7cb67690be123c0bc0f29797",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7938",
"triggerID" : "c411bedee151888a7cb67690be123c0bc0f29797",
"triggerType" : "PUSH"
}, {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 784c8cf617b0f4eca028205f87067e08bb55457b UNKNOWN
* c411bedee151888a7cb67690be123c0bc0f29797 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7938)
* 6f8d22ccc5efbd87ff993a46ea1977355842602f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1093748687
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7937",
"triggerID" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"triggerType" : "PUSH"
}, {
"hash" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"triggerType" : "PUSH"
}, {
"hash" : "c411bedee151888a7cb67690be123c0bc0f29797",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7938",
"triggerID" : "c411bedee151888a7cb67690be123c0bc0f29797",
"triggerType" : "PUSH"
}, {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 784c8cf617b0f4eca028205f87067e08bb55457b UNKNOWN
* c411bedee151888a7cb67690be123c0bc0f29797 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7938)
* 6f8d22ccc5efbd87ff993a46ea1977355842602f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1261302470
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
}, {
"hash" : "c5709b6c45cda7e0a0253319246b10da11668873",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853",
"triggerID" : "c5709b6c45cda7e0a0253319246b10da11668873",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5709b6c45cda7e0a0253319246b10da11668873 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1273928997
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
}, {
"hash" : "c5709b6c45cda7e0a0253319246b10da11668873",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853",
"triggerID" : "c5709b6c45cda7e0a0253319246b10da11668873",
"triggerType" : "PUSH"
}, {
"hash" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12114",
"triggerID" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5709b6c45cda7e0a0253319246b10da11668873 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853)
* d763b34664c25cdf07d1d2eb818ddbf462a8bd7f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12114)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaking sink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1284826838
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
}, {
"hash" : "c5709b6c45cda7e0a0253319246b10da11668873",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853",
"triggerID" : "c5709b6c45cda7e0a0253319246b10da11668873",
"triggerType" : "PUSH"
}, {
"hash" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12114",
"triggerID" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"triggerType" : "PUSH"
}, {
"hash" : "5c5dde77caa0524e38ba5def698b490c0a35a6c4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12193",
"triggerID" : "5c5dde77caa0524e38ba5def698b490c0a35a6c4",
"triggerType" : "PUSH"
}, {
"hash" : "96aa0891caaf95715b4877c8f2444bfc3b59386f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96aa0891caaf95715b4877c8f2444bfc3b59386f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c5dde77caa0524e38ba5def698b490c0a35a6c4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12193)
* 96aa0891caaf95715b4877c8f2444bfc3b59386f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #5269:
URL: https://github.com/apache/hudi/pull/5269#discussion_r995223267
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -78,13 +81,26 @@ protected Pair<CompletableFuture, ExecutorService> startService() {
while (!isShutdownRequested()) {
final HoodieInstant instant = fetchNextAsyncServiceInstant();
-
if (null != instant) {
LOG.info("Starting Compaction for instant " + instant);
+ synchronized (writeConfigUpdateLock) {
Review Comment:
we can't cancel existing async service if its inflight. And so we might have to create N such instances and need to find a way to clean those up. Wouldn't this be easier than that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1093606372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7515d1f98a1299f5a42e242d3e36d84a9a38bee UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1093646283
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7937",
"triggerID" : "d7515d1f98a1299f5a42e242d3e36d84a9a38bee",
"triggerType" : "PUSH"
}, {
"hash" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "784c8cf617b0f4eca028205f87067e08bb55457b",
"triggerType" : "PUSH"
}, {
"hash" : "c411bedee151888a7cb67690be123c0bc0f29797",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7938",
"triggerID" : "c411bedee151888a7cb67690be123c0bc0f29797",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d7515d1f98a1299f5a42e242d3e36d84a9a38bee Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7937)
* 784c8cf617b0f4eca028205f87067e08bb55457b UNKNOWN
* c411bedee151888a7cb67690be123c0bc0f29797 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7938)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaking sink
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #5269:
URL: https://github.com/apache/hudi/pull/5269#discussion_r998645726
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -20,24 +20,37 @@
package org.apache.hudi.async;
import org.apache.hudi.client.RunsTableService;
+import org.apache.hudi.client.embedded.EmbeddedTimelineServerHelper;
+import org.apache.hudi.client.embedded.EmbeddedTimelineService;
+import org.apache.hudi.common.util.Option;
import org.apache.hudi.config.HoodieWriteConfig;
+import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.Function;
public abstract class HoodieAsyncTableService extends HoodieAsyncService implements RunsTableService {
+ protected final Object writeConfigUpdateLock = new Object();
protected HoodieWriteConfig writeConfig;
+ protected Option<EmbeddedTimelineService> embeddedTimelineService;
+ protected AtomicBoolean isWriteConfigUpdated = new AtomicBoolean(false);
protected HoodieAsyncTableService() {
}
- protected HoodieAsyncTableService(HoodieWriteConfig writeConfig) {
+ protected HoodieAsyncTableService(HoodieWriteConfig writeConfig, Option<EmbeddedTimelineService> embeddedTimelineService) {
this.writeConfig = writeConfig;
+ this.embeddedTimelineService = embeddedTimelineService;
Review Comment:
Let's chain the ctors (so that we don't need to duplicate the init logic
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -47,4 +60,11 @@ public void start(Function<Boolean, Boolean> onShutdownCallback) {
}
super.start(onShutdownCallback);
}
+
+ public void updateWriteConfig(HoodieWriteConfig writeConfig) {
+ synchronized (writeConfigUpdateLock) {
+ this.writeConfig = EmbeddedTimelineServerHelper.updateWriteConfigWithTimelineServer(embeddedTimelineService.get(), writeConfig);
+ isWriteConfigUpdated.set(true);
Review Comment:
Flag setup is quite brittle: it makes resetting a 2-step process (first update the flag, then update needs to be handled asynchronously), which is easy to miss during impl.
Instead i'd suggest we
- Create abstract hook `handleWriteConfigUpdate` (hence make every inheritor implement it)
- Invoke it here under lock (that way we keep the lock private and avoid this logic spilling into inheritors)
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -20,24 +20,37 @@
package org.apache.hudi.async;
import org.apache.hudi.client.RunsTableService;
+import org.apache.hudi.client.embedded.EmbeddedTimelineServerHelper;
+import org.apache.hudi.client.embedded.EmbeddedTimelineService;
+import org.apache.hudi.common.util.Option;
import org.apache.hudi.config.HoodieWriteConfig;
+import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.Function;
public abstract class HoodieAsyncTableService extends HoodieAsyncService implements RunsTableService {
+ protected final Object writeConfigUpdateLock = new Object();
protected HoodieWriteConfig writeConfig;
+ protected Option<EmbeddedTimelineService> embeddedTimelineService;
+ protected AtomicBoolean isWriteConfigUpdated = new AtomicBoolean(false);
protected HoodieAsyncTableService() {
}
- protected HoodieAsyncTableService(HoodieWriteConfig writeConfig) {
+ protected HoodieAsyncTableService(HoodieWriteConfig writeConfig, Option<EmbeddedTimelineService> embeddedTimelineService) {
this.writeConfig = writeConfig;
+ this.embeddedTimelineService = embeddedTimelineService;
}
- protected HoodieAsyncTableService(HoodieWriteConfig writeConfig, boolean runInDaemonMode) {
+ protected HoodieAsyncTableService(HoodieWriteConfig writeConfig, Option<EmbeddedTimelineService> embeddedTimelineService, boolean runInDaemonMode) {
super(runInDaemonMode);
- this.writeConfig = writeConfig;
+ this.embeddedTimelineService = embeddedTimelineService;
+ if (embeddedTimelineService.isPresent()) {
Review Comment:
Why the same logic isn't applied in the other ctor?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -78,13 +81,26 @@ protected Pair<CompletableFuture, ExecutorService> startService() {
while (!isShutdownRequested()) {
final HoodieInstant instant = fetchNextAsyncServiceInstant();
-
if (null != instant) {
LOG.info("Starting Compaction for instant " + instant);
+ synchronized (writeConfigUpdateLock) {
Review Comment:
Please check my comment below regarding the flag
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #5269:
URL: https://github.com/apache/hudi/pull/5269#discussion_r980459872
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -732,31 +732,31 @@ public void setupWriteClient() throws IOException {
if ((null != schemaProvider)) {
Schema sourceSchema = schemaProvider.getSourceSchema();
Schema targetSchema = schemaProvider.getTargetSchema();
- reInitWriteClient(sourceSchema, targetSchema);
+ createNewWriteClientWithSchema(sourceSchema, targetSchema);
}
}
- private void reInitWriteClient(Schema sourceSchema, Schema targetSchema) throws IOException {
+ private void createNewWriteClientWithSchema(Schema sourceSchema, Schema targetSchema) throws IOException {
LOG.info("Setting up new Hoodie Write Client");
if (isDropPartitionColumns()) {
targetSchema = HoodieAvroUtils.removeFields(targetSchema, getPartitionColumns(keyGenerator, props));
}
registerAvroSchemas(sourceSchema, targetSchema);
- HoodieWriteConfig hoodieCfg = getHoodieClientConfig(targetSchema);
- if (hoodieCfg.isEmbeddedTimelineServerEnabled()) {
+ HoodieWriteConfig writeConfig = getHoodieClientConfig(targetSchema);
+ if (writeConfig.isEmbeddedTimelineServerEnabled()) {
if (!embeddedTimelineService.isPresent()) {
- embeddedTimelineService = EmbeddedTimelineServerHelper.createEmbeddedTimelineService(new HoodieSparkEngineContext(jssc), hoodieCfg);
+ embeddedTimelineService = EmbeddedTimelineServerHelper.createEmbeddedTimelineService(new HoodieSparkEngineContext(jssc), writeConfig);
Review Comment:
We should keep one long-running timeline service in DeltaSync, instead of shutting down (by closing the write client) and restarting the timeline server.
In the write client, we should change the behavior of shutting down the timeline server. If the timeline server instance is passed into the constructor externally (`public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig, Option<EmbeddedTimelineService> timelineService)`), the write client should not shut it down, and should let the caller to handle the lifecycle.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -81,8 +82,13 @@ protected Pair<CompletableFuture, ExecutorService> startService() {
if (null != instant) {
LOG.info("Starting Compaction for instant " + instant);
+ synchronized (writeConfigUpdateLock) {
+ compactor = createCompactor();
Review Comment:
We should only reinitialize the async compactor/clusterer when the write config is updated, not for every execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1260288169
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807)
* b987d84d2c18e5e3cb15268a2689a6294842b2c8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814)
* e7966fbbde59b43876542fc39135157835b6c861 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1260118937
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a360d286f9a9bff3f60cc7231bc0abfe86675a88 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240)
* 460ce44922c50af95bf6c68f111ceb61d07c0289 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804)
* f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1260104128
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a360d286f9a9bff3f60cc7231bc0abfe86675a88 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240)
* 460ce44922c50af95bf6c68f111ceb61d07c0289 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1260175625
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 460ce44922c50af95bf6c68f111ceb61d07c0289 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806)
* f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer and Spark streaking sink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1278371947
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
}, {
"hash" : "c5709b6c45cda7e0a0253319246b10da11668873",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853",
"triggerID" : "c5709b6c45cda7e0a0253319246b10da11668873",
"triggerType" : "PUSH"
}, {
"hash" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12114",
"triggerID" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"triggerType" : "PUSH"
}, {
"hash" : "5c5dde77caa0524e38ba5def698b490c0a35a6c4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12193",
"triggerID" : "5c5dde77caa0524e38ba5def698b490c0a35a6c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d763b34664c25cdf07d1d2eb818ddbf462a8bd7f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12114)
* 5c5dde77caa0524e38ba5def698b490c0a35a6c4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12193)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1240183216
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6f8d22ccc5efbd87ff993a46ea1977355842602f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944)
* a360d286f9a9bff3f60cc7231bc0abfe86675a88 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5269:
URL: https://github.com/apache/hudi/pull/5269#issuecomment-1273925102
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7944",
"triggerID" : "6f8d22ccc5efbd87ff993a46ea1977355842602f",
"triggerType" : "PUSH"
}, {
"hash" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11240",
"triggerID" : "a360d286f9a9bff3f60cc7231bc0abfe86675a88",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11804",
"triggerID" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"triggerType" : "PUSH"
}, {
"hash" : "460ce44922c50af95bf6c68f111ceb61d07c0289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11806",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"triggerType" : "PUSH"
}, {
"hash" : "f82d7aecb0d5f6b687a3e8083b562dd8fd5601c9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11807",
"triggerID" : "1260167190",
"triggerType" : "MANUAL"
}, {
"hash" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11814",
"triggerID" : "b987d84d2c18e5e3cb15268a2689a6294842b2c8",
"triggerType" : "PUSH"
}, {
"hash" : "e7966fbbde59b43876542fc39135157835b6c861",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11816",
"triggerID" : "e7966fbbde59b43876542fc39135157835b6c861",
"triggerType" : "PUSH"
}, {
"hash" : "c5709b6c45cda7e0a0253319246b10da11668873",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853",
"triggerID" : "c5709b6c45cda7e0a0253319246b10da11668873",
"triggerType" : "PUSH"
}, {
"hash" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d763b34664c25cdf07d1d2eb818ddbf462a8bd7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5709b6c45cda7e0a0253319246b10da11668873 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11853)
* d763b34664c25cdf07d1d2eb818ddbf462a8bd7f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org