You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/05 08:44:31 UTC
[GitHub] [hudi] jadireddi opened a new pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
jadireddi opened a new pull request #3929:
URL: https://github.com/apache/hudi/pull/3929
Jira: https://issues.apache.org/jira/browse/HUDI-1881
## What is the purpose of the pull request
Currently, HoodieMultiTableDeltaStreamer is not designed to run in continuous mode as the execution is done serially in continuous mode
Implemented a thread pool and let the sync() for each table happen asynchronously without blocking.
## Brief change log
- Added fixed thread pool executor to HoodieMultiTableDeltaStreamer to run sync tables asynchronously.
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [X] Has a corresponding JIRA in PR title & commit
- [X] Commit message is descriptive of the change
- [ ] CI is green
- [x] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] watermelon12138 commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
watermelon12138 commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-1010577806
Hello, I hava some questions. The target tables are different, so synchronous write does not have any impact. Why do you want to implement asynchronous? The sync method of the DeltaStreamer has started the thread. Does the MultiTableDeltaStreamer need to use the thread pool to start new threads again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-965974390
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163",
"triggerID" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"triggerType" : "PUSH"
}, {
"hash" : "1fda22afc404a8a474e41b74d256aa6ddb8f2e95",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1fda22afc404a8a474e41b74d256aa6ddb8f2e95",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8b346f81a853b6cddf84a7e770b3f6de496fcfd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163)
* 1fda22afc404a8a474e41b74d256aa6ddb8f2e95 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-961757552
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163",
"triggerID" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8b346f81a853b6cddf84a7e770b3f6de496fcfd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jadireddi edited a comment on pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
jadireddi edited a comment on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-961720413
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-1061000958
planning to take this up next, now that the cleaner PRs are done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-961720122
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-961720122
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a change in pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on a change in pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#discussion_r782921008
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
##########
@@ -378,16 +383,23 @@ private static String resetTarget(Config configuration, String database, String
/**
* Creates actual HoodieDeltaStreamer objects for every table/topic and does incremental sync.
*/
- public void sync() {
- for (TableExecutionContext context : tableExecutionContexts) {
Review comment:
@watermelon12138 the execution is happening serially in the current implementation. Please see this for loop here. We intend to change it to parallel execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-1010259152
@jadireddi Gentle ping. In case you are not having bandwidth to finish this, I can drive this through. Please LMK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-1072983613
Closing in favour of https://github.com/apache/hudi/pull/5071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-961720122
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8b346f81a853b6cddf84a7e770b3f6de496fcfd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jadireddi commented on pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
jadireddi commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-963813120
Thank you Vinoth. Done unit testing only.
Will make changes to trigger based on data availability .
It would be really helpful if we have a test environment. @pratyakshsharma can you please help me with the steps to set it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-961721536
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163",
"triggerID" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8b346f81a853b6cddf84a7e770b3f6de496fcfd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jadireddi commented on pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
jadireddi commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-961720413
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-998543878
@jadireddi Were you able to test it out?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-1050515572
this will definitely be useful for the community. lets get this in for 0.11.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jadireddi edited a comment on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
jadireddi edited a comment on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-1011950891
> @jadireddi Gentle ping. In case you are not having bandwidth to finish this, I can drive this through. Please LMK.
Hi @pratyakshsharma. I have done the testing and be available to push the changes. Sorry stuck with other work and not able to reply earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-1010709857
@watermelon12138 thread pool is being used since we intend to make the ingestion run in parallel for all the configured tables. I hope that answers your question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-1046216929
@jadireddi gentle ping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma closed pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
pratyakshsharma closed pull request #3929:
URL: https://github.com/apache/hudi/pull/3929
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jadireddi edited a comment on pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
jadireddi edited a comment on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-961720413
This is WIP PR.
@vinothchandar Implemented a thread pool and made the sync() for each table happen asynchronously.
Do we need to set even `executor awaitTermination`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jadireddi edited a comment on pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
jadireddi edited a comment on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-961720413
This is WIP PR.
@vinothchandar Implemented a thread pool and made the sync() for each table happen asynchronously.
Do we need to set even `executor timeout`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-961720122
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-965993030
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163",
"triggerID" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"triggerType" : "PUSH"
}, {
"hash" : "1fda22afc404a8a474e41b74d256aa6ddb8f2e95",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3299",
"triggerID" : "1fda22afc404a8a474e41b74d256aa6ddb8f2e95",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fda22afc404a8a474e41b74d256aa6ddb8f2e95 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3299)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-965699607
@jadireddi Thank you for raising this. Changes look good.
Regarding the setup, if you are able to run the DeltaStreamer on your local machine, we can test it there itself by setting up a mysql, debezium and kafka instances. That way we will be able to do end to end testing based on data availability as well. Will be great if you join our slack. We will connect there and I can help set up everything for you. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] watermelon12138 commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
watermelon12138 commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-1010741433
> @watermelon12138 thread pool is being used since we intend to make the ingestion run in parallel for all the configured tables. I hope that answers your question.
Hello, All configured tables start a thread by executing the sync method of DeltaStreamer, which means that the injection process of all configured tables is parallel. Isn't that so?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jadireddi commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
jadireddi commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-1011950891
> @jadireddi Gentle ping. In case you are not having bandwidth to finish this, I can drive this through. Please LMK.
Hi @pratyakshsharma. I have done the testing and be available to push the changes. Sorry stuck with other work and able to reply earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-1053439441
Ack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-965975329
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163",
"triggerID" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"triggerType" : "PUSH"
}, {
"hash" : "1fda22afc404a8a474e41b74d256aa6ddb8f2e95",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3299",
"triggerID" : "1fda22afc404a8a474e41b74d256aa6ddb8f2e95",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8b346f81a853b6cddf84a7e770b3f6de496fcfd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163)
* 1fda22afc404a8a474e41b74d256aa6ddb8f2e95 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3299)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#discussion_r789367919
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
##########
@@ -378,16 +383,23 @@ private static String resetTarget(Config configuration, String database, String
/**
* Creates actual HoodieDeltaStreamer objects for every table/topic and does incremental sync.
*/
- public void sync() {
- for (TableExecutionContext context : tableExecutionContexts) {
- try {
- new HoodieDeltaStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties())).sync();
- successTables.add(Helpers.getTableWithDatabase(context));
- } catch (Exception e) {
- logger.error("error while running MultiTableDeltaStreamer for table: " + context.getTableName(), e);
- failedTables.add(Helpers.getTableWithDatabase(context));
- }
- }
+ public void sync() throws InterruptedException {
+ ExecutorService executorService = Executors.newFixedThreadPool(tableExecutionContexts.size());
+ tableExecutionContexts.forEach(context -> {
+ executorService.execute(new Runnable() {
+ @Override
+ public void run() {
+ try {
+ new HoodieDeltaStreamer(context.getConfig(), jssc, Option.ofNullable(context.getProperties())).sync();
+ successTables.add(Helpers.getTableWithDatabase(context));
+ } catch (Exception e) {
+ logger.error("error while running MultiTableDeltaStreamer for table: " + context.getTableName(), e);
+ failedTables.add(Helpers.getTableWithDatabase(context));
+ }
+ }
+ });
+ });
+ executorService.shutdown();
Review comment:
should we add awaitTermination here ? we can't proceed w/ next batch until all tables in current batch is completed right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-961720122
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8b346f81a853b6cddf84a7e770b3f6de496fcfd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-961721536
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163",
"triggerID" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8b346f81a853b6cddf84a7e770b3f6de496fcfd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jadireddi commented on pull request #3929: [HUDI-2697] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
jadireddi commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-961720413
This is WIP PR.
@vinothchandar Implemented a thread pool and made the sync() for each table happen asynchronously.
Should we have even `executor timeout`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-1050515420
@pratyakshsharma : can you take this up please. Test it out by yourself, address any feedbacks and push updates. Once done, let me know. I can take one final pass and merge it in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-965975329
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163",
"triggerID" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"triggerType" : "PUSH"
}, {
"hash" : "1fda22afc404a8a474e41b74d256aa6ddb8f2e95",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3299",
"triggerID" : "1fda22afc404a8a474e41b74d256aa6ddb8f2e95",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8b346f81a853b6cddf84a7e770b3f6de496fcfd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163)
* 1fda22afc404a8a474e41b74d256aa6ddb8f2e95 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3299)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-965974390
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163",
"triggerID" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"triggerType" : "PUSH"
}, {
"hash" : "1fda22afc404a8a474e41b74d256aa6ddb8f2e95",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1fda22afc404a8a474e41b74d256aa6ddb8f2e95",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8b346f81a853b6cddf84a7e770b3f6de496fcfd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163)
* 1fda22afc404a8a474e41b74d256aa6ddb8f2e95 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3929: [HUDI-1881] Make multi table delta streamer to use thread pool for table sync asynchronously.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3929:
URL: https://github.com/apache/hudi/pull/3929#issuecomment-961757552
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163",
"triggerID" : "d8b346f81a853b6cddf84a7e770b3f6de496fcfd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d8b346f81a853b6cddf84a7e770b3f6de496fcfd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3163)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org