You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/01/19 11:19:40 UTC
[GitHub] [hudi] watermelon12138 opened a new pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
watermelon12138 opened a new pull request #4637:
URL: https://github.com/apache/hudi/pull/4637
## What is the purpose of the pull request
The purpose of pull request is to update a single target table from multiple source tables.
## Brief change log
- *Modify the HoodieMultiTableDeltaStreamer file so that it can generate the execution context of table based on source tables.*
- *Modify the DeltaSync.java file so that the source table can associate with other tables and the source can configure independent checkpoint.*
- *add UT*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] watermelon12138 commented on pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
Posted by GitBox <gi...@apache.org>.
watermelon12138 commented on pull request #4637:
URL: https://github.com/apache/hudi/pull/4637#issuecomment-1017540649
> oh I see. my bad, I did not look closely. sorry about that. We can keep it open. give me a day or two. I will take a detailed look and leave comments.
OK, thank you very much. You can see my new pull request. I want to close the current pull request.
https://github.com/apache/hudi/pull/4645
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
Posted by GitBox <gi...@apache.org>.
nsivabalan closed pull request #4637:
URL: https://github.com/apache/hudi/pull/4637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] watermelon12138 closed pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
Posted by GitBox <gi...@apache.org>.
watermelon12138 closed pull request #4637:
URL: https://github.com/apache/hudi/pull/4637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] watermelon12138 commented on pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
Posted by GitBox <gi...@apache.org>.
watermelon12138 commented on pull request #4637:
URL: https://github.com/apache/hudi/pull/4637#issuecomment-1017048479
@nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] watermelon12138 removed a comment on pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
Posted by GitBox <gi...@apache.org>.
watermelon12138 removed a comment on pull request #4637:
URL: https://github.com/apache/hudi/pull/4637#issuecomment-1017048479
@nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] watermelon12138 removed a comment on pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
Posted by GitBox <gi...@apache.org>.
watermelon12138 removed a comment on pull request #4637:
URL: https://github.com/apache/hudi/pull/4637#issuecomment-1017540649
> oh I see. my bad, I did not look closely. sorry about that. We can keep it open. give me a day or two. I will take a detailed look and leave comments.
OK, thank you very much. You can see my new pull request. I want to close the current pull request.
https://github.com/apache/hudi/pull/4645
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] watermelon12138 commented on pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
Posted by GitBox <gi...@apache.org>.
watermelon12138 commented on pull request #4637:
URL: https://github.com/apache/hudi/pull/4637#issuecomment-1017541355
> oh I see. my bad, I did not look closely. sorry about that. We can keep it open. give me a day or two. I will take a detailed look and leave comments.
@nsivabalan
OK, thank you very much. You can see my new pull request. I want to close the current pull request.
#4645
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4637:
URL: https://github.com/apache/hudi/pull/4637#issuecomment-1016364646
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b7bcc6e93091769c5ea8d440a2620f5344181719",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5345",
"triggerID" : "b7bcc6e93091769c5ea8d440a2620f5344181719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7bcc6e93091769c5ea8d440a2620f5344181719 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5345)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4637:
URL: https://github.com/apache/hudi/pull/4637#issuecomment-1016368058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b7bcc6e93091769c5ea8d440a2620f5344181719",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5345",
"triggerID" : "b7bcc6e93091769c5ea8d440a2620f5344181719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7bcc6e93091769c5ea8d440a2620f5344181719 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5345)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] watermelon12138 commented on pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
Posted by GitBox <gi...@apache.org>.
watermelon12138 commented on pull request #4637:
URL: https://github.com/apache/hudi/pull/4637#issuecomment-1017058557
@nsivabalan
Thank you for the advice.
However, the resumeCheckpointStr calculation method in DeltaSync applies only to updating a single target table by a single source. If multiple sources update a single target table, this calculation method does not work. If multiple sources update a single target, set an independent checkpoint for each source so that each source can recover from any checkpoint. I only changed the methods for calculating resumeCheckpointStr and saving checkpoints to checkpointCommitMetadata in DeltaSync. Specifically, I added these methods. This does not affect the calculation logic when a single source updates a single target.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4637:
URL: https://github.com/apache/hudi/pull/4637#issuecomment-1017413206
oh I see. my bad, I did not look closely. sorry about that. We can keep it open. give me a day or two. I will take a detailed look and leave comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4637:
URL: https://github.com/apache/hudi/pull/4637#issuecomment-1016364646
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b7bcc6e93091769c5ea8d440a2620f5344181719",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5345",
"triggerID" : "b7bcc6e93091769c5ea8d440a2620f5344181719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7bcc6e93091769c5ea8d440a2620f5344181719 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5345)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4637:
URL: https://github.com/apache/hudi/pull/4637#issuecomment-1016361126
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b7bcc6e93091769c5ea8d440a2620f5344181719",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b7bcc6e93091769c5ea8d440a2620f5344181719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7bcc6e93091769c5ea8d440a2620f5344181719 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4637: [HUDI-3103] Enable MultiTableDeltaStreamer to update a single target table from multiple source tables.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4637:
URL: https://github.com/apache/hudi/pull/4637#issuecomment-1016361126
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b7bcc6e93091769c5ea8d440a2620f5344181719",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b7bcc6e93091769c5ea8d440a2620f5344181719",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7bcc6e93091769c5ea8d440a2620f5344181719 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org