You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "SteNicholas (via GitHub)" <gi...@apache.org> on 2023/05/18 09:37:05 UTC
[GitHub] [hudi] SteNicholas opened a new pull request, #8753: [HUDI-5095] Flink: Stores a special watermark(flag) to identify the current progress of writing data
SteNicholas opened a new pull request, #8753:
URL: https://github.com/apache/hudi/pull/8753
### Change Logs
`AbstractStreamWriteFunction` could process watermark from `StreamWriteOperator` and send `WriteMetadataEvent` with the watermark of write function to `StreamWriteOperatorCoordinator`. `StreamWriteOperatorCoordinator` stores the min watermark of `WriteMetadataEvent`s from subtasks into the extra metadata of `HoodieCommitMetadata`. Meanwhile, `StreamWriteOperatorCoordinator` advances the min watermark for subtasks in which no data has written, and does not advance watermark for commit on empty batch.
### Impact
`StreamWriteOperatorCoordinator` stores the min watermark of write function into the extra metadata of `HoodieCommitMetadata` to identify the current progress of writing.
### Risk level (write none, low medium or high below)
None.
### Documentation Update
None.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8753: [HUDI-5095] Flink: Stores a special watermark(flag) to identify the current progress of writing data
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8753:
URL: https://github.com/apache/hudi/pull/8753#issuecomment-1552812693
Can you elaborate a little more what is the end-2-end user story here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8753: [HUDI-5095] Flink: Stores a special watermark(flag) to identify the current progress of writing data
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8753:
URL: https://github.com/apache/hudi/pull/8753#issuecomment-1552825940
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "45446c0cbf27d46589e8de1e9cc66221c420e353",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "45446c0cbf27d46589e8de1e9cc66221c420e353",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45446c0cbf27d46589e8de1e9cc66221c420e353 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] SteNicholas commented on pull request #8753: [HUDI-5095] Flink: Stores a special watermark(flag) to identify the current progress of writing data
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #8753:
URL: https://github.com/apache/hudi/pull/8753#issuecomment-1552823258
@danny0405, in the data warehouse incremental scenario, whether partition completes to write is depended on the progress of writing, which is used to commit partition for trigger downstream task to read the committed partition data. The watermark could identify the current progress of writing data and infer the committed partition. In BiliBili, incremental ETL uses the watermark to identify the partition committed and downstream ETL tasks could only query the committed partition data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8753: [HUDI-5095] Flink: Stores a special watermark(flag) to identify the current progress of writing data
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8753:
URL: https://github.com/apache/hudi/pull/8753#issuecomment-1552831796
> The watermark could identify the current progress of writing data and infer the committed partition
Can you sketch the inference details a little more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] SteNicholas commented on pull request #8753: [HUDI-5095] Flink: Stores a special watermark(flag) to identify the current progress of writing data
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #8753:
URL: https://github.com/apache/hudi/pull/8753#issuecomment-1552838233
@danny0405, we could use watermark to infer the partitions need to commit and need to create as follows:
```
hiveBeforeCommitEndTime = waterMarkToCurrentPartitionTime(hiveBeforeCommitEndTime, commitType);
hiveAfterCommitEndTime = waterMarkToCurrentPartitionTime(convertReadyCommitTime(hiveAfterCommitEndTime, watermark, partitionInterval), commitType);
List<List<String>> hiveBeforeNeedCommit = getNeedCommitList(hiveBeforeCommitStartTime, hiveBeforeCommitEndTime, partitionInterval, commitType);
List<List<String>> hiveAfterNeedCommit = getNeedCommitList(hiveAfterCommitStartTime, hiveAfterCommitEndTime, partitionInterval, commitType);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8753: [HUDI-5095] Flink: Stores a special watermark(flag) to identify the current progress of writing data
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8753:
URL: https://github.com/apache/hudi/pull/8753#issuecomment-1553767810
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "45446c0cbf27d46589e8de1e9cc66221c420e353",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17188",
"triggerID" : "45446c0cbf27d46589e8de1e9cc66221c420e353",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45446c0cbf27d46589e8de1e9cc66221c420e353 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8753: [HUDI-5095] Flink: Stores a special watermark(flag) to identify the current progress of writing data
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8753:
URL: https://github.com/apache/hudi/pull/8753#issuecomment-1552836348
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "45446c0cbf27d46589e8de1e9cc66221c420e353",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17188",
"triggerID" : "45446c0cbf27d46589e8de1e9cc66221c420e353",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45446c0cbf27d46589e8de1e9cc66221c420e353 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] SteNicholas commented on pull request #8753: [HUDI-5095] Flink: Stores a special watermark(flag) to identify the current progress of writing data
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #8753:
URL: https://github.com/apache/hudi/pull/8753#issuecomment-1560628076
@danny0405, could you help to review this implementation again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org