You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "stream2000 (via GitHub)" <gi...@apache.org> on 2023/09/08 06:44:06 UTC
[GitHub] [hudi] stream2000 opened a new pull request, #9651: [HUDI-6336] Support flink timeline-based ckp metadata
stream2000 opened a new pull request, #9651:
URL: https://github.com/apache/hudi/pull/9651
### Change Logs
Support flink timeline-based ckp metadata to reduce file system listing
### Impact
taskmanagers will read ckp metadata from timeline server
### Risk level (write none, low medium or high below)
low
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329482075
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/CkpMetadata.java:
##########
@@ -233,12 +210,18 @@ private Path fullPath(String fileName) {
return new Path(path, fileName);
}
- private List<CkpMessage> scanCkpMetadata(Path ckpMetaPath) throws IOException {
+ protected Stream<CkpMessage> fetchCkpMessages(Path ckpMetaPath) throws IOException {
// This is required when the storage is minio
if (!this.fs.exists(ckpMetaPath)) {
- return new ArrayList<>();
+ return Stream.empty();
}
- return Arrays.stream(this.fs.listStatus(ckpMetaPath)).map(CkpMessage::new)
+ Stream<CkpMessage> ckpMessageStream;
+ ckpMessageStream = Arrays.stream(this.fs.listStatus(ckpMetaPath)).map(CkpMessage::new);
+ return ckpMessageStream;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1725009334
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* da4c89e3baf151b650b54d1ad608c7d8fa4c13ca Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009)
* 0acc78ec78cc1d99a62a2d8c7514110a56deadd1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018)
* d9c59809dade4617f208b81d683bf496206cca50 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1727082089
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026",
"triggerID" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046",
"triggerID" : "1727079993",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 427392c696b285afb6d4a093059c2b9a3c190f2a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1328394158
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -505,6 +516,24 @@ private void registerMarkerAPI() {
}, false));
}
+ private void registerCkpMetadataAPI() {
+ app.get(InstantStateHandler.ALL_INSTANT_STATE_URL, new ViewHandler(ctx -> {
Review Comment:
Fix the method name, also the constant name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1713103723
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2acae42a85a382a3c1b30d4aee7f2f70f35d7b10 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729)
* 7d5ae2f19b99845990588ccfd74bba3b63bb75c0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1321266607
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -418,6 +418,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.sinceVersion("0.9.0")
.withDocumentation(MarkerType.class);
+ public static final ConfigProperty<Boolean> TIMELINE_SERVER_BASED_FLINK_CKP_METADATA = ConfigProperty
+ .key("hoodie.flink.ckp_metadata.timeline_server_based.enable")
+ .defaultValue(true)
Review Comment:
Thanks for your review~
> Can we remove the flink keywork here
Removing the `flink` keyword is OK.
> do we need an option for it, just add a new URI on the timeline server should be fine.
I'm mainly following the implementation of timeline-based markers which have an option for enabling timeline-based markers or not, too.
> you need to handle the diffferent request from various clients in good manner.
The parameter of Ckp metadata Handler is the full ckp metadata path, which already contains the uniqueId of one specific job. I think it's enough for the multi-client scenario, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1328048590
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java:
##########
@@ -349,8 +351,8 @@ private static void initMetadataTable(HoodieFlinkWriteClient<?> writeClient) {
writeClient.initMetadataTable();
}
- private static CkpMetadata initCkpMetadata(HoodieTableMetaClient metaClient, Configuration conf) throws IOException {
- CkpMetadata ckpMetadata = CkpMetadata.getInstance(metaClient, conf.getString(FlinkOptions.WRITE_CLIENT_ID));
+ private static CkpMetadata initCkpMetadata(HoodieTableMetaClient metaClient, HoodieWriteConfig writeConfig, Configuration conf) throws IOException {
+ CkpMetadata ckpMetadata = CkpMetadataFactory.getCkpMetadata(metaClient, writeConfig, conf);
ckpMetadata.bootstrap();
return ckpMetadata;
Review Comment:
You are right, we can't ensure that the refreshing by the coordinator is successful. I added a new mechanism that the instant states will be refresh if client request count reaches a threshold, with this mechanism we can ensure that the writer won't be blocked infinitely.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/util/TimelineServerHelper.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.util;
+
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+
+/**
+ * Helper class for executing timeline server requests.
+ */
+public class TimelineServerHelper {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TimelineServerHelper.class);
+ private final ObjectMapper mapper;
+ private final String timelineServerHost;
+ private final int timelineServerPort;
+ private final int timeoutSecs;
+
+ public TimelineServerHelper(HoodieWriteConfig writeConfig) {
+ this(writeConfig.getViewStorageConfig().getRemoteViewServerHost(),
+ writeConfig.getViewStorageConfig().getRemoteViewServerPort(),
+ writeConfig.getViewStorageConfig().getRemoteTimelineClientTimeoutSecs());
+ }
+
+ public TimelineServerHelper(String timelineServerHost, int timelineServerPort, int timeoutSecs) {
+ this.mapper = new ObjectMapper();
+ this.timelineServerHost = timelineServerHost;
+ this.timelineServerPort = timelineServerPort;
+ this.timeoutSecs = timeoutSecs;
+ }
+
+ public <T> T executeRequestToTimelineServerWithRetry(String requestPath, Map<String, String> queryParameters,
+ TypeReference reference, RequestMethod method) {
+ int retry = 5;
+ while (--retry >= 0) {
+ long start = System.currentTimeMillis();
+ try {
+ return executeRequestToTimelineServer(requestPath, queryParameters, reference, method);
+ } catch (IOException e) {
+ LOG.warn("Failed to execute request (" + requestPath + ") to timeline server", e);
+ } finally {
+ LOG.info("Execute request : (" + requestPath + "), costs: " + (System.currentTimeMillis() - start) + " ms");
Review Comment:
Yes sure. Reusing the retry config of remote file system view retry to config the retry count. And removed the noisy log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329808388
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/TestWriteBase.java:
##########
@@ -137,14 +140,17 @@ public static TestHarness instance() {
private String lastPending;
private String lastComplete;
+ private HoodieFlinkWriteClient writeClient;
+
public TestHarness preparePipeline(File basePath, Configuration conf) throws Exception {
this.baseFile = basePath;
this.basePath = this.baseFile.getAbsolutePath();
this.conf = conf;
this.pipeline = TestData.getWritePipeline(this.basePath, conf);
// open the function and ingest data
this.pipeline.openFunction();
- this.ckpMetadata = CkpMetadata.getInstance(conf);
+ this.writeClient = FlinkWriteClients.createWriteClient(conf);
+ this.ckpMetadata = CkpMetadataFactory.getCkpMetadata(StreamerUtil.createMetaClient(conf), writeClient.getConfig(), conf);
Review Comment:
Thanks for your suggestion~ Used write config from coordinator to imitate the reality cases, will use timeline server coordinator started now.
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/meta/TestCkpMetadata.java:
##########
@@ -94,9 +107,17 @@ void testBootstrap() throws Exception {
metadata1.getInstantCache(), is(Collections.singletonList("4")));
}
- private CkpMetadata getCkpMetadata(String uniqueId) {
- String basePath = tempFile.getAbsolutePath();
- FileSystem fs = FSUtils.getFs(basePath, HadoopConfigurations.getHadoopConf(new Configuration()));
- return CkpMetadata.getInstance(fs, basePath, uniqueId);
+ protected CkpMetadata getCkpMetadata(String uniqueId) {
+ conf.set(FlinkOptions.WRITE_CLIENT_ID, uniqueId);
+ return CkpMetadataFactory.getCkpMetadata(writeClient.getHoodieTable().getMetaClient(), writeClient.getConfig(), conf);
+ }
Review Comment:
Using a metaclient and a writeConfig to initialize the CkpMetadata is suitable for Coordinator and write functions. Maybe we don't need to add a new constructor for `CkpMetadata` and use the constructor that write functions used when testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1320946482
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -418,6 +418,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.sinceVersion("0.9.0")
.withDocumentation(MarkerType.class);
+ public static final ConfigProperty<Boolean> TIMELINE_SERVER_BASED_FLINK_CKP_METADATA = ConfigProperty
+ .key("hoodie.flink.ckp_metadata.timeline_server_based.enable")
+ .defaultValue(true)
Review Comment:
Can we remove the flink keywork here, do we need an option for it, just add a new URI on the timeline server should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1324012388
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -418,6 +418,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.sinceVersion("0.9.0")
.withDocumentation(MarkerType.class);
+ public static final ConfigProperty<Boolean> TIMELINE_SERVER_BASED_FLINK_CKP_METADATA = ConfigProperty
+ .key("hoodie.flink.ckp_metadata.timeline_server_based.enable")
+ .defaultValue(true)
Review Comment:
@danny0405 It's ready to review now, PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722438104
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* e64dbf44f7883bbdc1cca9dd56094cc871bded6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722479031
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1326770097
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -418,6 +418,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.sinceVersion("0.9.0")
.withDocumentation(MarkerType.class);
+ public static final ConfigProperty<Boolean> INSTANT_STATE_TIMELINE_SERVER_BASED = ConfigProperty
+ .key("hoodie.instant_state.timeline_server_based.enabled")
+ .defaultValue(true)
+ .sinceVersion("1.0.0")
+ .withDocumentation("If enabled, Flink writer get instant state from timeline server rather than requesting DFS directly");
Review Comment:
Remove the Flink keyword.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722408865
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfe38802d7a9a9c5bc677b19ecaefa72d4335553 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957)
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 6bca9994c9dd051503795aa417d3018140140c1a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976)
* e64dbf44f7883bbdc1cca9dd56094cc871bded6e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1720826880
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908)
* 026b65991bcb167f2a361c47fa87c29bc329ef9a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1724755763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* d73e1951739c48af6d0d0387c51c048dfdec9978 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993)
* da4c89e3baf151b650b54d1ad608c7d8fa4c13ca UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329642431
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/meta/TestCkpMetadata.java:
##########
@@ -94,9 +107,17 @@ void testBootstrap() throws Exception {
metadata1.getInstantCache(), is(Collections.singletonList("4")));
}
- private CkpMetadata getCkpMetadata(String uniqueId) {
- String basePath = tempFile.getAbsolutePath();
- FileSystem fs = FSUtils.getFs(basePath, HadoopConfigurations.getHadoopConf(new Configuration()));
- return CkpMetadata.getInstance(fs, basePath, uniqueId);
+ protected CkpMetadata getCkpMetadata(String uniqueId) {
+ conf.set(FlinkOptions.WRITE_CLIENT_ID, uniqueId);
+ return CkpMetadataFactory.getCkpMetadata(writeClient.getHoodieTable().getMetaClient(), writeClient.getConfig(), conf);
+ }
Review Comment:
Create a hoodie table each time is costly.
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/meta/TestCkpMetadata.java:
##########
@@ -94,9 +107,17 @@ void testBootstrap() throws Exception {
metadata1.getInstantCache(), is(Collections.singletonList("4")));
}
- private CkpMetadata getCkpMetadata(String uniqueId) {
- String basePath = tempFile.getAbsolutePath();
- FileSystem fs = FSUtils.getFs(basePath, HadoopConfigurations.getHadoopConf(new Configuration()));
- return CkpMetadata.getInstance(fs, basePath, uniqueId);
+ protected CkpMetadata getCkpMetadata(String uniqueId) {
+ conf.set(FlinkOptions.WRITE_CLIENT_ID, uniqueId);
+ return CkpMetadataFactory.getCkpMetadata(writeClient.getHoodieTable().getMetaClient(), writeClient.getConfig(), conf);
+ }
+
+ @AfterEach
+ public void cleanup() {
+ if (writeClient != null) {
+ writeClient.close();
+ writeClient = null;
+ }
}
+
Review Comment:
Unnecessary change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1725362955
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* d9c59809dade4617f208b81d683bf496206cca50 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023)
* 047da0779ab20c6c074285da9cdb0b12c18bf230 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025)
* 427392c696b285afb6d4a093059c2b9a3c190f2a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1730773952
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026",
"triggerID" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046",
"triggerID" : "1727079993",
"triggerType" : "MANUAL"
}, {
"hash" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20086",
"triggerID" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 427392c696b285afb6d4a093059c2b9a3c190f2a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046)
* 4a23e89b908b573b4c01a0406be5a36115ed8b3e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20086)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722516789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* e64dbf44f7883bbdc1cca9dd56094cc871bded6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329478535
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -425,6 +425,18 @@ public class HoodieWriteConfig extends HoodieConfig {
.sinceVersion("0.9.0")
.withDocumentation(MarkerType.class);
+ public static final ConfigProperty<Boolean> INSTANT_STATE_TIMELINE_SERVER_BASED = ConfigProperty
+ .key("hoodie.instant_state.timeline_server_based.enabled")
+ .defaultValue(true)
+ .sinceVersion("1.0.0")
+ .withDocumentation("If enabled, writers get instant state from timeline server rather than requesting DFS directly");
+
+ public static final ConfigProperty<Integer> INSTANT_STATE_TIMELINE_SERVER_BASED_REFRESH_THRESHOLD = ConfigProperty
+ .key("hoodie.instant_state.timeline_server_based.refresh.threshold")
+ .defaultValue(100)
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329670934
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -160,7 +171,7 @@ private boolean isLocalViewBehind(Context ctx) {
if (LOG.isDebugEnabled()) {
LOG.debug("Client [ LastTs=" + lastKnownInstantFromClient + ", TimelineHash=" + timelineHashFromClient
+ "], localTimeline=" + localTimeline.getInstants());
- }
+ }
Review Comment:
Modified this when formatting code and it's OK to keep it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1719025629
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ee73069669c2da523179f15a3d9119a34fcaaf0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904)
* c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1324112506
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/CkpMetadata.java:
##########
@@ -20,14 +20,22 @@
import org.apache.hudi.common.fs.FSUtils;
import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.dto.CkpMetadataDTO;
Review Comment:
And do we have some fallback logic when the HTTP request timed out or throws ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1717039772
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852)
* 91b363807661444af082389b9979ea59b5be4972 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1718716775
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91b363807661444af082389b9979ea59b5be4972 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883)
* 0ee73069669c2da523179f15a3d9119a34fcaaf0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1721052739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 026b65991bcb167f2a361c47fa87c29bc329ef9a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947)
* bfe38802d7a9a9c5bc677b19ecaefa72d4335553 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1328394158
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -505,6 +516,24 @@ private void registerMarkerAPI() {
}, false));
}
+ private void registerCkpMetadataAPI() {
+ app.get(InstantStateHandler.ALL_INSTANT_STATE_URL, new ViewHandler(ctx -> {
Review Comment:
Fix the method name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722988989
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1340795555
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/InstantStateHandler.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.timeline.service.handlers;
+
+import org.apache.hudi.common.table.timeline.dto.InstantStateDTO;
+import org.apache.hudi.common.table.view.FileSystemViewManager;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.timeline.service.TimelineService;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+
+/**
+ * REST Handler servicing instant state requests.
+ * <p>
+ * The instant states are cached in timeline server and will be refreshed after the instant states in file system were changed.
+ */
+public class InstantStateHandler extends Handler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(InstantStateHandler.class);
+
+ /**
+ * Base url for instant state requests.
+ */
+ private static final String BASE_URL = "/v1/hoodie/instantstate";
+
+ /**
+ * Param for instant state requests, which contains a uniqueId for different writers.
+ */
+ public static final String INSTANT_STATE_DIR_PATH_PARAM = "instantstatedirpath";
+
+ /**
+ * GET requests. Returns all the instant states under instant state path.
+ */
+ public static final String ALL_INSTANT_STATE_URL = String.format("%s/%s", BASE_URL, "all");
+
+ /**
+ * POST requests. Refresh the instant state data cached in memory.
+ */
+ public static final String REFRESH_INSTANT_STATE = String.format("%s/%s", BASE_URL, "refresh/");
+
+ /**
+ * Cached instant state data, instant state path -> list of instant states in fs.
+ */
+ private final ConcurrentHashMap<String, List<InstantStateDTO>> cachedInstantStates;
+
+ /**
+ * Number of requests after the last refresh.
+ */
+ private final AtomicLong requestCount;
+
+ public InstantStateHandler(Configuration conf, TimelineService.Config timelineServiceConfig, FileSystem fileSystem,
+ FileSystemViewManager viewManager) throws IOException {
+ super(conf, timelineServiceConfig, fileSystem, viewManager);
+ this.cachedInstantStates = new ConcurrentHashMap<>();
+ this.requestCount = new AtomicLong();
+ }
+
+ /**
+ * Read instant states from cache of file system.
+ *
+ * @return Instant states under the input instant state path.
+ */
+ public List<InstantStateDTO> getAllInstantStates(String instantStatePath) {
+ if (requestCount.incrementAndGet() >= timelineServiceConfig.instantStateForceRefreshRequestNumber) {
+ // Do refresh for every N requests to ensure the writers won't be blocked forever
+ refresh(instantStatePath);
+ }
+ return cachedInstantStates.computeIfAbsent(instantStatePath, k -> scanInstantState(new Path(k)));
+ }
+
+ /**
+ * Refresh the checkpoint messages cached. Will be called when coordinator start/commit/abort instant.
+ *
+ * @return Whether refreshing is successful.
+ */
+ public boolean refresh(String instantStatePath) {
+ try {
+ cachedInstantStates.put(instantStatePath, scanInstantState(new Path(instantStatePath)));
+ requestCount.set(0);
+ } catch (Exception e) {
+ LOG.error("Failed to load instant states, path: " + instantStatePath, e);
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Scan the instant states from file system.
+ */
+ public List<InstantStateDTO> scanInstantState(Path instantStatePath) {
+ try {
+ // Check instantStatePath exists before list status, see HUDI-5915
+ if (this.fileSystem.exists(instantStatePath)) {
+ return Arrays.stream(this.fileSystem.listStatus(instantStatePath)).map(InstantStateDTO::fromFileStatus).collect(Collectors.toList());
Review Comment:
Please fire a JIRA to track this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722687782
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* e64dbf44f7883bbdc1cca9dd56094cc871bded6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978)
* d1afbf2d4c5d8558caa1574f7c8bc688b39cca93 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722847513
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722923278
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722747446
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* e64dbf44f7883bbdc1cca9dd56094cc871bded6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978)
* d1afbf2d4c5d8558caa1574f7c8bc688b39cca93 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982)
* 71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1727079993
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1333848877
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriteFunction.java:
##########
@@ -113,7 +116,8 @@ public BulkInsertWriteFunction(Configuration config, RowType rowType) {
public void open(Configuration parameters) throws IOException {
this.taskID = getRuntimeContext().getIndexOfThisSubtask();
this.writeClient = FlinkWriteClients.createWriteClient(this.config, getRuntimeContext());
- this.ckpMetadata = CkpMetadata.getInstance(config);
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(this.config);
+ this.ckpMetadata = CkpMetadataFactory.getCkpMetadata(metaClient, writeClient.getConfig(), config);
Review Comment:
Yes, we can create the `CkpMetadata` only by `HoodieWriteConfig ` and `Configuration `. I've updated the code as you suggested.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/CkpMetadata.java:
##########
@@ -208,21 +200,6 @@ public List<String> getInstantCache() {
return this.instantCache;
}
- // -------------------------------------------------------------------------
- // Utilities
- // -------------------------------------------------------------------------
Review Comment:
done
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/CkpMetadata.java:
##########
@@ -249,4 +230,5 @@ private List<CkpMessage> scanCkpMetadata(Path ckpMetaPath) throws IOException {
}).get())
.sorted().collect(Collectors.toList());
}
+
Review Comment:
done
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/meta/TestCkpMetadata.java:
##########
@@ -49,11 +53,24 @@ public class TestCkpMetadata {
@TempDir
File tempFile;
+ protected Configuration conf;
+
+ protected HoodieFlinkWriteClient writeClient;
+
+ protected HoodieTableMetaClient metaClient;
+
@BeforeEach
public void beforeEach() throws Exception {
+ setup();
+ }
+
+ protected void setup() throws IOException {
String basePath = tempFile.getAbsolutePath();
- Configuration conf = TestConfigurations.getDefaultConf(basePath);
+ this.conf = TestConfigurations.getDefaultConf(basePath);
+ conf.setString(HoodieWriteConfig.INSTANT_STATE_TIMELINE_SERVER_BASED.key(), "false");
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329716979
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/TestWriteBase.java:
##########
@@ -137,14 +140,17 @@ public static TestHarness instance() {
private String lastPending;
private String lastComplete;
+ private HoodieFlinkWriteClient writeClient;
+
public TestHarness preparePipeline(File basePath, Configuration conf) throws Exception {
this.baseFile = basePath;
this.basePath = this.baseFile.getAbsolutePath();
this.conf = conf;
this.pipeline = TestData.getWritePipeline(this.basePath, conf);
// open the function and ingest data
this.pipeline.openFunction();
- this.ckpMetadata = CkpMetadata.getInstance(conf);
+ this.writeClient = FlinkWriteClients.createWriteClient(conf);
+ this.ckpMetadata = CkpMetadataFactory.getCkpMetadata(StreamerUtil.createMetaClient(conf), writeClient.getConfig(), conf);
Review Comment:
`checkInflightInstant` and `lastPendingInstant` should request as a client to the server started by the coordinator, you can not set another server there and request from it, it works, but the workflow is not in line with the reality cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1711587412
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2acae42a85a382a3c1b30d4aee7f2f70f35d7b10 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1717094386
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852)
* 91b363807661444af082389b9979ea59b5be4972 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1324109341
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/CkpMetadata.java:
##########
@@ -20,14 +20,22 @@
import org.apache.hudi.common.fs.FSUtils;
import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.dto.CkpMetadataDTO;
Review Comment:
Can we extend a sub-class instead of messing up the existing logic then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1325313091
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/CkpMetadata.java:
##########
@@ -20,14 +20,22 @@
import org.apache.hudi.common.fs.FSUtils;
import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.dto.CkpMetadataDTO;
Review Comment:
> Can we extend a sub-class instead of messing up the existing logic then?
Yes sure, already moved to a sub class. And I have refactor the constructors of `CkpMetadata`, now we have only one entry for creating ckpmetadata.
> And do we have some fallback logic when the HTTP request timed out or throws ?
Yes, when requesting timeline server failed we will read from file system directly. Added tests about this.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -418,6 +418,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.sinceVersion("0.9.0")
.withDocumentation(MarkerType.class);
+ public static final ConfigProperty<Boolean> TIMELINE_SERVER_BASED_FLINK_CKP_METADATA = ConfigProperty
+ .key("hoodie.flink.ckp_metadata.timeline_server_based.enable")
+ .defaultValue(true)
Review Comment:
> hoodie.ckp_metadata.timeline_server_based.enable -> hoodie.instant_state.timeline_server_based.enabled
done
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -505,6 +516,24 @@ private void registerMarkerAPI() {
}, false));
}
+ private void registerFlinkCkpMetadataAPI() {
+ app.get(FlinkCkpMetadataHandler.ALL_CKP_METADATA_URL, new ViewHandler(ctx -> {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1713109095
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2acae42a85a382a3c1b30d4aee7f2f70f35d7b10 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729)
* 7d5ae2f19b99845990588ccfd74bba3b63bb75c0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1715357576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d5ae2f19b99845990588ccfd74bba3b63bb75c0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787)
* 2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329645320
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/TestWriteBase.java:
##########
@@ -137,14 +140,17 @@ public static TestHarness instance() {
private String lastPending;
private String lastComplete;
+ private HoodieFlinkWriteClient writeClient;
+
public TestHarness preparePipeline(File basePath, Configuration conf) throws Exception {
this.baseFile = basePath;
this.basePath = this.baseFile.getAbsolutePath();
this.conf = conf;
this.pipeline = TestData.getWritePipeline(this.basePath, conf);
// open the function and ingest data
this.pipeline.openFunction();
- this.ckpMetadata = CkpMetadata.getInstance(conf);
+ this.writeClient = FlinkWriteClients.createWriteClient(conf);
+ this.ckpMetadata = CkpMetadataFactory.getCkpMetadata(StreamerUtil.createMetaClient(conf), writeClient.getConfig(), conf);
Review Comment:
Not sure what is the purpose of the write config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329717475
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/meta/TestCkpMetadata.java:
##########
@@ -94,9 +107,17 @@ void testBootstrap() throws Exception {
metadata1.getInstantCache(), is(Collections.singletonList("4")));
}
- private CkpMetadata getCkpMetadata(String uniqueId) {
- String basePath = tempFile.getAbsolutePath();
- FileSystem fs = FSUtils.getFs(basePath, HadoopConfigurations.getHadoopConf(new Configuration()));
- return CkpMetadata.getInstance(fs, basePath, uniqueId);
+ protected CkpMetadata getCkpMetadata(String uniqueId) {
+ conf.set(FlinkOptions.WRITE_CLIENT_ID, uniqueId);
+ return CkpMetadataFactory.getCkpMetadata(writeClient.getHoodieTable().getMetaClient(), writeClient.getConfig(), conf);
+ }
Review Comment:
Maybe just pass around the write config should be enough,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1725185249
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 0acc78ec78cc1d99a62a2d8c7514110a56deadd1 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018)
* d9c59809dade4617f208b81d683bf496206cca50 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023)
* 047da0779ab20c6c074285da9cdb0b12c18bf230 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1724867600
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* da4c89e3baf151b650b54d1ad608c7d8fa4c13ca Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 closed pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 closed pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
URL: https://github.com/apache/hudi/pull/9651
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1323960307
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -418,6 +418,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.sinceVersion("0.9.0")
.withDocumentation(MarkerType.class);
+ public static final ConfigProperty<Boolean> TIMELINE_SERVER_BASED_FLINK_CKP_METADATA = ConfigProperty
+ .key("hoodie.flink.ckp_metadata.timeline_server_based.enable")
+ .defaultValue(true)
Review Comment:
Ping me again if it is in good shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1718722485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91b363807661444af082389b9979ea59b5be4972 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883)
* 0ee73069669c2da523179f15a3d9119a34fcaaf0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1320946482
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -418,6 +418,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.sinceVersion("0.9.0")
.withDocumentation(MarkerType.class);
+ public static final ConfigProperty<Boolean> TIMELINE_SERVER_BASED_FLINK_CKP_METADATA = ConfigProperty
+ .key("hoodie.flink.ckp_metadata.timeline_server_based.enable")
+ .defaultValue(true)
Review Comment:
Can we remove the flink keywork here, do we need an option for it, just add a new URI on the timeline server should be fine. The URI is used for fetching the current pending instant, but you need to handle the diffferent request from various clients in good manner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1711221120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2acae42a85a382a3c1b30d4aee7f2f70f35d7b10 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1732506934
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026",
"triggerID" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046",
"triggerID" : "1727079993",
"triggerType" : "MANUAL"
}, {
"hash" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20086",
"triggerID" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"triggerType" : "PUSH"
}, {
"hash" : "9c30218283e77bc1df4bc88f230085156c81a204",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20090",
"triggerID" : "9c30218283e77bc1df4bc88f230085156c81a204",
"triggerType" : "PUSH"
}, {
"hash" : "5c93e9909dc30664db9417fccd2f99ef5a23d528",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20117",
"triggerID" : "5c93e9909dc30664db9417fccd2f99ef5a23d528",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 9c30218283e77bc1df4bc88f230085156c81a204 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20090)
* 5c93e9909dc30664db9417fccd2f99ef5a23d528 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20117)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1338115806
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/InstantStateHandler.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.timeline.service.handlers;
+
+import org.apache.hudi.common.table.timeline.dto.InstantStateDTO;
+import org.apache.hudi.common.table.view.FileSystemViewManager;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.timeline.service.TimelineService;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+
+/**
+ * REST Handler servicing instant state requests.
+ * <p>
+ * The instant states are cached in timeline server and will be refreshed after the instant states in file system were changed.
+ */
+public class InstantStateHandler extends Handler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(InstantStateHandler.class);
+
+ /**
+ * Base url for instant state requests.
+ */
+ private static final String BASE_URL = "/v1/hoodie/instantstate";
+
+ /**
+ * Param for instant state requests, which contains a uniqueId for different writers.
+ */
+ public static final String INSTANT_STATE_DIR_PATH_PARAM = "instantstatedirpath";
+
+ /**
+ * GET requests. Returns all the instant states under instant state path.
+ */
+ public static final String ALL_INSTANT_STATE_URL = String.format("%s/%s", BASE_URL, "all");
+
+ /**
+ * POST requests. Refresh the instant state data cached in memory.
+ */
+ public static final String REFRESH_INSTANT_STATE = String.format("%s/%s", BASE_URL, "refresh/");
+
+ /**
+ * Cached instant state data, instant state path -> list of instant states in fs.
+ */
+ private final ConcurrentHashMap<String, List<InstantStateDTO>> cachedInstantStates;
+
+ /**
+ * Number of requests after the last refresh.
+ */
+ private final AtomicLong requestCount;
+
+ public InstantStateHandler(Configuration conf, TimelineService.Config timelineServiceConfig, FileSystem fileSystem,
+ FileSystemViewManager viewManager) throws IOException {
+ super(conf, timelineServiceConfig, fileSystem, viewManager);
+ this.cachedInstantStates = new ConcurrentHashMap<>();
+ this.requestCount = new AtomicLong();
+ }
+
+ /**
+ * Read instant states from cache of file system.
+ *
+ * @return Instant states under the input instant state path.
+ */
+ public List<InstantStateDTO> getAllInstantStates(String instantStatePath) {
+ if (requestCount.incrementAndGet() >= timelineServiceConfig.instantStateForceRefreshRequestNumber) {
+ // Do refresh for every N requests to ensure the writers won't be blocked forever
+ refresh(instantStatePath);
+ }
+ return cachedInstantStates.computeIfAbsent(instantStatePath, k -> scanInstantState(new Path(k)));
+ }
+
+ /**
+ * Refresh the checkpoint messages cached. Will be called when coordinator start/commit/abort instant.
+ *
+ * @return Whether refreshing is successful.
+ */
+ public boolean refresh(String instantStatePath) {
+ try {
+ cachedInstantStates.put(instantStatePath, scanInstantState(new Path(instantStatePath)));
+ requestCount.set(0);
+ } catch (Exception e) {
+ LOG.error("Failed to load instant states, path: " + instantStatePath, e);
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Scan the instant states from file system.
+ */
+ public List<InstantStateDTO> scanInstantState(Path instantStatePath) {
+ try {
+ // Check instantStatePath exists before list status, see HUDI-5915
+ if (this.fileSystem.exists(instantStatePath)) {
+ return Arrays.stream(this.fileSystem.listStatus(instantStatePath)).map(InstantStateDTO::fromFileStatus).collect(Collectors.toList());
Review Comment:
You are right, we can implement a complete timeline server memory-based checkpoint metadata since we will delete all previous ckp metadata when bootstrapping(so we do not actually rely on a durable filesystem state). And I think the timeline server is actually stable enough to serve the checkpoint message requests.
For now, I think it's OK to keep the filesystem-backed checkpoint messages and make the timeline server a stateless reader for the following reasons:
1. We will have a safe fallback option when the timeline server is not available during runtime.
2. The filesystem-backed checkpoint code is still useful since we need to turn off the timeline server sometimes. And creating ckp messages files is not expensive, we can reuse the code.
3. There will be no conflict if we want to move the start/abort/commit request to the timeline server in the future, and this PR has already implemented the important optimization that reduces the massive file listing. Moving the write part to the timeline server needs extra code and some more careful tests, we can move on this PR.
We can merge this PR and create a Jira to track any possible optimization, what do you think? And thanks again for your review~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1338025400
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/InstantStateHandler.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.timeline.service.handlers;
+
+import org.apache.hudi.common.table.timeline.dto.InstantStateDTO;
+import org.apache.hudi.common.table.view.FileSystemViewManager;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.timeline.service.TimelineService;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+
+/**
+ * REST Handler servicing instant state requests.
+ * <p>
+ * The instant states are cached in timeline server and will be refreshed after the instant states in file system were changed.
+ */
+public class InstantStateHandler extends Handler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(InstantStateHandler.class);
+
+ /**
+ * Base url for instant state requests.
+ */
+ private static final String BASE_URL = "/v1/hoodie/instantstate";
+
+ /**
+ * Param for instant state requests, which contains a uniqueId for different writers.
+ */
+ public static final String INSTANT_STATE_DIR_PATH_PARAM = "instantstatedirpath";
+
+ /**
+ * GET requests. Returns all the instant states under instant state path.
+ */
+ public static final String ALL_INSTANT_STATE_URL = String.format("%s/%s", BASE_URL, "all");
+
+ /**
+ * POST requests. Refresh the instant state data cached in memory.
+ */
+ public static final String REFRESH_INSTANT_STATE = String.format("%s/%s", BASE_URL, "refresh/");
+
+ /**
+ * Cached instant state data, instant state path -> list of instant states in fs.
+ */
+ private final ConcurrentHashMap<String, List<InstantStateDTO>> cachedInstantStates;
+
+ /**
+ * Number of requests after the last refresh.
+ */
+ private final AtomicLong requestCount;
+
+ public InstantStateHandler(Configuration conf, TimelineService.Config timelineServiceConfig, FileSystem fileSystem,
+ FileSystemViewManager viewManager) throws IOException {
+ super(conf, timelineServiceConfig, fileSystem, viewManager);
+ this.cachedInstantStates = new ConcurrentHashMap<>();
+ this.requestCount = new AtomicLong();
+ }
+
+ /**
+ * Read instant states from cache of file system.
+ *
+ * @return Instant states under the input instant state path.
+ */
+ public List<InstantStateDTO> getAllInstantStates(String instantStatePath) {
+ if (requestCount.incrementAndGet() >= timelineServiceConfig.instantStateForceRefreshRequestNumber) {
+ // Do refresh for every N requests to ensure the writers won't be blocked forever
+ refresh(instantStatePath);
+ }
+ return cachedInstantStates.computeIfAbsent(instantStatePath, k -> scanInstantState(new Path(k)));
+ }
+
+ /**
+ * Refresh the checkpoint messages cached. Will be called when coordinator start/commit/abort instant.
+ *
+ * @return Whether refreshing is successful.
+ */
+ public boolean refresh(String instantStatePath) {
+ try {
+ cachedInstantStates.put(instantStatePath, scanInstantState(new Path(instantStatePath)));
+ requestCount.set(0);
+ } catch (Exception e) {
+ LOG.error("Failed to load instant states, path: " + instantStatePath, e);
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Scan the instant states from file system.
+ */
+ public List<InstantStateDTO> scanInstantState(Path instantStatePath) {
+ try {
+ // Check instantStatePath exists before list status, see HUDI-5915
+ if (this.fileSystem.exists(instantStatePath)) {
+ return Arrays.stream(this.fileSystem.listStatus(instantStatePath)).map(InstantStateDTO::fromFileStatus).collect(Collectors.toList());
Review Comment:
So the server based ckp metadata is still backed by the fs ckp message files ? Can you eliminate that part, just accepts the request from coordinator then stores the instant metadata in memory should be enought ? There is no need to write into the fs anymore I think. Or do you think the http server is not stable enough so it will ack with bad response frequently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1319439232
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -418,6 +418,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.sinceVersion("0.9.0")
.withDocumentation(MarkerType.class);
+ public static final ConfigProperty<Boolean> TIMELINE_SERVER_BASED_FLINK_CKP_METADATA = ConfigProperty
Review Comment:
make it defaults to true to run IT and UT tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1713249854
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d5ae2f19b99845990588ccfd74bba3b63bb75c0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1321266607
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -418,6 +418,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.sinceVersion("0.9.0")
.withDocumentation(MarkerType.class);
+ public static final ConfigProperty<Boolean> TIMELINE_SERVER_BASED_FLINK_CKP_METADATA = ConfigProperty
+ .key("hoodie.flink.ckp_metadata.timeline_server_based.enable")
+ .defaultValue(true)
Review Comment:
Thanks for your review~
> Can we remove the flink keywork here
Removing the `flink` keyword is OK.
> do we need an option for it, just add a new URI on the timeline server should be fine.
I'm mainly following the implementation of timeline-based markers which have an option for enabling timeline-based markers or not, too.
> you need to handle the diffferent request from various clients in good manner.
The parameter of Ckp metadata Handler is the full ckp metadata path, which already contains the uniqueId of one specific job. I think it's enough for the multi-client scenario, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1719095391
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ee73069669c2da523179f15a3d9119a34fcaaf0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904)
* c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1328225735
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/dto/InstantStateDTO.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline.dto;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.hadoop.fs.FileStatus;
+
+/**
+ * Data transfer object for instant state.
+ *
+ * see org.apache.hudi.sink.meta.CkpMessage.
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class InstantStateDTO {
+
+ /**
+ * Ckp message instant.
Review Comment:
Fix the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722415744
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 6bca9994c9dd051503795aa417d3018140140c1a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976)
* e64dbf44f7883bbdc1cca9dd56094cc871bded6e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1731373459
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026",
"triggerID" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046",
"triggerID" : "1727079993",
"triggerType" : "MANUAL"
}, {
"hash" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20086",
"triggerID" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"triggerType" : "PUSH"
}, {
"hash" : "9c30218283e77bc1df4bc88f230085156c81a204",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c30218283e77bc1df4bc88f230085156c81a204",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 4a23e89b908b573b4c01a0406be5a36115ed8b3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20086)
* 9c30218283e77bc1df4bc88f230085156c81a204 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1732505592
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026",
"triggerID" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046",
"triggerID" : "1727079993",
"triggerType" : "MANUAL"
}, {
"hash" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20086",
"triggerID" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"triggerType" : "PUSH"
}, {
"hash" : "9c30218283e77bc1df4bc88f230085156c81a204",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20090",
"triggerID" : "9c30218283e77bc1df4bc88f230085156c81a204",
"triggerType" : "PUSH"
}, {
"hash" : "5c93e9909dc30664db9417fccd2f99ef5a23d528",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5c93e9909dc30664db9417fccd2f99ef5a23d528",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 9c30218283e77bc1df4bc88f230085156c81a204 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20090)
* 5c93e9909dc30664db9417fccd2f99ef5a23d528 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1732535924
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026",
"triggerID" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046",
"triggerID" : "1727079993",
"triggerType" : "MANUAL"
}, {
"hash" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20086",
"triggerID" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"triggerType" : "PUSH"
}, {
"hash" : "9c30218283e77bc1df4bc88f230085156c81a204",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20090",
"triggerID" : "9c30218283e77bc1df4bc88f230085156c81a204",
"triggerType" : "PUSH"
}, {
"hash" : "5c93e9909dc30664db9417fccd2f99ef5a23d528",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20117",
"triggerID" : "5c93e9909dc30664db9417fccd2f99ef5a23d528",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 5c93e9909dc30664db9417fccd2f99ef5a23d528 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20117)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329453699
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/TimelineBasedCkpMetadata.java:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.meta;
+
+import org.apache.hudi.common.table.timeline.dto.InstantStateDTO;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.timeline.service.handlers.InstantStateHandler;
+import org.apache.hudi.util.HttpRequestClient;
+import org.apache.hudi.util.HttpRequestClient.RequestMethod;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Stream;
+
+/**
+ * Timeline server based CkpMetadata, will read ckpMessages from timeline-server instead of from file system directly.
+ */
+public class TimelineBasedCkpMetadata extends CkpMetadata {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TimelineBasedCkpMetadata.class);
+
+ private final HttpRequestClient httpRequestClient;
+
+ public TimelineBasedCkpMetadata(FileSystem fs, String basePath, String uniqueId, HoodieWriteConfig writeConfig) {
+ super(fs, basePath, uniqueId);
+ this.httpRequestClient = new HttpRequestClient(writeConfig);
+ LOG.info("Timeline server based CkpMetadata enabled");
+ }
+
+ @Override
+ public void startInstant(String instant) {
+ super.startInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void commitInstant(String instant) {
+ super.commitInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void abortInstant(String instant) {
+ super.abortInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ protected Stream<CkpMessage> fetchCkpMessages(Path ckpMetaPath) throws IOException {
+ // Read ckp messages from timeline server
+ Stream<CkpMessage> ckpMessageStream;
+ try {
+ List<InstantStateDTO> instantStateDTOList = httpRequestClient.executeRequestWithRetry(
+ InstantStateHandler.ALL_INSTANT_STATE_URL, getRequestParams(ckpMetaPath.toString()),
+ new TypeReference<List<InstantStateDTO>>() {
+ }, RequestMethod.GET);
+ ckpMessageStream = instantStateDTOList.stream().map(c -> new CkpMessage(c.getInstant(), c.getState()));
+ } catch (Exception e) {
+ LOG.error("Failed to execute scan ckp metadata, fall back to read from file system...", e);
+ // If we failed to request timeline server, read ckp messages from file system directly.
+ ckpMessageStream = super.fetchCkpMessages(ckpMetaPath);
+ }
+ return ckpMessageStream;
+ }
+
+ private Map<String, String> getRequestParams(String dirPath) {
+ return Collections.singletonMap(InstantStateHandler.INSTANT_STATE_DIR_PATH_PARAM, dirPath);
Review Comment:
Do we need to pass the whole dir path or just the client ID ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329452890
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/CkpMetadata.java:
##########
@@ -233,12 +210,18 @@ private Path fullPath(String fileName) {
return new Path(path, fileName);
}
- private List<CkpMessage> scanCkpMetadata(Path ckpMetaPath) throws IOException {
+ protected Stream<CkpMessage> fetchCkpMessages(Path ckpMetaPath) throws IOException {
// This is required when the storage is minio
if (!this.fs.exists(ckpMetaPath)) {
- return new ArrayList<>();
+ return Stream.empty();
}
- return Arrays.stream(this.fs.listStatus(ckpMetaPath)).map(CkpMessage::new)
+ Stream<CkpMessage> ckpMessageStream;
+ ckpMessageStream = Arrays.stream(this.fs.listStatus(ckpMetaPath)).map(CkpMessage::new);
+ return ckpMessageStream;
Review Comment:
Can we inline `ckpMessageStream` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329606169
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/embedded/EmbeddedTimelineService.java:
##########
@@ -100,6 +100,12 @@ public void startServer() throws IOException {
* writeConfig.getHoodieClientHeartbeatTolerableMisses());
}
+ if (writeConfig.isTimelineServerBasedInstantStateEnabled()) {
+ timelineServiceConfBuilder
+ .instantStateRefreshThreshold(writeConfig.getTimelineServerBasedInstantStateForceRefreshRequestNumber())
+ .enableInstantStateRequests(true);
Review Comment:
`instantStateRefreshThreshold` also needs a rename.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1332367160
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/CkpMetadata.java:
##########
@@ -208,21 +200,6 @@ public List<String> getInstantCache() {
return this.instantCache;
}
- // -------------------------------------------------------------------------
- // Utilities
- // -------------------------------------------------------------------------
Review Comment:
Recover the comments `Utilities`
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/meta/TestCkpMetadata.java:
##########
@@ -49,11 +53,24 @@ public class TestCkpMetadata {
@TempDir
File tempFile;
+ protected Configuration conf;
+
+ protected HoodieFlinkWriteClient writeClient;
+
+ protected HoodieTableMetaClient metaClient;
+
@BeforeEach
public void beforeEach() throws Exception {
+ setup();
+ }
+
+ protected void setup() throws IOException {
String basePath = tempFile.getAbsolutePath();
- Configuration conf = TestConfigurations.getDefaultConf(basePath);
+ this.conf = TestConfigurations.getDefaultConf(basePath);
+ conf.setString(HoodieWriteConfig.INSTANT_STATE_TIMELINE_SERVER_BASED.key(), "false");
Review Comment:
The default value should be false.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/CkpMetadata.java:
##########
@@ -249,4 +230,5 @@ private List<CkpMessage> scanCkpMetadata(Path ckpMetaPath) throws IOException {
}).get())
.sorted().collect(Collectors.toList());
}
+
Review Comment:
Revert this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1730894298
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026",
"triggerID" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046",
"triggerID" : "1727079993",
"triggerType" : "MANUAL"
}, {
"hash" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20086",
"triggerID" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 4a23e89b908b573b4c01a0406be5a36115ed8b3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20086)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1325398892
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/TimelineBasedCkpMetadata.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.meta;
+
+import org.apache.hudi.common.table.timeline.dto.CkpMetadataDTO;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.timeline.service.handlers.CkpMetadataHandler;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Stream;
+
+/**
+ * Timeline server based CkpMetadata, will read ckpMessages from timeline-server instead of from file system directly.
+ */
+public class TimelineBasedCkpMetadata extends CkpMetadata {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TimelineBasedCkpMetadata.class);
+
+ private final HoodieWriteConfig writeConfig;
+ private ObjectMapper mapper;
+
+ public TimelineBasedCkpMetadata(FileSystem fs, String basePath, String uniqueId, HoodieWriteConfig writeConfig) {
+ super(fs, basePath, uniqueId);
+ this.writeConfig = writeConfig;
+ mapper = new ObjectMapper();
+ LOG.info("Timeline server based CkpMetadata enabled");
+ }
+
+ @Override
+ public void startInstant(String instant) {
+ super.startInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void commitInstant(String instant) {
+ super.commitInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void abortInstant(String instant) {
+ super.abortInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ protected Stream<CkpMessage> readCkpMessages(Path ckpMetaPath) throws IOException {
Review Comment:
fetchCkpMessages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1325396538
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/CkpMetadataFactory.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.meta;
+
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.flink.configuration.Configuration;
+
+/**
+ * A factory to generate {@link CkpMetadata} instance based on whether {@link HoodieWriteConfig#INSTANT_STATE_TIMELINE_SERVER_BASED} enabled.
+ */
+public class CkpMetadataFactory {
+ public static CkpMetadata get(HoodieTable table, Configuration config) {
Review Comment:
get -> getCkpMetadata, looks like there is no need to pass the hoodie table as param, a `HoodieTableMetaClient` should be enough, constructing a HoodieTable on the flay is kind of costly, should avoid to do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1721213785
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfe38802d7a9a9c5bc677b19ecaefa72d4335553 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1326772840
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/TimelineBasedCkpMetadata.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.meta;
+
+import org.apache.hudi.common.table.timeline.dto.InstantStateDTO;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.timeline.service.handlers.InstantStateHandler;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Stream;
+
+/**
+ * Timeline server based CkpMetadata, will read ckpMessages from timeline-server instead of from file system directly.
+ */
+public class TimelineBasedCkpMetadata extends CkpMetadata {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TimelineBasedCkpMetadata.class);
+
+ private final HoodieWriteConfig writeConfig;
+ private final ObjectMapper mapper;
+
+ public TimelineBasedCkpMetadata(FileSystem fs, String basePath, String uniqueId, HoodieWriteConfig writeConfig) {
+ super(fs, basePath, uniqueId);
+ this.writeConfig = writeConfig;
+ mapper = new ObjectMapper();
+ LOG.info("Timeline server based CkpMetadata enabled");
+ }
+
+ @Override
+ public void startInstant(String instant) {
+ super.startInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void commitInstant(String instant) {
+ super.commitInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void abortInstant(String instant) {
+ super.abortInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ protected Stream<CkpMessage> fetchCkpMessages(Path ckpMetaPath) throws IOException {
+ // Read ckp messages from timeline server
+ Stream<CkpMessage> ckpMessageStream;
+ try {
+ List<InstantStateDTO> instantStateDTOList = executeRequestToTimelineServerWithRetry(
+ InstantStateHandler.ALL_INSTANT_STATE_URL, getRequestParams(ckpMetaPath.toString()),
+ new TypeReference<List<InstantStateDTO>>() {
+ }, RequestMethod.GET);
+ ckpMessageStream = instantStateDTOList.stream().map(c -> new CkpMessage(c.getInstant(), c.getState()));
+ } catch (Exception e) {
+ LOG.error("Failed to execute scan ckp metadata, fall back to read from file system...", e);
+ // If we failed to request timeline server, read ckp messages from file system directly.
+ ckpMessageStream = super.fetchCkpMessages(ckpMetaPath);
+ }
+ return ckpMessageStream;
+ }
+
+ private Map<String, String> getRequestParams(String dirPath) {
+ return Collections.singletonMap(InstantStateHandler.INSTANT_STATE_DIR_PATH_PARAM, dirPath);
+ }
+
+ /**
+ * Refresh the ckp messages that cached in timeline server.
+ */
+ private void sendRefreshRequest() {
+ try {
+ boolean success = executeRequestToTimelineServerWithRetry(
+ InstantStateHandler.REFRESH_INSTANT_STATE, getRequestParams(path.toString()),
+ new TypeReference<Boolean>() {
+ }, RequestMethod.POST);
+ if (!success) {
+ LOG.warn("Timeline server responses with failed refresh");
+ }
+ } catch (Exception e) {
+ // Do not propagate the exception because the server will also do auto refresh
+ LOG.error("Failed to execute refresh", e);
+ }
+ }
+
+ private <T> T executeRequestToTimelineServerWithRetry(String requestPath, Map<String, String> queryParameters,
+ TypeReference reference, RequestMethod method) {
+ int retry = 5;
+ while (--retry >= 0) {
+ long start = System.currentTimeMillis();
+ try {
+ return executeRequestToTimelineServer(requestPath, queryParameters, reference, method);
+ } catch (IOException e) {
+ LOG.warn("Failed to execute ckp request (" + requestPath + ") to timeline server", e);
+ } finally {
+ LOG.info("Execute request : (" + requestPath + "), costs: " + (System.currentTimeMillis() - start) + " ms");
+ }
+ }
+ throw new HoodieException("Failed to execute ckp request (" + requestPath + ")");
+ }
+
+ private <T> T executeRequestToTimelineServer(String requestPath, Map<String, String> queryParameters,
+ TypeReference reference, RequestMethod method) throws IOException {
+ URIBuilder builder =
Review Comment:
Can this code be reused?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1326770479
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/dto/InstantStateDTO.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline.dto;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.hadoop.fs.FileStatus;
+
+/**
+ * Data transfer object for instant state.
+ *
+ * see org.apache.hudi.sink.meta.CkpMessage.
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class InstantStateDTO {
+
+ /**
+ * Ckp message instant.
+ *
+ * see org.apache.hudi.sink.meta.CkpMessage#instant
Review Comment:
Use `@see` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722407602
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfe38802d7a9a9c5bc677b19ecaefa72d4335553 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957)
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 6bca9994c9dd051503795aa417d3018140140c1a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1718840542
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ee73069669c2da523179f15a3d9119a34fcaaf0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1327885439
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java:
##########
@@ -349,8 +351,8 @@ private static void initMetadataTable(HoodieFlinkWriteClient<?> writeClient) {
writeClient.initMetadataTable();
}
- private static CkpMetadata initCkpMetadata(HoodieTableMetaClient metaClient, Configuration conf) throws IOException {
- CkpMetadata ckpMetadata = CkpMetadata.getInstance(metaClient, conf.getString(FlinkOptions.WRITE_CLIENT_ID));
+ private static CkpMetadata initCkpMetadata(HoodieTableMetaClient metaClient, HoodieWriteConfig writeConfig, Configuration conf) throws IOException {
+ CkpMetadata ckpMetadata = CkpMetadataFactory.getCkpMetadata(metaClient, writeConfig, conf);
ckpMetadata.bootstrap();
return ckpMetadata;
Review Comment:
So this action must be succeed, or the writers would be blocked infinitely, how could we ensure that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722753132
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* d1afbf2d4c5d8558caa1574f7c8bc688b39cca93 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982)
* 71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1715373582
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d5ae2f19b99845990588ccfd74bba3b63bb75c0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787)
* 2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #9651:
URL: https://github.com/apache/hudi/pull/9651
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1740538797
JIRA fired for future improvement: https://issues.apache.org/jira/browse/HUDI-6906
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1725200068
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* d9c59809dade4617f208b81d683bf496206cca50 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023)
* 047da0779ab20c6c074285da9cdb0b12c18bf230 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1725669733
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026",
"triggerID" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 427392c696b285afb6d4a093059c2b9a3c190f2a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1720837721
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908)
* 026b65991bcb167f2a361c47fa87c29bc329ef9a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1724918931
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* da4c89e3baf151b650b54d1ad608c7d8fa4c13ca Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009)
* 0acc78ec78cc1d99a62a2d8c7514110a56deadd1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329641881
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/meta/TestCkpMetadata.java:
##########
@@ -49,11 +52,21 @@ public class TestCkpMetadata {
@TempDir
File tempFile;
+ protected Configuration conf;
+
+ protected HoodieFlinkWriteClient writeClient;
+
@BeforeEach
public void beforeEach() throws Exception {
+ setup();
+ }
+
+ protected void setup() throws IOException {
String basePath = tempFile.getAbsolutePath();
- Configuration conf = TestConfigurations.getDefaultConf(basePath);
+ this.conf = TestConfigurations.getDefaultConf(basePath);
+ conf.setString(HoodieWriteConfig.INSTANT_STATE_TIMELINE_SERVER_BASED.key(), "false");
StreamerUtil.initTableIfNotExists(conf);
+ this.writeClient = FlinkWriteClients.createWriteClient(conf);
}
Review Comment:
Not sure why we instantiate a server write client here, did you know the code would start an embedded timeline server there? Actually the coordinator would also start an embedded timeline server, where all the requests should be rootted too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1725022812
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 0acc78ec78cc1d99a62a2d8c7514110a56deadd1 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018)
* d9c59809dade4617f208b81d683bf496206cca50 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1325396538
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/CkpMetadataFactory.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.meta;
+
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.flink.configuration.Configuration;
+
+/**
+ * A factory to generate {@link CkpMetadata} instance based on whether {@link HoodieWriteConfig#INSTANT_STATE_TIMELINE_SERVER_BASED} enabled.
+ */
+public class CkpMetadataFactory {
+ public static CkpMetadata get(HoodieTable table, Configuration config) {
Review Comment:
get -> getCkpMetadata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1325392309
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/dto/CkpMetadataDTO.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline.dto;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.hadoop.fs.FileStatus;
+
+/**
+ * Data transfer object for CkpMessage.
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class CkpMetadataDTO {
+
Review Comment:
InstantStateDTO ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1721362417
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1326772399
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/TimelineBasedCkpMetadata.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.meta;
+
+import org.apache.hudi.common.table.timeline.dto.InstantStateDTO;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.timeline.service.handlers.InstantStateHandler;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Stream;
+
+/**
+ * Timeline server based CkpMetadata, will read ckpMessages from timeline-server instead of from file system directly.
+ */
+public class TimelineBasedCkpMetadata extends CkpMetadata {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TimelineBasedCkpMetadata.class);
+
+ private final HoodieWriteConfig writeConfig;
+ private final ObjectMapper mapper;
+
+ public TimelineBasedCkpMetadata(FileSystem fs, String basePath, String uniqueId, HoodieWriteConfig writeConfig) {
+ super(fs, basePath, uniqueId);
+ this.writeConfig = writeConfig;
+ mapper = new ObjectMapper();
+ LOG.info("Timeline server based CkpMetadata enabled");
+ }
+
+ @Override
+ public void startInstant(String instant) {
+ super.startInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void commitInstant(String instant) {
+ super.commitInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void abortInstant(String instant) {
+ super.abortInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ protected Stream<CkpMessage> fetchCkpMessages(Path ckpMetaPath) throws IOException {
+ // Read ckp messages from timeline server
+ Stream<CkpMessage> ckpMessageStream;
+ try {
+ List<InstantStateDTO> instantStateDTOList = executeRequestToTimelineServerWithRetry(
+ InstantStateHandler.ALL_INSTANT_STATE_URL, getRequestParams(ckpMetaPath.toString()),
+ new TypeReference<List<InstantStateDTO>>() {
+ }, RequestMethod.GET);
+ ckpMessageStream = instantStateDTOList.stream().map(c -> new CkpMessage(c.getInstant(), c.getState()));
+ } catch (Exception e) {
+ LOG.error("Failed to execute scan ckp metadata, fall back to read from file system...", e);
+ // If we failed to request timeline server, read ckp messages from file system directly.
+ ckpMessageStream = super.fetchCkpMessages(ckpMetaPath);
+ }
+ return ckpMessageStream;
+ }
+
+ private Map<String, String> getRequestParams(String dirPath) {
+ return Collections.singletonMap(InstantStateHandler.INSTANT_STATE_DIR_PATH_PARAM, dirPath);
+ }
+
+ /**
+ * Refresh the ckp messages that cached in timeline server.
+ */
+ private void sendRefreshRequest() {
+ try {
+ boolean success = executeRequestToTimelineServerWithRetry(
+ InstantStateHandler.REFRESH_INSTANT_STATE, getRequestParams(path.toString()),
+ new TypeReference<Boolean>() {
+ }, RequestMethod.POST);
+ if (!success) {
+ LOG.warn("Timeline server responses with failed refresh");
+ }
+ } catch (Exception e) {
+ // Do not propagate the exception because the server will also do auto refresh
+ LOG.error("Failed to execute refresh", e);
+ }
+ }
+
+ private <T> T executeRequestToTimelineServerWithRetry(String requestPath, Map<String, String> queryParameters,
+ TypeReference reference, RequestMethod method) {
+ int retry = 5;
+ while (--retry >= 0) {
+ long start = System.currentTimeMillis();
+ try {
+ return executeRequestToTimelineServer(requestPath, queryParameters, reference, method);
+ } catch (IOException e) {
+ LOG.warn("Failed to execute ckp request (" + requestPath + ") to timeline server", e);
+ } finally {
+ LOG.info("Execute request : (" + requestPath + "), costs: " + (System.currentTimeMillis() - start) + " ms");
+ }
+ }
+ throw new HoodieException("Failed to execute ckp request (" + requestPath + ")");
Review Comment:
Instead of throwing, can we fallback to file scanning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1326822299
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -418,6 +418,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.sinceVersion("0.9.0")
.withDocumentation(MarkerType.class);
+ public static final ConfigProperty<Boolean> INSTANT_STATE_TIMELINE_SERVER_BASED = ConfigProperty
+ .key("hoodie.instant_state.timeline_server_based.enabled")
+ .defaultValue(true)
+ .sinceVersion("1.0.0")
+ .withDocumentation("If enabled, Flink writer get instant state from timeline server rather than requesting DFS directly");
Review Comment:
done
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/TimelineBasedCkpMetadata.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.meta;
+
+import org.apache.hudi.common.table.timeline.dto.InstantStateDTO;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.timeline.service.handlers.InstantStateHandler;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Stream;
+
+/**
+ * Timeline server based CkpMetadata, will read ckpMessages from timeline-server instead of from file system directly.
+ */
+public class TimelineBasedCkpMetadata extends CkpMetadata {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TimelineBasedCkpMetadata.class);
+
+ private final HoodieWriteConfig writeConfig;
+ private final ObjectMapper mapper;
+
+ public TimelineBasedCkpMetadata(FileSystem fs, String basePath, String uniqueId, HoodieWriteConfig writeConfig) {
+ super(fs, basePath, uniqueId);
+ this.writeConfig = writeConfig;
+ mapper = new ObjectMapper();
+ LOG.info("Timeline server based CkpMetadata enabled");
+ }
+
+ @Override
+ public void startInstant(String instant) {
+ super.startInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void commitInstant(String instant) {
+ super.commitInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void abortInstant(String instant) {
+ super.abortInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ protected Stream<CkpMessage> fetchCkpMessages(Path ckpMetaPath) throws IOException {
+ // Read ckp messages from timeline server
+ Stream<CkpMessage> ckpMessageStream;
+ try {
+ List<InstantStateDTO> instantStateDTOList = executeRequestToTimelineServerWithRetry(
+ InstantStateHandler.ALL_INSTANT_STATE_URL, getRequestParams(ckpMetaPath.toString()),
+ new TypeReference<List<InstantStateDTO>>() {
+ }, RequestMethod.GET);
+ ckpMessageStream = instantStateDTOList.stream().map(c -> new CkpMessage(c.getInstant(), c.getState()));
+ } catch (Exception e) {
+ LOG.error("Failed to execute scan ckp metadata, fall back to read from file system...", e);
+ // If we failed to request timeline server, read ckp messages from file system directly.
+ ckpMessageStream = super.fetchCkpMessages(ckpMetaPath);
+ }
+ return ckpMessageStream;
+ }
+
+ private Map<String, String> getRequestParams(String dirPath) {
+ return Collections.singletonMap(InstantStateHandler.INSTANT_STATE_DIR_PATH_PARAM, dirPath);
+ }
+
+ /**
+ * Refresh the ckp messages that cached in timeline server.
+ */
+ private void sendRefreshRequest() {
+ try {
+ boolean success = executeRequestToTimelineServerWithRetry(
+ InstantStateHandler.REFRESH_INSTANT_STATE, getRequestParams(path.toString()),
+ new TypeReference<Boolean>() {
+ }, RequestMethod.POST);
+ if (!success) {
+ LOG.warn("Timeline server responses with failed refresh");
+ }
+ } catch (Exception e) {
+ // Do not propagate the exception because the server will also do auto refresh
+ LOG.error("Failed to execute refresh", e);
+ }
+ }
+
+ private <T> T executeRequestToTimelineServerWithRetry(String requestPath, Map<String, String> queryParameters,
+ TypeReference reference, RequestMethod method) {
+ int retry = 5;
+ while (--retry >= 0) {
+ long start = System.currentTimeMillis();
+ try {
+ return executeRequestToTimelineServer(requestPath, queryParameters, reference, method);
+ } catch (IOException e) {
+ LOG.warn("Failed to execute ckp request (" + requestPath + ") to timeline server", e);
+ } finally {
+ LOG.info("Execute request : (" + requestPath + "), costs: " + (System.currentTimeMillis() - start) + " ms");
+ }
+ }
+ throw new HoodieException("Failed to execute ckp request (" + requestPath + ")");
Review Comment:
We will catch any exceptions and fallback to file scanning in the caller side.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/TimelineBasedCkpMetadata.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.meta;
+
+import org.apache.hudi.common.table.timeline.dto.InstantStateDTO;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.timeline.service.handlers.InstantStateHandler;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Stream;
+
+/**
+ * Timeline server based CkpMetadata, will read ckpMessages from timeline-server instead of from file system directly.
+ */
+public class TimelineBasedCkpMetadata extends CkpMetadata {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TimelineBasedCkpMetadata.class);
+
+ private final HoodieWriteConfig writeConfig;
+ private final ObjectMapper mapper;
+
+ public TimelineBasedCkpMetadata(FileSystem fs, String basePath, String uniqueId, HoodieWriteConfig writeConfig) {
+ super(fs, basePath, uniqueId);
+ this.writeConfig = writeConfig;
+ mapper = new ObjectMapper();
+ LOG.info("Timeline server based CkpMetadata enabled");
+ }
+
+ @Override
+ public void startInstant(String instant) {
+ super.startInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void commitInstant(String instant) {
+ super.commitInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void abortInstant(String instant) {
+ super.abortInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ protected Stream<CkpMessage> fetchCkpMessages(Path ckpMetaPath) throws IOException {
+ // Read ckp messages from timeline server
+ Stream<CkpMessage> ckpMessageStream;
+ try {
+ List<InstantStateDTO> instantStateDTOList = executeRequestToTimelineServerWithRetry(
+ InstantStateHandler.ALL_INSTANT_STATE_URL, getRequestParams(ckpMetaPath.toString()),
+ new TypeReference<List<InstantStateDTO>>() {
+ }, RequestMethod.GET);
+ ckpMessageStream = instantStateDTOList.stream().map(c -> new CkpMessage(c.getInstant(), c.getState()));
+ } catch (Exception e) {
+ LOG.error("Failed to execute scan ckp metadata, fall back to read from file system...", e);
+ // If we failed to request timeline server, read ckp messages from file system directly.
+ ckpMessageStream = super.fetchCkpMessages(ckpMetaPath);
+ }
+ return ckpMessageStream;
+ }
+
+ private Map<String, String> getRequestParams(String dirPath) {
+ return Collections.singletonMap(InstantStateHandler.INSTANT_STATE_DIR_PATH_PARAM, dirPath);
+ }
+
+ /**
+ * Refresh the ckp messages that cached in timeline server.
+ */
+ private void sendRefreshRequest() {
+ try {
+ boolean success = executeRequestToTimelineServerWithRetry(
+ InstantStateHandler.REFRESH_INSTANT_STATE, getRequestParams(path.toString()),
+ new TypeReference<Boolean>() {
+ }, RequestMethod.POST);
+ if (!success) {
+ LOG.warn("Timeline server responses with failed refresh");
+ }
+ } catch (Exception e) {
+ // Do not propagate the exception because the server will also do auto refresh
+ LOG.error("Failed to execute refresh", e);
+ }
+ }
+
+ private <T> T executeRequestToTimelineServerWithRetry(String requestPath, Map<String, String> queryParameters,
+ TypeReference reference, RequestMethod method) {
+ int retry = 5;
+ while (--retry >= 0) {
+ long start = System.currentTimeMillis();
+ try {
+ return executeRequestToTimelineServer(requestPath, queryParameters, reference, method);
+ } catch (IOException e) {
+ LOG.warn("Failed to execute ckp request (" + requestPath + ") to timeline server", e);
+ } finally {
+ LOG.info("Execute request : (" + requestPath + "), costs: " + (System.currentTimeMillis() - start) + " ms");
+ }
+ }
+ throw new HoodieException("Failed to execute ckp request (" + requestPath + ")");
+ }
+
+ private <T> T executeRequestToTimelineServer(String requestPath, Map<String, String> queryParameters,
+ TypeReference reference, RequestMethod method) throws IOException {
+ URIBuilder builder =
Review Comment:
> Can this code be reused?
Sure. Will reuse the code in `TimelineServerBasedWriteMarkers` and add a new Class `TimelineServerHelper`.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java:
##########
@@ -349,8 +351,8 @@ private static void initMetadataTable(HoodieFlinkWriteClient<?> writeClient) {
writeClient.initMetadataTable();
}
- private static CkpMetadata initCkpMetadata(HoodieTableMetaClient metaClient, Configuration conf) throws IOException {
- CkpMetadata ckpMetadata = CkpMetadata.getInstance(metaClient, conf.getString(FlinkOptions.WRITE_CLIENT_ID));
+ private static CkpMetadata initCkpMetadata(HoodieTableMetaClient metaClient, HoodieWriteConfig writeConfig, Configuration conf) throws IOException {
+ CkpMetadata ckpMetadata = CkpMetadataFactory.getCkpMetadata(metaClient, writeConfig, conf);
ckpMetadata.bootstrap();
return ckpMetadata;
Review Comment:
when `org.apache.hudi.sink.meta.TimelineBasedCkpMetadata#sendRefreshRequest` is called. And it will be called after start/commit/abort instant.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/dto/InstantStateDTO.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline.dto;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.hadoop.fs.FileStatus;
+
+/**
+ * Data transfer object for instant state.
+ *
+ * see org.apache.hudi.sink.meta.CkpMessage.
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class InstantStateDTO {
+
+ /**
+ * Ckp message instant.
+ *
+ * see org.apache.hudi.sink.meta.CkpMessage#instant
Review Comment:
use @see will cause IDE error because we can't find class in the flink package. Should we just keep it, use @see or delete this line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1723270487
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991)
* d73e1951739c48af6d0d0387c51c048dfdec9978 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1727403829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026",
"triggerID" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046",
"triggerID" : "1727079993",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 427392c696b285afb6d4a093059c2b9a3c190f2a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1731388334
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026",
"triggerID" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046",
"triggerID" : "1727079993",
"triggerType" : "MANUAL"
}, {
"hash" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20086",
"triggerID" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"triggerType" : "PUSH"
}, {
"hash" : "9c30218283e77bc1df4bc88f230085156c81a204",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20090",
"triggerID" : "9c30218283e77bc1df4bc88f230085156c81a204",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 4a23e89b908b573b4c01a0406be5a36115ed8b3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20086)
* 9c30218283e77bc1df4bc88f230085156c81a204 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20090)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329481636
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/TimelineBasedCkpMetadata.java:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.meta;
+
+import org.apache.hudi.common.table.timeline.dto.InstantStateDTO;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.timeline.service.handlers.InstantStateHandler;
+import org.apache.hudi.util.HttpRequestClient;
+import org.apache.hudi.util.HttpRequestClient.RequestMethod;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Stream;
+
+/**
+ * Timeline server based CkpMetadata, will read ckpMessages from timeline-server instead of from file system directly.
+ */
+public class TimelineBasedCkpMetadata extends CkpMetadata {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TimelineBasedCkpMetadata.class);
+
+ private final HttpRequestClient httpRequestClient;
+
+ public TimelineBasedCkpMetadata(FileSystem fs, String basePath, String uniqueId, HoodieWriteConfig writeConfig) {
+ super(fs, basePath, uniqueId);
+ this.httpRequestClient = new HttpRequestClient(writeConfig);
+ LOG.info("Timeline server based CkpMetadata enabled");
+ }
+
+ @Override
+ public void startInstant(String instant) {
+ super.startInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void commitInstant(String instant) {
+ super.commitInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void abortInstant(String instant) {
+ super.abortInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ protected Stream<CkpMessage> fetchCkpMessages(Path ckpMetaPath) throws IOException {
+ // Read ckp messages from timeline server
+ Stream<CkpMessage> ckpMessageStream;
+ try {
+ List<InstantStateDTO> instantStateDTOList = httpRequestClient.executeRequestWithRetry(
+ InstantStateHandler.ALL_INSTANT_STATE_URL, getRequestParams(ckpMetaPath.toString()),
+ new TypeReference<List<InstantStateDTO>>() {
+ }, RequestMethod.GET);
+ ckpMessageStream = instantStateDTOList.stream().map(c -> new CkpMessage(c.getInstant(), c.getState()));
+ } catch (Exception e) {
+ LOG.error("Failed to execute scan ckp metadata, fall back to read from file system...", e);
+ // If we failed to request timeline server, read ckp messages from file system directly.
+ ckpMessageStream = super.fetchCkpMessages(ckpMetaPath);
+ }
+ return ckpMessageStream;
+ }
+
+ private Map<String, String> getRequestParams(String dirPath) {
+ return Collections.singletonMap(InstantStateHandler.INSTANT_STATE_DIR_PATH_PARAM, dirPath);
Review Comment:
We can make the timeline server as stateless as possible so I think it's better to pass the whole dir. Otherwise we may need a basepath information in the `InstantStateHandler`. The `MarkerHanlder` also use full path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722712005
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* e64dbf44f7883bbdc1cca9dd56094cc871bded6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978)
* d1afbf2d4c5d8558caa1574f7c8bc688b39cca93 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329452223
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -425,6 +425,18 @@ public class HoodieWriteConfig extends HoodieConfig {
.sinceVersion("0.9.0")
.withDocumentation(MarkerType.class);
+ public static final ConfigProperty<Boolean> INSTANT_STATE_TIMELINE_SERVER_BASED = ConfigProperty
+ .key("hoodie.instant_state.timeline_server_based.enabled")
+ .defaultValue(true)
+ .sinceVersion("1.0.0")
+ .withDocumentation("If enabled, writers get instant state from timeline server rather than requesting DFS directly");
+
+ public static final ConfigProperty<Integer> INSTANT_STATE_TIMELINE_SERVER_BASED_REFRESH_THRESHOLD = ConfigProperty
+ .key("hoodie.instant_state.timeline_server_based.refresh.threshold")
+ .defaultValue(100)
Review Comment:
hoodie.instant_state.timeline_server_based.refresh.threshold
-> hoodie.instant_state.timeline_server_based.force_refresh.request.number ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1724760758
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* d73e1951739c48af6d0d0387c51c048dfdec9978 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993)
* da4c89e3baf151b650b54d1ad608c7d8fa4c13ca Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1721626451
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* bfe38802d7a9a9c5bc677b19ecaefa72d4335553 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1328184747
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/util/TimelineServerHelper.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.util;
+
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+
+/**
+ * Helper class for executing timeline server requests.
+ */
+public class TimelineServerHelper {
Review Comment:
Rename to `HttpRequestClient`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1720995078
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908)
* 026b65991bcb167f2a361c47fa87c29bc329ef9a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947)
* bfe38802d7a9a9c5bc677b19ecaefa72d4335553 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1327886335
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/util/TimelineServerHelper.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.util;
+
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+
+/**
+ * Helper class for executing timeline server requests.
+ */
+public class TimelineServerHelper {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TimelineServerHelper.class);
+ private final ObjectMapper mapper;
+ private final String timelineServerHost;
+ private final int timelineServerPort;
+ private final int timeoutSecs;
+
+ public TimelineServerHelper(HoodieWriteConfig writeConfig) {
+ this(writeConfig.getViewStorageConfig().getRemoteViewServerHost(),
+ writeConfig.getViewStorageConfig().getRemoteViewServerPort(),
+ writeConfig.getViewStorageConfig().getRemoteTimelineClientTimeoutSecs());
+ }
+
+ public TimelineServerHelper(String timelineServerHost, int timelineServerPort, int timeoutSecs) {
+ this.mapper = new ObjectMapper();
+ this.timelineServerHost = timelineServerHost;
+ this.timelineServerPort = timelineServerPort;
+ this.timeoutSecs = timeoutSecs;
+ }
+
+ public <T> T executeRequestToTimelineServerWithRetry(String requestPath, Map<String, String> queryParameters,
+ TypeReference reference, RequestMethod method) {
+ int retry = 5;
+ while (--retry >= 0) {
+ long start = System.currentTimeMillis();
+ try {
+ return executeRequestToTimelineServer(requestPath, queryParameters, reference, method);
+ } catch (IOException e) {
+ LOG.warn("Failed to execute request (" + requestPath + ") to timeline server", e);
+ } finally {
+ LOG.info("Execute request : (" + requestPath + "), costs: " + (System.currentTimeMillis() - start) + " ms");
Review Comment:
Can we make the retry times not hard coded? Do we need to log for each request?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1721389341
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* bfe38802d7a9a9c5bc677b19ecaefa72d4335553 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1730769071
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026",
"triggerID" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046",
"triggerID" : "1727079993",
"triggerType" : "MANUAL"
}, {
"hash" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 427392c696b285afb6d4a093059c2b9a3c190f2a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046)
* 4a23e89b908b573b4c01a0406be5a36115ed8b3e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329720036
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/meta/TestCkpMetadata.java:
##########
@@ -49,11 +52,21 @@ public class TestCkpMetadata {
@TempDir
File tempFile;
+ protected Configuration conf;
+
+ protected HoodieFlinkWriteClient writeClient;
+
@BeforeEach
public void beforeEach() throws Exception {
+ setup();
+ }
+
+ protected void setup() throws IOException {
String basePath = tempFile.getAbsolutePath();
- Configuration conf = TestConfigurations.getDefaultConf(basePath);
+ this.conf = TestConfigurations.getDefaultConf(basePath);
+ conf.setString(HoodieWriteConfig.INSTANT_STATE_TIMELINE_SERVER_BASED.key(), "false");
StreamerUtil.initTableIfNotExists(conf);
+ this.writeClient = FlinkWriteClients.createWriteClient(conf);
}
Review Comment:
The tests may workout but you need to be cautious here, the CkpMetadata has read/write methods, the read methods should always be invoked from the client, while the write methods should be from driver, it is greate if you can imitate with the reality cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1338115806
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/InstantStateHandler.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.timeline.service.handlers;
+
+import org.apache.hudi.common.table.timeline.dto.InstantStateDTO;
+import org.apache.hudi.common.table.view.FileSystemViewManager;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.timeline.service.TimelineService;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+
+/**
+ * REST Handler servicing instant state requests.
+ * <p>
+ * The instant states are cached in timeline server and will be refreshed after the instant states in file system were changed.
+ */
+public class InstantStateHandler extends Handler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(InstantStateHandler.class);
+
+ /**
+ * Base url for instant state requests.
+ */
+ private static final String BASE_URL = "/v1/hoodie/instantstate";
+
+ /**
+ * Param for instant state requests, which contains a uniqueId for different writers.
+ */
+ public static final String INSTANT_STATE_DIR_PATH_PARAM = "instantstatedirpath";
+
+ /**
+ * GET requests. Returns all the instant states under instant state path.
+ */
+ public static final String ALL_INSTANT_STATE_URL = String.format("%s/%s", BASE_URL, "all");
+
+ /**
+ * POST requests. Refresh the instant state data cached in memory.
+ */
+ public static final String REFRESH_INSTANT_STATE = String.format("%s/%s", BASE_URL, "refresh/");
+
+ /**
+ * Cached instant state data, instant state path -> list of instant states in fs.
+ */
+ private final ConcurrentHashMap<String, List<InstantStateDTO>> cachedInstantStates;
+
+ /**
+ * Number of requests after the last refresh.
+ */
+ private final AtomicLong requestCount;
+
+ public InstantStateHandler(Configuration conf, TimelineService.Config timelineServiceConfig, FileSystem fileSystem,
+ FileSystemViewManager viewManager) throws IOException {
+ super(conf, timelineServiceConfig, fileSystem, viewManager);
+ this.cachedInstantStates = new ConcurrentHashMap<>();
+ this.requestCount = new AtomicLong();
+ }
+
+ /**
+ * Read instant states from cache of file system.
+ *
+ * @return Instant states under the input instant state path.
+ */
+ public List<InstantStateDTO> getAllInstantStates(String instantStatePath) {
+ if (requestCount.incrementAndGet() >= timelineServiceConfig.instantStateForceRefreshRequestNumber) {
+ // Do refresh for every N requests to ensure the writers won't be blocked forever
+ refresh(instantStatePath);
+ }
+ return cachedInstantStates.computeIfAbsent(instantStatePath, k -> scanInstantState(new Path(k)));
+ }
+
+ /**
+ * Refresh the checkpoint messages cached. Will be called when coordinator start/commit/abort instant.
+ *
+ * @return Whether refreshing is successful.
+ */
+ public boolean refresh(String instantStatePath) {
+ try {
+ cachedInstantStates.put(instantStatePath, scanInstantState(new Path(instantStatePath)));
+ requestCount.set(0);
+ } catch (Exception e) {
+ LOG.error("Failed to load instant states, path: " + instantStatePath, e);
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Scan the instant states from file system.
+ */
+ public List<InstantStateDTO> scanInstantState(Path instantStatePath) {
+ try {
+ // Check instantStatePath exists before list status, see HUDI-5915
+ if (this.fileSystem.exists(instantStatePath)) {
+ return Arrays.stream(this.fileSystem.listStatus(instantStatePath)).map(InstantStateDTO::fromFileStatus).collect(Collectors.toList());
Review Comment:
You are right, we can implement a complete timeline server memory-based checkpoint metadata since we will delete all previous ckp metadata when bootstrapping(so we do not actually rely on a durable filesystem state). And I think the timeline server is actually stable enough to serve the checkpoint message requests.
For now, I think it's OK to keep the filesystem-backed checkpoint messages and make the timeline server a stateless reader for the following reasons:
1. We will have a safe fallback option when the timeline server is not available during runtime.
2. The filesystem-backed checkpoint code is still useful since we need to turn off the timeline server sometimes. And creating ckp messages files is not expensive, we can reuse the code.
3. There will be no conflict if we want to move the start/abort/commit request to the timeline server in the future, and this PR has already implemented the important optimization that reduces the massive file listing. Moving the write part to the timeline server needs extra code and some more careful tests, we can move this PR.
We can merge this PR and create a Jira to track any possible optimization, what do you think? And thanks again for your review~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1711209798
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2acae42a85a382a3c1b30d4aee7f2f70f35d7b10 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1715733345
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1717287243
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91b363807661444af082389b9979ea59b5be4972 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1723610408
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* d73e1951739c48af6d0d0387c51c048dfdec9978 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1724929102
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* da4c89e3baf151b650b54d1ad608c7d8fa4c13ca Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009)
* 0acc78ec78cc1d99a62a2d8c7514110a56deadd1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1332366133
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bulk/BulkInsertWriteFunction.java:
##########
@@ -113,7 +116,8 @@ public BulkInsertWriteFunction(Configuration config, RowType rowType) {
public void open(Configuration parameters) throws IOException {
this.taskID = getRuntimeContext().getIndexOfThisSubtask();
this.writeClient = FlinkWriteClients.createWriteClient(this.config, getRuntimeContext());
- this.ckpMetadata = CkpMetadata.getInstance(config);
+ HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(this.config);
+ this.ckpMetadata = CkpMetadataFactory.getCkpMetadata(metaClient, writeClient.getConfig(), config);
Review Comment:
I think it shoud be feasible if we only maintains the interfafce as `getCkpMetadata(HoodieWriteConfig writeConfig, Configuration conf)`, then there is no need to create the meta client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1731660838
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026",
"triggerID" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20046",
"triggerID" : "1727079993",
"triggerType" : "MANUAL"
}, {
"hash" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20086",
"triggerID" : "4a23e89b908b573b4c01a0406be5a36115ed8b3e",
"triggerType" : "PUSH"
}, {
"hash" : "9c30218283e77bc1df4bc88f230085156c81a204",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20090",
"triggerID" : "9c30218283e77bc1df4bc88f230085156c81a204",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 9c30218283e77bc1df4bc88f230085156c81a204 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20090)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1725439585
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19993",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
}, {
"hash" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20009",
"triggerID" : "da4c89e3baf151b650b54d1ad608c7d8fa4c13ca",
"triggerType" : "PUSH"
}, {
"hash" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20018",
"triggerID" : "0acc78ec78cc1d99a62a2d8c7514110a56deadd1",
"triggerType" : "PUSH"
}, {
"hash" : "d9c59809dade4617f208b81d683bf496206cca50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20023",
"triggerID" : "d9c59809dade4617f208b81d683bf496206cca50",
"triggerType" : "PUSH"
}, {
"hash" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025",
"triggerID" : "047da0779ab20c6c074285da9cdb0b12c18bf230",
"triggerType" : "PUSH"
}, {
"hash" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026",
"triggerID" : "427392c696b285afb6d4a093059c2b9a3c190f2a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 047da0779ab20c6c074285da9cdb0b12c18bf230 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20025)
* 427392c696b285afb6d4a093059c2b9a3c190f2a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20026)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1723257424
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
}, {
"hash" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19982",
"triggerID" : "d1afbf2d4c5d8558caa1574f7c8bc688b39cca93",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983",
"triggerID" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"triggerType" : "PUSH"
}, {
"hash" : "71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991",
"triggerID" : "1722923278",
"triggerType" : "MANUAL"
}, {
"hash" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d73e1951739c48af6d0d0387c51c048dfdec9978",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* 71c1a0b48055b2bb7fec5244859d7d5c42fa5fc8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19983) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19991)
* d73e1951739c48af6d0d0387c51c048dfdec9978 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1329665072
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/meta/TestCkpMetadata.java:
##########
@@ -94,9 +107,17 @@ void testBootstrap() throws Exception {
metadata1.getInstantCache(), is(Collections.singletonList("4")));
}
- private CkpMetadata getCkpMetadata(String uniqueId) {
- String basePath = tempFile.getAbsolutePath();
- FileSystem fs = FSUtils.getFs(basePath, HadoopConfigurations.getHadoopConf(new Configuration()));
- return CkpMetadata.getInstance(fs, basePath, uniqueId);
+ protected CkpMetadata getCkpMetadata(String uniqueId) {
+ conf.set(FlinkOptions.WRITE_CLIENT_ID, uniqueId);
+ return CkpMetadataFactory.getCkpMetadata(writeClient.getHoodieTable().getMetaClient(), writeClient.getConfig(), conf);
+ }
Review Comment:
Changed to use metaClient created in `setup` method instead
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/TestWriteBase.java:
##########
@@ -137,14 +140,17 @@ public static TestHarness instance() {
private String lastPending;
private String lastComplete;
+ private HoodieFlinkWriteClient writeClient;
+
public TestHarness preparePipeline(File basePath, Configuration conf) throws Exception {
this.baseFile = basePath;
this.basePath = this.baseFile.getAbsolutePath();
this.conf = conf;
this.pipeline = TestData.getWritePipeline(this.basePath, conf);
// open the function and ingest data
this.pipeline.openFunction();
- this.ckpMetadata = CkpMetadata.getInstance(conf);
+ this.writeClient = FlinkWriteClients.createWriteClient(conf);
+ this.ckpMetadata = CkpMetadataFactory.getCkpMetadata(StreamerUtil.createMetaClient(conf), writeClient.getConfig(), conf);
Review Comment:
We maintain a ckpMetadata in `TestWriteBase` for `checkInflightInstant` and get `lastPendingInstant`. This metadata is not from coordinator so we need a timeline server for it.
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/meta/TestCkpMetadata.java:
##########
@@ -49,11 +52,21 @@ public class TestCkpMetadata {
@TempDir
File tempFile;
+ protected Configuration conf;
+
+ protected HoodieFlinkWriteClient writeClient;
+
@BeforeEach
public void beforeEach() throws Exception {
+ setup();
+ }
+
+ protected void setup() throws IOException {
String basePath = tempFile.getAbsolutePath();
- Configuration conf = TestConfigurations.getDefaultConf(basePath);
+ this.conf = TestConfigurations.getDefaultConf(basePath);
+ conf.setString(HoodieWriteConfig.INSTANT_STATE_TIMELINE_SERVER_BASED.key(), "false");
StreamerUtil.initTableIfNotExists(conf);
+ this.writeClient = FlinkWriteClients.createWriteClient(conf);
}
Review Comment:
In `TestCkpMetadata` there is no coordinator so we need to create a write client to start the timeline server.
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/meta/TestCkpMetadata.java:
##########
@@ -94,9 +107,17 @@ void testBootstrap() throws Exception {
metadata1.getInstantCache(), is(Collections.singletonList("4")));
}
- private CkpMetadata getCkpMetadata(String uniqueId) {
- String basePath = tempFile.getAbsolutePath();
- FileSystem fs = FSUtils.getFs(basePath, HadoopConfigurations.getHadoopConf(new Configuration()));
- return CkpMetadata.getInstance(fs, basePath, uniqueId);
+ protected CkpMetadata getCkpMetadata(String uniqueId) {
+ conf.set(FlinkOptions.WRITE_CLIENT_ID, uniqueId);
+ return CkpMetadataFactory.getCkpMetadata(writeClient.getHoodieTable().getMetaClient(), writeClient.getConfig(), conf);
+ }
+
+ @AfterEach
+ public void cleanup() {
+ if (writeClient != null) {
+ writeClient.close();
+ writeClient = null;
+ }
}
+
Review Comment:
removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1338025400
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/InstantStateHandler.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.timeline.service.handlers;
+
+import org.apache.hudi.common.table.timeline.dto.InstantStateDTO;
+import org.apache.hudi.common.table.view.FileSystemViewManager;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.timeline.service.TimelineService;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+
+/**
+ * REST Handler servicing instant state requests.
+ * <p>
+ * The instant states are cached in timeline server and will be refreshed after the instant states in file system were changed.
+ */
+public class InstantStateHandler extends Handler {
+
+ private static final Logger LOG = LoggerFactory.getLogger(InstantStateHandler.class);
+
+ /**
+ * Base url for instant state requests.
+ */
+ private static final String BASE_URL = "/v1/hoodie/instantstate";
+
+ /**
+ * Param for instant state requests, which contains a uniqueId for different writers.
+ */
+ public static final String INSTANT_STATE_DIR_PATH_PARAM = "instantstatedirpath";
+
+ /**
+ * GET requests. Returns all the instant states under instant state path.
+ */
+ public static final String ALL_INSTANT_STATE_URL = String.format("%s/%s", BASE_URL, "all");
+
+ /**
+ * POST requests. Refresh the instant state data cached in memory.
+ */
+ public static final String REFRESH_INSTANT_STATE = String.format("%s/%s", BASE_URL, "refresh/");
+
+ /**
+ * Cached instant state data, instant state path -> list of instant states in fs.
+ */
+ private final ConcurrentHashMap<String, List<InstantStateDTO>> cachedInstantStates;
+
+ /**
+ * Number of requests after the last refresh.
+ */
+ private final AtomicLong requestCount;
+
+ public InstantStateHandler(Configuration conf, TimelineService.Config timelineServiceConfig, FileSystem fileSystem,
+ FileSystemViewManager viewManager) throws IOException {
+ super(conf, timelineServiceConfig, fileSystem, viewManager);
+ this.cachedInstantStates = new ConcurrentHashMap<>();
+ this.requestCount = new AtomicLong();
+ }
+
+ /**
+ * Read instant states from cache of file system.
+ *
+ * @return Instant states under the input instant state path.
+ */
+ public List<InstantStateDTO> getAllInstantStates(String instantStatePath) {
+ if (requestCount.incrementAndGet() >= timelineServiceConfig.instantStateForceRefreshRequestNumber) {
+ // Do refresh for every N requests to ensure the writers won't be blocked forever
+ refresh(instantStatePath);
+ }
+ return cachedInstantStates.computeIfAbsent(instantStatePath, k -> scanInstantState(new Path(k)));
+ }
+
+ /**
+ * Refresh the checkpoint messages cached. Will be called when coordinator start/commit/abort instant.
+ *
+ * @return Whether refreshing is successful.
+ */
+ public boolean refresh(String instantStatePath) {
+ try {
+ cachedInstantStates.put(instantStatePath, scanInstantState(new Path(instantStatePath)));
+ requestCount.set(0);
+ } catch (Exception e) {
+ LOG.error("Failed to load instant states, path: " + instantStatePath, e);
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Scan the instant states from file system.
+ */
+ public List<InstantStateDTO> scanInstantState(Path instantStatePath) {
+ try {
+ // Check instantStatePath exists before list status, see HUDI-5915
+ if (this.fileSystem.exists(instantStatePath)) {
+ return Arrays.stream(this.fileSystem.listStatus(instantStatePath)).map(InstantStateDTO::fromFileStatus).collect(Collectors.toList());
Review Comment:
So the server based ckp metadata still backed by the fs ckp message files ? Can you eliminate that part, just accepts the request from coordinator the store the instant metadata in memory should be enought ? There is no need to write into the fs anymore I think. Or do you think the http server is not stable so it will ack with bad response frequently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1324111208
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -418,6 +418,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.sinceVersion("0.9.0")
.withDocumentation(MarkerType.class);
+ public static final ConfigProperty<Boolean> TIMELINE_SERVER_BASED_FLINK_CKP_METADATA = ConfigProperty
+ .key("hoodie.flink.ckp_metadata.timeline_server_based.enable")
+ .defaultValue(true)
Review Comment:
`hoodie.ckp_metadata.timeline_server_based.enable` -> `hoodie.instant_state.timeline_server_based.enabled`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1324111772
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -505,6 +516,24 @@ private void registerMarkerAPI() {
}, false));
}
+ private void registerFlinkCkpMetadataAPI() {
+ app.get(FlinkCkpMetadataHandler.ALL_CKP_METADATA_URL, new ViewHandler(ctx -> {
Review Comment:
Remove the Flink keyword.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1325391712
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/embedded/EmbeddedTimelineService.java:
##########
@@ -100,6 +100,10 @@ public void startServer() throws IOException {
* writeConfig.getHoodieClientHeartbeatTolerableMisses());
}
+ if (writeConfig.isTimelineServerBasedInstantStateEnabled()) {
+ timelineServiceConfBuilder.enableCkpMetadataRequests(true);
Review Comment:
Fix the method name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1325472907
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/embedded/EmbeddedTimelineService.java:
##########
@@ -100,6 +100,10 @@ public void startServer() throws IOException {
* writeConfig.getHoodieClientHeartbeatTolerableMisses());
}
+ if (writeConfig.isTimelineServerBasedInstantStateEnabled()) {
+ timelineServiceConfBuilder.enableCkpMetadataRequests(true);
Review Comment:
done. And should we rename all `ckpMetadata` in timeline server to `InstantState`? for example in `CkpMetadataHandler` there are a lot of ckpmetadata keywords.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/CkpMetadataFactory.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.meta;
+
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.flink.configuration.Configuration;
+
+/**
+ * A factory to generate {@link CkpMetadata} instance based on whether {@link HoodieWriteConfig#INSTANT_STATE_TIMELINE_SERVER_BASED} enabled.
+ */
+public class CkpMetadataFactory {
+ public static CkpMetadata get(HoodieTable table, Configuration config) {
Review Comment:
done and use `HoodieTableMetaClient` and `HoodieWriteConfig` as param instead of constructing hoodie table
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/meta/TimelineBasedCkpMetadata.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.meta;
+
+import org.apache.hudi.common.table.timeline.dto.CkpMetadataDTO;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.timeline.service.handlers.CkpMetadataHandler;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Stream;
+
+/**
+ * Timeline server based CkpMetadata, will read ckpMessages from timeline-server instead of from file system directly.
+ */
+public class TimelineBasedCkpMetadata extends CkpMetadata {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TimelineBasedCkpMetadata.class);
+
+ private final HoodieWriteConfig writeConfig;
+ private ObjectMapper mapper;
+
+ public TimelineBasedCkpMetadata(FileSystem fs, String basePath, String uniqueId, HoodieWriteConfig writeConfig) {
+ super(fs, basePath, uniqueId);
+ this.writeConfig = writeConfig;
+ mapper = new ObjectMapper();
+ LOG.info("Timeline server based CkpMetadata enabled");
+ }
+
+ @Override
+ public void startInstant(String instant) {
+ super.startInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void commitInstant(String instant) {
+ super.commitInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ public void abortInstant(String instant) {
+ super.abortInstant(instant);
+ sendRefreshRequest();
+ }
+
+ @Override
+ protected Stream<CkpMessage> readCkpMessages(Path ckpMetaPath) throws IOException {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1719360017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1326773541
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java:
##########
@@ -349,8 +351,8 @@ private static void initMetadataTable(HoodieFlinkWriteClient<?> writeClient) {
writeClient.initMetadataTable();
}
- private static CkpMetadata initCkpMetadata(HoodieTableMetaClient metaClient, Configuration conf) throws IOException {
- CkpMetadata ckpMetadata = CkpMetadata.getInstance(metaClient, conf.getString(FlinkOptions.WRITE_CLIENT_ID));
+ private static CkpMetadata initCkpMetadata(HoodieTableMetaClient metaClient, HoodieWriteConfig writeConfig, Configuration conf) throws IOException {
+ CkpMetadata ckpMetadata = CkpMetadataFactory.getCkpMetadata(metaClient, writeConfig, conf);
ckpMetadata.bootstrap();
return ckpMetadata;
Review Comment:
When the instant state was updated ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1323960093
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -418,6 +418,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.sinceVersion("0.9.0")
.withDocumentation(MarkerType.class);
+ public static final ConfigProperty<Boolean> TIMELINE_SERVER_BASED_FLINK_CKP_METADATA = ConfigProperty
+ .key("hoodie.flink.ckp_metadata.timeline_server_based.enable")
+ .defaultValue(true)
Review Comment:
Yeah, should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722400858
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfe38802d7a9a9c5bc677b19ecaefa72d4335553 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957)
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9651:
URL: https://github.com/apache/hudi/pull/9651#issuecomment-1722479764
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19729",
"triggerID" : "2acae42a85a382a3c1b30d4aee7f2f70f35d7b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19787",
"triggerID" : "7d5ae2f19b99845990588ccfd74bba3b63bb75c0",
"triggerType" : "PUSH"
}, {
"hash" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19852",
"triggerID" : "2903cb1ada8ffddfa24dbf6ec2120f58b4b68e78",
"triggerType" : "PUSH"
}, {
"hash" : "91b363807661444af082389b9979ea59b5be4972",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19883",
"triggerID" : "91b363807661444af082389b9979ea59b5be4972",
"triggerType" : "PUSH"
}, {
"hash" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19904",
"triggerID" : "0ee73069669c2da523179f15a3d9119a34fcaaf0",
"triggerType" : "PUSH"
}, {
"hash" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19908",
"triggerID" : "c2b6bfd20f4b6ec9a4ffb1060c9df467941d2687",
"triggerType" : "PUSH"
}, {
"hash" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19947",
"triggerID" : "026b65991bcb167f2a361c47fa87c29bc329ef9a",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19952",
"triggerID" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"triggerType" : "PUSH"
}, {
"hash" : "bfe38802d7a9a9c5bc677b19ecaefa72d4335553",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19957",
"triggerID" : "1721362417",
"triggerType" : "MANUAL"
}, {
"hash" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "451ef2808e72665d212eabc9079bd8a25bb502a4",
"triggerType" : "PUSH"
}, {
"hash" : "6bca9994c9dd051503795aa417d3018140140c1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19976",
"triggerID" : "6bca9994c9dd051503795aa417d3018140140c1a",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977",
"triggerID" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"triggerType" : "PUSH"
}, {
"hash" : "e64dbf44f7883bbdc1cca9dd56094cc871bded6e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978",
"triggerID" : "1722479031",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 451ef2808e72665d212eabc9079bd8a25bb502a4 UNKNOWN
* e64dbf44f7883bbdc1cca9dd56094cc871bded6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19977) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19978)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9651: [HUDI-6336] Support flink timeline-based ckp metadata
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9651:
URL: https://github.com/apache/hudi/pull/9651#discussion_r1328225815
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/dto/InstantStateDTO.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline.dto;
+
+import org.apache.hudi.common.util.ValidationUtils;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.hadoop.fs.FileStatus;
+
+/**
+ * Data transfer object for instant state.
+ *
+ * see org.apache.hudi.sink.meta.CkpMessage.
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class InstantStateDTO {
+
+ /**
+ * Ckp message instant.
+ *
+ * see org.apache.hudi.sink.meta.CkpMessage#instant
+ */
+ @JsonProperty("instant")
+ String instant;
+
+ /**
+ * Ckp message state.
+ *
Review Comment:
Fix the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org