You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "fhan688 (via GitHub)" <gi...@apache.org> on 2024/10/25 08:32:25 UTC
[PR] [HUDI-8433] Fix not update IssuedInstant when stream read empty commits [hudi]
fhan688 opened a new pull request, #12166:
URL: https://github.com/apache/hudi/pull/12166
### Change Logs
When empty commits are allowed , if number of continuous empty commits exceeds 'read.commits.limit' , IssuedInstant will not be updated which will block stream read.
### Impact
hudi-flink-datasource
### Risk level (write none, low medium or high below)
low
### Documentation Update
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2449494051
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
}, {
"hash" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414",
"triggerID" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"triggerType" : "PUSH"
}, {
"hash" : "a88e8d14a78a088853051917a0c302ca9bead107",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1473",
"triggerID" : "a88e8d14a78a088853051917a0c302ca9bead107",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a88e8d14a78a088853051917a0c302ca9bead107 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1473)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on code in PR #12166:
URL: https://github.com/apache/hudi/pull/12166#discussion_r1824140876
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/StreamReadMonitoringFunction.java:
##########
@@ -214,8 +214,9 @@ public void monitorDirAndForwardSplits(SourceContext<MergeOnReadInputSplit> cont
}
IncrementalInputSplits.Result result =
incrementalInputSplits.inputSplits(metaClient, this.issuedOffset, this.cdcEnabled);
- if (result.isEmpty()) {
+ if (result.equals(IncrementalInputSplits.Result.EMPTY)) {
Review Comment:
This bug is easy to be reproduced. Let us take UT `TestStreamReadMonitoringFunction#testConsumeForSpeedLimitWhenEmptyCommitExists()` as an example:
Step1: create 4 empty commit
Step2: trigger streaming read from first instant and set READ_COMMITS_LIMIT 2
Step3: assert current IssuedOffset couldn't be null.
Base on `IncrementalInputSplits#inputSplits => .startCompletionTime(issuedOffset != null ? issuedOffset : this.conf.getString(FlinkOptions.READ_START_COMMIT))`
If IssuedOffset still was null, hudi would take FlinkOptions.READ_START_COMMIT again, which means streaming read is blocked.
Without this PR this UT will be failed which means streaming reading empty commits will be blocked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update IssuedInstant when stream read empty commits [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12166:
URL: https://github.com/apache/hudi/pull/12166#discussion_r1817558367
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/StreamReadMonitoringFunction.java:
##########
@@ -214,8 +214,9 @@ public void monitorDirAndForwardSplits(SourceContext<MergeOnReadInputSplit> cont
}
IncrementalInputSplits.Result result =
incrementalInputSplits.inputSplits(metaClient, this.issuedOffset, this.cdcEnabled);
- if (result.isEmpty()) {
+ if (result.equals(IncrementalInputSplits.Result.EMPTY)) {
Review Comment:
it is not updated but can still be used to reset the variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update IssuedInstant when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2437355137
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2451506798
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
}, {
"hash" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414",
"triggerID" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"triggerType" : "PUSH"
}, {
"hash" : "a88e8d14a78a088853051917a0c302ca9bead107",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1473",
"triggerID" : "a88e8d14a78a088853051917a0c302ca9bead107",
"triggerType" : "PUSH"
}, {
"hash" : "25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a88e8d14a78a088853051917a0c302ca9bead107 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1473)
* 25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2451623576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
}, {
"hash" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414",
"triggerID" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"triggerType" : "PUSH"
}, {
"hash" : "a88e8d14a78a088853051917a0c302ca9bead107",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1473",
"triggerID" : "a88e8d14a78a088853051917a0c302ca9bead107",
"triggerType" : "PUSH"
}, {
"hash" : "25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1476",
"triggerID" : "25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1476)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2458719589
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
}, {
"hash" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414",
"triggerID" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"triggerType" : "PUSH"
}, {
"hash" : "a88e8d14a78a088853051917a0c302ca9bead107",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1473",
"triggerID" : "a88e8d14a78a088853051917a0c302ca9bead107",
"triggerType" : "PUSH"
}, {
"hash" : "25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1476",
"triggerID" : "25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093",
"triggerType" : "PUSH"
}, {
"hash" : "f20d0e028c9e1461a0a70f4a7715d5fa1254b5d5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1517",
"triggerID" : "f20d0e028c9e1461a0a70f4a7715d5fa1254b5d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f20d0e028c9e1461a0a70f4a7715d5fa1254b5d5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1517)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2449362833
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
}, {
"hash" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414",
"triggerID" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"triggerType" : "PUSH"
}, {
"hash" : "a88e8d14a78a088853051917a0c302ca9bead107",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1473",
"triggerID" : "a88e8d14a78a088853051917a0c302ca9bead107",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31c047b6750f2dfabee1846e9f24ede07d181d74 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414)
* a88e8d14a78a088853051917a0c302ca9bead107 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1473)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update IssuedInstant when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2437211916
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update IssuedInstant when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2437209089
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update IssuedInstant when stream read empty commits [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12166:
URL: https://github.com/apache/hudi/pull/12166#discussion_r1822381780
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/StreamReadMonitoringFunction.java:
##########
@@ -214,8 +214,9 @@ public void monitorDirAndForwardSplits(SourceContext<MergeOnReadInputSplit> cont
}
IncrementalInputSplits.Result result =
incrementalInputSplits.inputSplits(metaClient, this.issuedOffset, this.cdcEnabled);
- if (result.isEmpty()) {
+ if (result.equals(IncrementalInputSplits.Result.EMPTY)) {
Review Comment:
I meant just to use `instant4.completionTime` to update the `issuedInstant` in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12166:
URL: https://github.com/apache/hudi/pull/12166#discussion_r1826399863
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/source/TestStreamReadMonitoringFunction.java:
##########
@@ -143,6 +150,48 @@ public void testConsumeFromLastCommit() throws Exception {
}
}
+ @Test
+ public void testConsumeForSpeedLimitWhenEmptyCommitExists() throws Exception {
+ // Step1 : create 4 empty commit
+ Configuration conf = new Configuration(this.conf);
+ conf.set(FlinkOptions.TABLE_TYPE, FlinkOptions.TABLE_TYPE_COPY_ON_WRITE);
+ conf.setBoolean(HoodieWriteConfig.ALLOW_EMPTY_COMMIT.key(), true);
+
+ TestData.writeData(Collections.EMPTY_LIST, conf);
+ TestData.writeData(Collections.EMPTY_LIST, conf);
+ TestData.writeData(Collections.EMPTY_LIST, conf);
+ TestData.writeData(Collections.EMPTY_LIST, conf);
+
+ HoodieTableMetaClient metaClient = HoodieTestUtils.init(conf.get(FlinkOptions.PATH), HoodieTableType.COPY_ON_WRITE);
+ HoodieTimeline commitsTimeline = metaClient.reloadActiveTimeline()
+ .filter(hoodieInstant -> hoodieInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION));
+ HoodieInstant firstInstant = commitsTimeline.firstInstant().get();
+
+ // Step2: trigger streaming read from first instant and set READ_COMMITS_LIMIT 2
+ conf.set(FlinkOptions.READ_AS_STREAMING, true);
+ conf.set(FlinkOptions.READ_STREAMING_SKIP_CLUSTERING, true);
+ conf.set(FlinkOptions.READ_STREAMING_SKIP_COMPACT, true);
+ conf.set(FlinkOptions.READ_COMMITS_LIMIT, 2);
+ conf.set(FlinkOptions.READ_START_COMMIT, String.valueOf((Long.valueOf(firstInstant.getTimestamp()) - 100)));
+ StreamReadMonitoringFunction function = TestUtils.getMonitorFunc(conf);
+ try (AbstractStreamOperatorTestHarness<MergeOnReadInputSplit> harness = createHarness(function)) {
+ harness.setup();
+ harness.open();
+
+ CountDownLatch latch = new CountDownLatch(0);
+ CollectingSourceContext sourceContext = new CollectingSourceContext(latch);
+ function.monitorDirAndForwardSplits(sourceContext);
+ assertEquals(0, sourceContext.splits.size(), "There should be no inputSplits");
+
+ // Step3: assert current IssuedOffset couldn't be null.
+ // Base on "IncrementalInputSplits#inputSplits => .startCompletionTime(issuedOffset != null ? issuedOffset : this.conf.getString(FlinkOptions.READ_START_COMMIT))"
+ // If IssuedOffset still was null, hudi would take FlinkOptions.READ_START_COMMIT again, which means streaming read is blocked.
+ assertTrue(function.getIssuedOffset() != null);
Review Comment:
Use assertNotNull
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update IssuedInstant when stream read empty commits [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on code in PR #12166:
URL: https://github.com/apache/hudi/pull/12166#discussion_r1816375117
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/StreamReadMonitoringFunction.java:
##########
@@ -214,8 +214,9 @@ public void monitorDirAndForwardSplits(SourceContext<MergeOnReadInputSplit> cont
}
IncrementalInputSplits.Result result =
incrementalInputSplits.inputSplits(metaClient, this.issuedOffset, this.cdcEnabled);
- if (result.isEmpty()) {
+ if (result.equals(IncrementalInputSplits.Result.EMPTY)) {
Review Comment:
If value of `read.speed.limit` is smaller than number of empty instants , result.isEmpty is true and issuedInstant is not updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update IssuedInstant when stream read empty commits [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12166:
URL: https://github.com/apache/hudi/pull/12166#discussion_r1816325241
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/StreamReadMonitoringFunction.java:
##########
@@ -214,8 +214,9 @@ public void monitorDirAndForwardSplits(SourceContext<MergeOnReadInputSplit> cont
}
IncrementalInputSplits.Result result =
incrementalInputSplits.inputSplits(metaClient, this.issuedOffset, this.cdcEnabled);
- if (result.isEmpty()) {
+ if (result.equals(IncrementalInputSplits.Result.EMPTY)) {
Review Comment:
should we always update the `issuedInstant` if it is not null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update IssuedInstant when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2437384906
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
}, {
"hash" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414",
"triggerID" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412)
* 31c047b6750f2dfabee1846e9f24ede07d181d74 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on code in PR #12166:
URL: https://github.com/apache/hudi/pull/12166#discussion_r1830344871
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/source/TestStreamReadMonitoringFunction.java:
##########
@@ -438,6 +487,20 @@ private void runAsync(
task.start();
}
+ private Integer intervalBetween2Instants(HoodieTimeline timeline, String instant1, String instant2) {
+ Integer idxInstant1 = getInstantIdxInTimeline(timeline, instant1);
Review Comment:
not called, just remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update IssuedInstant when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2437381994
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
}, {
"hash" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412)
* 31c047b6750f2dfabee1846e9f24ede07d181d74 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2451508741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
}, {
"hash" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414",
"triggerID" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"triggerType" : "PUSH"
}, {
"hash" : "a88e8d14a78a088853051917a0c302ca9bead107",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1473",
"triggerID" : "a88e8d14a78a088853051917a0c302ca9bead107",
"triggerType" : "PUSH"
}, {
"hash" : "25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1476",
"triggerID" : "25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a88e8d14a78a088853051917a0c302ca9bead107 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1473)
* 25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1476)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12166:
URL: https://github.com/apache/hudi/pull/12166#discussion_r1826399591
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/source/TestStreamReadMonitoringFunction.java:
##########
@@ -438,6 +487,20 @@ private void runAsync(
task.start();
}
+ private Integer intervalBetween2Instants(HoodieTimeline timeline, String instant1, String instant2) {
+ Integer idxInstant1 = getInstantIdxInTimeline(timeline, instant1);
Review Comment:
Can we just filter the timeline with proper range and count the instants instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2462172029
> The Azure failure is not related.
got it. thank you so much for Teacher Danny's advice and is there anything esle to revise?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12166:
URL: https://github.com/apache/hudi/pull/12166#discussion_r1825322880
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/StreamReadMonitoringFunction.java:
##########
@@ -214,8 +214,9 @@ public void monitorDirAndForwardSplits(SourceContext<MergeOnReadInputSplit> cont
}
IncrementalInputSplits.Result result =
incrementalInputSplits.inputSplits(metaClient, this.issuedOffset, this.cdcEnabled);
- if (result.isEmpty()) {
+ if (result.equals(IncrementalInputSplits.Result.EMPTY)) {
Review Comment:
We only update the `issuedOffset` with non-null `issuedInstant`, does it make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2449360013
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
}, {
"hash" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414",
"triggerID" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"triggerType" : "PUSH"
}, {
"hash" : "a88e8d14a78a088853051917a0c302ca9bead107",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a88e8d14a78a088853051917a0c302ca9bead107",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31c047b6750f2dfabee1846e9f24ede07d181d74 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414)
* a88e8d14a78a088853051917a0c302ca9bead107 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update IssuedInstant when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2437533013
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
}, {
"hash" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414",
"triggerID" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31c047b6750f2dfabee1846e9f24ede07d181d74 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update IssuedInstant when stream read empty commits [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on code in PR #12166:
URL: https://github.com/apache/hudi/pull/12166#discussion_r1822195797
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/StreamReadMonitoringFunction.java:
##########
@@ -214,8 +214,9 @@ public void monitorDirAndForwardSplits(SourceContext<MergeOnReadInputSplit> cont
}
IncrementalInputSplits.Result result =
incrementalInputSplits.inputSplits(metaClient, this.issuedOffset, this.cdcEnabled);
- if (result.isEmpty()) {
+ if (result.equals(IncrementalInputSplits.Result.EMPTY)) {
Review Comment:
In speed limit scenario, it can not be updated any more. I try to give an example to explain this:
assume current active timeline is instant1、instant2、instant3、instant4、instant5、instant6 ...
If the issuedInstant is instant2 and read.speed.limit=2 and instant3 and instant4 are empty commits, then the inputSplits method return `Result.instance(inputSplits, endInstant, offsetToIssue)`.
In this return, inputSplits is Collections.emptyList() , endInstant is instant4, offsetToIssue is max(instant3 completionTime, instant4.completionTime).
So result.isEmpty() is true and monitorDirAndForwardSplits() returned, but offsetToIssue and issuedInstant are not updated. Next time when monitorDirAndForwardSplits() is called, the inputSplits method return the same as above, endless loop.
@danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2461952818
The Azure failure is not related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12166:
URL: https://github.com/apache/hudi/pull/12166
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "fhan688 (via GitHub)" <gi...@apache.org>.
fhan688 commented on code in PR #12166:
URL: https://github.com/apache/hudi/pull/12166#discussion_r1825583037
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/StreamReadMonitoringFunction.java:
##########
@@ -214,8 +214,9 @@ public void monitorDirAndForwardSplits(SourceContext<MergeOnReadInputSplit> cont
}
IncrementalInputSplits.Result result =
incrementalInputSplits.inputSplits(metaClient, this.issuedOffset, this.cdcEnabled);
- if (result.isEmpty()) {
+ if (result.equals(IncrementalInputSplits.Result.EMPTY)) {
Review Comment:
make sense. PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2458660941
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
}, {
"hash" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414",
"triggerID" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"triggerType" : "PUSH"
}, {
"hash" : "a88e8d14a78a088853051917a0c302ca9bead107",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1473",
"triggerID" : "a88e8d14a78a088853051917a0c302ca9bead107",
"triggerType" : "PUSH"
}, {
"hash" : "25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1476",
"triggerID" : "25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093",
"triggerType" : "PUSH"
}, {
"hash" : "f20d0e028c9e1461a0a70f4a7715d5fa1254b5d5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1517",
"triggerID" : "f20d0e028c9e1461a0a70f4a7715d5fa1254b5d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1476)
* f20d0e028c9e1461a0a70f4a7715d5fa1254b5d5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1517)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12166:
URL: https://github.com/apache/hudi/pull/12166#issuecomment-2458659749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1412",
"triggerID" : "20ddbf1ae8aeae055aaf58c9ce58a8712e96eefb",
"triggerType" : "PUSH"
}, {
"hash" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1414",
"triggerID" : "31c047b6750f2dfabee1846e9f24ede07d181d74",
"triggerType" : "PUSH"
}, {
"hash" : "a88e8d14a78a088853051917a0c302ca9bead107",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1473",
"triggerID" : "a88e8d14a78a088853051917a0c302ca9bead107",
"triggerType" : "PUSH"
}, {
"hash" : "25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1476",
"triggerID" : "25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093",
"triggerType" : "PUSH"
}, {
"hash" : "f20d0e028c9e1461a0a70f4a7715d5fa1254b5d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f20d0e028c9e1461a0a70f4a7715d5fa1254b5d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25f8bc5efc86fe5bd2cbfd4037bcf552cd56a093 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1476)
* f20d0e028c9e1461a0a70f4a7715d5fa1254b5d5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8433] Fix not update issuedOffset when stream read empty commits [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12166:
URL: https://github.com/apache/hudi/pull/12166#discussion_r1826400118
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/StreamReadMonitoringFunction.java:
##########
@@ -214,8 +215,10 @@ public void monitorDirAndForwardSplits(SourceContext<MergeOnReadInputSplit> cont
}
IncrementalInputSplits.Result result =
incrementalInputSplits.inputSplits(metaClient, this.issuedOffset, this.cdcEnabled);
- if (result.isEmpty()) {
+
+ if (result.isEmpty() && StringUtils.isNullOrEmpty(result.getEndInstant())) {
// no new instants, returns early
+ LOG.info("result is empty, do not update issuedInstant.");
Review Comment:
result -> Result，and use log.warn instead, maybe we just eliminate the log message because it's almost useless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org