You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Davis-Zhang-Onehouse (via GitHub)" <gi...@apache.org> on 2026/07/21 01:37:32 UTC
[PR] logging: add SQS backlog / in-flight visibility to S3 events source [hudi]
Davis-Zhang-Onehouse opened a new pull request, #19333:
URL: https://github.com/apache/hudi/pull/19333
### Change Logs
Adds a lifecycle log line to `S3EventsSource.onCommit` recording how many SQS messages are deleted at commit and the checkpoint, improving operational visibility into the SQS-backed S3 events ingestion path. First increment of a broader observability change (receive-loop plan/summary + in-flight queue counters) that will follow.
### Impact
Logging only; no behavior or API change.
### Risk level
none
### Documentation Update
None.
### Contributor's checklist
- [x] Change Logs and Impact were stated clearly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] logging: add SQS backlog / in-flight visibility to S3 events source [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19333:
URL: https://github.com/apache/hudi/pull/19333#issuecomment-5031768271
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "45b5166f907b27226dd6f740de7427feda0a10bf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15581",
"triggerID" : "45b5166f907b27226dd6f740de7427feda0a10bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45b5166f907b27226dd6f740de7427feda0a10bf Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15581)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] logging: add SQS backlog / in-flight visibility to S3 events source [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19333:
URL: https://github.com/apache/hudi/pull/19333#issuecomment-5029328165
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "45b5166f907b27226dd6f740de7427feda0a10bf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15581",
"triggerID" : "45b5166f907b27226dd6f740de7427feda0a10bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45b5166f907b27226dd6f740de7427feda0a10bf Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15581)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] logging: add SQS backlog / in-flight visibility to S3 events source [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19333:
URL: https://github.com/apache/hudi/pull/19333#discussion_r3624595094
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/S3EventsSource.java:
##########
@@ -109,6 +113,8 @@ public void close() throws IOException {
@Override
public void onCommit(String lastCkptStr) {
+ LOG.info("S3EventsSource.onCommit: deleting {} processed messages from queue, checkpoint={}.",
+ processedMessages.size(), lastCkptStr);
Review Comment:
🤖 nit: the `S3EventsSource.onCommit:` prefix is redundant — SLF4J already includes the class name in the log output. Could you simplify to something like `"Deleting {} processed messages from SQS queue, checkpoint={}."`?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(utilities): add SQS backlog / in-flight visibility to S3 events source [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous merged PR #19333:
URL: https://github.com/apache/hudi/pull/19333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] logging: add SQS backlog / in-flight visibility to S3 events source [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19333:
URL: https://github.com/apache/hudi/pull/19333#discussion_r3628579375
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/S3EventsSource.java:
##########
@@ -109,6 +113,8 @@ public void close() throws IOException {
@Override
public void onCommit(String lastCkptStr) {
+ LOG.info("S3EventsSource.onCommit: deleting {} processed messages from queue, checkpoint={}.",
+ processedMessages.size(), lastCkptStr);
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(utilities): add SQS backlog / in-flight visibility to S3 events source [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19333:
URL: https://github.com/apache/hudi/pull/19333#issuecomment-5043724733
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "45b5166f907b27226dd6f740de7427feda0a10bf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15581",
"triggerID" : "45b5166f907b27226dd6f740de7427feda0a10bf",
"triggerType" : "PUSH"
}, {
"hash" : "c31f679ec0f91a21bb801600d573ae0dac1fb410",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15617",
"triggerID" : "c31f679ec0f91a21bb801600d573ae0dac1fb410",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45b5166f907b27226dd6f740de7427feda0a10bf Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15581)
* c31f679ec0f91a21bb801600d573ae0dac1fb410 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15617)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(utilities): add SQS backlog / in-flight visibility to S3 events source [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19333:
URL: https://github.com/apache/hudi/pull/19333#issuecomment-5044726871
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "45b5166f907b27226dd6f740de7427feda0a10bf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15581",
"triggerID" : "45b5166f907b27226dd6f740de7427feda0a10bf",
"triggerType" : "PUSH"
}, {
"hash" : "c31f679ec0f91a21bb801600d573ae0dac1fb410",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15617",
"triggerID" : "c31f679ec0f91a21bb801600d573ae0dac1fb410",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c31f679ec0f91a21bb801600d573ae0dac1fb410 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15617)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(utilities): add SQS backlog / in-flight visibility to S3 events source [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19333:
URL: https://github.com/apache/hudi/pull/19333#discussion_r3628623845
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/S3EventsSource.java:
##########
@@ -109,6 +113,8 @@ public void close() throws IOException {
@Override
public void onCommit(String lastCkptStr) {
+ LOG.info("S3EventsSource.onCommit: deleting {} processed messages from queue, checkpoint={}.",
+ processedMessages.size(), lastCkptStr);
Review Comment:
Applied. Checked the claim first -- every log4j2 pattern in the repo carries `%c` (`%-4r [%t] %-5p %c %x - %m%n`, `%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n`), so the class name was indeed duplicated.
Worth noting for future instances of this nit: `%c` covers the *class*, not the *method* -- none of the patterns use `%M`. Dropping `onCommit` is fine here only because it is the sole place in this class that deletes processed messages. It would not be safe to strip a method-name prefix by reflex.
Now reads:
```java
LOG.info("Deleting {} processed messages from SQS queue, checkpoint={}.",
processedMessages.size(), lastCkptStr);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(utilities): add SQS backlog / in-flight visibility to S3 events source [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19333:
URL: https://github.com/apache/hudi/pull/19333#discussion_r3628623845
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/S3EventsSource.java:
##########
@@ -109,6 +113,8 @@ public void close() throws IOException {
@Override
public void onCommit(String lastCkptStr) {
+ LOG.info("S3EventsSource.onCommit: deleting {} processed messages from queue, checkpoint={}.",
+ processedMessages.size(), lastCkptStr);
Review Comment:
Changed to:
```java
LOG.info("Deleting {} processed messages from SQS queue, checkpoint={}.",
processedMessages.size(), lastCkptStr);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(utilities): add SQS backlog / in-flight visibility to S3 events source [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19333:
URL: https://github.com/apache/hudi/pull/19333#issuecomment-5043670424
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "45b5166f907b27226dd6f740de7427feda0a10bf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15581",
"triggerID" : "45b5166f907b27226dd6f740de7427feda0a10bf",
"triggerType" : "PUSH"
}, {
"hash" : "c31f679ec0f91a21bb801600d573ae0dac1fb410",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c31f679ec0f91a21bb801600d573ae0dac1fb410",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45b5166f907b27226dd6f740de7427feda0a10bf Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15581)
* c31f679ec0f91a21bb801600d573ae0dac1fb410 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] logging: add SQS backlog / in-flight visibility to S3 events source [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19333:
URL: https://github.com/apache/hudi/pull/19333#issuecomment-5029297385
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "45b5166f907b27226dd6f740de7427feda0a10bf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "45b5166f907b27226dd6f740de7427feda0a10bf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45b5166f907b27226dd6f740de7427feda0a10bf UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org