You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "devdeegaw (via GitHub)" <gi...@apache.org> on 2023/06/09 15:01:12 UTC
[GitHub] [camel] devdeegaw opened a new pull request, #10315: Camel 19433 handle error while consuming records
devdeegaw opened a new pull request, #10315:
URL: https://github.com/apache/camel/pull/10315
# Description
In the event of an error occurring during record consumption from a topic, the Kafka consumer will persistently send LeaveGroup requests to the coordinator due to consumer poll timeout has expired.
The camel Kafka consumer silently ignores it and throws no exceptions when fetching data for the topics.
Handling the error safely unsubscribe the consumer, accompanied by an appropriate log message
# Target
- [X] I checked that the commit is targeting the correct branch (note that Camel 3 uses `camel-3.x`, whereas Camel 4 uses the `main` branch)
# Tracking
- [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it).
<!--
# *Note*: trivial changes like, typos, minor documentation fixes and other small items do not require a JIRA issue. In this case your pull request should address just this issue, without pulling in other changes.
-->
# Apache Camel coding standards and style
- [x] I checked that each commit in the pull request has a meaningful subject line and body.
<!--
If you're unsure, you can format the pull request title like `[CAMEL-XXX] Fixes bug in camel-file component`, where you replace `CAMEL-XXX` with the appropriate JIRA issue.
-->
- [x] I formatted the code using `mvn -Pformat,fastinstall install && mvn -Psourcecheck`
<!--
You can run the aforementioned command in your module so that the build auto-formats your code and the source check verifies that is complies with our coding style. This will also be verified as part of the checks and your PR may be rejected if the checkstyle does not pass.
You can learn more about the contribution guidelines at https://github.com/apache/camel/blob/main/CONTRIBUTING.md
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] orpiske closed pull request #10315: CAMEL-19433: Handle error while consuming records from topic
Posted by "orpiske (via GitHub)" <gi...@apache.org>.
orpiske closed pull request #10315: CAMEL-19433: Handle error while consuming records from topic
URL: https://github.com/apache/camel/pull/10315
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] devdeegaw commented on a diff in pull request #10315: CAMEL-19433: Handle error while consuming records from topic
Posted by "devdeegaw (via GitHub)" <gi...@apache.org>.
devdeegaw commented on code in PR #10315:
URL: https://github.com/apache/camel/pull/10315#discussion_r1226322727
##########
components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaFetchRecords.java:
##########
@@ -358,6 +358,10 @@ protected void startPolling() {
if (LOG.isTraceEnabled()) {
LOG.trace("The kafka consumer was woken up while polling on thread {} for {}", threadId, getPrintableTopic());
}
+ } catch (Error e) { // NOSONAR - rethrown
Review Comment:
@orpiske We found the root cause during our usage of the Spring Kafka framework (KafkaMessageListenerContainer as the listener for message consumption). However, thanks to the error handling and logging capabilities provided by the framework, we were able to capture and log the errors, despite not yet determining the root cause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] orpiske commented on pull request #10315: CAMEL-19433: Handle error while consuming records from topic
Posted by "orpiske (via GitHub)" <gi...@apache.org>.
orpiske commented on PR #10315:
URL: https://github.com/apache/camel/pull/10315#issuecomment-1609902340
Closing this PR as it is stalled and the problem needs to be solved differently ([work-around described here](https://issues.apache.org/jira/browse/CAMEL-19433?focusedCommentId=17737789&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17737789))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] orpiske commented on pull request #10315: CAMEL-19433: Handle error while consuming records from topic
Posted by "orpiske (via GitHub)" <gi...@apache.org>.
orpiske commented on PR #10315:
URL: https://github.com/apache/camel/pull/10315#issuecomment-1586804469
Also, I think it would be good to check if the [StopErrorStrategy](https://github.com/apache/camel/blob/2925e9694c521de3b78b7cb288037d321cf59717/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/errorhandler/StopErrorStrategy.java) can do this before implementing a new one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] github-actions[bot] commented on pull request #10315: CAMEL-19433: Handle error while consuming records from topic
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10315:
URL: https://github.com/apache/camel/pull/10315#issuecomment-1587279287
### Core test results:
| Tested | Failed :x: | Passed :white_check_mark: |
| --- | --- | --- |
| 1 | 1 | 0 |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] orpiske commented on a diff in pull request #10315: CAMEL-19433: Handle error while consuming records from topic
Posted by "orpiske (via GitHub)" <gi...@apache.org>.
orpiske commented on code in PR #10315:
URL: https://github.com/apache/camel/pull/10315#discussion_r1226337521
##########
components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaFetchRecords.java:
##########
@@ -358,6 +358,10 @@ protected void startPolling() {
if (LOG.isTraceEnabled()) {
LOG.trace("The kafka consumer was woken up while polling on thread {} for {}", threadId, getPrintableTopic());
}
+ } catch (Error e) { // NOSONAR - rethrown
Review Comment:
The problem is: just because it worked on your particular scenario, there's no guarantee it would work reliably (or work at all) on all others that fall on the same category (i.e.: all other sub-classes of Error).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] devdeegaw commented on a diff in pull request #10315: CAMEL-19433: Handle error while consuming records from topic
Posted by "devdeegaw (via GitHub)" <gi...@apache.org>.
devdeegaw commented on code in PR #10315:
URL: https://github.com/apache/camel/pull/10315#discussion_r1226311268
##########
components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaFetchRecords.java:
##########
@@ -358,6 +358,10 @@ protected void startPolling() {
if (LOG.isTraceEnabled()) {
LOG.trace("The kafka consumer was woken up while polling on thread {} for {}", threadId, getPrintableTopic());
}
+ } catch (Error e) { // NOSONAR - rethrown
Review Comment:
@orpiske Although it would not resolve the problem entirely, capturing the error and implementing appropriate logging would provide valuable insights to the developer regarding the occurrences of poll timeouts and the consumer leaving the group.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] orpiske commented on a diff in pull request #10315: CAMEL-19433: Handle error while consuming records from topic
Posted by "orpiske (via GitHub)" <gi...@apache.org>.
orpiske commented on code in PR #10315:
URL: https://github.com/apache/camel/pull/10315#discussion_r1226251174
##########
components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaFetchRecords.java:
##########
@@ -358,6 +358,10 @@ protected void startPolling() {
if (LOG.isTraceEnabled()) {
LOG.trace("The kafka consumer was woken up while polling on thread {} for {}", threadId, getPrintableTopic());
}
+ } catch (Error e) { // NOSONAR - rethrown
Review Comment:
Yeah, if we were to implement this ... it would have to be the last one, indeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] github-actions[bot] commented on pull request #10315: CAMEL-19433: Handle error while consuming records from topic
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10315:
URL: https://github.com/apache/camel/pull/10315#issuecomment-1586888997
:no_entry_sign: There are (likely) no changes in core core to be tested in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] davsclaus commented on a diff in pull request #10315: CAMEL-19433: Handle error while consuming records from topic
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on code in PR #10315:
URL: https://github.com/apache/camel/pull/10315#discussion_r1226198162
##########
components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaFetchRecords.java:
##########
@@ -358,6 +358,10 @@ protected void startPolling() {
if (LOG.isTraceEnabled()) {
LOG.trace("The kafka consumer was woken up while polling on thread {} for {}", threadId, getPrintableTopic());
}
+ } catch (Error e) { // NOSONAR - rethrown
Review Comment:
I kinda forgot whether the order of the catch matters, eg I would assume this should be last, so regular exception can be caught for the normal exceptions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] github-actions[bot] commented on pull request #10315: CAMEL-19433: Handle error while consuming records from topic
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10315:
URL: https://github.com/apache/camel/pull/10315#issuecomment-1587279246
### Components test results:
| Total | Tested | Failed :x: | Passed :white_check_mark: |
| --- | --- | --- | --- |
| 1 | 1 | 0 | 1 |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] github-actions[bot] commented on pull request #10315: Camel 19433 handle error while consuming records
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10315:
URL: https://github.com/apache/camel/pull/10315#issuecomment-1584727912
:star2: Thank you for your contribution to the Apache Camel project! :star2:
:camel: Maintainers, please note that first-time contributors *require manual approval* for the GitHub Actions to run.
:warning: Please note that the changes on this PR may be **tested automatically** if they change components.
:robot: Use the command `/component-test (camel-)component-name1 (camel-)component-name2..` to request a test from the test bot.
If necessary Apache Camel Committers may access logs and test results in the job summaries!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] github-actions[bot] commented on pull request #10315: CAMEL-19433: Handle error while consuming records from topic
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10315:
URL: https://github.com/apache/camel/pull/10315#issuecomment-1586888901
### Components test results:
| Total | Tested | Failed :x: | Passed :white_check_mark: |
| --- | --- | --- | --- |
| 1 | 1 | 0 | 1 |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org