You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "davsclaus (via GitHub)" <gi...@apache.org> on 2024/07/05 10:13:15 UTC
[PR] CAMEL-20227: camel-kafka - Kafka offset advances when using Pausable … [camel]
davsclaus opened a new pull request, #14745:
URL: https://github.com/apache/camel/pull/14745
…EIP and messages are lost when resumed
# Description
<!--
- Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
-->
# Target
- [ ] I checked that the commit is targeting the correct branch (note that Camel 3 uses `camel-3.x`, whereas Camel 4 uses the `main` branch)
# Tracking
- [ ] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it).
<!--
# *Note*: trivial changes like, typos, minor documentation fixes and other small items do not require a JIRA issue. In this case your pull request should address just this issue, without pulling in other changes.
-->
# Apache Camel coding standards and style
- [ ] I checked that each commit in the pull request has a meaningful subject line and body.
<!--
If you're unsure, you can format the pull request title like `[CAMEL-XXX] Fixes bug in camel-file component`, where you replace `CAMEL-XXX` with the appropriate JIRA issue.
-->
- [ ] I have run `mvn clean install -DskipTests` locally and I have committed all auto-generated changes
<!--
You can run the aforementioned command in your module so that the build auto-formats your code. This will also be verified as part of the checks and your PR may be rejected if if there are uncommited changes after running `mvn clean install -DskipTests`.
You can learn more about the contribution guidelines at https://github.com/apache/camel/blob/main/CONTRIBUTING.md
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-20227: camel-kafka - Kafka offset advances when using Pausable … [camel]
Posted by "orpiske (via GitHub)" <gi...@apache.org>.
orpiske commented on PR #14745:
URL: https://github.com/apache/camel/pull/14745#issuecomment-2210657575
Cool. It's a holiday here, but I will try to take a look later today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-20227: camel-kafka - Kafka offset advances when using Pausable … [camel]
Posted by "orpiske (via GitHub)" <gi...@apache.org>.
orpiske commented on PR #14745:
URL: https://github.com/apache/camel/pull/14745#issuecomment-2212350669
@davsclaus actually, this is different from pausing the route per se. The idea for the `pausable` API is to pause the Kafka consumer (so as not to cause a rebalance when there is a transient issue).
Nonetheless, let's see if we can brainstorm some improvements to this once you are back. There are some specific requirements for pausing the KafkaConsumer (such as: it must be done from the same thread as the one calling 'poll'), but if we could do this using the `RoutePolicy`, we could retire this API and make it simpler for the users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-20227: camel-kafka - Kafka offset advances when using Pausable … [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #14745:
URL: https://github.com/apache/camel/pull/14745#issuecomment-2212345822
Yeah okay that can be better explain in the docs / javadoc / and the API is using a predicate so an exception is not thrown.
If the intend is to pause/resume kafka via some logic then use `RoutePolicy` that is intended for that. There is a throttling policy out of the box, but you can make your own as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-20227: camel-kafka - Kafka offset advances when using Pausable … [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #14745:
URL: https://github.com/apache/camel/pull/14745#issuecomment-2210607107
Argh yeah this test is a bit wrong as you want to be able to consume all 15 messages, but in the middle the consumer is paused
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-20227: camel-kafka - Kafka offset advances when using Pausable … [camel]
Posted by "orpiske (via GitHub)" <gi...@apache.org>.
orpiske commented on PR #14745:
URL: https://github.com/apache/camel/pull/14745#issuecomment-2210972636
At first it wasn't very clear to me how the user would be losing messages after the pause.
The KafkaConsumer documentation says that [`pause`](https://kafka.apache.org/22/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#pause-java.util.Collection-) _"Suspend fetching from the requested partitions. Future calls to [poll(Duration)](https://kafka.apache.org/22/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll-java.time.Duration-) will not return any records from these partitions until they have been resumed using [resume(Collection)](https://kafka.apache.org/22/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#resume-java.util.Collection-)"_.
But looking more closely to the reproducer, I think I kinda understand the problem ... It's in part because of a misuse of the API (and, poor documentation for this particular feature).
In particular, our documentation states:
_"The pausable EIP is meant to be used as a support mechanism when there is an **exception** somewhere in the route that prevents the exchange from being processed. More specifically, the check called by the pausable EIP should be used to test for transient conditions preventing the exchange from being processed."_
Specifically, the trigger for those transient problems should be pulled by an exception, so that [we have a failure in the processing result](https://github.com/apache/camel/blob/main/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/support/streaming/KafkaRecordStreamingProcessorFacade.java#L97-L117). Otherwise, [we never cause the pause](https://github.com/apache/camel/blob/main/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/errorhandler/KafkaConsumerListener.java#L71-L88), thus leading to them losing the messages.
So, to make it short: the reproducer from the user is wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-20227: camel-kafka - Kafka offset advances when using Pausable … [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus closed pull request #14745: CAMEL-20227: camel-kafka - Kafka offset advances when using Pausable …
URL: https://github.com/apache/camel/pull/14745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-20227: camel-kafka - Kafka offset advances when using Pausable … [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14745:
URL: https://github.com/apache/camel/pull/14745#issuecomment-2210604647
:star2: Thank you for your contribution to the Apache Camel project! :star2:
:robot: CI automation will test this PR automatically.
:camel: Apache Camel Committers, please review the following items:
* First-time contributors **require MANUAL approval** for the GitHub Actions to run
* You can use the command `/component-test (camel-)component-name1 (camel-)component-name2..` to request a test from the test bot.
* You can label PRs using `build-all`, `build-dependents`, `skip-tests` and `test-dependents` to fine-tune the checks executed by this PR.
* Build and test logs are available in the Summary page. **Only** [Apache Camel committers](https://camel.apache.org/community/team/#committers) have access to the summary.
* :warning: Be careful when sharing logs. Review their contents before sharing them publicly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-20227: camel-kafka - Kafka offset advances when using Pausable … [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #14745:
URL: https://github.com/apache/camel/pull/14745#issuecomment-2448203554
closing old PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-20227: camel-kafka - Kafka offset advances when using Pausable … [camel]
Posted by "orpiske (via GitHub)" <gi...@apache.org>.
orpiske commented on PR #14745:
URL: https://github.com/apache/camel/pull/14745#issuecomment-2210976615
That said, I think we do have a room for improvement on the API. I am not sure if we should be [allowing the code to resume the assignment if pause hasn't been called](https://github.com/apache/camel/blob/main/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/consumer/errorhandler/KafkaConsumerListener.java#L58-L68).
Maybe we could throw an exception explaining that this is a misuse of the API, and trying to force a resume without and underlying transient problem is not supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-20227: camel-kafka - Kafka offset advances when using Pausable … [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #14745:
URL: https://github.com/apache/camel/pull/14745#issuecomment-2210644805
okay I improved this so the pause and resume will not lose messages and that the pause is waiting a poll cycle, otherwise it keeps looping asap which is not desireable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org