You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "allthingssecurity (via GitHub)" <gi...@apache.org> on 2026/10/16 00:04:11 UTC
[PR] CAMEL-25679: camel-mongodb - keep the consumer thread when a new cursor cannot be opened [camel]
allthingssecurity opened a new pull request, #27918:
URL: https://github.com/apache/camel/pull/27918
# Description
[CAMEL-25679](https://issues.apache.org/jira/browse/CAMEL-25679)
The tailable cursor consumer (the default `consumerType`) and the change streams consumer read from a cursor on one thread, `MongoAbstractConsumerThread.run()`. When a read fails (for example the connection to the server is lost) the loop logs a WARN ("Will try again on next poll"), closes the cursor, waits `cursorRegenerationDelay` (1000 ms by default) and opens a new cursor.
CAMEL-16025 put `doRun()` inside a `try`, but the new cursor is still opened after it (`cursor = initializeCursor()`, line 130 on main). While the server is still unreachable, opening the cursor fails once the driver's server selection timeout (30 seconds by default) is over, with `MongoTimeoutException`. The exception left `run()` and ended the consumer thread. The route stayed started and consumed nothing until it was restarted, and the only trace was the uncaught exception printed on stderr by the thread. With default options this happens whenever the server cannot be reached for longer than about 31 seconds (restart, slow failover, network outage).
Changes (`MongoAbstractConsumerThread` only, so both consumers get it):
- The new cursor is opened at the start of the next iteration, inside the `try`: a failure is logged like a failed read and tried again after `cursorRegenerationDelay`, until the server is back or the consumer is stopped. The order of the steps (close, wait, open, read) is the same as before.
- A cursor that is opened while the consumer is being stopped is closed instead of read: `stop()` closes the cursor it sees, and on main a cursor opened right after that was left open.
- The `cursor` field is volatile, as `stop()` reads it from another thread.
What users can notice: after an outage the consumer reads again once the server is back. While the server is down there is one WARN with stack trace per attempt (about every 31 seconds with the defaults), the same WARN that main already logs for a failed read. No option or default changes, so no upgrade guide entry.
Tests:
- New `MongoDbConsumerCursorRegenerationTest` (3 tests), Mockito mocks of the driver objects, no server, every wait bounded, no sleeps:
- tailable consumer: the cursor of the start fails its read (`MongoSocketReadException`), the first new cursor cannot be opened (`MongoTimeoutException`), the next one returns a document, which must be routed;
- the same for the change streams consumer;
- the consumer thread is stopped while it opens a new cursor: that cursor must be closed and not read. This test runs the tailing thread of the consumer on a thread of the test (no route), so that it can call `stop()` while the cursor is being opened without reaching into private fields.
- Without the change (main's `MongoAbstractConsumerThread`, same tests) all three fail: `mock://result Received message count. Expected: <1> but was: <0>` for both consumers (the output shows `Exception in thread "Camel (camel-1) thread #1 - mongodb://myDb" com.mongodb.MongoTimeoutException: Timed out while waiting for a server`), and `Wanted but not invoked: mongoCursor.close()` for the third.
- With the change the camel-mongodb unit tests pass: 21 run, 0 failures, 0 skipped.
What is not changed:
- The tailable consumer still ends quietly when the driver throws `IllegalStateException` from `hasNext()` (that branch is meant for a cursor closed by `stop()`).
- The change streams consumer still logs a failed read at DEBUG only.
- No backoff beyond `cursorRegenerationDelay`; opening a cursor while the server is down already blocks for the server selection timeout.
What was not run: the module's integration tests (they need a MongoDB container), a real server outage.
# Target
- [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch)
# Tracking
- [ ] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it).
(The ticket is drafted and not filed yet; CAMEL-25679 is a placeholder until it is.)
# Apache Camel coding standards and style
- [x] I checked that each commit in the pull request has a meaningful subject line and body.
- [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes.
(I built and tested camel-mongodb, including the formatter and import-sort plugins. I did not run the full root build.)
# AI-assisted contributions
- [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used.
This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer.
_Claude Code on behalf of allthingssecurity_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25679: camel-mongodb - keep the consumer thread when a new cursor cannot be opened [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27918:
URL: https://github.com/apache/camel/pull/27918#issuecomment-6211154192
:star2: Thank you for your contribution to the Apache Camel project! :star2:
:robot: CI automation will test this PR automatically.
:camel: Apache Camel Committers, please review the following items:
* First-time contributors **require MANUAL approval** for the GitHub Actions to run
* You can use the command `/component-test (camel-)component-name1 (camel-)component-name2..` to request a test from the test bot although they are normally detected and executed by CI.
* You can label PRs using `skip-tests` and `test-dependents` to fine-tune the checks executed by this PR.
* Build and test logs are available in the summary page. **Only** [Apache Camel committers](https://camel.apache.org/community/team/#committers) have access to the summary.
:warning: Be careful when sharing logs. Review their contents before sharing them publicly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25679: camel-mongodb - keep the consumer thread when a new cursor cannot be opened [camel]
Posted by "gnodet-bot (via GitHub)" <gi...@apache.org>.
gnodet-bot commented on code in PR #27918:
URL: https://github.com/apache/camel/pull/27918#discussion_r4291286993
##########
components/camel-mongodb/src/main/java/org/apache/camel/component/mongodb/MongoAbstractConsumerThread.java:
##########
@@ -99,8 +99,20 @@ protected boolean processExchange(Exchange exchange) {
public void run() {
stoppedLatch = new CountDownLatch(1);
try {
- while (keepRunning) {
+ // an interrupt ends the thread, as it did before the cursor was reopened inside the try
+ while (keepRunning && !Thread.currentThread().isInterrupted()) {
try {
+ if (cursor == null) {
+ // a new cursor after the previous one ended. It is opened here, inside the try, so that a
+ // failure to open it (the server cannot be reached yet) is reported and tried again after
+ // the delay, instead of ending this thread while the consumer stays started
Review Comment:
⚠️ **Log volume concern (low severity):** `initializeCursor()` is now inside the inner `try` block, so permanent MongoDB failures will log a WARN with full stack trace on every `cursorRegenerationDelay` cycle. Consider rate-limiting: e.g. always log the first failure, then every Nth retry only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25679: camel-mongodb - keep the consumer thread when a new cursor cannot be opened [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27918:
URL: https://github.com/apache/camel/pull/27918#issuecomment-6212267417
<!-- ci-tested-modules -->
:test_tube: **CI tested the following changed modules:**
- `components/camel-mongodb`
:repeat: **1 test passed only after a retry on JDK 25** (1 retried attempt)
<details><summary>Recovered flaky tests on JDK 25 (1)</summary>
| Module | Test | Failed attempts | First failure |
| --- | --- | --- | --- |
| `dsl/camel-jbang/camel-jbang-plugin-tui` | `LaunchManagerOutcomeTest.startedWhenTheStatusFileSaysSo` | 1 | expected: null but was: LaunchOutcome[ok=false, log=null] |
> :information_source: These tests did **not** fail the build. Surefire retried them and they passed.
> Retries are enabled project-wide by `surefire.rerunFailingTestsCount` in `parent/pom.xml`.
</details>
---
<details><summary>:microscope: Scalpel shadow comparison — Scalpel: 8 of 706 tested, 27 compile-only — current: 8 all tested</summary>
[Maveniverse Scalpel](https://github.com/maveniverse/scalpel) detected **8 affected modules** (current approach: 8).
**Skip-tests mode would test 8 modules** (1 direct + 7 downstream), **skip tests for 27** (generated code, meta-modules)
<details><summary>Modules Scalpel would test (8)</summary>
- `camel-jbang-mcp` ← downstream of org.apache.camel:camel-catalog
- `camel-jbang-plugin-mcp` ← downstream of org.apache.camel:camel-jbang-core
- `camel-jbang-plugin-tui` ← downstream of org.apache.camel:camel-catalog
- `camel-jbang-plugin-validate` ← downstream of org.apache.camel:camel-yaml-dsl-validator
- `camel-launcher-container` ← downstream of org.apache.camel:camel-launcher
- `camel-mongodb` ← components/camel-mongodb/src/main/java/org/apache/camel/component/mongodb/MongoAbstractConsumerThread.java, components/camel-mongodb/src/test/java/org/apache/camel/component/mongodb/MongoDbConsumerCursorRegenerationTest.java
- `camel-yaml-dsl-validator` ← downstream of org.apache.camel:camel-catalog
- `camel-yaml-dsl-validator-maven-plugin` ← downstream of org.apache.camel:camel-yaml-dsl-validator
</details>
<details><summary>Modules with tests skipped (27)</summary>
- `apache-camel`
- `camel-allcomponents`
- `camel-catalog`
- `camel-catalog-console`
- `camel-catalog-maven`
- `camel-catalog-suggest`
- `camel-componentdsl`
- `camel-endpointdsl`
- `camel-endpointdsl-support`
- `camel-itest`
- `camel-jbang-core`
- `camel-jbang-it`
- `camel-jbang-main`
- `camel-jbang-plugin-edit`
- `camel-jbang-plugin-generate`
- `camel-jbang-plugin-kubernetes`
- `camel-jbang-plugin-test`
- `camel-kamelet-main`
- `camel-launcher`
- `camel-report-maven-plugin`
- `camel-route-parser`
- `camel-yaml-dsl`
- `camel-yaml-dsl-deserializers`
- `camel-yaml-dsl-maven-plugin`
- `coverage`
- `docs`
- `dummy-component`
</details>
> :information_source: Shadow mode — Scalpel observes but does not affect test execution. [Learn more](https://github.com/maveniverse/scalpel)
</details>
<details><summary>All tested modules (35 modules, 5m 9s total)</summary>
**Total reactor time:** 5m 9s
| Module | Duration | Status |
| --- | --- | --- |
| Camel :: Launcher | 48.9s | SUCCESS |
| Camel :: JBang :: MCP | 40.5s | SUCCESS |
| Camel :: JBang :: Plugin :: Validate | 33.4s | SUCCESS |
| Camel :: Catalog :: Camel Catalog | 22.4s | SUCCESS |
| Camel :: YAML DSL | 20.3s | SUCCESS |
| Camel :: YAML DSL :: Validator | 19.6s | SUCCESS |
| Camel :: Component DSL | 18.2s | SUCCESS |
| Camel :: Docs | 14.9s | SUCCESS |
| Camel :: Kamelet Main | 14.8s | SUCCESS |
| Camel :: Catalog :: Camel Report Maven Plugin | 12.6s | SUCCESS |
| Camel :: JBang :: Plugin :: Kubernetes | 11.6s | SUCCESS |
| Camel :: JBang :: Plugin :: Testing | 8.3s | SUCCESS |
| Camel :: YAML DSL :: Deserializers | 7.5s | SUCCESS |
| Camel :: Catalog :: Camel Route Parser | 7.0s | SUCCESS |
| Camel :: All Components Sync point | 5.2s | SUCCESS |
| Camel :: YAML DSL :: Validator Maven Plugin | 3.9s | SUCCESS |
| Camel :: YAML DSL :: Maven Plugins | 3.3s | SUCCESS |
| Camel :: Catalog :: Suggest (deprecated) | 2.4s | SUCCESS |
| Camel :: Catalog :: Maven | 2.3s | SUCCESS |
| Camel :: Assembly | 2.0s | SUCCESS |
| Camel :: Coverage | 1.5s | SUCCESS |
| Camel :: JBang :: Plugin :: Edit | 1.4s | SUCCESS |
| Camel :: JBang :: Plugin :: Generate | 1.2s | SUCCESS |
| Camel :: Catalog :: Dummy Component | 1.1s | SUCCESS |
| Camel :: Catalog :: Console | 1.0s | SUCCESS |
| Camel :: Endpoint DSL :: Support | 0.9s | SUCCESS |
| Camel :: JBang :: Integration tests | 0.9s | SUCCESS |
| Camel :: Launcher :: Container | 0.7s | SUCCESS |
| Camel :: JBang :: Main | 0.7s | SUCCESS |
| Camel :: JBang :: Plugin :: MCP | 0.5s | SUCCESS |
| Camel :: Endpoint DSL | n/a | |
| Camel :: Integration Tests | n/a | |
| Camel :: JBang :: Core | n/a | |
| Camel :: JBang :: Plugin :: TUI | n/a | |
| Camel :: MongoDB | n/a | |
**Top 20 slowest modules:**
- `Camel :: Launcher` (48.9s)
- `Camel :: JBang :: MCP` (40.5s)
- `Camel :: JBang :: Plugin :: Validate` (33.4s)
- `Camel :: Catalog :: Camel Catalog` (22.4s)
- `Camel :: YAML DSL` (20.3s)
- `Camel :: YAML DSL :: Validator` (19.6s)
- `Camel :: Component DSL` (18.2s)
- `Camel :: Docs` (14.9s)
- `Camel :: Kamelet Main` (14.8s)
- `Camel :: Catalog :: Camel Report Maven Plugin` (12.6s)
- `Camel :: JBang :: Plugin :: Kubernetes` (11.6s)
- `Camel :: JBang :: Plugin :: Testing` (8.3s)
- `Camel :: YAML DSL :: Deserializers` (7.5s)
- `Camel :: Catalog :: Camel Route Parser` (7.0s)
- `Camel :: All Components Sync point` (5.2s)
- `Camel :: YAML DSL :: Validator Maven Plugin` (3.9s)
- `Camel :: YAML DSL :: Maven Plugins` (3.3s)
- `Camel :: Catalog :: Suggest (deprecated)` (2.4s)
- `Camel :: Catalog :: Maven` (2.3s)
- `Camel :: Assembly` (2.0s)
</details>
---
:gear: [View full build and test results](https://github.com/apache/camel/actions/runs/38856303536)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25679: camel-mongodb - keep the consumer thread when a new cursor cannot be opened [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus merged PR #27918:
URL: https://github.com/apache/camel/pull/27918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25679: camel-mongodb - keep the consumer thread when a new cursor cannot be opened [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on code in PR #27918:
URL: https://github.com/apache/camel/pull/27918#discussion_r4302183176
##########
components/camel-mongodb/src/main/java/org/apache/camel/component/mongodb/MongoAbstractConsumerThread.java:
##########
@@ -99,8 +99,20 @@ protected boolean processExchange(Exchange exchange) {
public void run() {
stoppedLatch = new CountDownLatch(1);
try {
- while (keepRunning) {
+ // an interrupt ends the thread, as it did before the cursor was reopened inside the try
+ while (keepRunning && !Thread.currentThread().isInterrupted()) {
try {
+ if (cursor == null) {
+ // a new cursor after the previous one ended. It is opened here, inside the try, so that a
+ // failure to open it (the server cannot be reached yet) is reported and tried again after
+ // the delay, instead of ending this thread while the consumer stays started
Review Comment:
@allthingssecurity offered to do this as a follow-up with its own test (the stack trace only for the first of consecutive failures), so this PR stays about the consumer thread surviving. Resolving to merge; @allthingssecurity, please open a JIRA for the follow-up so it is not lost.
_Claude Code on behalf of Claus Ibsen_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25679: camel-mongodb - keep the consumer thread when a new cursor cannot be opened [camel]
Posted by "allthingssecurity (via GitHub)" <gi...@apache.org>.
allthingssecurity commented on code in PR #27918:
URL: https://github.com/apache/camel/pull/27918#discussion_r4302723738
##########
components/camel-mongodb/src/main/java/org/apache/camel/component/mongodb/MongoAbstractConsumerThread.java:
##########
@@ -99,8 +99,20 @@ protected boolean processExchange(Exchange exchange) {
public void run() {
stoppedLatch = new CountDownLatch(1);
try {
- while (keepRunning) {
+ // an interrupt ends the thread, as it did before the cursor was reopened inside the try
+ while (keepRunning && !Thread.currentThread().isInterrupted()) {
try {
+ if (cursor == null) {
+ // a new cursor after the previous one ended. It is opened here, inside the try, so that a
+ // failure to open it (the server cannot be reached yet) is reported and tried again after
+ // the delay, instead of ending this thread while the consumer stays started
Review Comment:
Filed as CAMEL-25731 (stack trace only for the first of consecutive cursor failures, with a test).
_Claude Code on behalf of allthingssecurity_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25679: camel-mongodb - keep the consumer thread when a new cursor cannot be opened [camel]
Posted by "allthingssecurity (via GitHub)" <gi...@apache.org>.
allthingssecurity commented on code in PR #27918:
URL: https://github.com/apache/camel/pull/27918#discussion_r4291252661
##########
components/camel-mongodb/src/main/java/org/apache/camel/component/mongodb/MongoAbstractConsumerThread.java:
##########
@@ -101,6 +101,17 @@ public void run() {
try {
while (keepRunning) {
Review Comment:
Thanks, good catch. Done in 057e95c631ca: the loop now stops when the thread is interrupted (`while (keepRunning && !Thread.currentThread().isInterrupted())`), as you suggested, so an interrupt ends the thread again instead of making it spin. New test `interruptEndsTheThreadWhileTheCursorCannotBeOpened`: the first cursor breaks, every new cursor fails to open, the thread is interrupted; it fails on the previous commit (the thread is still alive 5 s after the interrupt) and passes now. camel-mongodb unit tests: 20 run, 0 failures.
_Claude Code on behalf of allthingssecurity_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25679: camel-mongodb - keep the consumer thread when a new cursor cannot be opened [camel]
Posted by "allthingssecurity (via GitHub)" <gi...@apache.org>.
allthingssecurity commented on PR #27918:
URL: https://github.com/apache/camel/pull/27918#issuecomment-6208846461
On the non-blocking question about a cursor that can never be opened again (resume token gone, permissions, dropped collection): I agree a WARN with stack trace every `cursorRegenerationDelay` is noisy there. I would rather do it as a follow-up with its own test, logging the stack trace only for the first of consecutive failures (and a WARN line without it after that), so this PR stays about the thread not dying. The description is updated: the ticket line now links CAMEL-25679, and it mentions the interrupt change.
_Claude Code on behalf of allthingssecurity_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25679: camel-mongodb - keep the consumer thread when a new cursor cannot be opened [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on code in PR #27918:
URL: https://github.com/apache/camel/pull/27918#discussion_r4291166265
##########
components/camel-mongodb/src/main/java/org/apache/camel/component/mongodb/MongoAbstractConsumerThread.java:
##########
@@ -101,6 +101,17 @@ public void run() {
try {
while (keepRunning) {
Review Comment:
One edge case this opens: an interrupt that arrives while `keepRunning` is still true now makes the loop spin. The sleep catch re-sets the flag, the next `initializeCursor()` fails at once with `MongoInterruptedException` (the driver's server selection waits interruptibly), that is now caught and logged as a WARN with stack trace, and `Thread.sleep` throws straight away again. Before, the same interrupt ended the thread. Could the loop stop on interrupt?
```suggestion
while (keepRunning && !Thread.currentThread().isInterrupted()) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org