You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "allthingssecurity (via GitHub)" <gi...@apache.org> on 2026/10/11 00:04:29 UTC
[PR] CAMEL-25520: camel-google-mail - Do not consume a message again while its exchange is in flight [camel]
allthingssecurity opened a new pull request, #27671:
URL: https://github.com/apache/camel/pull/27671
# Description
[CAMEL-25520](https://issues.apache.org/jira/browse/CAMEL-25520)
The `google-mail-stream` consumer hands each exchange to the route with an empty callback and does not wait; the message is marked as read (`markAsRead`, the default) by the exchange's on completion. With an asynchronous route (`seda`, which hands the on completion over, kafka or another async producer, `threads()`), the next poll lists the still unread message again (the default query is `is:unread`) and routes it a second time. If one of the duplicates fails, its rollback adds the UNREAD label back, so a message that was already processed successfully is consumed once more.
This is the defect fixed for aws2-s3 in CAMEL-17110. The consumer now keeps the ids of the messages whose exchanges are in progress: a poll skips them, and an id is released when its exchange completes or fails (after marking the message read or unread), or when the poll does not process it (get failure, consumer stopping). No new option: a plain set in the consumer, not cleared on route stop (in-flight ids are released when their exchanges complete). With `markAsRead=false` a message is now consumed again only after its previous exchange completed; 4.23 upgrade-guide note.
Tests (`FakeGmailClientFactory`: a Gmail client whose `MockHttpTransport` answers from an in-memory mailbox, no network):
- `GoogleMailStreamConsumerInProgressTest.messageInFlightIsNotConsumedAgain`: the seda route holds the exchange until the consumer has completed a second poll (counted with a `pollStrategy`). Without the change: `m1 was consumed again by a later poll while its first exchange was still in flight ==> expected: <1> but was: <2>`.
- `failedMessageIsConsumedAgain`: a message whose exchange failed is consumed again and then marked as read; `messageIsConsumedAfterAFailedGet`: the get of the first poll fails (status 500) and a later poll consumes the message; `notProcessedMessageIsConsumedAgain`: a message whose exchange a stopping consumer did not process is not left in progress. All three pass without the change too (there is no in-progress set on main); they guard the releases on failure, on a failed get and on stop.
- With the change, the camel-google-mail unit tests pass (48).
Found with a TLA+ model of the poll and the asynchronous completion: "a message is routed again only after an exchange for it failed" is violated in 10 steps (poll, get, dispatch, poll again, get, dispatch again). I then reproduced it with the real consumer.
# Target
- [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch)
# Tracking
- [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it).
# Apache Camel coding standards and style
- [x] I checked that each commit in the pull request has a meaningful subject line and body.
- [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes.
(I built and tested camel-google-mail, including the formatter and import-sort plugins; the build did not change any generated file. I did not run the full root build.)
# AI-assisted contributions
- [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used.
This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer.
_Claude Code on behalf of allthingssecurity_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25520: camel-google-mail - Do not consume a message again while its exchange is in flight [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27671:
URL: https://github.com/apache/camel/pull/27671#issuecomment-6105966970
:star2: Thank you for your contribution to the Apache Camel project! :star2:
:robot: CI automation will test this PR automatically.
:camel: Apache Camel Committers, please review the following items:
* First-time contributors **require MANUAL approval** for the GitHub Actions to run
* You can use the command `/component-test (camel-)component-name1 (camel-)component-name2..` to request a test from the test bot although they are normally detected and executed by CI.
* You can label PRs using `skip-tests` and `test-dependents` to fine-tune the checks executed by this PR.
* Build and test logs are available in the summary page. **Only** [Apache Camel committers](https://camel.apache.org/community/team/#committers) have access to the summary.
:warning: Be careful when sharing logs. Review their contents before sharing them publicly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25520: camel-google-mail - Do not consume a message again while its exchange is in flight [camel]
Posted by "allthingssecurity (via GitHub)" <gi...@apache.org>.
allthingssecurity commented on PR #27671:
URL: https://github.com/apache/camel/pull/27671#issuecomment-6106529180
Rebased on `main` (587012a4bf20); the only conflict was the 4.23 upgrade guide, where both entries are kept. The code is unchanged from the reviewed commits.
_Claude Code on behalf of allthingssecurity_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25520: camel-google-mail - Do not consume a message again while its exchange is in flight [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27671:
URL: https://github.com/apache/camel/pull/27671#issuecomment-6106371808
<!-- ci-tested-modules -->
:test_tube: **CI tested the following changed modules:**
- `components/camel-google/camel-google-mail`
- `docs`
---
<details><summary>:microscope: Scalpel shadow comparison — Scalpel: 9 of 704 tested, 26 compile-only — current: 9 all tested</summary>
[Maveniverse Scalpel](https://github.com/maveniverse/scalpel) detected **9 affected modules** (current approach: 9).
**Skip-tests mode would test 9 modules** (2 direct + 8 downstream), **skip tests for 26** (generated code, meta-modules)
<details><summary>Modules Scalpel would test (9)</summary>
- `camel-google-mail` ← components/camel-google/camel-google-mail/src/main/java/org/apache/camel/component/google/mail/stream/GoogleMailStreamConsumer.java, components/camel-google/camel-google-mail/src/test/java/org/apache/camel/component/google/mail/stream/FakeGmailClientFactory.java, components/camel-google/camel-google-mail/src/test/java/org/apache/camel/component/google/mail/stream/GoogleMailStreamConsumerInProgressTest.java
- `camel-jbang-mcp` ← downstream of org.apache.camel:camel-catalog
- `camel-jbang-plugin-mcp` ← downstream of org.apache.camel:camel-jbang-core
- `camel-jbang-plugin-route-parser` ← downstream of org.apache.camel:camel-route-parser
- `camel-jbang-plugin-tui` ← downstream of org.apache.camel:camel-catalog
- `camel-jbang-plugin-validate` ← downstream of org.apache.camel:camel-yaml-dsl-validator
- `camel-launcher-container` ← downstream of org.apache.camel:camel-launcher
- `camel-yaml-dsl-validator` ← downstream of org.apache.camel:camel-catalog
- `camel-yaml-dsl-validator-maven-plugin` ← downstream of org.apache.camel:camel-yaml-dsl-validator
</details>
<details><summary>Modules with tests skipped (26)</summary>
- `apache-camel`
- `camel-allcomponents`
- `camel-catalog`
- `camel-catalog-console`
- `camel-catalog-maven`
- `camel-catalog-suggest`
- `camel-componentdsl`
- `camel-endpointdsl`
- `camel-endpointdsl-support`
- `camel-itest`
- `camel-jbang-core`
- `camel-jbang-it`
- `camel-jbang-main`
- `camel-jbang-plugin-edit`
- `camel-jbang-plugin-generate`
- `camel-jbang-plugin-kubernetes`
- `camel-jbang-plugin-test`
- `camel-kamelet-main`
- `camel-launcher`
- `camel-report-maven-plugin`
- `camel-route-parser`
- `camel-yaml-dsl`
- `camel-yaml-dsl-deserializers`
- `camel-yaml-dsl-maven-plugin`
- `coverage`
- `dummy-component`
</details>
> :information_source: Shadow mode — Scalpel observes but does not affect test execution. [Learn more](https://github.com/maveniverse/scalpel)
</details>
<details><summary>All tested modules (36 modules, 5m 47s total)</summary>
**Total reactor time:** 5m 47s
| Module | Duration | Status |
| --- | --- | --- |
| Camel :: Launcher | 47.0s | SUCCESS |
| Camel :: JBang :: MCP | 41.7s | SUCCESS |
| Camel :: Google :: Mail | 26.3s | SUCCESS |
| Camel :: Catalog :: Camel Catalog | 25.7s | SUCCESS |
| Camel :: JBang :: Plugin :: Kubernetes | 25.1s | SUCCESS |
| Camel :: Component DSL | 22.9s | SUCCESS |
| Camel :: YAML DSL :: Validator | 22.7s | SUCCESS |
| Camel :: YAML DSL | 21.6s | SUCCESS |
| Camel :: JBang :: Plugin :: Validate | 16.9s | SUCCESS |
| Camel :: Docs | 16.7s | SUCCESS |
| Camel :: Kamelet Main | 11.8s | SUCCESS |
| Camel :: JBang :: Plugin :: Testing | 9.3s | SUCCESS |
| Camel :: Catalog :: Camel Route Parser | 8.8s | SUCCESS |
| Camel :: YAML DSL :: Deserializers | 8.4s | SUCCESS |
| Camel :: Catalog :: Camel Report Maven Plugin | 7.0s | SUCCESS |
| Camel :: YAML DSL :: Validator Maven Plugin | 6.8s | SUCCESS |
| Camel :: All Components Sync point | 5.6s | SUCCESS |
| Camel :: Catalog :: Maven | 3.3s | SUCCESS |
| Camel :: YAML DSL :: Maven Plugins | 3.2s | SUCCESS |
| Camel :: Catalog :: Suggest (deprecated) | 2.9s | SUCCESS |
| Camel :: Assembly | 2.0s | SUCCESS |
| Camel :: JBang :: Plugin :: MCP | 1.5s | SUCCESS |
| Camel :: JBang :: Plugin :: Edit | 1.3s | SUCCESS |
| Camel :: Coverage | 1.3s | SUCCESS |
| Camel :: Catalog :: Dummy Component | 1.2s | SUCCESS |
| Camel :: JBang :: Integration tests | 1.1s | SUCCESS |
| Camel :: JBang :: Plugin :: Generate | 1.0s | SUCCESS |
| Camel :: Catalog :: Console | 0.9s | SUCCESS |
| Camel :: Endpoint DSL :: Support | 0.8s | SUCCESS |
| Camel :: JBang :: Main | 0.7s | SUCCESS |
| Camel :: Launcher :: Container | 0.6s | SUCCESS |
| Camel :: JBang :: Plugin :: Route Parser | 0.5s | SUCCESS |
| Camel :: Endpoint DSL | n/a | |
| Camel :: Integration Tests | n/a | |
| Camel :: JBang :: Core | n/a | |
| Camel :: JBang :: Plugin :: TUI | n/a | |
**Top 20 slowest modules:**
- `Camel :: Launcher` (47.0s)
- `Camel :: JBang :: MCP` (41.7s)
- `Camel :: Google :: Mail` (26.3s)
- `Camel :: Catalog :: Camel Catalog` (25.7s)
- `Camel :: JBang :: Plugin :: Kubernetes` (25.1s)
- `Camel :: Component DSL` (22.9s)
- `Camel :: YAML DSL :: Validator` (22.7s)
- `Camel :: YAML DSL` (21.6s)
- `Camel :: JBang :: Plugin :: Validate` (16.9s)
- `Camel :: Docs` (16.7s)
- `Camel :: Kamelet Main` (11.8s)
- `Camel :: JBang :: Plugin :: Testing` (9.3s)
- `Camel :: Catalog :: Camel Route Parser` (8.8s)
- `Camel :: YAML DSL :: Deserializers` (8.4s)
- `Camel :: Catalog :: Camel Report Maven Plugin` (7.0s)
- `Camel :: YAML DSL :: Validator Maven Plugin` (6.8s)
- `Camel :: All Components Sync point` (5.6s)
- `Camel :: Catalog :: Maven` (3.3s)
- `Camel :: YAML DSL :: Maven Plugins` (3.2s)
- `Camel :: Catalog :: Suggest (deprecated)` (2.9s)
</details>
---
:gear: [View full build and test results](https://github.com/apache/camel/actions/runs/38117102715)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25520: camel-google-mail - Do not consume a message again while its exchange is in flight [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus merged PR #27671:
URL: https://github.com/apache/camel/pull/27671
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25520: camel-google-mail - Do not consume a message again while its exchange is in flight [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #27671:
URL: https://github.com/apache/camel/pull/27671#issuecomment-6106507504
@allthingssecurity can you resolve the merge conflict
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25520: camel-google-mail - Do not consume a message again while its exchange is in flight [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on code in PR #27671:
URL: https://github.com/apache/camel/pull/27671#discussion_r4240436925
##########
components/camel-google/camel-google-mail/src/main/java/org/apache/camel/component/google/mail/stream/GoogleMailStreamConsumer.java:
##########
@@ -91,17 +95,41 @@ protected int poll() throws Exception {
forceConsumerAsReady();
if (c.getMessages() != null) {
- for (Message message : c.getMessages()) {
- Message mess
- = getClient().users().messages().get("me", message.getId()).setFormat(messageFormat()).execute();
- Exchange exchange = createExchange(getEndpoint().getExchangePattern(), mess);
- answer.add(exchange);
+ try {
+ for (Message message : c.getMessages()) {
+ addExchange(message.getId(), answer);
+ }
+ } catch (Exception e) {
+ // none of the messages of this poll is processed
+ releaseInProgress(answer);
+ throw e;
}
}
return processBatch(CastUtils.cast(answer));
}
+ private void addExchange(String id, Queue<Exchange> answer) throws Exception {
+ // skip the message if an exchange of a previous poll is still processing it
+ if (!inProgress.add(id)) {
Review Comment:
Suggest snapshotting the in-progress ids before listing and skipping anything that was in flight at listing time:
```java
Set<String> busy = Set.copyOf(inProgress); // before request.execute()
...
if (busy.contains(id) || !inProgress.add(id)) {
// skip
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25520: camel-google-mail - Do not consume a message again while its exchange is in flight [camel]
Posted by "allthingssecurity (via GitHub)" <gi...@apache.org>.
allthingssecurity commented on code in PR #27671:
URL: https://github.com/apache/camel/pull/27671#discussion_r4240520036
##########
components/camel-google/camel-google-mail/src/main/java/org/apache/camel/component/google/mail/stream/GoogleMailStreamConsumer.java:
##########
@@ -91,17 +95,41 @@ protected int poll() throws Exception {
forceConsumerAsReady();
if (c.getMessages() != null) {
- for (Message message : c.getMessages()) {
- Message mess
- = getClient().users().messages().get("me", message.getId()).setFormat(messageFormat()).execute();
- Exchange exchange = createExchange(getEndpoint().getExchangePattern(), mess);
- answer.add(exchange);
+ try {
+ for (Message message : c.getMessages()) {
+ addExchange(message.getId(), answer);
+ }
+ } catch (Exception e) {
+ // none of the messages of this poll is processed
+ releaseInProgress(answer);
+ throw e;
}
}
return processBatch(CastUtils.cast(answer));
}
+ private void addExchange(String id, Queue<Exchange> answer) throws Exception {
+ // skip the message if an exchange of a previous poll is still processing it
+ if (!inProgress.add(id)) {
Review Comment:
Confirmed and fixed in c31224e63271, with the suggested shape: `Set.copyOf(inProgress)` is taken before
`request.execute()` (`GoogleMailStreamConsumer.java:95`) and `addExchange` skips an id that is in the snapshot or
already in progress (line 119). New test `messageCompletedAfterTheListingIsNotConsumedAgain`: the fake Gmail transport
holds the listing response until the in-flight exchange has completed (marked as read, id released), gated on its
`ExchangeCompletedEvent`, no sleeps. Against the previous commit `m1` is got twice (`expected: <1> but was: <2>`), now
once. The snapshot is per poll, so nothing is skipped for longer than one poll: with `markAsRead=false`, or after a
failed exchange, a message that was in flight when the poll started is consumed by the next poll instead of this one.
The exchanges that are never routed (stopping consumer, failed poll) are now also released to the exchange factory
(line 137). Module suite: 49 tests, 0 failures (17 integration tests skipped, they need credentials).
_Claude Code on behalf of allthingssecurity_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org