You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "allthingssecurity (via GitHub)" <gi...@apache.org> on 2026/10/11 00:04:11 UTC
[PR] CAMEL-25519: camel-google-storage - Do not consume an object again while its exchange is in flight [camel]
allthingssecurity opened a new pull request, #27670:
URL: https://github.com/apache/camel/pull/27670
# Description
[CAMEL-25519](https://issues.apache.org/jira/browse/CAMEL-25519)
The google-storage consumer hands each exchange to the route with an empty callback and does not wait; the object is deleted (`deleteAfterRead`, the default) or moved by the exchange's on completion. With an asynchronous route (`seda`, which hands the on completion over, kafka or another async producer, `threads()`), the next poll lists and downloads the same object again and routes it a second time. If the first exchange deletes the object after the next poll listed it, the download of that poll fails with a 404 and the whole poll ends with an exception. The same happens with `objectName` set.
This is the defect fixed for aws2-s3 in CAMEL-17110 (and for minio in CAMEL-25512, #27645, merged). The consumer now keeps the names of the objects whose exchanges are in progress: a poll skips them (listing and `objectName` mode), and a name is released when its exchange completes or fails (after the delete/move), or when the poll does not process it (download failure, consumer stopping). A listed object that no longer exists when its content is downloaded is skipped instead of failing the poll; the exchanges are created before the batch is routed, so `CamelBatchSize` only counts the routed objects. No new option, as in the minio fix: a plain set in the consumer, not cleared on route stop (in-flight names are released when their exchanges complete). With `deleteAfterRead=false` an object is now consumed again only after its previous exchange completed. 4.23 upgrade-guide note for both behaviour changes.
Tests (module `LocalStorageHelper` in-memory storage, no container):
- `GoogleCloudStorageConsumerInProgressTest`: the seda route holds the exchange until the consumer has completed a second poll (counted with a `pollStrategy`); listing and `objectName` mode. Without the change: `[a.txt was consumed again by a later poll while its first exchange was still in flight] expected: 1 but was: 2` (both modes).
- `GoogleCloudStorageConsumerDeletedObjectTest`: the storage deletes `b.txt` right after the listing; `a.txt` and `c.txt` are routed by the same poll with `CamelBatchSize` 2. Without the change the poll fails (`[the poll failed because b.txt was deleted between the listing and the download]`).
- `GoogleCloudStorageConsumerInProgressFailureTest`: a failed object is consumed again by a later poll; `notProcessedObjectIsConsumedAgain`: an object whose exchange a stopping consumer did not process is not left in progress. Both pass without the change too (there is no in-progress set on main); they guard the releases on failure and on stop.
- With the change, the camel-google-storage unit tests pass (58).
Found with a TLA+ model of the poll and the asynchronous completion: "with deleteAfterRead every object is routed once" is violated in 10 steps (poll, download, dispatch, poll again, download, dispatch again). I then reproduced it with the real consumer.
# Target
- [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch)
# Tracking
- [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it).
# Apache Camel coding standards and style
- [x] I checked that each commit in the pull request has a meaningful subject line and body.
- [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes.
(I built and tested camel-google-storage, including the formatter and import-sort plugins; the build did not change any generated file. I did not run the full root build.)
# AI-assisted contributions
- [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used.
This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer.
_Claude Code on behalf of allthingssecurity_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25519: camel-google-storage - Do not consume an object again while its exchange is in flight [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27670:
URL: https://github.com/apache/camel/pull/27670#issuecomment-6105966801
:star2: Thank you for your contribution to the Apache Camel project! :star2:
:robot: CI automation will test this PR automatically.
:camel: Apache Camel Committers, please review the following items:
* First-time contributors **require MANUAL approval** for the GitHub Actions to run
* You can use the command `/component-test (camel-)component-name1 (camel-)component-name2..` to request a test from the test bot although they are normally detected and executed by CI.
* You can label PRs using `skip-tests` and `test-dependents` to fine-tune the checks executed by this PR.
* Build and test logs are available in the summary page. **Only** [Apache Camel committers](https://camel.apache.org/community/team/#committers) have access to the summary.
:warning: Be careful when sharing logs. Review their contents before sharing them publicly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25519: camel-google-storage - Do not consume an object again while its exchange is in flight [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27670:
URL: https://github.com/apache/camel/pull/27670#issuecomment-6106373653
<!-- ci-tested-modules -->
:test_tube: **CI tested the following changed modules:**
- `components/camel-google/camel-google-storage`
- `docs`
---
<details><summary>:microscope: Scalpel shadow comparison — Scalpel: 9 of 704 tested, 26 compile-only — current: 9 all tested</summary>
[Maveniverse Scalpel](https://github.com/maveniverse/scalpel) detected **9 affected modules** (current approach: 9).
**Skip-tests mode would test 9 modules** (2 direct + 8 downstream), **skip tests for 26** (generated code, meta-modules)
<details><summary>Modules Scalpel would test (9)</summary>
- `camel-google-storage` ← components/camel-google/camel-google-storage/src/main/java/org/apache/camel/component/google/storage/GoogleCloudStorageConsumer.java, components/camel-google/camel-google-storage/src/test/java/org/apache/camel/component/google/storage/GoogleCloudStorageConsumerDeletedObjectTest.java, components/camel-google/camel-google-storage/src/test/java/org/apache/camel/component/google/storage/GoogleCloudStorageConsumerInProgressFailureTest.java, components/camel-google/camel-google-storage/src/test/java/org/apache/camel/component/google/storage/GoogleCloudStorageConsumerInProgressTest.java, components/camel-google/camel-google-storage/src/test/java/org/apache/camel/component/google/storage/GoogleCloudStorageConsumerListingWindowTest.java
- `camel-jbang-mcp` ← downstream of org.apache.camel:camel-catalog
- `camel-jbang-plugin-mcp` ← downstream of org.apache.camel:camel-jbang-core
- `camel-jbang-plugin-route-parser` ← downstream of org.apache.camel:camel-route-parser
- `camel-jbang-plugin-tui` ← downstream of org.apache.camel:camel-catalog
- `camel-jbang-plugin-validate` ← downstream of org.apache.camel:camel-yaml-dsl-validator
- `camel-launcher-container` ← downstream of org.apache.camel:camel-launcher
- `camel-yaml-dsl-validator` ← downstream of org.apache.camel:camel-catalog
- `camel-yaml-dsl-validator-maven-plugin` ← downstream of org.apache.camel:camel-yaml-dsl-validator
</details>
<details><summary>Modules with tests skipped (26)</summary>
- `apache-camel`
- `camel-allcomponents`
- `camel-catalog`
- `camel-catalog-console`
- `camel-catalog-maven`
- `camel-catalog-suggest`
- `camel-componentdsl`
- `camel-endpointdsl`
- `camel-endpointdsl-support`
- `camel-itest`
- `camel-jbang-core`
- `camel-jbang-it`
- `camel-jbang-main`
- `camel-jbang-plugin-edit`
- `camel-jbang-plugin-generate`
- `camel-jbang-plugin-kubernetes`
- `camel-jbang-plugin-test`
- `camel-kamelet-main`
- `camel-launcher`
- `camel-report-maven-plugin`
- `camel-route-parser`
- `camel-yaml-dsl`
- `camel-yaml-dsl-deserializers`
- `camel-yaml-dsl-maven-plugin`
- `coverage`
- `dummy-component`
</details>
> :information_source: Shadow mode — Scalpel observes but does not affect test execution. [Learn more](https://github.com/maveniverse/scalpel)
</details>
<details><summary>All tested modules (36 modules, 5m 53s total)</summary>
**Total reactor time:** 5m 53s
| Module | Duration | Status |
| --- | --- | --- |
| Camel :: Launcher | 48.9s | SUCCESS |
| Camel :: JBang :: MCP | 41.9s | SUCCESS |
| Camel :: Google :: Storage | 32.2s | SUCCESS |
| Camel :: JBang :: Plugin :: Kubernetes | 26.1s | SUCCESS |
| Camel :: Catalog :: Camel Catalog | 24.8s | SUCCESS |
| Camel :: YAML DSL :: Validator | 21.6s | SUCCESS |
| Camel :: YAML DSL | 19.6s | SUCCESS |
| Camel :: Component DSL | 18.6s | SUCCESS |
| Camel :: JBang :: Plugin :: Validate | 17.3s | SUCCESS |
| Camel :: Docs | 16.3s | SUCCESS |
| Camel :: Kamelet Main | 14.3s | SUCCESS |
| Camel :: Catalog :: Camel Report Maven Plugin | 11.8s | SUCCESS |
| Camel :: JBang :: Plugin :: Testing | 10.6s | SUCCESS |
| Camel :: YAML DSL :: Deserializers | 8.7s | SUCCESS |
| Camel :: Catalog :: Camel Route Parser | 7.8s | SUCCESS |
| Camel :: All Components Sync point | 5.1s | SUCCESS |
| Camel :: YAML DSL :: Validator Maven Plugin | 4.4s | SUCCESS |
| Camel :: Catalog :: Maven | 3.5s | SUCCESS |
| Camel :: YAML DSL :: Maven Plugins | 3.2s | SUCCESS |
| Camel :: Catalog :: Suggest (deprecated) | 2.9s | SUCCESS |
| Camel :: Assembly | 1.7s | SUCCESS |
| Camel :: JBang :: Plugin :: MCP | 1.6s | SUCCESS |
| Camel :: JBang :: Plugin :: Edit | 1.5s | SUCCESS |
| Camel :: Coverage | 1.4s | SUCCESS |
| Camel :: Catalog :: Dummy Component | 1.2s | SUCCESS |
| Camel :: Endpoint DSL :: Support | 1.1s | SUCCESS |
| Camel :: JBang :: Plugin :: Generate | 1.0s | SUCCESS |
| Camel :: JBang :: Integration tests | 0.9s | SUCCESS |
| Camel :: Catalog :: Console | 0.8s | SUCCESS |
| Camel :: Launcher :: Container | 0.7s | SUCCESS |
| Camel :: JBang :: Main | 0.7s | SUCCESS |
| Camel :: JBang :: Plugin :: Route Parser | 0.6s | SUCCESS |
| Camel :: Endpoint DSL | n/a | |
| Camel :: Integration Tests | n/a | |
| Camel :: JBang :: Core | n/a | |
| Camel :: JBang :: Plugin :: TUI | n/a | |
**Top 20 slowest modules:**
- `Camel :: Launcher` (48.9s)
- `Camel :: JBang :: MCP` (41.9s)
- `Camel :: Google :: Storage` (32.2s)
- `Camel :: JBang :: Plugin :: Kubernetes` (26.1s)
- `Camel :: Catalog :: Camel Catalog` (24.8s)
- `Camel :: YAML DSL :: Validator` (21.6s)
- `Camel :: YAML DSL` (19.6s)
- `Camel :: Component DSL` (18.6s)
- `Camel :: JBang :: Plugin :: Validate` (17.3s)
- `Camel :: Docs` (16.3s)
- `Camel :: Kamelet Main` (14.3s)
- `Camel :: Catalog :: Camel Report Maven Plugin` (11.8s)
- `Camel :: JBang :: Plugin :: Testing` (10.6s)
- `Camel :: YAML DSL :: Deserializers` (8.7s)
- `Camel :: Catalog :: Camel Route Parser` (7.8s)
- `Camel :: All Components Sync point` (5.1s)
- `Camel :: YAML DSL :: Validator Maven Plugin` (4.4s)
- `Camel :: Catalog :: Maven` (3.5s)
- `Camel :: YAML DSL :: Maven Plugins` (3.2s)
- `Camel :: Catalog :: Suggest (deprecated)` (2.9s)
</details>
---
:gear: [View full build and test results](https://github.com/apache/camel/actions/runs/38117001079)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25519: camel-google-storage - Do not consume an object again while its exchange is in flight [camel]
Posted by "allthingssecurity (via GitHub)" <gi...@apache.org>.
allthingssecurity commented on code in PR #27670:
URL: https://github.com/apache/camel/pull/27670#discussion_r4240519860
##########
components/camel-google/camel-google-storage/src/main/java/org/apache/camel/component/google/storage/GoogleCloudStorageConsumer.java:
##########
@@ -169,12 +173,44 @@ protected Queue<Exchange> createExchanges(List<Blob> blobList) {
}
} catch (Exception e) {
LOG.warn("Error getting object due: {}", e.getMessage(), e);
+ // none of the objects of this poll is processed
+ releaseInProgress(answer);
throw e;
}
return answer;
}
+ private void addExchange(Blob blob, String key, Queue<Exchange> answer) {
+ // skip the object if an exchange of a previous poll is still processing it
+ if (!inProgress.add(key)) {
Review Comment:
Confirmed and fixed in ec35d39b3e42. `poll()` now takes `Set.copyOf(inProgress)` before `list` / `get`
(`GoogleCloudStorageConsumer.java:103`) and `addExchange` skips a key that is in that snapshot or already in progress
(line 194). New `GoogleCloudStorageConsumerListingWindowTest` holds the listing (and, with `objectName`, the get) open
with a `Storage` proxy until the in-flight exchange has completed (object deleted, name released), with
`includeBody=false`: against the previous commit both cases route `a.txt` a second time, now once. No sleeps: the gate
is the `ExchangeCompletedEvent` of the held exchange. The snapshot is per poll, so an object is never skipped for longer
than one poll: with `deleteAfterRead=false`, or after a failed exchange, an object that was in flight when the poll
started is consumed by the next poll instead of this one. Module suite: 61 tests, 0 failures.
_Claude Code on behalf of allthingssecurity_
##########
docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_23.adoc:
##########
@@ -3692,6 +3692,14 @@ When `downloadFileName` is fully dynamic and has no static directory prefix (for
`downloadFileName=${file:name}`), a relative destination must stay within the working directory. An absolute
destination that comes from the route's own expression is not confined.
+=== camel-google-storage - objects whose exchange is in flight
Review Comment:
Done in ec35d39b3e42: the second heading is gone and the text is now the last paragraph of the existing
`=== camel-google-storage` section.
_Claude Code on behalf of allthingssecurity_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25519: camel-google-storage - Do not consume an object again while its exchange is in flight [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on code in PR #27670:
URL: https://github.com/apache/camel/pull/27670#discussion_r4240437004
##########
components/camel-google/camel-google-storage/src/main/java/org/apache/camel/component/google/storage/GoogleCloudStorageConsumer.java:
##########
@@ -169,12 +173,44 @@ protected Queue<Exchange> createExchanges(List<Blob> blobList) {
}
} catch (Exception e) {
LOG.warn("Error getting object due: {}", e.getMessage(), e);
+ // none of the objects of this poll is processed
+ releaseInProgress(answer);
throw e;
}
return answer;
}
+ private void addExchange(Blob blob, String key, Queue<Exchange> answer) {
+ // skip the object if an exchange of a previous poll is still processing it
+ if (!inProgress.add(key)) {
Review Comment:
Same window as in the google-mail PR: snapshot the set before listing (`Set<String> busy = Set.copyOf(inProgress);`) and skip keys that were in flight at listing time.
##########
docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_23.adoc:
##########
@@ -3692,6 +3692,14 @@ When `downloadFileName` is fully dynamic and has no static directory prefix (for
`downloadFileName=${file:name}`), a relative destination must stay within the working directory. An absolute
destination that comes from the route's own expression is not confined.
+=== camel-google-storage - objects whose exchange is in flight
Review Comment:
There's already a `=== camel-google-storage` section a few lines above (line 3672); please fold this text into it rather than adding a second heading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25519: camel-google-storage - Do not consume an object again while its exchange is in flight [camel]
Posted by "allthingssecurity (via GitHub)" <gi...@apache.org>.
allthingssecurity commented on PR #27670:
URL: https://github.com/apache/camel/pull/27670#issuecomment-6106216309
All three points are addressed in ec35d39b3e42:
- Snapshot before `list` / `get`: see the inline reply; deterministic test for both the listing and the `objectName`
path with `includeBody=false`.
- Exchange not released on a failed download: `createExchange` now releases the exchange for any failure while it is
populated (`GoogleCloudStorageConsumer.java:368-376`), which covers the 404 skip, other download failures and the
rejected object name. `exchangeOfAnObjectThatNoLongerExistsIsReleased` fails on the previous commit. The exchanges a
stopping consumer does not route, or that are dropped when the poll fails, are released as well (line 221).
- Upgrade guide: one `=== camel-google-storage` section.
Module suite: 61 tests, 0 failures, 0 errors.
_Claude Code on behalf of allthingssecurity_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25519: camel-google-storage - Do not consume an object again while its exchange is in flight [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus merged PR #27670:
URL: https://github.com/apache/camel/pull/27670
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org