You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "allthingssecurity (via GitHub)" <gi...@apache.org> on 2026/10/20 00:10:09 UTC
[PR] CAMEL-25770: camel-azure-storage-datalake - skip a listed file that cannot be got instead of failing the poll [camel]
allthingssecurity opened a new pull request, #28131:
URL: https://github.com/apache/camel/pull/28131
# Description
[CAMEL-25770](https://issues.apache.org/jira/browse/CAMEL-25770)
The datalake consumer lists the whole file system at every poll and gets every listed file when it creates the exchanges, with no catch for one file. A failure of one get (access denied by the ACL of that file, a file encrypted with a key the consumer does not have) failed the poll, and a file deleted after the listing (404) ended the poll with no exchange at all. As every poll lists the file system again, one file that cannot be read stopped the consumer from consuming any file. In Data Lake Storage Gen2 every file has its own ACL, so a file the consumer may list but not read is a normal state.
This change does what camel-aws2-s3 does since CAMEL-25558 (and what CAMEL-25760 does for camel-alibaba-oss and camel-huaweicloud-obs):
- a failed get of one listed file is collected, the file is skipped for this poll and reported to the consumer's exception handler with its name, and the other listed files are consumed;
- a file deleted after the listing is skipped at `DEBUG` instead of emptying the poll;
- the poll still fails (with the first failure) when none of the listed files could be got, as for expired credentials, so the error handling and the backoff of the scheduled consumer apply;
- a failure with `fileName` set, and a failure of the listing, are not changed.
The behaviour changes are in the 4.24 upgrade guide.
Tests:
- New `DataLakeConsumerFileThatCannotBeGotTest` (3 tests) with a mocked `DataLakeServiceClient`; the read of a forbidden file throws a `DataLakeStorageException` with status 403, that of a deleted file status 404. Without the fix `fileThatCannotBeGotDoesNotBlockTheOtherFiles` (a.txt, forbidden b.txt, c.txt) and `fileDeletedAfterTheListingIsSkipped` (a.txt, deleted b.txt, c.txt) receive no file at all. `pollFailsWhenNoListedFileCanBeGot` passes with and without the fix; it is there because the fix must not swallow that case.
- camel-azure-storage-datalake unit tests: 19 run, 0 failures (formatter and import sort run in the same build).
- Not run: against Azure Storage.
How it was found: variant analysis of the camel-aws2-s3 fix CAMEL-25558 across the object-store consumers (a Semgrep rule for a batch consumer that creates the exchange of every listed item, which downloads it, with no per-item catch). camel-minio, camel-google-storage and camel-azure-storage-blob have the same defect and are fixed in separate PRs.
# Target
- [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch)
# Tracking
- [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it).
# Apache Camel coding standards and style
- [x] I checked that each commit in the pull request has a meaningful subject line and body.
- [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes.
(I built and tested camel-azure-storage-datalake, including the formatter and import-sort plugins. I did not run the full root build.)
# AI-assisted contributions
- [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used.
This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer.
_Claude Code on behalf of allthingssecurity_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25770: camel-azure-storage-datalake - skip a listed file that cannot be got instead of failing the poll [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28131:
URL: https://github.com/apache/camel/pull/28131#issuecomment-6284335322
:star2: Thank you for your contribution to the Apache Camel project! :star2:
:robot: CI automation will test this PR automatically.
:camel: Apache Camel Committers, please review the following items:
* First-time contributors **require MANUAL approval** for the GitHub Actions to run
* You can use the command `/component-test (camel-)component-name1 (camel-)component-name2..` to request a test from the test bot although they are normally detected and executed by CI.
* You can label PRs using `skip-tests` and `test-dependents` to fine-tune the checks executed by this PR.
* Build and test logs are available in the summary page. **Only** [Apache Camel committers](https://camel.apache.org/community/team/#committers) have access to the summary.
:warning: Be careful when sharing logs. Review their contents before sharing them publicly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25770: camel-azure-storage-datalake - skip a listed file that cannot be got instead of failing the poll [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28131:
URL: https://github.com/apache/camel/pull/28131#issuecomment-6284629187
<!-- ci-tested-modules -->
:test_tube: **CI tested the following changed modules:**
- `components/camel-azure/camel-azure-storage-datalake`
- `docs`
:repeat: **1 test passed only after a retry on JDK 25** (1 retried attempt)
<details><summary>Recovered flaky tests on JDK 25 (1)</summary>
| Module | Test | Failed attempts | First failure |
| --- | --- | --- | --- |
| `dsl/camel-jbang/camel-jbang-plugin-tui` | `LaunchManagerOutcomeTest.startedWhenTheStatusFileSaysSo` | 1 | expected: null but was: LaunchOutcome[ok=false, log=null] |
> :information_source: These tests did **not** fail the build. Surefire retried them and they passed.
> Retries are enabled project-wide by `surefire.rerunFailingTestsCount` in `parent/pom.xml`.
</details>
---
<details><summary>:microscope: Scalpel shadow comparison — Scalpel: 8 of 707 tested, 26 compile-only — current: 8 all tested</summary>
[Maveniverse Scalpel](https://github.com/maveniverse/scalpel) detected **8 affected modules** (current approach: 8).
**Skip-tests mode would test 8 modules** (2 direct + 7 downstream), **skip tests for 26** (generated code, meta-modules)
<details><summary>Modules Scalpel would test (8)</summary>
- `camel-azure-storage-datalake` ← components/camel-azure/camel-azure-storage-datalake/src/main/java/org/apache/camel/component/azure/storage/datalake/DataLakeConsumer.java, components/camel-azure/camel-azure-storage-datalake/src/test/java/org/apache/camel/component/azure/storage/datalake/DataLakeConsumerFileThatCannotBeGotTest.java
- `camel-jbang-mcp` ← downstream of org.apache.camel:camel-catalog
- `camel-jbang-plugin-mcp` ← downstream of org.apache.camel:camel-jbang-core
- `camel-jbang-plugin-tui` ← downstream of org.apache.camel:camel-catalog
- `camel-jbang-plugin-validate` ← downstream of org.apache.camel:camel-yaml-dsl-validator
- `camel-launcher-container` ← downstream of org.apache.camel:camel-launcher
- `camel-yaml-dsl-validator` ← downstream of org.apache.camel:camel-catalog
- `camel-yaml-dsl-validator-maven-plugin` ← downstream of org.apache.camel:camel-yaml-dsl-validator
</details>
<details><summary>Modules with tests skipped (26)</summary>
- `apache-camel`
- `camel-allcomponents`
- `camel-catalog`
- `camel-catalog-console`
- `camel-catalog-maven`
- `camel-catalog-suggest`
- `camel-componentdsl`
- `camel-endpointdsl`
- `camel-endpointdsl-support`
- `camel-itest`
- `camel-jbang-core`
- `camel-jbang-it`
- `camel-jbang-main`
- `camel-jbang-plugin-edit`
- `camel-jbang-plugin-generate`
- `camel-jbang-plugin-kubernetes`
- `camel-jbang-plugin-test`
- `camel-kamelet-main`
- `camel-launcher`
- `camel-report-maven-plugin`
- `camel-route-parser`
- `camel-yaml-dsl`
- `camel-yaml-dsl-deserializers`
- `camel-yaml-dsl-maven-plugin`
- `coverage`
- `dummy-component`
</details>
> :information_source: Shadow mode — Scalpel observes but does not affect test execution. [Learn more](https://github.com/maveniverse/scalpel)
</details>
<details><summary>All tested modules (35 modules, 4m 29s total)</summary>
**Total reactor time:** 4m 29s
| Module | Duration | Status |
| --- | --- | --- |
| Camel :: JBang :: Plugin :: Validate | 28.1s | SUCCESS |
| Camel :: YAML DSL | 25.8s | SUCCESS |
| Camel :: YAML DSL :: Validator | 25.1s | SUCCESS |
| Camel :: JBang :: MCP | 23.9s | SUCCESS |
| Camel :: Catalog :: Camel Catalog | 23.1s | SUCCESS |
| Camel :: Component DSL | 19.7s | SUCCESS |
| Camel :: Azure :: Storage Datalake | 15.8s | SUCCESS |
| Camel :: Docs | 15.6s | SUCCESS |
| Camel :: Kamelet Main | 14.6s | SUCCESS |
| Camel :: JBang :: Plugin :: Testing | 12.0s | SUCCESS |
| Camel :: JBang :: Plugin :: Kubernetes | 11.9s | SUCCESS |
| Camel :: Catalog :: Camel Report Maven Plugin | 8.6s | SUCCESS |
| Camel :: YAML DSL :: Deserializers | 8.3s | SUCCESS |
| Camel :: Catalog :: Camel Route Parser | 8.2s | SUCCESS |
| Camel :: All Components Sync point | 5.1s | SUCCESS |
| Camel :: YAML DSL :: Validator Maven Plugin | 4.0s | SUCCESS |
| Camel :: YAML DSL :: Maven Plugins | 3.6s | SUCCESS |
| Camel :: Catalog :: Maven | 2.9s | SUCCESS |
| Camel :: Catalog :: Suggest (deprecated) | 2.6s | SUCCESS |
| Camel :: Coverage | 1.6s | SUCCESS |
| Camel :: JBang :: Plugin :: Edit | 1.4s | SUCCESS |
| Camel :: Assembly | 1.2s | SUCCESS |
| Camel :: Catalog :: Dummy Component | 1.2s | SUCCESS |
| Camel :: Catalog :: Console | 1.1s | SUCCESS |
| Camel :: JBang :: Plugin :: Generate | 1.1s | SUCCESS |
| Camel :: Endpoint DSL :: Support | 1.0s | SUCCESS |
| Camel :: JBang :: Main | 0.7s | SUCCESS |
| Camel :: JBang :: Plugin :: MCP | 0.6s | SUCCESS |
| Camel :: Endpoint DSL | n/a | |
| Camel :: Integration Tests | n/a | |
| Camel :: JBang :: Core | n/a | |
| Camel :: JBang :: Integration tests | n/a | |
| Camel :: JBang :: Plugin :: TUI | n/a | |
| Camel :: Launcher | n/a | |
| Camel :: Launcher :: Container | n/a | |
**Top 20 slowest modules:**
- `Camel :: JBang :: Plugin :: Validate` (28.1s)
- `Camel :: YAML DSL` (25.8s)
- `Camel :: YAML DSL :: Validator` (25.1s)
- `Camel :: JBang :: MCP` (23.9s)
- `Camel :: Catalog :: Camel Catalog` (23.1s)
- `Camel :: Component DSL` (19.7s)
- `Camel :: Azure :: Storage Datalake` (15.8s)
- `Camel :: Docs` (15.6s)
- `Camel :: Kamelet Main` (14.6s)
- `Camel :: JBang :: Plugin :: Testing` (12.0s)
- `Camel :: JBang :: Plugin :: Kubernetes` (11.9s)
- `Camel :: Catalog :: Camel Report Maven Plugin` (8.6s)
- `Camel :: YAML DSL :: Deserializers` (8.3s)
- `Camel :: Catalog :: Camel Route Parser` (8.2s)
- `Camel :: All Components Sync point` (5.1s)
- `Camel :: YAML DSL :: Validator Maven Plugin` (4.0s)
- `Camel :: YAML DSL :: Maven Plugins` (3.6s)
- `Camel :: Catalog :: Maven` (2.9s)
- `Camel :: Catalog :: Suggest (deprecated)` (2.6s)
- `Camel :: Coverage` (1.6s)
</details>
---
:gear: [View full build and test results](https://github.com/apache/camel/actions/runs/39346024182)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25770: camel-azure-storage-datalake - skip a listed file that cannot be got instead of failing the poll [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus merged PR #28131:
URL: https://github.com/apache/camel/pull/28131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org