You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "allthingssecurity (via GitHub)" <gi...@apache.org> on 2026/10/19 00:11:56 UTC
[PR] CAMEL-25760: camel-alibaba-oss - skip a listed object that cannot be got instead of failing the poll [camel]
allthingssecurity opened a new pull request, #28075:
URL: https://github.com/apache/camel/pull/28075
# Description
[CAMEL-25760](https://issues.apache.org/jira/browse/CAMEL-25760)
The OSS consumer lists one page of the bucket and gets every listed object. An object that no longer exists is skipped (CAMEL-25551); any other failure of a get was rethrown, so the whole poll failed, and the continuation token was only moved on after the batch was processed. The next poll therefore listed the same page again. One object whose get always fails (access denied for that object, an Archive or Cold Archive object that is not restored, a disabled KMS key) blocked every other object of its page and all the following pages; with a bucket that fits in one page the consumer consumed nothing, until someone removed or restored the object.
This change does what camel-aws2-s3 does since CAMEL-25558:
- a failed get of one listed object is collected, the object is skipped for this poll and reported to the consumer's exception handler with its key, and the other listed objects are consumed;
- the poll still fails (with the first failure) when none of the listed objects could be got, as for expired credentials or no permission to get objects, so the error handling and the backoff of the scheduled consumer apply;
- the continuation token is taken from the listing before the objects are got, as camel-aws2-s3 and camel-huaweicloud-obs do with their marker, so a page whose objects cannot be got does not hold the consumer back. Its objects are listed again when the listing starts over.
An object deleted after the listing is still skipped at `DEBUG`. The behaviour change (a skipped object is reported at every poll that lists it; a failed poll no longer repeats the same page) is in the 4.24 upgrade guide.
Tests:
- New `OSSConsumerObjectThatCannotBeGotTest` (3 tests) with a fake bucket that lists its keys in order, `maxKeys` at a time with a continuation token, and answers the get of a forbidden object with 403. Without the fix `objectThatCannotBeGotDoesNotBlockTheOtherObjects` (a.txt, forbidden b.txt, c.txt) and `pageOfObjectsThatCannotBeGotDoesNotBlockTheNextPage` (`maxMessagesPerPoll=1`, forbidden a.txt) receive no object at all. `pollFailsWhenNoListedObjectCanBeGot` passes with and without the fix; it is there because the fix must not swallow that case.
- camel-alibaba-oss unit tests: 22 run, 0 failures (formatter and import sort run in the same build).
- Not run: against a real OSS bucket. The status codes for forbidden and archived objects are from the OSS documentation.
How it was found: comparing the sibling object-store consumers (camel-aws2-s3 after CAMEL-25558, camel-huaweicloud-obs, camel-alibaba-oss) on the same listing, get and commit steps. camel-huaweicloud-obs also fails the poll for one object, but moves its marker on, so it stops for good only when the bucket fits in one page (fixed separately). camel-alibaba-oss also keeps its listing position, so it stops for good whatever the size of the bucket.
Not changed here: camel-minio (a failed get in `MinioConsumer.processBatch` stops the rest of the batch and fails the poll) and camel-google-storage (`GoogleCloudStorageConsumer.addExchange` rethrows, and the poll fails) have the same kind of defect; they are follow-ups.
# Target
- [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch)
# Tracking
- [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it).
# Apache Camel coding standards and style
- [x] I checked that each commit in the pull request has a meaningful subject line and body.
- [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes.
(I built and tested camel-alibaba-oss, including the formatter and import-sort plugins. I did not run the full root build.)
# AI-assisted contributions
- [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used.
This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer.
_Claude Code on behalf of allthingssecurity_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25760: camel-alibaba-oss - skip a listed object that cannot be got instead of failing the poll [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on code in PR #28075:
URL: https://github.com/apache/camel/pull/28075#discussion_r4311100721
##########
components/camel-alibaba/camel-alibaba-oss/src/test/java/org/apache/camel/component/alibaba/oss/OSSConsumerObjectThatCannotBeGotTest.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.alibaba.oss;
+
+import java.io.ByteArrayInputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.aliyun.sdk.service.oss2.OSSClient;
+import com.aliyun.sdk.service.oss2.exceptions.OperationException;
+import com.aliyun.sdk.service.oss2.exceptions.ServiceException;
+import com.aliyun.sdk.service.oss2.models.DeleteObjectRequest;
+import com.aliyun.sdk.service.oss2.models.GetObjectRequest;
+import com.aliyun.sdk.service.oss2.models.GetObjectResult;
+import com.aliyun.sdk.service.oss2.models.ListObjectsV2Request;
+import com.aliyun.sdk.service.oss2.models.ListObjectsV2Result;
+import com.aliyun.sdk.service.oss2.models.ObjectSummary;
+import org.apache.camel.CamelContext;
+import org.apache.camel.Consumer;
+import org.apache.camel.Endpoint;
+import org.apache.camel.Exchange;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.spi.ExceptionHandler;
+import org.apache.camel.spi.PollingConsumerPollStrategy;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.awaitility.Awaitility.await;
+
+/**
+ * A listed object that cannot be got (access denied, an archived object that is not restored) must not stop the
+ * consumer: the other listed objects are consumed, the next page of the listing is read, and the poll fails only when
+ * none of the listed objects could be got. The fake bucket lists its keys in order, a page of maxKeys keys at a time,
+ * with the last key of a page as continuation token, and answers the get of a forbidden object with 403.
+ */
+class OSSConsumerObjectThatCannotBeGotTest extends CamelTestSupport {
+
+ private final Set<String> objects = new ConcurrentSkipListSet<>();
+ private final Set<String> forbidden = ConcurrentHashMap.newKeySet();
+ private final AtomicInteger failedPolls = new AtomicInteger();
+ private final List<String> handled = new CopyOnWriteArrayList<>();
+
+ @Override
+ protected CamelContext createCamelContext() throws Exception {
+ CamelContext context = super.createCamelContext();
+ context.getRegistry().bind("ossClient", ossClient());
+ context.getRegistry().bind("countPolls", new PollingConsumerPollStrategy() {
+ @Override
+ public boolean begin(Consumer consumer, Endpoint endpoint) {
+ return true;
+ }
+
+ @Override
+ public void commit(Consumer consumer, Endpoint endpoint, int polledMessages) {
+ // noop
+ }
+
+ @Override
+ public boolean rollback(Consumer consumer, Endpoint endpoint, int retryCounter, Exception e) {
+ failedPolls.incrementAndGet();
+ return false;
+ }
+ });
+ context.getRegistry().bind("recordErrors", new ExceptionHandler() {
+ @Override
+ public void handleException(Throwable exception) {
+ handled.add(String.valueOf(exception.getMessage()));
+ }
+
+ @Override
+ public void handleException(String message, Throwable exception) {
+ handled.add(message);
+ }
+
+ @Override
+ public void handleException(String message, Exchange exchange, Throwable exception) {
+ handled.add(message);
+ }
+ });
+ return context;
+ }
+
+ @Test
+ void objectThatCannotBeGotDoesNotBlockTheOtherObjects() throws Exception {
+ objects.add("a.txt");
+ objects.add("b.txt");
+ objects.add("c.txt");
+ forbidden.add("b.txt");
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceivedInAnyOrder("a.txt", "c.txt");
+
+ addConsumerRoute("&deleteAfterRead=true&exceptionHandler=#recordErrors");
+
+ mock.assertIsSatisfied();
+ assertThat(objects).as("a.txt and c.txt are consumed and deleted, b.txt stays").containsExactly("b.txt");
Review Comment:
With `deleteAfterRead=true` the object is deleted when the exchange completes, after it reached `mock:result`, so `assertIsSatisfied()` can return before the last object is deleted. This passed in CI, but the same test in the OBS PR (#28076) failed on exactly this race. Waiting for the bucket closes it:
```suggestion
await().atMost(20, TimeUnit.SECONDS).untilAsserted(
() -> assertThat(objects).as("a.txt and c.txt are consumed and deleted, b.txt stays").containsExactly("b.txt"));
```
##########
components/camel-alibaba/camel-alibaba-oss/src/test/java/org/apache/camel/component/alibaba/oss/OSSConsumerObjectThatCannotBeGotTest.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.alibaba.oss;
+
+import java.io.ByteArrayInputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.aliyun.sdk.service.oss2.OSSClient;
+import com.aliyun.sdk.service.oss2.exceptions.OperationException;
+import com.aliyun.sdk.service.oss2.exceptions.ServiceException;
+import com.aliyun.sdk.service.oss2.models.DeleteObjectRequest;
+import com.aliyun.sdk.service.oss2.models.GetObjectRequest;
+import com.aliyun.sdk.service.oss2.models.GetObjectResult;
+import com.aliyun.sdk.service.oss2.models.ListObjectsV2Request;
+import com.aliyun.sdk.service.oss2.models.ListObjectsV2Result;
+import com.aliyun.sdk.service.oss2.models.ObjectSummary;
+import org.apache.camel.CamelContext;
+import org.apache.camel.Consumer;
+import org.apache.camel.Endpoint;
+import org.apache.camel.Exchange;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.spi.ExceptionHandler;
+import org.apache.camel.spi.PollingConsumerPollStrategy;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.awaitility.Awaitility.await;
+
+/**
+ * A listed object that cannot be got (access denied, an archived object that is not restored) must not stop the
+ * consumer: the other listed objects are consumed, the next page of the listing is read, and the poll fails only when
+ * none of the listed objects could be got. The fake bucket lists its keys in order, a page of maxKeys keys at a time,
+ * with the last key of a page as continuation token, and answers the get of a forbidden object with 403.
+ */
+class OSSConsumerObjectThatCannotBeGotTest extends CamelTestSupport {
+
+ private final Set<String> objects = new ConcurrentSkipListSet<>();
+ private final Set<String> forbidden = ConcurrentHashMap.newKeySet();
+ private final AtomicInteger failedPolls = new AtomicInteger();
+ private final List<String> handled = new CopyOnWriteArrayList<>();
+
+ @Override
+ protected CamelContext createCamelContext() throws Exception {
+ CamelContext context = super.createCamelContext();
+ context.getRegistry().bind("ossClient", ossClient());
+ context.getRegistry().bind("countPolls", new PollingConsumerPollStrategy() {
+ @Override
+ public boolean begin(Consumer consumer, Endpoint endpoint) {
+ return true;
+ }
+
+ @Override
+ public void commit(Consumer consumer, Endpoint endpoint, int polledMessages) {
+ // noop
+ }
+
+ @Override
+ public boolean rollback(Consumer consumer, Endpoint endpoint, int retryCounter, Exception e) {
+ failedPolls.incrementAndGet();
+ return false;
+ }
+ });
+ context.getRegistry().bind("recordErrors", new ExceptionHandler() {
+ @Override
+ public void handleException(Throwable exception) {
+ handled.add(String.valueOf(exception.getMessage()));
+ }
+
+ @Override
+ public void handleException(String message, Throwable exception) {
+ handled.add(message);
+ }
+
+ @Override
+ public void handleException(String message, Exchange exchange, Throwable exception) {
+ handled.add(message);
+ }
+ });
+ return context;
+ }
+
+ @Test
+ void objectThatCannotBeGotDoesNotBlockTheOtherObjects() throws Exception {
+ objects.add("a.txt");
+ objects.add("b.txt");
+ objects.add("c.txt");
+ forbidden.add("b.txt");
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceivedInAnyOrder("a.txt", "c.txt");
+
+ addConsumerRoute("&deleteAfterRead=true&exceptionHandler=#recordErrors");
+
+ mock.assertIsSatisfied();
+ assertThat(objects).as("a.txt and c.txt are consumed and deleted, b.txt stays").containsExactly("b.txt");
+ await().atMost(20, TimeUnit.SECONDS).untilAsserted(
+ () -> assertThat(handled).as("the skipped object is reported").anyMatch(m -> m.contains("b.txt")));
+ }
+
+ @Test
+ void pageOfObjectsThatCannotBeGotDoesNotBlockTheNextPage() throws Exception {
+ objects.add("a.txt");
+ objects.add("b.txt");
+ objects.add("c.txt");
+ forbidden.add("a.txt");
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceivedInAnyOrder("b.txt", "c.txt");
+
+ // one object per page: the first page holds only a.txt
+ addConsumerRoute("&deleteAfterRead=true&maxMessagesPerPoll=1&pollStrategy=#countPolls");
+
+ mock.assertIsSatisfied();
+ assertThat(objects).as("b.txt and c.txt are consumed and deleted, a.txt stays").containsExactly("a.txt");
Review Comment:
Same race as above.
```suggestion
await().atMost(20, TimeUnit.SECONDS).untilAsserted(
() -> assertThat(objects).as("b.txt and c.txt are consumed and deleted, a.txt stays").containsExactly("a.txt"));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25760: camel-alibaba-oss - skip a listed object that cannot be got instead of failing the poll [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus merged PR #28075:
URL: https://github.com/apache/camel/pull/28075
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25760: camel-alibaba-oss - skip a listed object that cannot be got instead of failing the poll [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus closed pull request #28075: CAMEL-25760: camel-alibaba-oss - skip a listed object that cannot be got instead of failing the poll
URL: https://github.com/apache/camel/pull/28075
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25760: camel-alibaba-oss - skip a listed object that cannot be got instead of failing the poll [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus closed pull request #28075: CAMEL-25760: camel-alibaba-oss - skip a listed object that cannot be got instead of failing the poll
URL: https://github.com/apache/camel/pull/28075
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25760: camel-alibaba-oss - skip a listed object that cannot be got instead of failing the poll [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #28075:
URL: https://github.com/apache/camel/pull/28075#issuecomment-6261557314
Closing and reopening to re-run CI against main: the `CatalogSamplesTest.callingAServiceIsTheHttpComponentNotEnrich` failure in camel-jbang-core came from main and is fixed there (bbd6f750aa00). Nothing to change in this PR.
_Claude Code on behalf of Claus Ibsen_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25760: camel-alibaba-oss - skip a listed object that cannot be got instead of failing the poll [camel]
Posted by "allthingssecurity (via GitHub)" <gi...@apache.org>.
allthingssecurity commented on code in PR #28075:
URL: https://github.com/apache/camel/pull/28075#discussion_r4313704996
##########
components/camel-alibaba/camel-alibaba-oss/src/test/java/org/apache/camel/component/alibaba/oss/OSSConsumerObjectThatCannotBeGotTest.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.alibaba.oss;
+
+import java.io.ByteArrayInputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.aliyun.sdk.service.oss2.OSSClient;
+import com.aliyun.sdk.service.oss2.exceptions.OperationException;
+import com.aliyun.sdk.service.oss2.exceptions.ServiceException;
+import com.aliyun.sdk.service.oss2.models.DeleteObjectRequest;
+import com.aliyun.sdk.service.oss2.models.GetObjectRequest;
+import com.aliyun.sdk.service.oss2.models.GetObjectResult;
+import com.aliyun.sdk.service.oss2.models.ListObjectsV2Request;
+import com.aliyun.sdk.service.oss2.models.ListObjectsV2Result;
+import com.aliyun.sdk.service.oss2.models.ObjectSummary;
+import org.apache.camel.CamelContext;
+import org.apache.camel.Consumer;
+import org.apache.camel.Endpoint;
+import org.apache.camel.Exchange;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.spi.ExceptionHandler;
+import org.apache.camel.spi.PollingConsumerPollStrategy;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.awaitility.Awaitility.await;
+
+/**
+ * A listed object that cannot be got (access denied, an archived object that is not restored) must not stop the
+ * consumer: the other listed objects are consumed, the next page of the listing is read, and the poll fails only when
+ * none of the listed objects could be got. The fake bucket lists its keys in order, a page of maxKeys keys at a time,
+ * with the last key of a page as continuation token, and answers the get of a forbidden object with 403.
+ */
+class OSSConsumerObjectThatCannotBeGotTest extends CamelTestSupport {
+
+ private final Set<String> objects = new ConcurrentSkipListSet<>();
+ private final Set<String> forbidden = ConcurrentHashMap.newKeySet();
+ private final AtomicInteger failedPolls = new AtomicInteger();
+ private final List<String> handled = new CopyOnWriteArrayList<>();
+
+ @Override
+ protected CamelContext createCamelContext() throws Exception {
+ CamelContext context = super.createCamelContext();
+ context.getRegistry().bind("ossClient", ossClient());
+ context.getRegistry().bind("countPolls", new PollingConsumerPollStrategy() {
+ @Override
+ public boolean begin(Consumer consumer, Endpoint endpoint) {
+ return true;
+ }
+
+ @Override
+ public void commit(Consumer consumer, Endpoint endpoint, int polledMessages) {
+ // noop
+ }
+
+ @Override
+ public boolean rollback(Consumer consumer, Endpoint endpoint, int retryCounter, Exception e) {
+ failedPolls.incrementAndGet();
+ return false;
+ }
+ });
+ context.getRegistry().bind("recordErrors", new ExceptionHandler() {
+ @Override
+ public void handleException(Throwable exception) {
+ handled.add(String.valueOf(exception.getMessage()));
+ }
+
+ @Override
+ public void handleException(String message, Throwable exception) {
+ handled.add(message);
+ }
+
+ @Override
+ public void handleException(String message, Exchange exchange, Throwable exception) {
+ handled.add(message);
+ }
+ });
+ return context;
+ }
+
+ @Test
+ void objectThatCannotBeGotDoesNotBlockTheOtherObjects() throws Exception {
+ objects.add("a.txt");
+ objects.add("b.txt");
+ objects.add("c.txt");
+ forbidden.add("b.txt");
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceivedInAnyOrder("a.txt", "c.txt");
+
+ addConsumerRoute("&deleteAfterRead=true&exceptionHandler=#recordErrors");
+
+ mock.assertIsSatisfied();
+ assertThat(objects).as("a.txt and c.txt are consumed and deleted, b.txt stays").containsExactly("b.txt");
Review Comment:
Thanks, right: the delete happens when the exchange completes, after `mock:result`. Applied as suggested in bd114c44ceba (both checks); the test class passed 3 runs in a row, camel-alibaba-oss 22 tests, 0 failures.
_Claude Code on behalf of allthingssecurity_
##########
components/camel-alibaba/camel-alibaba-oss/src/test/java/org/apache/camel/component/alibaba/oss/OSSConsumerObjectThatCannotBeGotTest.java:
##########
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.alibaba.oss;
+
+import java.io.ByteArrayInputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import com.aliyun.sdk.service.oss2.OSSClient;
+import com.aliyun.sdk.service.oss2.exceptions.OperationException;
+import com.aliyun.sdk.service.oss2.exceptions.ServiceException;
+import com.aliyun.sdk.service.oss2.models.DeleteObjectRequest;
+import com.aliyun.sdk.service.oss2.models.GetObjectRequest;
+import com.aliyun.sdk.service.oss2.models.GetObjectResult;
+import com.aliyun.sdk.service.oss2.models.ListObjectsV2Request;
+import com.aliyun.sdk.service.oss2.models.ListObjectsV2Result;
+import com.aliyun.sdk.service.oss2.models.ObjectSummary;
+import org.apache.camel.CamelContext;
+import org.apache.camel.Consumer;
+import org.apache.camel.Endpoint;
+import org.apache.camel.Exchange;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.spi.ExceptionHandler;
+import org.apache.camel.spi.PollingConsumerPollStrategy;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.awaitility.Awaitility.await;
+
+/**
+ * A listed object that cannot be got (access denied, an archived object that is not restored) must not stop the
+ * consumer: the other listed objects are consumed, the next page of the listing is read, and the poll fails only when
+ * none of the listed objects could be got. The fake bucket lists its keys in order, a page of maxKeys keys at a time,
+ * with the last key of a page as continuation token, and answers the get of a forbidden object with 403.
+ */
+class OSSConsumerObjectThatCannotBeGotTest extends CamelTestSupport {
+
+ private final Set<String> objects = new ConcurrentSkipListSet<>();
+ private final Set<String> forbidden = ConcurrentHashMap.newKeySet();
+ private final AtomicInteger failedPolls = new AtomicInteger();
+ private final List<String> handled = new CopyOnWriteArrayList<>();
+
+ @Override
+ protected CamelContext createCamelContext() throws Exception {
+ CamelContext context = super.createCamelContext();
+ context.getRegistry().bind("ossClient", ossClient());
+ context.getRegistry().bind("countPolls", new PollingConsumerPollStrategy() {
+ @Override
+ public boolean begin(Consumer consumer, Endpoint endpoint) {
+ return true;
+ }
+
+ @Override
+ public void commit(Consumer consumer, Endpoint endpoint, int polledMessages) {
+ // noop
+ }
+
+ @Override
+ public boolean rollback(Consumer consumer, Endpoint endpoint, int retryCounter, Exception e) {
+ failedPolls.incrementAndGet();
+ return false;
+ }
+ });
+ context.getRegistry().bind("recordErrors", new ExceptionHandler() {
+ @Override
+ public void handleException(Throwable exception) {
+ handled.add(String.valueOf(exception.getMessage()));
+ }
+
+ @Override
+ public void handleException(String message, Throwable exception) {
+ handled.add(message);
+ }
+
+ @Override
+ public void handleException(String message, Exchange exchange, Throwable exception) {
+ handled.add(message);
+ }
+ });
+ return context;
+ }
+
+ @Test
+ void objectThatCannotBeGotDoesNotBlockTheOtherObjects() throws Exception {
+ objects.add("a.txt");
+ objects.add("b.txt");
+ objects.add("c.txt");
+ forbidden.add("b.txt");
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceivedInAnyOrder("a.txt", "c.txt");
+
+ addConsumerRoute("&deleteAfterRead=true&exceptionHandler=#recordErrors");
+
+ mock.assertIsSatisfied();
+ assertThat(objects).as("a.txt and c.txt are consumed and deleted, b.txt stays").containsExactly("b.txt");
+ await().atMost(20, TimeUnit.SECONDS).untilAsserted(
+ () -> assertThat(handled).as("the skipped object is reported").anyMatch(m -> m.contains("b.txt")));
+ }
+
+ @Test
+ void pageOfObjectsThatCannotBeGotDoesNotBlockTheNextPage() throws Exception {
+ objects.add("a.txt");
+ objects.add("b.txt");
+ objects.add("c.txt");
+ forbidden.add("a.txt");
+ MockEndpoint mock = getMockEndpoint("mock:result");
+ mock.expectedBodiesReceivedInAnyOrder("b.txt", "c.txt");
+
+ // one object per page: the first page holds only a.txt
+ addConsumerRoute("&deleteAfterRead=true&maxMessagesPerPoll=1&pollStrategy=#countPolls");
+
+ mock.assertIsSatisfied();
+ assertThat(objects).as("b.txt and c.txt are consumed and deleted, a.txt stays").containsExactly("a.txt");
Review Comment:
Applied as well in bd114c44ceba.
_Claude Code on behalf of allthingssecurity_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25760: camel-alibaba-oss - skip a listed object that cannot be got instead of failing the poll [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28075:
URL: https://github.com/apache/camel/pull/28075#issuecomment-6260903424
<!-- ci-tested-modules -->
:test_tube: **CI tested the following changed modules:**
- `components/camel-alibaba/camel-alibaba-oss`
- `docs`
:repeat: **1 test passed only after a retry on JDK 17** (2 retried attempts)
<details><summary>Recovered flaky tests on JDK 17 (1)</summary>
| Module | Test | Failed attempts | First failure |
| --- | --- | --- | --- |
| `components/camel-alibaba/camel-alibaba-oss` | `OSSConsumerObjectThatCannotBeGotTest.objectThatCannotBeGotDoesNotBlockTheOtherObjects` | 2 | [a.txt and c.txt are consumed and deleted, b.txt stays] Expecting actual: ["b.txt"] to contain exactly (and in same order): ["b.txt"] but some elements were not expected: ["c.txt"] |
> :information_source: These tests did **not** fail the build. Surefire retried them and they passed.
> Retries are enabled project-wide by `surefire.rerunFailingTestsCount` in `parent/pom.xml`.
</details>
---
<details><summary>:microscope: Scalpel shadow comparison — Scalpel: 8 of 706 tested, 26 compile-only — current: 8 all tested</summary>
[Maveniverse Scalpel](https://github.com/maveniverse/scalpel) detected **8 affected modules** (current approach: 8).
**Skip-tests mode would test 8 modules** (2 direct + 7 downstream), **skip tests for 26** (generated code, meta-modules)
<details><summary>Modules Scalpel would test (8)</summary>
- `camel-alibaba-oss` ← components/camel-alibaba/camel-alibaba-oss/src/main/java/org/apache/camel/component/alibaba/oss/OSSConsumer.java, components/camel-alibaba/camel-alibaba-oss/src/test/java/org/apache/camel/component/alibaba/oss/OSSConsumerObjectThatCannotBeGotTest.java
- `camel-jbang-mcp` ← downstream of org.apache.camel:camel-catalog
- `camel-jbang-plugin-mcp` ← downstream of org.apache.camel:camel-jbang-core
- `camel-jbang-plugin-tui` ← downstream of org.apache.camel:camel-catalog
- `camel-jbang-plugin-validate` ← downstream of org.apache.camel:camel-yaml-dsl-validator
- `camel-launcher-container` ← downstream of org.apache.camel:camel-launcher
- `camel-yaml-dsl-validator` ← downstream of org.apache.camel:camel-catalog
- `camel-yaml-dsl-validator-maven-plugin` ← downstream of org.apache.camel:camel-yaml-dsl-validator
</details>
<details><summary>Modules with tests skipped (26)</summary>
- `apache-camel`
- `camel-allcomponents`
- `camel-catalog`
- `camel-catalog-console`
- `camel-catalog-maven`
- `camel-catalog-suggest`
- `camel-componentdsl`
- `camel-endpointdsl`
- `camel-endpointdsl-support`
- `camel-itest`
- `camel-jbang-core`
- `camel-jbang-it`
- `camel-jbang-main`
- `camel-jbang-plugin-edit`
- `camel-jbang-plugin-generate`
- `camel-jbang-plugin-kubernetes`
- `camel-jbang-plugin-test`
- `camel-kamelet-main`
- `camel-launcher`
- `camel-report-maven-plugin`
- `camel-route-parser`
- `camel-yaml-dsl`
- `camel-yaml-dsl-deserializers`
- `camel-yaml-dsl-maven-plugin`
- `coverage`
- `dummy-component`
</details>
> :information_source: Shadow mode — Scalpel observes but does not affect test execution. [Learn more](https://github.com/maveniverse/scalpel)
</details>
<details><summary>All tested modules (35 modules, 3m 16s total)</summary>
**Total reactor time:** 3m 16s
| Module | Duration | Status |
| --- | --- | --- |
| Camel :: YAML DSL | 25.0s | SUCCESS |
| Camel :: Alibaba Cloud :: OSS | 24.8s | SUCCESS |
| Camel :: Catalog :: Camel Catalog | 24.5s | SUCCESS |
| Camel :: YAML DSL :: Validator | 22.7s | SUCCESS |
| Camel :: Component DSL | 17.6s | SUCCESS |
| Camel :: Docs | 15.7s | SUCCESS |
| Camel :: Kamelet Main | 15.1s | SUCCESS |
| Camel :: YAML DSL :: Deserializers | 8.3s | SUCCESS |
| Camel :: Catalog :: Camel Route Parser | 8.0s | SUCCESS |
| Camel :: Catalog :: Camel Report Maven Plugin | 6.9s | SUCCESS |
| Camel :: All Components Sync point | 5.6s | SUCCESS |
| Camel :: YAML DSL :: Validator Maven Plugin | 5.6s | SUCCESS |
| Camel :: Assembly | 4.7s | SUCCESS |
| Camel :: YAML DSL :: Maven Plugins | 3.3s | SUCCESS |
| Camel :: Catalog :: Maven | 3.1s | SUCCESS |
| Camel :: Catalog :: Suggest (deprecated) | 2.6s | SUCCESS |
| Camel :: Catalog :: Dummy Component | 1.1s | SUCCESS |
| Camel :: Catalog :: Console | 0.9s | SUCCESS |
| Camel :: Endpoint DSL :: Support | 0.9s | SUCCESS |
| Camel :: Coverage | n/a | |
| Camel :: Endpoint DSL | n/a | |
| Camel :: Integration Tests | n/a | |
| Camel :: JBang :: Core | n/a | |
| Camel :: JBang :: Integration tests | n/a | |
| Camel :: JBang :: MCP | n/a | |
| Camel :: JBang :: Main | n/a | |
| Camel :: JBang :: Plugin :: Edit | n/a | |
| Camel :: JBang :: Plugin :: Generate | n/a | |
| Camel :: JBang :: Plugin :: Kubernetes | n/a | |
| Camel :: JBang :: Plugin :: MCP | n/a | |
| Camel :: JBang :: Plugin :: TUI | n/a | |
| Camel :: JBang :: Plugin :: Testing | n/a | |
| Camel :: JBang :: Plugin :: Validate | n/a | |
| Camel :: Launcher | n/a | |
| Camel :: Launcher :: Container | n/a | |
**Top 20 slowest modules:**
- `Camel :: YAML DSL` (25.0s)
- `Camel :: Alibaba Cloud :: OSS` (24.8s)
- `Camel :: Catalog :: Camel Catalog` (24.5s)
- `Camel :: YAML DSL :: Validator` (22.7s)
- `Camel :: Component DSL` (17.6s)
- `Camel :: Docs` (15.7s)
- `Camel :: Kamelet Main` (15.1s)
- `Camel :: YAML DSL :: Deserializers` (8.3s)
- `Camel :: Catalog :: Camel Route Parser` (8.0s)
- `Camel :: Catalog :: Camel Report Maven Plugin` (6.9s)
- `Camel :: All Components Sync point` (5.6s)
- `Camel :: YAML DSL :: Validator Maven Plugin` (5.6s)
- `Camel :: Assembly` (4.7s)
- `Camel :: YAML DSL :: Maven Plugins` (3.3s)
- `Camel :: Catalog :: Maven` (3.1s)
- `Camel :: Catalog :: Suggest (deprecated)` (2.6s)
- `Camel :: Catalog :: Dummy Component` (1.1s)
- `Camel :: Catalog :: Console` (0.9s)
- `Camel :: Endpoint DSL :: Support` (0.9s)
</details>
---
:gear: [View full build and test results](https://github.com/apache/camel/actions/runs/39179690308)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25760: camel-alibaba-oss - skip a listed object that cannot be got instead of failing the poll [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28075:
URL: https://github.com/apache/camel/pull/28075#issuecomment-6260619033
:star2: Thank you for your contribution to the Apache Camel project! :star2:
:robot: CI automation will test this PR automatically.
:camel: Apache Camel Committers, please review the following items:
* First-time contributors **require MANUAL approval** for the GitHub Actions to run
* You can use the command `/component-test (camel-)component-name1 (camel-)component-name2..` to request a test from the test bot although they are normally detected and executed by CI.
* You can label PRs using `skip-tests` and `test-dependents` to fine-tune the checks executed by this PR.
* Build and test logs are available in the summary page. **Only** [Apache Camel committers](https://camel.apache.org/community/team/#committers) have access to the summary.
:warning: Be careful when sharing logs. Review their contents before sharing them publicly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org