You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "bartoszpop (via GitHub)" <gi...@apache.org> on 2026/01/28 11:17:28 UTC
[PR] Enhance S3 producer to stream the payload of type GenericFile [camel]
bartoszpop opened a new pull request, #21109:
URL: https://github.com/apache/camel/pull/21109
# Description
When moving a big file from SFTP to S3 in the streaming fashion we experience OutOfMemoryError. This is because `org.apache.camel.component.aws2.s3.AWS2S3Producer#processSingleOp` consumes a stream to determine the content length. However, for instance of `org.apache.camel.component.file.remote.RemoteFile` and in general `org.apache.camel.component.file.GenericFile`, the `org.apache.camel.component.file.GenericFile#getFileLength` method can be used for that purpose.
# Target
- [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch)
# Tracking
- [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it).
<!--
# *Note*: trivial changes like, typos, minor documentation fixes and other small items do not require a JIRA issue. In this case your pull request should address just this issue, without pulling in other changes.
-->
# Apache Camel coding standards and style
- [x] I checked that each commit in the pull request has a meaningful subject line and body.
<!--
If you're unsure, you can format the pull request title like `[CAMEL-XXX] Fixes bug in camel-file component`, where you replace `CAMEL-XXX` with the appropriate JIRA issue.
-->
- [x] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes.
<!--
You can run the aforementioned command in your module so that the build auto-formats your code. This will also be verified as part of the checks and your PR may be rejected if if there are uncommited changes after running `mvn clean install -DskipTests`.
You can learn more about the contribution guidelines at https://github.com/apache/camel/blob/main/CONTRIBUTING.md
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance S3 producer to stream the payload of type GenericFile [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #21109:
URL: https://github.com/apache/camel/pull/21109#issuecomment-3812938768
This would be good to backport to 4.14.x so lets create a JIRA to have a record of this in the release notes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance S3 producer to stream the payload of type GenericFile [camel]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on code in PR #21109:
URL: https://github.com/apache/camel/pull/21109#discussion_r2736331039
##########
components/camel-aws/camel-aws2-s3/pom.xml:
##########
@@ -64,6 +64,10 @@
<artifactId>apache-client</artifactId>
<version>${aws-java-sdk2-version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.camel</groupId>
+ <artifactId>camel-file</artifactId>
Review Comment:
yes, I think it would make more sense, instead of adding the dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance S3 producer to stream the payload of type GenericFile [camel]
Posted by "Croway (via GitHub)" <gi...@apache.org>.
Croway commented on code in PR #21109:
URL: https://github.com/apache/camel/pull/21109#discussion_r2736311421
##########
components/camel-aws/camel-aws2-s3/pom.xml:
##########
@@ -64,6 +64,10 @@
<artifactId>apache-client</artifactId>
<version>${aws-java-sdk2-version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.camel</groupId>
+ <artifactId>camel-file</artifactId>
Review Comment:
@davsclaus @bartoszpop instead of adding this dependency, what if we improve `WrappedFile` with a new API `getFileLength()` ?
This way we can get rid of camel-file, and use the API instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance S3 producer to stream the payload of type GenericFile [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on code in PR #21109:
URL: https://github.com/apache/camel/pull/21109#discussion_r2736451024
##########
components/camel-aws/camel-aws2-s3/pom.xml:
##########
@@ -64,6 +64,10 @@
<artifactId>apache-client</artifactId>
<version>${aws-java-sdk2-version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.camel</groupId>
+ <artifactId>camel-file</artifactId>
Review Comment:
Yeah we can do that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance S3 producer to stream the payload of type GenericFile [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21109:
URL: https://github.com/apache/camel/pull/21109#issuecomment-3810705613
:star2: Thank you for your contribution to the Apache Camel project! :star2:
:robot: CI automation will test this PR automatically.
:camel: Apache Camel Committers, please review the following items:
* First-time contributors **require MANUAL approval** for the GitHub Actions to run
* You can use the command `/component-test (camel-)component-name1 (camel-)component-name2..` to request a test from the test bot.
* You can label PRs using `build-all`, `build-dependents`, `skip-tests` and `test-dependents` to fine-tune the checks executed by this PR.
* Build and test logs are available in the Summary page. **Only** [Apache Camel committers](https://camel.apache.org/community/team/#committers) have access to the summary.
* :warning: Be careful when sharing logs. Review their contents before sharing them publicly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enhance S3 producer to stream the payload of type GenericFile [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on code in PR #21109:
URL: https://github.com/apache/camel/pull/21109#discussion_r2736578544
##########
components/camel-aws/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Producer.java:
##########
@@ -325,7 +325,9 @@ public void processSingleOp(final Exchange exchange) throws Exception {
try {
// Need to check if the message body is WrappedFile
if (obj instanceof WrappedFile) {
Review Comment:
Avoid ugly type cast below and use modern java style
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-22925: Enhance S3 producer to stream the payload of type GenericFile [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus merged PR #21109:
URL: https://github.com/apache/camel/pull/21109
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org