You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by KARTHIK PRABHU N <ka...@gmail.com> on 2025/05/06 02:35:37 UTC
Camel's File Stream Limitation for Large Files
Hello Team,
Good morning & hope you are doing fine !
I am working on a use-case / concept to generate, download and upload very
large file ( > 1GB file size <100GB) should support both binary and
non-binary file type. To achieve this concept, I tried Camel’s File or
Stream component to read / write file as stream and REST component that
plugins to servlet component which exposes routes as REST endpoint.
The download route designed to searches for given file and once found,
streams the data to target file location ( intermediate step we can ignore)
as well as http response body. While testing the route using curl tool, we
observed multiple issues as listed below.
1. Route streams data to client perfectly unless the file size <= 2GB,
beyond this size the route will throw below error. The exception clearly
indicates that JVM is trying to allocate an array with the size of about
2.13 GB and java arrays are limited to max size of array index around 2GB .
I tried to understand how streams work internally but somehow, I am lost.
Could you please assist how I can handle this scenario may be read and
publish data as chucks ?
*java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too
large at
java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649)
*
*at
java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642)
*
*at
java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
*
*at
java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)*
*at
org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:108)
*
*at java.base/java.io.InputStream.transferTo(InputStream.java:783) at
org.apache.camel.util.IOHelper.copy(IOHelper.java:165)*
2. How to increase the download rate ? As the file size is large, the
total time taken to complete the download is very long. I tried with higher
buffer range for file component & max response size in the servlet consumer
level but didn’t work.
Could you please assist. Let us know if you need more details form myside.
Thank you,
Karthik Prabhu
Re: Camel's File Stream Limitation for Large Files
Posted by KARTHIK PRABHU N <ka...@gmail.com>.
Hello Claus,
First issue got fixed after enabling the stream cache pool.
Could you please assist for 2nd one ?
Thank you,
Karthik
On Tue, 6 May 2025, 10:35 am Claus Ibsen, <cl...@gmail.com> wrote:
> Hi
>
> Enable spool on stream caching
> https://camel.apache.org/manual/stream-caching.html
>
>
> On Tue, May 6, 2025 at 4:37 AM KARTHIK PRABHU N <karthikprabhu77@gmail.com
> >
> wrote:
>
> > Hello Team,
> >
> > Good morning & hope you are doing fine !
> >
> > I am working on a use-case / concept to generate, download and upload
> very
> > large file ( > 1GB file size <100GB) should support both binary and
> > non-binary file type. To achieve this concept, I tried Camel’s File or
> > Stream component to read / write file as stream and REST component that
> > plugins to servlet component which exposes routes as REST endpoint.
> >
> >
> >
> > The download route designed to searches for given file and once found,
> > streams the data to target file location ( intermediate step we can
> ignore)
> > as well as http response body. While testing the route using curl tool,
> we
> > observed multiple issues as listed below.
> >
> >
> >
> > 1. Route streams data to client perfectly unless the file size <= 2GB,
> > beyond this size the route will throw below error. The exception
> clearly
> > indicates that JVM is trying to allocate an array with the size of
> about
> > 2.13 GB and java arrays are limited to max size of array index around
> 2GB .
> > I tried to understand how streams work internally but somehow, I am
> lost.
> > Could you please assist how I can handle this scenario may be read and
> > publish data as chucks ?
> >
> >
> >
> >
> > *java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too
> > large at
> >
> java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649)
> > *
> >
> > *at
> >
> java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642)
> > *
> >
> > *at
> > java.base/java.io
> .ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
> > *
> >
> > *at
> > java.base/java.io
> .ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)*
> >
> > *at
> >
> org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:108)
> > *
> >
> > *at java.base/java.io.InputStream.transferTo(InputStream.java:783) at
> > org.apache.camel.util.IOHelper.copy(IOHelper.java:165)*
> >
> >
> >
> >
> >
> > 2. How to increase the download rate ? As the file size is large, the
> > total time taken to complete the download is very long. I tried with
> higher
> > buffer range for file component & max response size in the servlet
> consumer
> > level but didn’t work.
> >
> >
> >
> >
> >
> >
> >
> > Could you please assist. Let us know if you need more details form
> myside.
> >
> >
> >
> > Thank you,
> >
> > Karthik Prabhu
> >
> >
> >
> >
> >
>
> --
> Claus Ibsen
> -----------------
> @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
Re: Camel's File Stream Limitation for Large Files
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Enable spool on stream caching
https://camel.apache.org/manual/stream-caching.html
On Tue, May 6, 2025 at 4:37 AM KARTHIK PRABHU N <ka...@gmail.com>
wrote:
> Hello Team,
>
> Good morning & hope you are doing fine !
>
> I am working on a use-case / concept to generate, download and upload very
> large file ( > 1GB file size <100GB) should support both binary and
> non-binary file type. To achieve this concept, I tried Camel’s File or
> Stream component to read / write file as stream and REST component that
> plugins to servlet component which exposes routes as REST endpoint.
>
>
>
> The download route designed to searches for given file and once found,
> streams the data to target file location ( intermediate step we can ignore)
> as well as http response body. While testing the route using curl tool, we
> observed multiple issues as listed below.
>
>
>
> 1. Route streams data to client perfectly unless the file size <= 2GB,
> beyond this size the route will throw below error. The exception clearly
> indicates that JVM is trying to allocate an array with the size of about
> 2.13 GB and java arrays are limited to max size of array index around 2GB .
> I tried to understand how streams work internally but somehow, I am lost.
> Could you please assist how I can handle this scenario may be read and
> publish data as chucks ?
>
>
>
>
> *java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too
> large at
> java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649)
> *
>
> *at
> java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642)
> *
>
> *at
> java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
> *
>
> *at
> java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)*
>
> *at
> org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:108)
> *
>
> *at java.base/java.io.InputStream.transferTo(InputStream.java:783) at
> org.apache.camel.util.IOHelper.copy(IOHelper.java:165)*
>
>
>
>
>
> 2. How to increase the download rate ? As the file size is large, the
> total time taken to complete the download is very long. I tried with higher
> buffer range for file component & max response size in the servlet consumer
> level but didn’t work.
>
>
>
>
>
>
>
> Could you please assist. Let us know if you need more details form myside.
>
>
>
> Thank you,
>
> Karthik Prabhu
>
>
>
>
>
--
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2