You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Max Bridgewater <ma...@gmail.com> on 2014/04/14 12:11:53 UTC
OutOfMemoryError when handling large file
Hi,
I am trying to process a large file (900MB). I need to send it to a HTTP
remote location: B. But before I send it to B, I need to post some metadata
to HTTP location A.
Because I do not want the content of the file to be sent to A, I move the
file from the body to a header named XomBody. Unfortunately, when posting
to A, the Camel HTTP producer converts all headers to string. Which causes
the 900MB file to result into OutOfmemoryError.
How can I prevent this?
I was hoping that by using a header filtering strategy to exclude all
headers with name starting with Xom, this would resolve my problem. This
does not seem to be the case. Looking at the HttpProducer, it appears that
the filtering strategy is applied after type conversion.
So, in general: Should the type conversion be applied only when all tests
are done and the header is really a candidate for transfer?
In my specific case: what is the recommended way to do what I am trying to
do?
Thanks,
Max.
AW: OutOfMemoryError when handling large file
Posted by "Jan Matèrne (jhm)" <ap...@materne.de>.
> I am trying to process a large file (900MB). I need to send it to a
> HTTP remote location: B. But before I send it to B, I need to post some
> metadata to HTTP location A.
>
> Because I do not want the content of the file to be sent to A, I move
> the file from the body to a header named XomBody. Unfortunately, when
> posting to A, the Camel HTTP producer converts all headers to string.
> Which causes the 900MB file to result into OutOfmemoryError.
>
> How can I prevent this?
idea 1: store the original as property instead of header
idea 2: store the original in a database and the db-key as header (EIP
"Claim Check"
http://camel.apache.org/claim-check.html)
> I was hoping that by using a header filtering strategy to exclude all
> headers with name starting with Xom, this would resolve my problem.
> This does not seem to be the case. Looking at the HttpProducer, it
> appears that the filtering strategy is applied after type conversion.
idea 3: an intermediate processor which deletes all headers matching your
criteria?
Jan
Re: OutOfMemoryError when handling large file
Posted by Henryk Konsek <he...@gmail.com>.
Hi Max,
> How can I prevent this?
What about storing only a path to the file (File#absolutePath()) in
the header instead of the java.io.File instance?
Cheers.
--
Henryk Konsek
http://henryk-konsek.blogspot.com
Re: OutOfMemoryError when handling large file
Posted by Max Bridgewater <ma...@gmail.com>.
Awesome. property works great.
Thanks a bunch.
On Mon, Apr 14, 2014 at 8:58 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> You can store the message body as a exchange property, and then
> replace the body with what meta-data you want to send, and then
> afterwards restore the message body with the exchange property to get
> the file back.
>
> On Mon, Apr 14, 2014 at 3:11 AM, Max Bridgewater
> <ma...@gmail.com> wrote:
> > Hi,
> >
> > I am trying to process a large file (900MB). I need to send it to a HTTP
> > remote location: B. But before I send it to B, I need to post some
> metadata
> > to HTTP location A.
> >
> > Because I do not want the content of the file to be sent to A, I move the
> > file from the body to a header named XomBody. Unfortunately, when posting
> > to A, the Camel HTTP producer converts all headers to string. Which
> causes
> > the 900MB file to result into OutOfmemoryError.
> >
> > How can I prevent this?
> >
> > I was hoping that by using a header filtering strategy to exclude all
> > headers with name starting with Xom, this would resolve my problem. This
> > does not seem to be the case. Looking at the HttpProducer, it appears
> that
> > the filtering strategy is applied after type conversion.
> >
> > So, in general: Should the type conversion be applied only when all tests
> > are done and the header is really a candidate for transfer?
> >
> > In my specific case: what is the recommended way to do what I am trying
> to
> > do?
> >
> > Thanks,
> > Max.
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/
>
Re: OutOfMemoryError when handling large file
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
You can store the message body as a exchange property, and then
replace the body with what meta-data you want to send, and then
afterwards restore the message body with the exchange property to get
the file back.
On Mon, Apr 14, 2014 at 3:11 AM, Max Bridgewater
<ma...@gmail.com> wrote:
> Hi,
>
> I am trying to process a large file (900MB). I need to send it to a HTTP
> remote location: B. But before I send it to B, I need to post some metadata
> to HTTP location A.
>
> Because I do not want the content of the file to be sent to A, I move the
> file from the body to a header named XomBody. Unfortunately, when posting
> to A, the Camel HTTP producer converts all headers to string. Which causes
> the 900MB file to result into OutOfmemoryError.
>
> How can I prevent this?
>
> I was hoping that by using a header filtering strategy to exclude all
> headers with name starting with Xom, this would resolve my problem. This
> does not seem to be the case. Looking at the HttpProducer, it appears that
> the filtering strategy is applied after type conversion.
>
> So, in general: Should the type conversion be applied only when all tests
> are done and the header is really a candidate for transfer?
>
> In my specific case: what is the recommended way to do what I am trying to
> do?
>
> Thanks,
> Max.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/