You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Tim Dudgeon <td...@gmail.com> on 2016/10/25 12:20:49 UTC
MIME-Mutipart DataFormat streaming?
I was looking at the MIME-Mutipart DataFormat and it looks very useful:
https://camel.apache.org/mime-multipart.html
But I was not sure if it fully streams the data for the body and
attachments, or whether it holds the content in memory (this is for both
marshalling and unmarshalling).
Thanks
Tim
RE: MIME-Mutipart DataFormat streaming?
Posted by "Siano, Stephan" <st...@sap.com>.
Hi Tim,
There is a library named mimepull that is supposed to support streaming for attachments, however I never tried that.
Best regards
Stephan
-----Original Message-----
From: Tim Dudgeon [mailto:tdudgeon.ml@gmail.com]
Sent: Samstag, 29. Oktober 2016 10:52
To: users@camel.apache.org
Subject: Re: MIME-Mutipart DataFormat streaming?
Thanks Stephan, that's what I feared.
Are there any suggested alternatives to this?
In my case I have Camel at both ends of an HTTP connection so hope there's some solution.
I would typically have a small body, and a small number of attachments, but these could potentially be very large. Seems like I should be able to use multipart form, but I'm not sure how to ensure its 100% streamed at both ends?
Tim
On 26/10/2016 06:43, Siano, Stephan wrote:
> Hi Tim,
>
> The MIME-Multipart DataFormat uses javamail as the underlying technology. The DataFormat is using streaming as far as the DataFormat is concerned, but I fear that javamail as such keeps a lot of stuff in memory so there is very likely no full streaming with that component available. For marshalling this might work with streaming as long as the component providing the Camel Message with its attachments is stream capbable, but for unmarshalling, I think that the javamail component will create ByteArrayDataSources at least for the attachments.
>
> Best regards
> Stephan
>
> -----Original Message-----
> From: Tim Dudgeon [mailto:tdudgeon.ml@gmail.com]
> Sent: Dienstag, 25. Oktober 2016 14:21
> To: users@camel.apache.org
> Subject: MIME-Mutipart DataFormat streaming?
>
> I was looking at the MIME-Mutipart DataFormat and it looks very useful:
> https://camel.apache.org/mime-multipart.html
>
> But I was not sure if it fully streams the data for the body and
> attachments, or whether it holds the content in memory (this is for both
> marshalling and unmarshalling).
>
> Thanks
> Tim
>
Re: MIME-Mutipart DataFormat streaming?
Posted by Tim Dudgeon <td...@gmail.com>.
Thanks Stephan, that's what I feared.
Are there any suggested alternatives to this?
In my case I have Camel at both ends of an HTTP connection so hope
there's some solution.
I would typically have a small body, and a small number of attachments,
but these could potentially be very large. Seems like I should be able
to use multipart form, but I'm not sure how to ensure its 100% streamed
at both ends?
Tim
On 26/10/2016 06:43, Siano, Stephan wrote:
> Hi Tim,
>
> The MIME-Multipart DataFormat uses javamail as the underlying technology. The DataFormat is using streaming as far as the DataFormat is concerned, but I fear that javamail as such keeps a lot of stuff in memory so there is very likely no full streaming with that component available. For marshalling this might work with streaming as long as the component providing the Camel Message with its attachments is stream capbable, but for unmarshalling, I think that the javamail component will create ByteArrayDataSources at least for the attachments.
>
> Best regards
> Stephan
>
> -----Original Message-----
> From: Tim Dudgeon [mailto:tdudgeon.ml@gmail.com]
> Sent: Dienstag, 25. Oktober 2016 14:21
> To: users@camel.apache.org
> Subject: MIME-Mutipart DataFormat streaming?
>
> I was looking at the MIME-Mutipart DataFormat and it looks very useful:
> https://camel.apache.org/mime-multipart.html
>
> But I was not sure if it fully streams the data for the body and
> attachments, or whether it holds the content in memory (this is for both
> marshalling and unmarshalling).
>
> Thanks
> Tim
>
RE: MIME-Mutipart DataFormat streaming?
Posted by "Siano, Stephan" <st...@sap.com>.
Hi,
I actually don't understand that issue. Can you provide a simple test case for it? The stream involved in the code lines you are citing is actually the output stream, not the input stream. The multipart data format should also work on streams, so if there is an issue with that, that is probably a bug. If you want to open a JIRA issue for it, a unit test reproducing the issue would be great.
Best regards
Stephan
-----Original Message-----
From: Tim Dudgeon [mailto:tdudgeon.ml@gmail.com]
Sent: Montag, 11. Dezember 2017 12:45
To: users@camel.apache.org
Subject: Re: MIME-Mutipart DataFormat streaming?
Following up on this again (after a long delay).
I tried using the MIME-Multipart DataFormat to send and hit a problem
when the input stream for an attachment cannot be read multiple times.
It looks like in the MimeMultipartDataFormat.marshal( ... ) method [1]
this InputStream for an attachment is read on both of these lines:
Line 140: mm.saveChanges();
Line 154: mm.writeTo(stream, headers.toArray(new String[0]));
When using input that can be read repeatedly (e.g. ByteArrayInputStream)
it works fine. When using an InputStream that can only be consumed once
then it blows up the second time it is read.
I recognise that when reading a mulitpart message there can be
complications, but when writing one it should be possible to do this by
only reading the input once, so I believe this should be considered a bug.
Shall I raise an issue for this?
[1]
https://github.com/apache/camel/blob/master/components/camel-mail/src/main/java/org/apache/camel/dataformat/mime/multipart/MimeMultipartDataFormat.java#L92:L160
On 26/10/16 06:43, Siano, Stephan wrote:
> Hi Tim,
>
> The MIME-Multipart DataFormat uses javamail as the underlying technology. The DataFormat is using streaming as far as the DataFormat is concerned, but I fear that javamail as such keeps a lot of stuff in memory so there is very likely no full streaming with that component available. For marshalling this might work with streaming as long as the component providing the Camel Message with its attachments is stream capbable, but for unmarshalling, I think that the javamail component will create ByteArrayDataSources at least for the attachments.
>
> Best regards
> Stephan
>
> -----Original Message-----
> From: Tim Dudgeon [mailto:tdudgeon.ml@gmail.com]
> Sent: Dienstag, 25. Oktober 2016 14:21
> To: users@camel.apache.org
> Subject: MIME-Mutipart DataFormat streaming?
>
> I was looking at the MIME-Mutipart DataFormat and it looks very useful:
> https://camel.apache.org/mime-multipart.html
>
> But I was not sure if it fully streams the data for the body and
> attachments, or whether it holds the content in memory (this is for both
> marshalling and unmarshalling).
>
> Thanks
> Tim
>
Re: MIME-Mutipart DataFormat streaming?
Posted by Tim Dudgeon <td...@gmail.com>.
Following up on this again (after a long delay).
I tried using the MIME-Multipart DataFormat to send and hit a problem
when the input stream for an attachment cannot be read multiple times.
It looks like in the MimeMultipartDataFormat.marshal( ... ) method [1]
this InputStream for an attachment is read on both of these lines:
Line 140: mm.saveChanges();
Line 154: mm.writeTo(stream, headers.toArray(new String[0]));
When using input that can be read repeatedly (e.g. ByteArrayInputStream)
it works fine. When using an InputStream that can only be consumed once
then it blows up the second time it is read.
I recognise that when reading a mulitpart message there can be
complications, but when writing one it should be possible to do this by
only reading the input once, so I believe this should be considered a bug.
Shall I raise an issue for this?
[1]
https://github.com/apache/camel/blob/master/components/camel-mail/src/main/java/org/apache/camel/dataformat/mime/multipart/MimeMultipartDataFormat.java#L92:L160
On 26/10/16 06:43, Siano, Stephan wrote:
> Hi Tim,
>
> The MIME-Multipart DataFormat uses javamail as the underlying technology. The DataFormat is using streaming as far as the DataFormat is concerned, but I fear that javamail as such keeps a lot of stuff in memory so there is very likely no full streaming with that component available. For marshalling this might work with streaming as long as the component providing the Camel Message with its attachments is stream capbable, but for unmarshalling, I think that the javamail component will create ByteArrayDataSources at least for the attachments.
>
> Best regards
> Stephan
>
> -----Original Message-----
> From: Tim Dudgeon [mailto:tdudgeon.ml@gmail.com]
> Sent: Dienstag, 25. Oktober 2016 14:21
> To: users@camel.apache.org
> Subject: MIME-Mutipart DataFormat streaming?
>
> I was looking at the MIME-Mutipart DataFormat and it looks very useful:
> https://camel.apache.org/mime-multipart.html
>
> But I was not sure if it fully streams the data for the body and
> attachments, or whether it holds the content in memory (this is for both
> marshalling and unmarshalling).
>
> Thanks
> Tim
>
RE: MIME-Mutipart DataFormat streaming?
Posted by "Siano, Stephan" <st...@sap.com>.
Hi Tim,
The MIME-Multipart DataFormat uses javamail as the underlying technology. The DataFormat is using streaming as far as the DataFormat is concerned, but I fear that javamail as such keeps a lot of stuff in memory so there is very likely no full streaming with that component available. For marshalling this might work with streaming as long as the component providing the Camel Message with its attachments is stream capbable, but for unmarshalling, I think that the javamail component will create ByteArrayDataSources at least for the attachments.
Best regards
Stephan
-----Original Message-----
From: Tim Dudgeon [mailto:tdudgeon.ml@gmail.com]
Sent: Dienstag, 25. Oktober 2016 14:21
To: users@camel.apache.org
Subject: MIME-Mutipart DataFormat streaming?
I was looking at the MIME-Mutipart DataFormat and it looks very useful:
https://camel.apache.org/mime-multipart.html
But I was not sure if it fully streams the data for the body and
attachments, or whether it holds the content in memory (this is for both
marshalling and unmarshalling).
Thanks
Tim