You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by ch...@innogy.com on 2017/11/20 18:21:59 UTC
Camel Stream caching seems not to delete the temporary files
Hi there,
I have to download pretty large files (> 200 MB) from a remote server via http and process them with a very specific library. This library relies on using files and not streams, so I have a route looking like
from (...)
.toD("${body}")
.to("file:" + tempDirectoryName)
In the body, I had constructed the URI to the file to be downloaded. I quickly ran into memory leaks, so I changed the code to
from(...=
.toD("${body}")
.streamCaching()
.to("file:" + tempDirectoryName)
This helped with the memory leak because now the data are written twice on my disk: one to the temporary directory name that I had intended and one in the java.io.tempdir directory. The documentation on Camel Stream-caching states "Camel itself will handle deleting the temporary file once the cached stream is no longer necessary." But I can't see that happen. The large files remain where they are. Any ideas what went wrong?
Kind regards,
Christian Jacob
Innogy SE
Retail IT
Integration and Digital Solutions (AFS-IGI)
Rellinghauser Str. 37
45128 Essen
T intern: 70-20581
T extern: +49 (0)201 12 20582
T mobil: +49 (0)1622843981
Fax: +49 (0)201 12 24796
mailto: christian.jacob@innogy.com
----------------------------------------------------------------
innogy SE Vorsitzender des Aufsichtsrates: Dr. Werner Brandt
Vorstand: Peter Terium (Vorsitzender), Dr. Hans Buenting,
Dr. Bernhard Guenther, Martin Herrmann, Hildegard Mueller, Uwe Tigges
Sitz der Gesellschaft: Essen, Eingetragen beim Amtsgericht Essen,
Handelsregister-Nr. HRB 27091, USt-IdNr. DE304171711
AW: Camel Stream caching seems not to delete the temporary files
Posted by ch...@innogy.com.
Hi Claus,
it's Camel version 2.20.0 .
Yes, I do use custom processors, but they are only working with the library I have mentioned. And this library only works with my temporary file. Actually, the next statement after my
.to("file:...")
is
.setBody(simple("${null}")
to make sure everything is cleaned up.
Thanks in advance for your help.
Christian
-----Ursprüngliche Nachricht-----
Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
Gesendet: Dienstag, 21. November 2017 13:35
An: users@camel.apache.org
Betreff: Re: Camel Stream caching seems not to delete the temporary files
What version of Camel do you use?
Btw the message history does not copy the message body, so it should not be a problem.
The source code you posted is that all your did, are you not doing something more, such as a custom processor / bean that works on the message body.
On Tue, Nov 21, 2017 at 1:16 PM, Zoran Regvart <zo...@regvart.com> wrote:
> Hi Christian,
> if you're not going to read from the body of the message multiple
> times you can disable stream caching. I think that the memory increase
> you see is due to message history, try disabling that[1]. If that
> fails to help perhaps you could provide an example that we can
> reproduce this?
>
> zoran
>
> [1] https://camel.apache.org/message-history.html
>
> On Mon, Nov 20, 2017 at 7:21 PM, <ch...@innogy.com> wrote:
>> Hi there,
>>
>> I have to download pretty large files (> 200 MB) from a remote server
>> via http and process them with a very specific library. This library
>> relies on using files and not streams, so I have a route looking like
>>
>> from (...)
>> .toD("${body}")
>> .to("file:" + tempDirectoryName)
>>
>> In the body, I had constructed the URI to the file to be downloaded.
>> I quickly ran into memory leaks, so I changed the code to from(...=
>> .toD("${body}")
>> .streamCaching()
>> .to("file:" + tempDirectoryName)
>>
>> This helped with the memory leak because now the data are written twice on my disk: one to the temporary directory name that I had intended and one in the java.io.tempdir directory. The documentation on Camel Stream-caching states "Camel itself will handle deleting the temporary file once the cached stream is no longer necessary." But I can't see that happen. The large files remain where they are. Any ideas what went wrong?
>>
>> Kind regards,
>> Christian Jacob
>>
>> Innogy SE
>> Retail IT
>> Integration and Digital Solutions (AFS-IGI) Rellinghauser Str. 37
>> 45128 Essen
>> T intern: 70-20581
>> T extern: +49 (0)201 12 20582
>> T mobil: +49 (0)1622843981
>> Fax: +49 (0)201 12 24796
>> mailto: christian.jacob@innogy.com
>>
>> ----------------------------------------------------------------
>> innogy SE Vorsitzender des Aufsichtsrates: Dr. Werner Brandt
>> Vorstand: Peter Terium (Vorsitzender), Dr. Hans Buenting, Dr.
>> Bernhard Guenther, Martin Herrmann, Hildegard Mueller, Uwe Tigges
>> Sitz der Gesellschaft: Essen, Eingetragen beim Amtsgericht Essen,
>> Handelsregister-Nr. HRB 27091, USt-IdNr. DE304171711
>
>
>
> --
> Zoran Regvart
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
----------------------------------------------------------------
innogy SE Vorsitzender des Aufsichtsrates: Dr. Werner Brandt
Vorstand: Peter Terium (Vorsitzender), Dr. Hans Buenting,
Dr. Bernhard Guenther, Martin Herrmann, Hildegard Mueller, Uwe Tigges
Sitz der Gesellschaft: Essen, Eingetragen beim Amtsgericht Essen,
Handelsregister-Nr. HRB 27091, USt-IdNr. DE304171711
Re: Camel Stream caching seems not to delete the temporary files
Posted by Claus Ibsen <cl...@gmail.com>.
What version of Camel do you use?
Btw the message history does not copy the message body, so it should
not be a problem.
The source code you posted is that all your did, are you not doing
something more, such as a custom processor / bean that works on the
message body.
On Tue, Nov 21, 2017 at 1:16 PM, Zoran Regvart <zo...@regvart.com> wrote:
> Hi Christian,
> if you're not going to read from the body of the message multiple
> times you can disable stream caching. I think that the memory increase
> you see is due to message history, try disabling that[1]. If that
> fails to help perhaps you could provide an example that we can
> reproduce this?
>
> zoran
>
> [1] https://camel.apache.org/message-history.html
>
> On Mon, Nov 20, 2017 at 7:21 PM, <ch...@innogy.com> wrote:
>> Hi there,
>>
>> I have to download pretty large files (> 200 MB) from a remote server via http and process them with a very specific library. This library relies on using files and not streams, so I have a route looking like
>>
>> from (...)
>> .toD("${body}")
>> .to("file:" + tempDirectoryName)
>>
>> In the body, I had constructed the URI to the file to be downloaded. I quickly ran into memory leaks, so I changed the code to
>> from(...=
>> .toD("${body}")
>> .streamCaching()
>> .to("file:" + tempDirectoryName)
>>
>> This helped with the memory leak because now the data are written twice on my disk: one to the temporary directory name that I had intended and one in the java.io.tempdir directory. The documentation on Camel Stream-caching states "Camel itself will handle deleting the temporary file once the cached stream is no longer necessary." But I can't see that happen. The large files remain where they are. Any ideas what went wrong?
>>
>> Kind regards,
>> Christian Jacob
>>
>> Innogy SE
>> Retail IT
>> Integration and Digital Solutions (AFS-IGI)
>> Rellinghauser Str. 37
>> 45128 Essen
>> T intern: 70-20581
>> T extern: +49 (0)201 12 20582
>> T mobil: +49 (0)1622843981
>> Fax: +49 (0)201 12 24796
>> mailto: christian.jacob@innogy.com
>>
>> ----------------------------------------------------------------
>> innogy SE Vorsitzender des Aufsichtsrates: Dr. Werner Brandt
>> Vorstand: Peter Terium (Vorsitzender), Dr. Hans Buenting,
>> Dr. Bernhard Guenther, Martin Herrmann, Hildegard Mueller, Uwe Tigges
>> Sitz der Gesellschaft: Essen, Eingetragen beim Amtsgericht Essen,
>> Handelsregister-Nr. HRB 27091, USt-IdNr. DE304171711
>
>
>
> --
> Zoran Regvart
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Camel Stream caching seems not to delete the temporary files
Posted by Zoran Regvart <zo...@regvart.com>.
Hi Christian,
if you're not going to read from the body of the message multiple
times you can disable stream caching. I think that the memory increase
you see is due to message history, try disabling that[1]. If that
fails to help perhaps you could provide an example that we can
reproduce this?
zoran
[1] https://camel.apache.org/message-history.html
On Mon, Nov 20, 2017 at 7:21 PM, <ch...@innogy.com> wrote:
> Hi there,
>
> I have to download pretty large files (> 200 MB) from a remote server via http and process them with a very specific library. This library relies on using files and not streams, so I have a route looking like
>
> from (...)
> .toD("${body}")
> .to("file:" + tempDirectoryName)
>
> In the body, I had constructed the URI to the file to be downloaded. I quickly ran into memory leaks, so I changed the code to
> from(...=
> .toD("${body}")
> .streamCaching()
> .to("file:" + tempDirectoryName)
>
> This helped with the memory leak because now the data are written twice on my disk: one to the temporary directory name that I had intended and one in the java.io.tempdir directory. The documentation on Camel Stream-caching states "Camel itself will handle deleting the temporary file once the cached stream is no longer necessary." But I can't see that happen. The large files remain where they are. Any ideas what went wrong?
>
> Kind regards,
> Christian Jacob
>
> Innogy SE
> Retail IT
> Integration and Digital Solutions (AFS-IGI)
> Rellinghauser Str. 37
> 45128 Essen
> T intern: 70-20581
> T extern: +49 (0)201 12 20582
> T mobil: +49 (0)1622843981
> Fax: +49 (0)201 12 24796
> mailto: christian.jacob@innogy.com
>
> ----------------------------------------------------------------
> innogy SE Vorsitzender des Aufsichtsrates: Dr. Werner Brandt
> Vorstand: Peter Terium (Vorsitzender), Dr. Hans Buenting,
> Dr. Bernhard Guenther, Martin Herrmann, Hildegard Mueller, Uwe Tigges
> Sitz der Gesellschaft: Essen, Eingetragen beim Amtsgericht Essen,
> Handelsregister-Nr. HRB 27091, USt-IdNr. DE304171711
--
Zoran Regvart
AW: Camel Stream caching seems not to delete the temporary files
Posted by ch...@innogy.com.
Okay, one addition: the files on java.io.tempdir are emptied after a while, but they still are there.
-----Ursprüngliche Nachricht-----
Von: christian.jacob@innogy.com [mailto:christian.jacob@innogy.com]
Gesendet: Montag, 20. November 2017 19:22
An: users@camel.apache.org
Betreff: Camel Stream caching seems not to delete the temporary files
Hi there,
I have to download pretty large files (> 200 MB) from a remote server via http and process them with a very specific library. This library relies on using files and not streams, so I have a route looking like
from (...)
.toD("${body}")
.to("file:" + tempDirectoryName)
In the body, I had constructed the URI to the file to be downloaded. I quickly ran into memory leaks, so I changed the code to from(...=
.toD("${body}")
.streamCaching()
.to("file:" + tempDirectoryName)
This helped with the memory leak because now the data are written twice on my disk: one to the temporary directory name that I had intended and one in the java.io.tempdir directory. The documentation on Camel Stream-caching states "Camel itself will handle deleting the temporary file once the cached stream is no longer necessary." But I can't see that happen. The large files remain where they are. Any ideas what went wrong?
Kind regards,
Christian Jacob
Innogy SE
Retail IT
Integration and Digital Solutions (AFS-IGI) Rellinghauser Str. 37
45128 Essen
T intern: 70-20581
T extern: +49 (0)201 12 20582
T mobil: +49 (0)1622843981
Fax: +49 (0)201 12 24796
mailto: christian.jacob@innogy.com
----------------------------------------------------------------
innogy SE Vorsitzender des Aufsichtsrates: Dr. Werner Brandt
Vorstand: Peter Terium (Vorsitzender), Dr. Hans Buenting, Dr. Bernhard Guenther, Martin Herrmann, Hildegard Mueller, Uwe Tigges Sitz der Gesellschaft: Essen, Eingetragen beim Amtsgericht Essen, Handelsregister-Nr. HRB 27091, USt-IdNr. DE304171711
----------------------------------------------------------------
innogy SE Vorsitzender des Aufsichtsrates: Dr. Werner Brandt
Vorstand: Peter Terium (Vorsitzender), Dr. Hans Buenting,
Dr. Bernhard Guenther, Martin Herrmann, Hildegard Mueller, Uwe Tigges
Sitz der Gesellschaft: Essen, Eingetragen beim Amtsgericht Essen,
Handelsregister-Nr. HRB 27091, USt-IdNr. DE304171711