You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Daniel Pompa <po...@gmail.com> on 2016/06/02 09:29:48 UTC
Problem transfer from SMB to FTP OutMemory Java Heap Space
I have a problema with Fuse. I created a route,
<route xmlns="http://camel.apache.org/schema/spring"
id="sap-xrt-connect-interface-receive-20160414">
<from
uri="smb://xxxxx;xxxxx@xxxxx/APLICACIONES/EFACDES/xxxs/?password=xxxx&delete=false&delay=60000"/>
<onException>
<redeliveryPolicy maximumRedeliveries="2"
redeliveryDelay="20000" retryAttemptedLogLevel="WARN"/>
</onException>
<setProperty propertyName="CamelBatchComplete">
<simple>${property.CamelBatchComplete}</simple>
</setProperty>
<process/>
<filter>
<method>net.aqualogy.sap.xrt.connect.SapProcessor$1$2</method>
<process/>
<to
uri="sftp://xxxxx@xxxxx//DATOS/Interfaces/EXTRACTOS_xxx/extractosXRT?flatten=true&password=xxxxx&passiveMode=true&binary=true&autoCreate=false"/>
<log message="sent file ${file:name} to SAP"/>
</filter>
</route>
The route smb have 5000 files, when a start to tranfer I have the error
OutMemory Java Heap Space, and sockedRead(File Descriptor). I think is for
the amount of files in smb.
Thanks
--
View this message in context: http://camel.465427.n5.nabble.com/Problem-transfer-from-SMB-to-FTP-OutMemory-Java-Heap-Space-tp5783383.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Problem transfer from SMB to FTP OutMemory Java Heap Space
Posted by Brad Johnson <br...@mediadriver.com>.
Obviously I'm operating in the blind here as I can't see the logs or
operations or code. If you have Hawtio running you can log into the
console and see the memory space as your application is running and get
better detail as to what is happening.
You have a maximum memory of 512MB and are hitting the limit. If you look
at the memory of the running instance after start up how much heap space do
you have left? Also, which Java are you using? Java 1.8 no longer uses
the perm gen memory settings. Obviously a simple test is to just bump the
min/max and see if that fixes the problem or if it just delays it
(indicating a different problem).
Is your filter just looking at file names or is it opening the file? I'm
not co
I'm guessing your entire file is getting read into memory and it is hitting
the limit. The first thing I'd do would simply be to try a test with
changing:
<to uri="sftp://xxxxx@xxxxx
//DATOS/Interfaces/EXTRACTOS_xxx/extractosXRT?flatten=true&password=xxxxx&passiveMode=true&binary=true&autoCreate=false"/>
to
<to uri="file://temp/foo.txt"/>
and see what the memory in Hawtio does during that read.
On Mon, Jun 6, 2016 at 4:26 AM, Daniel P22 <po...@gmail.com> wrote:
> I have the default configuration, of karaf.bat
>
> if "%JAVA_MIN_MEM%" == "" (
> set JAVA_MIN_MEM=128M
> )
> if "%JAVA_MAX_MEM%" == "" (
> set JAVA_MAX_MEM=512M
> )
> if "%JAVA_PERM_MEM%" == "" (
> set JAVA_PERM_MEM=16M
> )
> if "%JAVA_MAX_PERM_MEM%" == "" (
> set JAVA_MAX_PERM_MEM=128M
> )
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Problem-transfer-from-SMB-to-FTP-OutMemory-Java-Heap-Space-tp5783383p5783488.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Problem transfer from SMB to FTP OutMemory Java Heap Space
Posted by Daniel P22 <po...@gmail.com>.
I have the default configuration, of karaf.bat
if "%JAVA_MIN_MEM%" == "" (
set JAVA_MIN_MEM=128M
)
if "%JAVA_MAX_MEM%" == "" (
set JAVA_MAX_MEM=512M
)
if "%JAVA_PERM_MEM%" == "" (
set JAVA_PERM_MEM=16M
)
if "%JAVA_MAX_PERM_MEM%" == "" (
set JAVA_MAX_PERM_MEM=128M
)
--
View this message in context: http://camel.465427.n5.nabble.com/Problem-transfer-from-SMB-to-FTP-OutMemory-Java-Heap-Space-tp5783383p5783488.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Problem transfer from SMB to FTP OutMemory Java Heap Space
Posted by Brad Johnson <br...@mediadriver.com>.
Were you able to bump the heap space in Fuse. The bin directory with
fuse/karaf sh or bat file is where that is located. I believe the heap
used to be small by default because OSGi/karaf were and are used in small
devices. However Fuse isn't.
Unfortunately I don't know smb/JFCIS. I know in similar circumstances with
FTP or files I will use either a flag of streaming or chunked splitter to
make sure the endpoint isn't loading everything into memory. In this case
I don't know the mechanics of SMB. Is it actually streaming the file to
you and your camel route and then you are pushing it to the FTP location?
Try breaking the problem down. Temporarily replace your push to FTP with a
write to a local file. Do you run into the problem still? If not, switch
it and run it from the local file to your FTP server. Do you run into the
problem that way?
On Fri, Jun 3, 2016 at 9:42 AM, Daniel P22 <po...@gmail.com> wrote:
> Thanks Hans and Pontus, let me try this of mount a share.
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Problem-transfer-from-SMB-to-FTP-OutMemory-Java-Heap-Space-tp5783383p5783453.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Problem transfer from SMB to FTP OutMemory Java Heap Space
Posted by Daniel P22 <po...@gmail.com>.
Thanks Hans and Pontus, let me try this of mount a share.
--
View this message in context: http://camel.465427.n5.nabble.com/Problem-transfer-from-SMB-to-FTP-OutMemory-Java-Heap-Space-tp5783383p5783453.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Problem transfer from SMB to FTP OutMemory Java Heap Space
Posted by Pontus Ullgren <ul...@gmail.com>.
Strange I have run applications where I transfer files up to 300MB without
problems using camel-jcifs.
Feel free to log a jira (
https://camel-extra.atlassian.net/projects/CAMEX/issues) with your findings.
However I do agree that if you use case is such that you can mount the
shares (either on windows or using smbmount on linux) then using the
file:// transport is probably a more stable and tested way to go.
Also if you are running from a Windows machine I do belive using file://
witth a UNC path is also possible. Provided that the user your JVM is
running as has access to the share.
Best regards
Pontus Ullgren
On Fri, 3 Jun 2016 at 15:17 Hans Orbaan <ha...@docdata.eu> wrote:
> Hi,
>
> You should mount your share and use the file:// component to avoid out of
> memory errors. There is something wrong with camel-jcifs. It will go out of
> memory (threads I guess, not memory, but system resources) will retrieving
> a file.
> This already happens from 5 MB or more we found. Could not really locate
> the issue though.
>
> With kind regards,
>
> Hans Orbaan
>
>
> -----Oorspronkelijk bericht-----
> Van: Pontus Ullgren [mailto:ullgren@gmail.com]
> Verzonden: Friday 3 June 2016 13:15
> Aan: users@camel.apache.org
> Onderwerp: Re: Problem transfer from SMB to FTP OutMemory Java Heap Space
>
> localWorkDirectory is a directory on the server where your camel route run
> (in Fuse as you say),
>
> localWorkDirectory is also supported in camel-jcifs (
> http://camel.apache.org/jcifs.html).
>
>
> // Pontus
>
> On Fri, 3 Jun 2016 at 08:12 Daniel Pompa <po...@gmail.com> wrote:
>
> > Hi Pontus, sorry Im new in Fuse. The Camel version is 2.12.0
> >
> > how I can defined the heap size to the Java Process?
> >
> > I add localWorkDirectory into the URI, but fail again, this
> > localWorkDirectory is a directory in Fuse or in the smb server? I put
> > in the localWorkDirectory in smb.
> >
> > I saw localWorkDirectory is just for FTP.
> >
> > Thanks.
> >
> >
> >
> > --
> > View this message in context:
> > http://camel.465427.n5.nabble.com/Problem-transfer-from-SMB-to-FTP-Out
> > Memory-Java-Heap-Space-tp5783383p5783388.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
> >
>
RE: Problem transfer from SMB to FTP OutMemory Java Heap Space
Posted by Hans Orbaan <ha...@docdata.eu>.
Hi,
You should mount your share and use the file:// component to avoid out of memory errors. There is something wrong with camel-jcifs. It will go out of memory (threads I guess, not memory, but system resources) will retrieving a file.
This already happens from 5 MB or more we found. Could not really locate the issue though.
With kind regards,
Hans Orbaan
-----Oorspronkelijk bericht-----
Van: Pontus Ullgren [mailto:ullgren@gmail.com]
Verzonden: Friday 3 June 2016 13:15
Aan: users@camel.apache.org
Onderwerp: Re: Problem transfer from SMB to FTP OutMemory Java Heap Space
localWorkDirectory is a directory on the server where your camel route run (in Fuse as you say),
localWorkDirectory is also supported in camel-jcifs ( http://camel.apache.org/jcifs.html).
// Pontus
On Fri, 3 Jun 2016 at 08:12 Daniel Pompa <po...@gmail.com> wrote:
> Hi Pontus, sorry Im new in Fuse. The Camel version is 2.12.0
>
> how I can defined the heap size to the Java Process?
>
> I add localWorkDirectory into the URI, but fail again, this
> localWorkDirectory is a directory in Fuse or in the smb server? I put
> in the localWorkDirectory in smb.
>
> I saw localWorkDirectory is just for FTP.
>
> Thanks.
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Problem-transfer-from-SMB-to-FTP-Out
> Memory-Java-Heap-Space-tp5783383p5783388.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Problem transfer from SMB to FTP OutMemory Java Heap Space
Posted by Pontus Ullgren <ul...@gmail.com>.
localWorkDirectory is a directory on the server where your camel route run
(in Fuse as you say),
localWorkDirectory is also supported in camel-jcifs (
http://camel.apache.org/jcifs.html).
// Pontus
On Fri, 3 Jun 2016 at 08:12 Daniel Pompa <po...@gmail.com> wrote:
> Hi Pontus, sorry Im new in Fuse. The Camel version is 2.12.0
>
> how I can defined the heap size to the Java Process?
>
> I add localWorkDirectory into the URI, but fail again, this
> localWorkDirectory is a directory in Fuse or in the smb server? I put in
> the
> localWorkDirectory in smb.
>
> I saw localWorkDirectory is just for FTP.
>
> Thanks.
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Problem-transfer-from-SMB-to-FTP-OutMemory-Java-Heap-Space-tp5783383p5783388.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Problem transfer from SMB to FTP OutMemory Java Heap Space
Posted by Daniel Pompa <po...@gmail.com>.
Hi Pontus, sorry Im new in Fuse. The Camel version is 2.12.0
how I can defined the heap size to the Java Process?
I add localWorkDirectory into the URI, but fail again, this
localWorkDirectory is a directory in Fuse or in the smb server? I put in the
localWorkDirectory in smb.
I saw localWorkDirectory is just for FTP.
Thanks.
--
View this message in context: http://camel.465427.n5.nabble.com/Problem-transfer-from-SMB-to-FTP-OutMemory-Java-Heap-Space-tp5783383p5783388.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Problem transfer from SMB to FTP OutMemory Java Heap Space
Posted by Pontus Ullgren <ul...@gmail.com>.
Which version of Camel and Camel Extra are you using ?
What heap size have you allocated to the Java process ?
Have you tried adding the localWorkDirectory URI option to the smb URI to
avoid loading the file content into memory ?
Best regard
Pontus Ullgren
On Thu, 2 Jun 2016 at 11:40 Daniel Pompa <po...@gmail.com> wrote:
> I have a problema with Fuse. I created a route,
>
> <route xmlns="http://camel.apache.org/schema/spring"
> id="sap-xrt-connect-interface-receive-20160414">
> <from
> uri="smb://xxxxx;xxxxx@xxxxx
> /APLICACIONES/EFACDES/xxxs/?password=xxxx&delete=false&delay=60000"/>
> <onException>
> <redeliveryPolicy maximumRedeliveries="2"
> redeliveryDelay="20000" retryAttemptedLogLevel="WARN"/>
> </onException>
> <setProperty propertyName="CamelBatchComplete">
> <simple>${property.CamelBatchComplete}</simple>
> </setProperty>
> <process/>
> <filter>
> <method>net.aqualogy.sap.xrt.connect.SapProcessor$1$2</method>
> <process/>
> <to
> uri="sftp://xxxxx@xxxxx
> //DATOS/Interfaces/EXTRACTOS_xxx/extractosXRT?flatten=true&password=xxxxx&passiveMode=true&binary=true&autoCreate=false"/>
> <log message="sent file ${file:name} to SAP"/>
> </filter>
> </route>
>
> The route smb have 5000 files, when a start to tranfer I have the error
> OutMemory Java Heap Space, and sockedRead(File Descriptor). I think is for
> the amount of files in smb.
>
> Thanks
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Problem-transfer-from-SMB-to-FTP-OutMemory-Java-Heap-Space-tp5783383.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>