You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by "Karl Palsson (JIRA)" <ji...@apache.org> on 2010/07/02 13:51:51 UTC
[jira] Created: (CAMEL-2899) out of heap space if remote FTP site
has too many files to pick up
out of heap space if remote FTP site has too many files to pick up
------------------------------------------------------------------
Key: CAMEL-2899
URL: https://issues.apache.org/activemq/browse/CAMEL-2899
Project: Apache Camel
Issue Type: Bug
Components: camel-ftp
Affects Versions: 2.3.0
Reporter: Karl Palsson
2010-07-02 11:38:07,439 FATAL [org.apache.camel.component.file.remote.FtpConsumer:CamelThread 10] - <Consumer Consumer[my_ftp_URI_here caused by: Java heap space>
java.lang.OutOfMemoryError: Java heap space
My remote FTP server has ~60k 100 byte files, and the camel endpoint consumer falls over and doesn't start again. I can use JMX to stop/start the consumer, (it still has status "started") and it will log in to the remote server again, but then fall over with the out of heap space.
I can work around this by increasing the heap, or by moving some of the files aside, but I don't think camel should care how many files there are, or at least, I think it should deal with it more gracefully.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-2899) out of heap space if remote FTP site
has too many files to pick up
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=60412#action_60412 ]
Claus Ibsen commented on CAMEL-2899:
------------------------------------
We could either
a)
Use lightweight record structure for the 160K list of files instead of turning them into Exchange objects which carries extra load
b)
Alter so the maxMessagesPerPoll will be effective earlier. Currently it loads the entire list, then you can filter, sort etc. And after all this it will cut down the list.
If we apply the maxMessagesPerPoll earlier we can't really be able to sort the entire list etc. And the filter could also cut out some from the max list.
Or in other words we will have to re-add files until we hit that limit if there are more files.
In your case I don't think you care about sorting/filtering etc. You just want to poll X number of files per go to avoid excessive memory overhead. So I think we could add an option in the uri for solution b.
Anyone got a good name for such an option?
Any thoughts?
> out of heap space if remote FTP site has too many files to pick up
> ------------------------------------------------------------------
>
> Key: CAMEL-2899
> URL: https://issues.apache.org/activemq/browse/CAMEL-2899
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-ftp
> Affects Versions: 2.3.0
> Reporter: Karl Palsson
> Fix For: 2.4.0
>
>
> 2010-07-02 11:38:07,439 FATAL [org.apache.camel.component.file.remote.FtpConsumer:CamelThread 10] - <Consumer Consumer[my_ftp_URI_here caused by: Java heap space>
> java.lang.OutOfMemoryError: Java heap space
> My remote FTP server has ~60k 100 byte files, and the camel endpoint consumer falls over and doesn't start again. I can use JMX to stop/start the consumer, (it still has status "started") and it will log in to the remote server again, but then fall over with the out of heap space.
> I can work around this by increasing the heap, or by moving some of the files aside, but I don't think camel should care how many files there are, or at least, I think it should deal with it more gracefully.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CAMEL-2899) out of heap space if remote FTP site
has too many files to pick up
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen reassigned CAMEL-2899:
----------------------------------
Assignee: Claus Ibsen
> out of heap space if remote FTP site has too many files to pick up
> ------------------------------------------------------------------
>
> Key: CAMEL-2899
> URL: https://issues.apache.org/activemq/browse/CAMEL-2899
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-ftp
> Affects Versions: 2.3.0
> Reporter: Karl Palsson
> Assignee: Claus Ibsen
> Fix For: 2.4.0
>
>
> 2010-07-02 11:38:07,439 FATAL [org.apache.camel.component.file.remote.FtpConsumer:CamelThread 10] - <Consumer Consumer[my_ftp_URI_here caused by: Java heap space>
> java.lang.OutOfMemoryError: Java heap space
> My remote FTP server has ~60k 100 byte files, and the camel endpoint consumer falls over and doesn't start again. I can use JMX to stop/start the consumer, (it still has status "started") and it will log in to the remote server again, but then fall over with the out of heap space.
> I can work around this by increasing the heap, or by moving some of the files aside, but I don't think camel should care how many files there are, or at least, I think it should deal with it more gracefully.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CAMEL-2899) out of heap space if remote FTP site
has too many files to pick up
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen resolved CAMEL-2899.
--------------------------------
Resolution: Fixed
The {{maxMessagesPerPoll}} option is now applied earlier for the file/ftp components so we avoid creating so many Exchange objects which are then afterwards discarded.
trunk: 960839.
Notice there is no limit in the java.io.File or FTP API to tell it to only list at most X number of files.
> out of heap space if remote FTP site has too many files to pick up
> ------------------------------------------------------------------
>
> Key: CAMEL-2899
> URL: https://issues.apache.org/activemq/browse/CAMEL-2899
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-ftp
> Affects Versions: 2.3.0
> Reporter: Karl Palsson
> Assignee: Claus Ibsen
> Fix For: 2.4.0
>
>
> 2010-07-02 11:38:07,439 FATAL [org.apache.camel.component.file.remote.FtpConsumer:CamelThread 10] - <Consumer Consumer[my_ftp_URI_here caused by: Java heap space>
> java.lang.OutOfMemoryError: Java heap space
> My remote FTP server has ~60k 100 byte files, and the camel endpoint consumer falls over and doesn't start again. I can use JMX to stop/start the consumer, (it still has status "started") and it will log in to the remote server again, but then fall over with the out of heap space.
> I can work around this by increasing the heap, or by moving some of the files aside, but I don't think camel should care how many files there are, or at least, I think it should deal with it more gracefully.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-2899) out of heap space if remote FTP site
has too many files to pick up
Posted by "Gert Vanthienen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=60406#action_60406 ]
Gert Vanthienen commented on CAMEL-2899:
----------------------------------------
I think you can already configure that with the *{{maxMessagesPerPoll}}* URI parameter to limit the number of files being processing in a single poll (cfr. http://camel.apache.org/file2.html) - could you give that a try to see if it solves your problem?
> out of heap space if remote FTP site has too many files to pick up
> ------------------------------------------------------------------
>
> Key: CAMEL-2899
> URL: https://issues.apache.org/activemq/browse/CAMEL-2899
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-ftp
> Affects Versions: 2.3.0
> Reporter: Karl Palsson
>
> 2010-07-02 11:38:07,439 FATAL [org.apache.camel.component.file.remote.FtpConsumer:CamelThread 10] - <Consumer Consumer[my_ftp_URI_here caused by: Java heap space>
> java.lang.OutOfMemoryError: Java heap space
> My remote FTP server has ~60k 100 byte files, and the camel endpoint consumer falls over and doesn't start again. I can use JMX to stop/start the consumer, (it still has status "started") and it will log in to the remote server again, but then fall over with the out of heap space.
> I can work around this by increasing the heap, or by moving some of the files aside, but I don't think camel should care how many files there are, or at least, I think it should deal with it more gracefully.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-2899) out of heap space if remote FTP site
has too many files to pick up
Posted by "Karl Palsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=60411#action_60411 ]
Karl Palsson commented on CAMEL-2899:
-------------------------------------
Some stack traces...
This is without having a file name pattern match filter
FATAL [org.apache.camel.component.file.remote.FtpConsumer:CamelThread 4] - <Consumer Consumer[ftp://sms02@localhost:2121/autreq/ok?delay=5000&maxMessagesPerPoll=50&move=.processed&passiveMode=true&password=sms02] could not poll endpoint: ftp://sms02@localhost:2121/autreq/ok?delay=5000&maxMessagesPerPoll=50&move=.processed&passiveMode=true&password=sms02 caused by: Java heap space>
java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.ConcurrentHashMap.<init>(ConcurrentHashMap.java:613)
at java.util.concurrent.ConcurrentHashMap.<init>(ConcurrentHashMap.java:652)
at org.apache.camel.impl.DefaultExchange.getProperties(DefaultExchange.java:168)
at org.apache.camel.impl.DefaultExchange.setProperty(DefaultExchange.java:153)
at org.apache.camel.component.file.GenericFile.bindToExchange(GenericFile.java:96)
at org.apache.camel.component.file.remote.RemoteFileEndpoint.createExchange(RemoteFileEndpoint.java:50)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:87)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:98)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
The two below here were when I had an ant pattern match filter. From the debug logs, it filtered _every_ file, regardless of the max messages per poll
FATAL [org.apache.camel.component.file.remote.FtpConsumer:CamelThread 4] - <Consumer Consumer[ftp://sms02@localhost:2121/autreq/ok?delay=5000&filter=%23conaxRequestFilter&maxMessagesPerPoll=50&move=.processed&passiveMode=true&password=sms02] could not poll endpoint: ftp://sms02@localhost:2121/autreq/ok?delay=5000&filter=%23conaxRequestFilter&maxMessagesPerPoll=50&move=.processed&passiveMode=true&password=sms02 caused by: Java heap space>
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at org.apache.camel.component.file.GenericFile.populateHeaders(GenericFile.java:118)
at org.apache.camel.component.file.remote.RemoteFile.populateHeaders(RemoteFile.java:39)
at org.apache.camel.component.file.GenericFile.bindToExchange(GenericFile.java:99)
at org.apache.camel.component.file.remote.RemoteFileEndpoint.createExchange(RemoteFileEndpoint.java:50)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:87)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:98)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
FATAL [org.apache.camel.component.file.remote.FtpConsumer:CamelThread 4] - <Consumer Consumer[ftp://sms02@localhost:2121/autreq/ok?delay=5000&filter=%23conaxRequestFilter&maxMessagesPerPoll=50&move=.processed&passiveMode=true&password=sms02] could not poll endpoint: ftp://sms02@localhost:2121/autreq/ok?delay=5000&filter=%23conaxRequestFilter&maxMessagesPerPoll=50&move=.processed&passiveMode=true&password=sms02 caused by: Java heap space>
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at org.apache.camel.component.file.GenericFile.populateHeaders(GenericFile.java:118)
at org.apache.camel.component.file.remote.RemoteFile.populateHeaders(RemoteFile.java:39)
at org.apache.camel.component.file.GenericFile.bindToExchange(GenericFile.java:99)
at org.apache.camel.component.file.remote.RemoteFileEndpoint.createExchange(RemoteFileEndpoint.java:50)
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:87)
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:98)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
> out of heap space if remote FTP site has too many files to pick up
> ------------------------------------------------------------------
>
> Key: CAMEL-2899
> URL: https://issues.apache.org/activemq/browse/CAMEL-2899
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-ftp
> Affects Versions: 2.3.0
> Reporter: Karl Palsson
>
> 2010-07-02 11:38:07,439 FATAL [org.apache.camel.component.file.remote.FtpConsumer:CamelThread 10] - <Consumer Consumer[my_ftp_URI_here caused by: Java heap space>
> java.lang.OutOfMemoryError: Java heap space
> My remote FTP server has ~60k 100 byte files, and the camel endpoint consumer falls over and doesn't start again. I can use JMX to stop/start the consumer, (it still has status "started") and it will log in to the remote server again, but then fall over with the out of heap space.
> I can work around this by increasing the heap, or by moving some of the files aside, but I don't think camel should care how many files there are, or at least, I think it should deal with it more gracefully.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CAMEL-2899) out of heap space if remote FTP site
has too many files to pick up
Posted by "Karl Palsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=60410#action_60410 ]
Karl Palsson commented on CAMEL-2899:
-------------------------------------
Seemed very reasonable, but unfortunately, it doesn't help....
2010-07-02 17:02:42,327 FATAL Consumer Consumer[ftp://sms02@localhost:2121/autre
q/ok?delay=5000&filter=%23conaxRequestFilter&maxMessagesPerPoll=50&move=.process
ed&passiveMode=true&password=sms02] could not poll endpoint: ftp://sms02@localho
st:2121/autreq/ok?delay=5000&filter=%23conaxRequestFilter&maxMessagesPerPoll=50&
move=.processed&passiveMode=true&password=sms02 caused by: Java heap space
java.lang.OutOfMemoryError: Java heap space
I tried again without file filter, (just an ant pattern name matcher) in case that was causing any problems, but it still fails.
> out of heap space if remote FTP site has too many files to pick up
> ------------------------------------------------------------------
>
> Key: CAMEL-2899
> URL: https://issues.apache.org/activemq/browse/CAMEL-2899
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-ftp
> Affects Versions: 2.3.0
> Reporter: Karl Palsson
>
> 2010-07-02 11:38:07,439 FATAL [org.apache.camel.component.file.remote.FtpConsumer:CamelThread 10] - <Consumer Consumer[my_ftp_URI_here caused by: Java heap space>
> java.lang.OutOfMemoryError: Java heap space
> My remote FTP server has ~60k 100 byte files, and the camel endpoint consumer falls over and doesn't start again. I can use JMX to stop/start the consumer, (it still has status "started") and it will log in to the remote server again, but then fall over with the out of heap space.
> I can work around this by increasing the heap, or by moving some of the files aside, but I don't think camel should care how many files there are, or at least, I think it should deal with it more gracefully.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CAMEL-2899) out of heap space if remote FTP site
has too many files to pick up
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/CAMEL-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen updated CAMEL-2899:
-------------------------------
Fix Version/s: 2.4.0
> out of heap space if remote FTP site has too many files to pick up
> ------------------------------------------------------------------
>
> Key: CAMEL-2899
> URL: https://issues.apache.org/activemq/browse/CAMEL-2899
> Project: Apache Camel
> Issue Type: Bug
> Components: camel-ftp
> Affects Versions: 2.3.0
> Reporter: Karl Palsson
> Fix For: 2.4.0
>
>
> 2010-07-02 11:38:07,439 FATAL [org.apache.camel.component.file.remote.FtpConsumer:CamelThread 10] - <Consumer Consumer[my_ftp_URI_here caused by: Java heap space>
> java.lang.OutOfMemoryError: Java heap space
> My remote FTP server has ~60k 100 byte files, and the camel endpoint consumer falls over and doesn't start again. I can use JMX to stop/start the consumer, (it still has status "started") and it will log in to the remote server again, but then fall over with the out of heap space.
> I can work around this by increasing the heap, or by moving some of the files aside, but I don't think camel should care how many files there are, or at least, I think it should deal with it more gracefully.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.