You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@servicemix.apache.org by "Gert Vanthienen (JIRA)" <ji...@apache.org> on 2007/04/03 09:17:36 UTC
[jira] Created: (SM-915) FTP poller stalls with threads waiting on
PlainSocketImpl.socketAccept()
FTP poller stalls with threads waiting on PlainSocketImpl.socketAccept()
------------------------------------------------------------------------
Key: SM-915
URL: https://issues.apache.org/activemq/browse/SM-915
Project: ServiceMix
Issue Type: Bug
Components: servicemix-ftp
Affects Versions: 3.1
Reporter: Gert Vanthienen
FTP poller threads stop working with the stack trace below, causing the FTP client pool to run out of connections
{code}
Name: pool-component.servicemix-ftp-thread-2
State: RUNNABLE
Total blocked: 78,430 Total waited: 4,771
Stack trace:
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
java.net.ServerSocket.implAccept(ServerSocket.java:450)
java.net.ServerSocket.accept(ServerSocket.java:421)
org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:502)
org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2390)
org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2364)
org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2141)
org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(FtpPollerEndpoint.java:211)
org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(FtpPollerEndpoint.java:203)
org.apache.servicemix.ftp.FtpPollerEndpoint.poll(FtpPollerEndpoint.java:78)
org.apache.servicemix.common.endpoints.PollingEndpoint$PollSchedulerTask$1.run(PollingEndpoint.java:155)
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
java.lang.Thread.run(Thread.java:595)
{code}
See http://www.nabble.com/FTP-poller-stalls...-tf3490690s12049.html for more information
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (SM-915) FTP poller stalls with threads waiting on PlainSocketImpl.socketAccept()
Posted by Guillaume Nodet <gn...@gmail.com>.
I'm all in favor of upgrading to commons-net if it can fix some issues.
Do they have any time frame for a release ?
On 4/13/07, Gert Vanthienen (JIRA) <ji...@apache.org> wrote:
>
>
> [
> https://issues.apache.org/activemq/browse/SM-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_39005]
>
> Gert Vanthienen commented on SM-915:
> ------------------------------------
>
> On the commons mailing list, Rory Winston suggested to upgrade to
> commons-net-2.0 RC, because it contains fixes for various threading
> issues. It is available at
> http://people.apache.org/~rwinston/commons-net-2.0/commons-net-ftp-2.0.0-SNAPSHOT.jar
>
> However, it requires JDK5/6 and it is not a simple replacement for the
> current commons-net-1.4.x.jar (API changes).
>
> > FTP poller stalls with threads waiting on PlainSocketImpl.socketAccept()
> > ------------------------------------------------------------------------
> >
> > Key: SM-915
> > URL: https://issues.apache.org/activemq/browse/SM-915
> > Project: ServiceMix
> > Issue Type: Bug
> > Components: servicemix-ftp
> > Affects Versions: 3.1
> > Reporter: Gert Vanthienen
> > Attachments: SM-915.patch
> >
> >
> > FTP poller threads stop working with the stack trace below, causing the
> FTP client pool to run out of connections
> > {code}
> > Name: pool-component.servicemix-ftp-thread-2
> > State: RUNNABLE
> > Total blocked: 78,430 Total waited: 4,771
> > Stack trace:
> > java.net.PlainSocketImpl.socketAccept(Native Method)
> > java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> > java.net.ServerSocket.implAccept(ServerSocket.java:450)
> > java.net.ServerSocket.accept(ServerSocket.java:421)
> > org.apache.commons.net.ftp.FTPClient._openDataConnection_(
> FTPClient.java:502)
> > org.apache.commons.net.ftp.FTPClient.initiateListParsing(
> FTPClient.java:2390)
> > org.apache.commons.net.ftp.FTPClient.initiateListParsing(
> FTPClient.java:2364)
> > org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2141)
> > org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(
> FtpPollerEndpoint.java:211)
> > org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(
> FtpPollerEndpoint.java:203)
> > org.apache.servicemix.ftp.FtpPollerEndpoint.poll(
> FtpPollerEndpoint.java:78)
> >
> org.apache.servicemix.common.endpoints.PollingEndpoint$PollSchedulerTask$1.run
> (PollingEndpoint.java:155)
> >
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> (ThreadPoolExecutor.java:665)
> >
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:690)
> > java.lang.Thread.run(Thread.java:595)
> > {code}
> > See http://www.nabble.com/FTP-poller-stalls...-tf3490690s12049.html for
> more information
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
--
Cheers,
Guillaume Nodet
------------------------
Principal Engineer, IONA
Blog: http://gnodet.blogspot.com/
[jira] Commented: (SM-915) FTP poller stalls with threads waiting
on PlainSocketImpl.socketAccept()
Posted by "Gert Vanthienen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_38944 ]
Gert Vanthienen commented on SM-915:
------------------------------------
Some additional observations - the problem only appears when:
* when a lot of files become available on the FTP server together
* some of them are being downloaded before they are complete, i.e. incomplete XML messages are being read and processing fails for these messages
In the FTP poller code, we find
{code}
InputStream in = ftp.retrieveFileStream(file);
InOnly exchange = getExchangeFactory().createInOnlyExchange();
configureExchangeTarget(exchange);
NormalizedMessage message = exchange.createMessage();
exchange.setInMessage(message);
marshaler.readMessage(exchange, message, in, file);
sendSync(exchange);
in.close();
ftp.completePendingCommand();
{code}
My question: Is it possible that {{sendSync(exchange)}} throws an Exception when an incomplete XML message is retrieved from the server? If it does, the inputstream is never closed and {{completePendingCommand()}} is never called, which might explain the problems
> FTP poller stalls with threads waiting on PlainSocketImpl.socketAccept()
> ------------------------------------------------------------------------
>
> Key: SM-915
> URL: https://issues.apache.org/activemq/browse/SM-915
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-ftp
> Affects Versions: 3.1
> Reporter: Gert Vanthienen
>
> FTP poller threads stop working with the stack trace below, causing the FTP client pool to run out of connections
> {code}
> Name: pool-component.servicemix-ftp-thread-2
> State: RUNNABLE
> Total blocked: 78,430 Total waited: 4,771
> Stack trace:
> java.net.PlainSocketImpl.socketAccept(Native Method)
> java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> java.net.ServerSocket.implAccept(ServerSocket.java:450)
> java.net.ServerSocket.accept(ServerSocket.java:421)
> org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:502)
> org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2390)
> org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2364)
> org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2141)
> org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(FtpPollerEndpoint.java:211)
> org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(FtpPollerEndpoint.java:203)
> org.apache.servicemix.ftp.FtpPollerEndpoint.poll(FtpPollerEndpoint.java:78)
> org.apache.servicemix.common.endpoints.PollingEndpoint$PollSchedulerTask$1.run(PollingEndpoint.java:155)
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
> java.lang.Thread.run(Thread.java:595)
> {code}
> See http://www.nabble.com/FTP-poller-stalls...-tf3490690s12049.html for more information
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SM-915) FTP poller stalls with threads waiting on
PlainSocketImpl.socketAccept()
Posted by "Gert Vanthienen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gert Vanthienen updated SM-915:
-------------------------------
Attachment: SM-915.patch
The patch file makes sure that the inputstream is always closed -- followed by the required call to {{completePendingCommand}}.
> FTP poller stalls with threads waiting on PlainSocketImpl.socketAccept()
> ------------------------------------------------------------------------
>
> Key: SM-915
> URL: https://issues.apache.org/activemq/browse/SM-915
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-ftp
> Affects Versions: 3.1
> Reporter: Gert Vanthienen
> Attachments: SM-915.patch
>
>
> FTP poller threads stop working with the stack trace below, causing the FTP client pool to run out of connections
> {code}
> Name: pool-component.servicemix-ftp-thread-2
> State: RUNNABLE
> Total blocked: 78,430 Total waited: 4,771
> Stack trace:
> java.net.PlainSocketImpl.socketAccept(Native Method)
> java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> java.net.ServerSocket.implAccept(ServerSocket.java:450)
> java.net.ServerSocket.accept(ServerSocket.java:421)
> org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:502)
> org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2390)
> org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2364)
> org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2141)
> org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(FtpPollerEndpoint.java:211)
> org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(FtpPollerEndpoint.java:203)
> org.apache.servicemix.ftp.FtpPollerEndpoint.poll(FtpPollerEndpoint.java:78)
> org.apache.servicemix.common.endpoints.PollingEndpoint$PollSchedulerTask$1.run(PollingEndpoint.java:155)
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
> java.lang.Thread.run(Thread.java:595)
> {code}
> See http://www.nabble.com/FTP-poller-stalls...-tf3490690s12049.html for more information
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-915) FTP poller stalls with threads waiting
on PlainSocketImpl.socketAccept()
Posted by "Gert Vanthienen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_39005 ]
Gert Vanthienen commented on SM-915:
------------------------------------
On the commons mailing list, Rory Winston suggested to upgrade to commons-net-2.0 RC, because it contains fixes for various threading issues. It is available at http://people.apache.org/~rwinston/commons-net-2.0/commons-net-ftp-2.0.0-SNAPSHOT.jar
However, it requires JDK5/6 and it is not a simple replacement for the current commons-net-1.4.x.jar (API changes).
> FTP poller stalls with threads waiting on PlainSocketImpl.socketAccept()
> ------------------------------------------------------------------------
>
> Key: SM-915
> URL: https://issues.apache.org/activemq/browse/SM-915
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-ftp
> Affects Versions: 3.1
> Reporter: Gert Vanthienen
> Attachments: SM-915.patch
>
>
> FTP poller threads stop working with the stack trace below, causing the FTP client pool to run out of connections
> {code}
> Name: pool-component.servicemix-ftp-thread-2
> State: RUNNABLE
> Total blocked: 78,430 Total waited: 4,771
> Stack trace:
> java.net.PlainSocketImpl.socketAccept(Native Method)
> java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> java.net.ServerSocket.implAccept(ServerSocket.java:450)
> java.net.ServerSocket.accept(ServerSocket.java:421)
> org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:502)
> org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2390)
> org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2364)
> org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2141)
> org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(FtpPollerEndpoint.java:211)
> org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(FtpPollerEndpoint.java:203)
> org.apache.servicemix.ftp.FtpPollerEndpoint.poll(FtpPollerEndpoint.java:78)
> org.apache.servicemix.common.endpoints.PollingEndpoint$PollSchedulerTask$1.run(PollingEndpoint.java:155)
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
> java.lang.Thread.run(Thread.java:595)
> {code}
> See http://www.nabble.com/FTP-poller-stalls...-tf3490690s12049.html for more information
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-915) FTP poller stalls with threads waiting
on PlainSocketImpl.socketAccept()
Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_38945 ]
Guillaume Nodet commented on SM-915:
------------------------------------
You're right. The sendSync command may throw an exception.
I'm not sure if this happens in this case, but it needs to be fixed anyway.
> FTP poller stalls with threads waiting on PlainSocketImpl.socketAccept()
> ------------------------------------------------------------------------
>
> Key: SM-915
> URL: https://issues.apache.org/activemq/browse/SM-915
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-ftp
> Affects Versions: 3.1
> Reporter: Gert Vanthienen
>
> FTP poller threads stop working with the stack trace below, causing the FTP client pool to run out of connections
> {code}
> Name: pool-component.servicemix-ftp-thread-2
> State: RUNNABLE
> Total blocked: 78,430 Total waited: 4,771
> Stack trace:
> java.net.PlainSocketImpl.socketAccept(Native Method)
> java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> java.net.ServerSocket.implAccept(ServerSocket.java:450)
> java.net.ServerSocket.accept(ServerSocket.java:421)
> org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:502)
> org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2390)
> org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2364)
> org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2141)
> org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(FtpPollerEndpoint.java:211)
> org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(FtpPollerEndpoint.java:203)
> org.apache.servicemix.ftp.FtpPollerEndpoint.poll(FtpPollerEndpoint.java:78)
> org.apache.servicemix.common.endpoints.PollingEndpoint$PollSchedulerTask$1.run(PollingEndpoint.java:155)
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
> java.lang.Thread.run(Thread.java:595)
> {code}
> See http://www.nabble.com/FTP-poller-stalls...-tf3490690s12049.html for more information
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-915) FTP poller stalls with threads waiting
on PlainSocketImpl.socketAccept()
Posted by "Gert Vanthienen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_38955 ]
Gert Vanthienen commented on SM-915:
------------------------------------
... but the original problem still isn't solved.
We get an additional stacktrace when ServiceMix is stopping and the problem only appears to exist when the FTP server is IIS.
{code}
java.io.IOException: Fatal thread interruption during read.
at org.apache.commons.net.telnet.TelnetInputStream.read(TelnetInputStream.java:344)
at org.apache.commons.net.telnet.TelnetInputStream.read(TelnetInputStream.java:466)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.readLine(BufferedReader.java:299)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:264)
at org.apache.commons.net.ftp.FTP.getReply(FTP.java:605)
at org.apache.commons.net.ftp.FTPClient.completePendingCommand(FTPClient.java:1253)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2400)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2364)
at org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2141)
at org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(FtpPollerEndpoint.java:210)
at org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(FtpPollerEndpoint.java:202)
at org.apache.servicemix.ftp.FtpPollerEndpoint.poll(FtpPollerEndpoint.java:77)
at org.apache.servicemix.common.endpoints.PollingEndpoint$PollSchedulerTask$1.run(PollingEndpoint.java:155)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
{code}
I've asked the commons mailing list for help on this one...
> FTP poller stalls with threads waiting on PlainSocketImpl.socketAccept()
> ------------------------------------------------------------------------
>
> Key: SM-915
> URL: https://issues.apache.org/activemq/browse/SM-915
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-ftp
> Affects Versions: 3.1
> Reporter: Gert Vanthienen
> Attachments: SM-915.patch
>
>
> FTP poller threads stop working with the stack trace below, causing the FTP client pool to run out of connections
> {code}
> Name: pool-component.servicemix-ftp-thread-2
> State: RUNNABLE
> Total blocked: 78,430 Total waited: 4,771
> Stack trace:
> java.net.PlainSocketImpl.socketAccept(Native Method)
> java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> java.net.ServerSocket.implAccept(ServerSocket.java:450)
> java.net.ServerSocket.accept(ServerSocket.java:421)
> org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:502)
> org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2390)
> org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2364)
> org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2141)
> org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(FtpPollerEndpoint.java:211)
> org.apache.servicemix.ftp.FtpPollerEndpoint.pollFileOrDirectory(FtpPollerEndpoint.java:203)
> org.apache.servicemix.ftp.FtpPollerEndpoint.poll(FtpPollerEndpoint.java:78)
> org.apache.servicemix.common.endpoints.PollingEndpoint$PollSchedulerTask$1.run(PollingEndpoint.java:155)
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665)
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690)
> java.lang.Thread.run(Thread.java:595)
> {code}
> See http://www.nabble.com/FTP-poller-stalls...-tf3490690s12049.html for more information
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.