You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Suvendu Sekhar Mondal <su...@gmail.com> on 2019/07/23 11:39:15 UTC
OT: Connection timeout with HttpClient
One of our legacy applications is using Apache Commons HttpClient 3.1.
POST call to one REST service is failing with
"java.net.ConnectException: Connection timed out: connect"
exception[1]. Timeout is occurring after one minute. To figure out
where thread is spending all the time, I took multiple thread dumps.
In all of them, thread is trying to create secure socket[2]. I am not
seeing any thread pool issue. Also the target REST service is working
properly with same payload while invoked from Postman. It is only
failing consistently when going through the HttpClient. I am trying to
figure out why it is taking so long in connecting to the socket. I am
looking for suggestion on how to attack this one.
Tomcat: 7.0.55
JRE: 1.8_92
-Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2
[1]
java.net.ConnectException: Connection timed out: connect
at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:472)
at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:153)
at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:82)
at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:127)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
[2]
"http-apr-18100-exec-5" #327 daemon prio=5 os_prio=0
tid=0x000000002efbf800 nid=0x2508 runnable [0x000000002e55d000]
java.lang.Thread.State: RUNNABLE
at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
- locked <0x000000071a984780> (a java.net.TwoStacksPlainSocketImpl)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:472)
at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:153)
at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:82)
at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:127)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: OT: Connection timeout with HttpClient
Posted by Suvendu Sekhar Mondal <su...@gmail.com>.
Hi Chris,
On Tue, Jul 23, 2019 at 6:00 PM Christopher Schultz
<ch...@christopherschultz.net> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Suvendu,
>
> On 7/23/19 07:39, Suvendu Sekhar Mondal wrote:
> > One of our legacy applications is using Apache Commons HttpClient
> > 3.1. POST call to one REST service is failing with
> > "java.net.ConnectException: Connection timed out: connect"
> > exception[1]. Timeout is occurring after one minute. To figure out
> > where thread is spending all the time, I took multiple thread
> > dumps. In all of them, thread is trying to create secure socket[2].
> > I am not seeing any thread pool issue. Also the target REST service
> > is working properly with same payload while invoked from Postman.
> > It is only failing consistently when going through the HttpClient.
> > I am trying to figure out why it is taking so long in connecting to
> > the socket. I am looking for suggestion on how to attack this one.
> >
> > Tomcat: 7.0.55 JRE: 1.8_92 -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2
>
> Your version is Tomcat is quite old and contains numerous "important"
> publicly-known vulnerabilities. You should upgrade ASAP.
>
Yes, I agree. There are some "technical debt" and upgrade Tomcat and
Apache are one of them. We have plans to do it on next release. :)
> The connection timeout you are seeing is on the client end: your
> client is connecting to another server and the server isn't responding
> fast enough. That's why your stack trace always shows the thread
> "creating a socket": it's trying to connect to the remote service and
> it's never completing.
>
> Since it's failing on "connect" and not "read", it's likely that there
> is a firewall between your client and the service you are trying to
> connect to which is dropping all packets instead of returning a
> "connection refused" response.
>
> So this isn't a problem with your code or with Tomcat. It's a problem
> between your client (which has the stack trace below) and the service
> that code is trying to call.
>
I am planning to use Wireshark to find out the root cause. Some weird
thing must be happening while making call through our client code.
Let's see.
> - -chris
>
> >
> > [1] java.net.ConnectException: Connection timed out: connect at
> > java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method) at
> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.jav
> a:350)
> >
> >
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImp
> l.java:206)
> > at
> > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:
> 188)
> >
> >
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
> > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
> > java.net.Socket.connect(Socket.java:589) at
> > sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) at
> > sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:472) at
> > sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImp
> l.java:153)
> >
> >
> at
> org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSo
> cket(SSLProtocolSocketFactory.java:82)
> > at
> > org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.create
> Socket(SSLProtocolSocketFactory.java:127)
> >
> >
> at
> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:70
> 7)
> > at
> > org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Http
> MethodDirector.java:387)
> >
> >
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMetho
> dDirector.java:171)
> > at
> > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java
> :397)
> >
> >
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:3
> 23)
> >
> > [2] "http-apr-18100-exec-5" #327 daemon prio=5 os_prio=0
> > tid=0x000000002efbf800 nid=0x2508 runnable [0x000000002e55d000]
> > java.lang.Thread.State: RUNNABLE at
> > java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method) at
> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.jav
> a:350)
> >
> >
> - - locked <0x000000071a984780> (a java.net.TwoStacksPlainSocketImpl)
> > at
> > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketI
> mpl.java:206)
> >
> >
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:18
> 8)
> > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) at
> > java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
> > java.net.Socket.connect(Socket.java:589) at
> > sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) at
> > sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:472) at
> > sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImp
> l.java:153)
> >
> >
> at
> org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSo
> cket(SSLProtocolSocketFactory.java:82)
> > at
> > org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.create
> Socket(SSLProtocolSocketFactory.java:127)
> >
> >
> at
> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:70
> 7)
> > at
> > org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Http
> MethodDirector.java:387)
> >
> >
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMetho
> dDirector.java:171)
> > at
> > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java
> :397)
> >
> >
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:3
> 23)
> >
> > ---------------------------------------------------------------------
> >
> >
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl02/eIACgkQHPApP6U8
> pFikGw/+JYcsOSeHO8zlyDnaclEAsFGaSgTZ0zm3zy/M97MoCV1eECM28cuWevJK
> uNk5WD+FtWI3eKZTnqOyFa+hAJhAy4d+B7NA7uHn0mIv9PE6NeMjlmfkMd5avta8
> eaxLfZWGlT5Xnx7egf0eFn4qCDVsMgljjl8GcS2KZJveVndlr7tpLRwnLzH7DY13
> mzzOEejgy8pKDvHP4drnJFHK/KYl2RkuoIFJHUmWeBvlUIuoZHi2nOdyyhy+WIRc
> akI6/i521CzgHXdo55g1GfoOuGdsnOiYMqRvD/USADbARVFzHbTrnh97GOxDCSly
> 2tXSUFtYLIBmzfDFEMa/RLvuPBJ6r3SJ1LnVP07an5rs+Ni+hprZn/DAkE1HhlGQ
> AqrRLLUa1/t0HYswtonG9zfw1KS00kpr4Sql4Jsof5gtYyfOBDXIbAHUJ2vxvXvq
> tvBpR48ES6zR35ssgvpr5+R/pV4ICKsYoz4TkAj9V4pX7eSSwd7zYioOlD0lrJrR
> Jatg9NlVTEjBzGsUAYQ9RRkRgAY2Grw0tHg+SSJkDiWV/OBj2wPMfStiNtnY16mx
> lkL7gTsAPg4sa7XsQ8xzM0rc4+LuEE4qjevL4Uu2lLBLMtIGLW22VgWvyYZy3S6X
> 5neYAkOCSJKuAq9EuRNBMD/JocAlAlt5uCX0neyXpbdFQmdJ0/A=
> =+eTp
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: OT: Connection timeout with HttpClient
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Suvendu,
On 7/23/19 07:39, Suvendu Sekhar Mondal wrote:
> One of our legacy applications is using Apache Commons HttpClient
> 3.1. POST call to one REST service is failing with
> "java.net.ConnectException: Connection timed out: connect"
> exception[1]. Timeout is occurring after one minute. To figure out
> where thread is spending all the time, I took multiple thread
> dumps. In all of them, thread is trying to create secure socket[2].
> I am not seeing any thread pool issue. Also the target REST service
> is working properly with same payload while invoked from Postman.
> It is only failing consistently when going through the HttpClient.
> I am trying to figure out why it is taking so long in connecting to
> the socket. I am looking for suggestion on how to attack this one.
>
> Tomcat: 7.0.55 JRE: 1.8_92 -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2
Your version is Tomcat is quite old and contains numerous "important"
publicly-known vulnerabilities. You should upgrade ASAP.
The connection timeout you are seeing is on the client end: your
client is connecting to another server and the server isn't responding
fast enough. That's why your stack trace always shows the thread
"creating a socket": it's trying to connect to the remote service and
it's never completing.
Since it's failing on "connect" and not "read", it's likely that there
is a firewall between your client and the service you are trying to
connect to which is dropping all packets instead of returning a
"connection refused" response.
So this isn't a problem with your code or with Tomcat. It's a problem
between your client (which has the stack trace below) and the service
that code is trying to call.
- -chris
>
> [1] java.net.ConnectException: Connection timed out: connect at
> java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method) at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.jav
a:350)
>
>
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImp
l.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:
188)
>
>
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
> java.net.Socket.connect(Socket.java:589) at
> sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) at
> sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:472) at
> sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImp
l.java:153)
>
>
at
org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSo
cket(SSLProtocolSocketFactory.java:82)
> at
> org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.create
Socket(SSLProtocolSocketFactory.java:127)
>
>
at
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:70
7)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Http
MethodDirector.java:387)
>
>
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMetho
dDirector.java:171)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java
:397)
>
>
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:3
23)
>
> [2] "http-apr-18100-exec-5" #327 daemon prio=5 os_prio=0
> tid=0x000000002efbf800 nid=0x2508 runnable [0x000000002e55d000]
> java.lang.Thread.State: RUNNABLE at
> java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method) at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.jav
a:350)
>
>
- - locked <0x000000071a984780> (a java.net.TwoStacksPlainSocketImpl)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketI
mpl.java:206)
>
>
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:18
8)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) at
> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
> java.net.Socket.connect(Socket.java:589) at
> sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) at
> sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:472) at
> sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImp
l.java:153)
>
>
at
org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSo
cket(SSLProtocolSocketFactory.java:82)
> at
> org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.create
Socket(SSLProtocolSocketFactory.java:127)
>
>
at
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:70
7)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Http
MethodDirector.java:387)
>
>
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMetho
dDirector.java:171)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java
:397)
>
>
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:3
23)
>
> ---------------------------------------------------------------------
>
>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl02/eIACgkQHPApP6U8
pFikGw/+JYcsOSeHO8zlyDnaclEAsFGaSgTZ0zm3zy/M97MoCV1eECM28cuWevJK
uNk5WD+FtWI3eKZTnqOyFa+hAJhAy4d+B7NA7uHn0mIv9PE6NeMjlmfkMd5avta8
eaxLfZWGlT5Xnx7egf0eFn4qCDVsMgljjl8GcS2KZJveVndlr7tpLRwnLzH7DY13
mzzOEejgy8pKDvHP4drnJFHK/KYl2RkuoIFJHUmWeBvlUIuoZHi2nOdyyhy+WIRc
akI6/i521CzgHXdo55g1GfoOuGdsnOiYMqRvD/USADbARVFzHbTrnh97GOxDCSly
2tXSUFtYLIBmzfDFEMa/RLvuPBJ6r3SJ1LnVP07an5rs+Ni+hprZn/DAkE1HhlGQ
AqrRLLUa1/t0HYswtonG9zfw1KS00kpr4Sql4Jsof5gtYyfOBDXIbAHUJ2vxvXvq
tvBpR48ES6zR35ssgvpr5+R/pV4ICKsYoz4TkAj9V4pX7eSSwd7zYioOlD0lrJrR
Jatg9NlVTEjBzGsUAYQ9RRkRgAY2Grw0tHg+SSJkDiWV/OBj2wPMfStiNtnY16mx
lkL7gTsAPg4sa7XsQ8xzM0rc4+LuEE4qjevL4Uu2lLBLMtIGLW22VgWvyYZy3S6X
5neYAkOCSJKuAq9EuRNBMD/JocAlAlt5uCX0neyXpbdFQmdJ0/A=
=+eTp
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org