You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Sadeep Jayasumana <ga...@gmail.com> on 2012/01/27 10:48:44 UTC
HTTPCore-NIO: Server rejecting socket connections?
Hi,
I have written an HTTP server using HTTPCore-NIO and currently testing it
with httpcore-ab [1]. I have been seeing very good TPS values so far. But
when I try to send 5KB HTTP messages with 2500 and concurrency level, 50
messages per thread, I'm getting the following exception several times from
httpcore-ab and it results in "Failed requests" in the httpcore-ab output.
No exceptions are displayed from the server side. Hence, I think the server
is rejecting some socket connections because it's already fully loaded.
I'm running the server and load generator on two server machines that are
connected with Gigabit Ethernet. Each server has 16 CPU cores, 16 GB memory
etc.
I have increased the socket timeout in both httpcore-ab and server side
ListeningIOReactor to 360000. 16 threads are allocated for the reactor.
Messages received by the server are processed in a separate worker pool
which has ~500 threads. I have also increased the open file limit of my OS
(Debian) to 65535.
Is there any tuning parameter of HTTPCore that I can used to get rid of
this exception? Perhaps, something similar to "acceptQueueSize" in Jetty
[2]?
[1]
https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
[2] http://docs.codehaus.org/display/JETTY/Configuring+Connectors
[3]
java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.<init>(Socket.java:375)
at java.net.Socket.<init>(Socket.java:189)
at org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.<init>(Socket.java:375)
at java.net.Socket.<init>(Socket.java:189)
at org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Thanks & Regards,
--
Sadeep Jayasumana****
Email: gayansadeep@gmail.com****
Phone: +94-77-2266507
Re: HTTPCore-NIO: Server rejecting socket connections?
Posted by Sadeep Jayasumana <ga...@gmail.com>.
Hi Oleg,
On Wed, Feb 1, 2012 at 5:38 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Wed, 2012-02-01 at 16:16 +0530, Sadeep Jayasumana wrote:
> > Hi Oleg,
> >
> > I tested the patch. When I haven't explicitly set the connection
> timeout, I
> > got the same exception, java.net.ConnectException: Connection timed out,
> at
> > concurrency = 3000, messages/thread = 100 intermittently. One time I got
> it
> > in less than 17 seconds. This is contrary to the Javadoc of
> > Socket#connect() method [1] which says "*A timeout of zero is interpreted
> > as an infinite timeout*".
> >
> > Then when I explicitly set it to 5 seconds as follows I was unable to
> > reproduce the exception although I tried several times.
> >
> > socket.connect(new InetSocketAddress(hostname, port), 1000 * 60 * 5);
> >
> > The other doubt I have is; although I'm getting a
> > java.net.ConnectException, according to [1], this method should throw
> > SocketTimeoutException in case of a timeout. Further, JavaDoc of
> > ConnectException (the one I'm actually getting) says it signals that *"an
> > error occurred while attempting to connect a socket to a remote address
> and
> > port. Typically, the connection was refused remotely (e.g., no process is
> > listening on the remote address/port)." *
> >
> > Therefore, I believe that the actual cause of this error is server's
> > ListeningIOReactor not being able to react to incoming connection
> requests.
> > I had allocated 16 threads (= no of CPU cores) for ListeningIOReactor.
> > Maybe increasing it further would solve this issue. I will test this
> > further and let you know.
> >
>
> ConnectException is a subclass of SocketTimeoutException and the queer
> thing the exception message clearly says "Connection timed out".
It turns out that ConnectException is not a subclass of
SocketTimeoutException. That's why I'm confused.
So, I
> am still not entirely convinced this rules out a client side issue.
>
> Please also note that in its current implementation the
> DefaultListeningIOReactor utilizes only one thread and only one i/o
> selector for all incoming requests no matter how many worker threads are
> allocated for handling i/o on active channels. If you suspect this could
> be the issue, you may need to consider writing a custom
> ListeningIOReactor
>
Got the idea. Thanks for the explanation.
Thanks,
Sadeep
>
> Oleg
>
> > [1]
> >
> http://docs.oracle.com/javase/6/docs/api/java/net/Socket.html#connect%28java.net.SocketAddress,%20int%29
> > [2]
> http://docs.oracle.com/javase/6/docs/api/java/net/ConnectException.html
> >
> > Thanks,
> > Sadeep
> >
> > On Tue, Jan 31, 2012 at 9:42 PM, Sadeep Jayasumana <
> gayansadeep@gmail.com>wrote:
> >
> > > Hi Oleg,
> > >
> > > On Tue, Jan 31, 2012 at 7:48 PM, Oleg Kalnichevski <olegk@apache.org
> >wrote:
> > >
> > >> On Mon, 2012-01-30 at 14:56 +0100, Oleg Kalnichevski wrote:
> > >> > On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote:
> > >> > > Hi,
> > >> > >
> > >> > > On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <
> olegk@apache.org>
> > >> wrote:
> > >> > >
> > >> > > > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote:
> > >> > > > > Hi,
> > >> > > > >
> > >> > > > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <
> > >> olegk@apache.org>
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
> > >> > > > > > > Hi,
> > >> > > > > > >
> > >> > > > > > > I have written an HTTP server using HTTPCore-NIO and
> currently
> > >> > > > testing it
> > >> > > > > > > with httpcore-ab [1]. I have been seeing very good TPS
> values
> > >> so
> > >> > > > far. But
> > >> > > > > > > when I try to send 5KB HTTP messages with 2500 and
> concurrency
> > >> > > > level, 50
> > >> > > > > > > messages per thread, I'm getting the following exception
> > >> several
> > >> > > > times
> > >> > > > > > from
> > >> > > > > > > httpcore-ab and it results in "Failed requests" in the
> > >> httpcore-ab
> > >> > > > > > output.
> > >> > > > > > > No exceptions are displayed from the server side. Hence, I
> > >> think the
> > >> > > > > > server
> > >> > > > > > > is rejecting some socket connections because it's already
> > >> fully
> > >> > > > loaded.
> > >> > > > > > >
> > >> > > > > > > I'm running the server and load generator on two server
> > >> machines
> > >> > > > that are
> > >> > > > > > > connected with Gigabit Ethernet. Each server has 16 CPU
> > >> cores, 16 GB
> > >> > > > > > memory
> > >> > > > > > > etc.
> > >> > > > > > >
> > >> > > > > > > I have increased the socket timeout in both httpcore-ab
> and
> > >> server
> > >> > > > side
> > >> > > > > > > ListeningIOReactor to 360000. 16 threads are allocated
> for the
> > >> > > > reactor.
> > >> > > > > > > Messages received by the server are processed in a
> separate
> > >> worker
> > >> > > > pool
> > >> > > > > > > which has ~500 threads. I have also increased the open
> file
> > >> limit of
> > >> > > > my
> > >> > > > > > OS
> > >> > > > > > > (Debian) to 65535.
> > >> > > > > > >
> > >> > > > > > > Is there any tuning parameter of HTTPCore that I can used
> to
> > >> get rid
> > >> > > > of
> > >> > > > > > > this exception? Perhaps, something similar to
> > >> "acceptQueueSize" in
> > >> > > > Jetty
> > >> > > > > > > [2]?
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > > Hi Sadeep
> > >> > > > > >
> > >> > > > > > A couple of points in no particular logical order.
> > >> > > > > >
> > >> > > > > > (1) HttpCore does not depend on a logging toolkit and does
> not
> > >> do any
> > >> > > > > > logging per default. If it ever encounters an abnormal
> > >> situation,
> > >> > > > > > instead of logging such an event it throws an exception and
> > >> terminates.
> > >> > > > > > Quite often this is not desirable, so one can provide an
> > >> exception
> > >> > > > > > handler that can log using a a logging toolkit of choice or
> > >> handle a
> > >> > > > > > particular type of exception in an application specific way.
> > >> > > > > >
> > >> > > > > > If the server did not terminate under with an exception, I
> kind
> > >> of
> > >> > > > > > suspect it was still trying to chew through the load without
> > >> > > > > > encountering anything abnormal. Unless you have added a
> custom
> > >> > > > exception
> > >> > > > > > handler.
> > >> > > > > >
> > >> > > > > > (2) HttpCore does not have a concept of accepted connection
> > >> queue. As
> > >> > > > > > soon as the listener accepts an incoming connection its
> channel
> > >> gets
> > >> > > > > > immediately added to the underlying i/o selector. No
> > >> intermediate
> > >> > > > > > queuing takes place.
> > >> > > > >
> > >> > > > >
> > >> > > > > > (3) Given that the connection times out on the client, this
> > >> sounds more
> > >> > > > > > like a client side issue. Was connection really suck in
> > >> > > > > > Socket#socketConnect for 360000?
> > >> > > > > >
> > >> > > > >
> > >> > > > > I'm using httpcore-ab [1] as the client. I have set socket
> > >> timeout of
> > >> > > > using
> > >> > > > > -t 360000, and changed the connection timeout with the patch
> > >> shown below.
> > >> > > > > But I'm starting to get this error well before 360000 millis
> past
> > >> the
> > >> > > > test
> > >> > > > > startup. Therefore, I there is no possibility that the
> connection
> > >> attempt
> > >> > > > > hangs for 360000 millis before printing this exception. I was
> > >> thinking
> > >> > > > > whether there are upper bounds for these limits that cause my
> > >> settings to
> > >> > > > > be neglected.
> > >> > > > >
> > >> > > >
> > >> > > > You see. If java.net.ConnectException: Connection timed out is
> > >> thrown by
> > >> > > > java.net.PlainSocketImpl.socketConnect before 360000 millis
> elapse,
> > >> > > > there must be something wrong with the client side.
> > >> > > >
> > >> > >
> > >> > > Could this be a possible bug in httpcore-ab code? I'm quite sure
> that
> > >> it
> > >> > > doesn't wait 360000 millis before throwing this exception.
> > >> > >
> > >> >
> > >>
> > >> Hi Sadeep
> > >>
> > >> It did turn out to be an issue with the way HttpCore AB initialized
> > >> connections. It was basically using an old (Java 1.0) socket
> constructor
> > >> whose behavior can differ between OS platforms and possibly between
> JRE
> > >> major releases with regards to connect timeout handling. The connect
> > >> timeout value set in the HttpParams simply had no effect and therefore
> > >> connections in your tests were timing out before 360000 milliseconds
> > >> elapsed (probably using some sort of a system default).
> > >>
> > >> I committed a fix for the problem
> > >>
> > >> http://svn.apache.org/viewvc?rev=1238566&view=rev
> > >>
> > >> Please review
> > >>
> > >
> > > Many thanks for looking into this. I will review this at my earliest
> and
> > > let you know the outcome.
> > >
> > > Thanks,
> > > Sadeep
> > >
> > >
> > >>
> > >> Oleg
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > >> For additional commands, e-mail: dev-help@hc.apache.org
> > >>
> > >>
> > >
> > >
> > > --
> > >
> > > Sadeep Jayasumana****
> > >
> > > Email: gayansadeep@gmail.com****
> > >
> > > Phone: +94-77-2266507
> > >
> > >
> > >
> > >
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
--
Sadeep Jayasumana****
Email: gayansadeep@gmail.com****
Phone: +94-77-2266507
Re: HTTPCore-NIO: Server rejecting socket connections?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2012-02-01 at 16:16 +0530, Sadeep Jayasumana wrote:
> Hi Oleg,
>
> I tested the patch. When I haven't explicitly set the connection timeout, I
> got the same exception, java.net.ConnectException: Connection timed out, at
> concurrency = 3000, messages/thread = 100 intermittently. One time I got it
> in less than 17 seconds. This is contrary to the Javadoc of
> Socket#connect() method [1] which says "*A timeout of zero is interpreted
> as an infinite timeout*".
>
> Then when I explicitly set it to 5 seconds as follows I was unable to
> reproduce the exception although I tried several times.
>
> socket.connect(new InetSocketAddress(hostname, port), 1000 * 60 * 5);
>
> The other doubt I have is; although I'm getting a
> java.net.ConnectException, according to [1], this method should throw
> SocketTimeoutException in case of a timeout. Further, JavaDoc of
> ConnectException (the one I'm actually getting) says it signals that *"an
> error occurred while attempting to connect a socket to a remote address and
> port. Typically, the connection was refused remotely (e.g., no process is
> listening on the remote address/port)." *
>
> Therefore, I believe that the actual cause of this error is server's
> ListeningIOReactor not being able to react to incoming connection requests.
> I had allocated 16 threads (= no of CPU cores) for ListeningIOReactor.
> Maybe increasing it further would solve this issue. I will test this
> further and let you know.
>
ConnectException is a subclass of SocketTimeoutException and the queer
thing the exception message clearly says "Connection timed out". So, I
am still not entirely convinced this rules out a client side issue.
Please also note that in its current implementation the
DefaultListeningIOReactor utilizes only one thread and only one i/o
selector for all incoming requests no matter how many worker threads are
allocated for handling i/o on active channels. If you suspect this could
be the issue, you may need to consider writing a custom
ListeningIOReactor
Oleg
> [1]
> http://docs.oracle.com/javase/6/docs/api/java/net/Socket.html#connect%28java.net.SocketAddress,%20int%29
> [2] http://docs.oracle.com/javase/6/docs/api/java/net/ConnectException.html
>
> Thanks,
> Sadeep
>
> On Tue, Jan 31, 2012 at 9:42 PM, Sadeep Jayasumana <ga...@gmail.com>wrote:
>
> > Hi Oleg,
> >
> > On Tue, Jan 31, 2012 at 7:48 PM, Oleg Kalnichevski <ol...@apache.org>wrote:
> >
> >> On Mon, 2012-01-30 at 14:56 +0100, Oleg Kalnichevski wrote:
> >> > On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote:
> >> > > Hi,
> >> > >
> >> > > On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <ol...@apache.org>
> >> wrote:
> >> > >
> >> > > > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote:
> >> > > > > Hi,
> >> > > > >
> >> > > > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <
> >> olegk@apache.org>
> >> > > > wrote:
> >> > > > >
> >> > > > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
> >> > > > > > > Hi,
> >> > > > > > >
> >> > > > > > > I have written an HTTP server using HTTPCore-NIO and currently
> >> > > > testing it
> >> > > > > > > with httpcore-ab [1]. I have been seeing very good TPS values
> >> so
> >> > > > far. But
> >> > > > > > > when I try to send 5KB HTTP messages with 2500 and concurrency
> >> > > > level, 50
> >> > > > > > > messages per thread, I'm getting the following exception
> >> several
> >> > > > times
> >> > > > > > from
> >> > > > > > > httpcore-ab and it results in "Failed requests" in the
> >> httpcore-ab
> >> > > > > > output.
> >> > > > > > > No exceptions are displayed from the server side. Hence, I
> >> think the
> >> > > > > > server
> >> > > > > > > is rejecting some socket connections because it's already
> >> fully
> >> > > > loaded.
> >> > > > > > >
> >> > > > > > > I'm running the server and load generator on two server
> >> machines
> >> > > > that are
> >> > > > > > > connected with Gigabit Ethernet. Each server has 16 CPU
> >> cores, 16 GB
> >> > > > > > memory
> >> > > > > > > etc.
> >> > > > > > >
> >> > > > > > > I have increased the socket timeout in both httpcore-ab and
> >> server
> >> > > > side
> >> > > > > > > ListeningIOReactor to 360000. 16 threads are allocated for the
> >> > > > reactor.
> >> > > > > > > Messages received by the server are processed in a separate
> >> worker
> >> > > > pool
> >> > > > > > > which has ~500 threads. I have also increased the open file
> >> limit of
> >> > > > my
> >> > > > > > OS
> >> > > > > > > (Debian) to 65535.
> >> > > > > > >
> >> > > > > > > Is there any tuning parameter of HTTPCore that I can used to
> >> get rid
> >> > > > of
> >> > > > > > > this exception? Perhaps, something similar to
> >> "acceptQueueSize" in
> >> > > > Jetty
> >> > > > > > > [2]?
> >> > > > > > >
> >> > > > > >
> >> > > > > > Hi Sadeep
> >> > > > > >
> >> > > > > > A couple of points in no particular logical order.
> >> > > > > >
> >> > > > > > (1) HttpCore does not depend on a logging toolkit and does not
> >> do any
> >> > > > > > logging per default. If it ever encounters an abnormal
> >> situation,
> >> > > > > > instead of logging such an event it throws an exception and
> >> terminates.
> >> > > > > > Quite often this is not desirable, so one can provide an
> >> exception
> >> > > > > > handler that can log using a a logging toolkit of choice or
> >> handle a
> >> > > > > > particular type of exception in an application specific way.
> >> > > > > >
> >> > > > > > If the server did not terminate under with an exception, I kind
> >> of
> >> > > > > > suspect it was still trying to chew through the load without
> >> > > > > > encountering anything abnormal. Unless you have added a custom
> >> > > > exception
> >> > > > > > handler.
> >> > > > > >
> >> > > > > > (2) HttpCore does not have a concept of accepted connection
> >> queue. As
> >> > > > > > soon as the listener accepts an incoming connection its channel
> >> gets
> >> > > > > > immediately added to the underlying i/o selector. No
> >> intermediate
> >> > > > > > queuing takes place.
> >> > > > >
> >> > > > >
> >> > > > > > (3) Given that the connection times out on the client, this
> >> sounds more
> >> > > > > > like a client side issue. Was connection really suck in
> >> > > > > > Socket#socketConnect for 360000?
> >> > > > > >
> >> > > > >
> >> > > > > I'm using httpcore-ab [1] as the client. I have set socket
> >> timeout of
> >> > > > using
> >> > > > > -t 360000, and changed the connection timeout with the patch
> >> shown below.
> >> > > > > But I'm starting to get this error well before 360000 millis past
> >> the
> >> > > > test
> >> > > > > startup. Therefore, I there is no possibility that the connection
> >> attempt
> >> > > > > hangs for 360000 millis before printing this exception. I was
> >> thinking
> >> > > > > whether there are upper bounds for these limits that cause my
> >> settings to
> >> > > > > be neglected.
> >> > > > >
> >> > > >
> >> > > > You see. If java.net.ConnectException: Connection timed out is
> >> thrown by
> >> > > > java.net.PlainSocketImpl.socketConnect before 360000 millis elapse,
> >> > > > there must be something wrong with the client side.
> >> > > >
> >> > >
> >> > > Could this be a possible bug in httpcore-ab code? I'm quite sure that
> >> it
> >> > > doesn't wait 360000 millis before throwing this exception.
> >> > >
> >> >
> >>
> >> Hi Sadeep
> >>
> >> It did turn out to be an issue with the way HttpCore AB initialized
> >> connections. It was basically using an old (Java 1.0) socket constructor
> >> whose behavior can differ between OS platforms and possibly between JRE
> >> major releases with regards to connect timeout handling. The connect
> >> timeout value set in the HttpParams simply had no effect and therefore
> >> connections in your tests were timing out before 360000 milliseconds
> >> elapsed (probably using some sort of a system default).
> >>
> >> I committed a fix for the problem
> >>
> >> http://svn.apache.org/viewvc?rev=1238566&view=rev
> >>
> >> Please review
> >>
> >
> > Many thanks for looking into this. I will review this at my earliest and
> > let you know the outcome.
> >
> > Thanks,
> > Sadeep
> >
> >
> >>
> >> Oleg
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> >> For additional commands, e-mail: dev-help@hc.apache.org
> >>
> >>
> >
> >
> > --
> >
> > Sadeep Jayasumana****
> >
> > Email: gayansadeep@gmail.com****
> >
> > Phone: +94-77-2266507
> >
> >
> >
> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: HTTPCore-NIO: Server rejecting socket connections?
Posted by Sadeep Jayasumana <ga...@gmail.com>.
Hi Oleg,
I tested the patch. When I haven't explicitly set the connection timeout, I
got the same exception, java.net.ConnectException: Connection timed out, at
concurrency = 3000, messages/thread = 100 intermittently. One time I got it
in less than 17 seconds. This is contrary to the Javadoc of
Socket#connect() method [1] which says "*A timeout of zero is interpreted
as an infinite timeout*".
Then when I explicitly set it to 5 seconds as follows I was unable to
reproduce the exception although I tried several times.
socket.connect(new InetSocketAddress(hostname, port), 1000 * 60 * 5);
The other doubt I have is; although I'm getting a
java.net.ConnectException, according to [1], this method should throw
SocketTimeoutException in case of a timeout. Further, JavaDoc of
ConnectException (the one I'm actually getting) says it signals that *"an
error occurred while attempting to connect a socket to a remote address and
port. Typically, the connection was refused remotely (e.g., no process is
listening on the remote address/port)." *
Therefore, I believe that the actual cause of this error is server's
ListeningIOReactor not being able to react to incoming connection requests.
I had allocated 16 threads (= no of CPU cores) for ListeningIOReactor.
Maybe increasing it further would solve this issue. I will test this
further and let you know.
[1]
http://docs.oracle.com/javase/6/docs/api/java/net/Socket.html#connect%28java.net.SocketAddress,%20int%29
[2] http://docs.oracle.com/javase/6/docs/api/java/net/ConnectException.html
Thanks,
Sadeep
On Tue, Jan 31, 2012 at 9:42 PM, Sadeep Jayasumana <ga...@gmail.com>wrote:
> Hi Oleg,
>
> On Tue, Jan 31, 2012 at 7:48 PM, Oleg Kalnichevski <ol...@apache.org>wrote:
>
>> On Mon, 2012-01-30 at 14:56 +0100, Oleg Kalnichevski wrote:
>> > On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote:
>> > > Hi,
>> > >
>> > > On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <ol...@apache.org>
>> wrote:
>> > >
>> > > > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote:
>> > > > > Hi,
>> > > > >
>> > > > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <
>> olegk@apache.org>
>> > > > wrote:
>> > > > >
>> > > > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
>> > > > > > > Hi,
>> > > > > > >
>> > > > > > > I have written an HTTP server using HTTPCore-NIO and currently
>> > > > testing it
>> > > > > > > with httpcore-ab [1]. I have been seeing very good TPS values
>> so
>> > > > far. But
>> > > > > > > when I try to send 5KB HTTP messages with 2500 and concurrency
>> > > > level, 50
>> > > > > > > messages per thread, I'm getting the following exception
>> several
>> > > > times
>> > > > > > from
>> > > > > > > httpcore-ab and it results in "Failed requests" in the
>> httpcore-ab
>> > > > > > output.
>> > > > > > > No exceptions are displayed from the server side. Hence, I
>> think the
>> > > > > > server
>> > > > > > > is rejecting some socket connections because it's already
>> fully
>> > > > loaded.
>> > > > > > >
>> > > > > > > I'm running the server and load generator on two server
>> machines
>> > > > that are
>> > > > > > > connected with Gigabit Ethernet. Each server has 16 CPU
>> cores, 16 GB
>> > > > > > memory
>> > > > > > > etc.
>> > > > > > >
>> > > > > > > I have increased the socket timeout in both httpcore-ab and
>> server
>> > > > side
>> > > > > > > ListeningIOReactor to 360000. 16 threads are allocated for the
>> > > > reactor.
>> > > > > > > Messages received by the server are processed in a separate
>> worker
>> > > > pool
>> > > > > > > which has ~500 threads. I have also increased the open file
>> limit of
>> > > > my
>> > > > > > OS
>> > > > > > > (Debian) to 65535.
>> > > > > > >
>> > > > > > > Is there any tuning parameter of HTTPCore that I can used to
>> get rid
>> > > > of
>> > > > > > > this exception? Perhaps, something similar to
>> "acceptQueueSize" in
>> > > > Jetty
>> > > > > > > [2]?
>> > > > > > >
>> > > > > >
>> > > > > > Hi Sadeep
>> > > > > >
>> > > > > > A couple of points in no particular logical order.
>> > > > > >
>> > > > > > (1) HttpCore does not depend on a logging toolkit and does not
>> do any
>> > > > > > logging per default. If it ever encounters an abnormal
>> situation,
>> > > > > > instead of logging such an event it throws an exception and
>> terminates.
>> > > > > > Quite often this is not desirable, so one can provide an
>> exception
>> > > > > > handler that can log using a a logging toolkit of choice or
>> handle a
>> > > > > > particular type of exception in an application specific way.
>> > > > > >
>> > > > > > If the server did not terminate under with an exception, I kind
>> of
>> > > > > > suspect it was still trying to chew through the load without
>> > > > > > encountering anything abnormal. Unless you have added a custom
>> > > > exception
>> > > > > > handler.
>> > > > > >
>> > > > > > (2) HttpCore does not have a concept of accepted connection
>> queue. As
>> > > > > > soon as the listener accepts an incoming connection its channel
>> gets
>> > > > > > immediately added to the underlying i/o selector. No
>> intermediate
>> > > > > > queuing takes place.
>> > > > >
>> > > > >
>> > > > > > (3) Given that the connection times out on the client, this
>> sounds more
>> > > > > > like a client side issue. Was connection really suck in
>> > > > > > Socket#socketConnect for 360000?
>> > > > > >
>> > > > >
>> > > > > I'm using httpcore-ab [1] as the client. I have set socket
>> timeout of
>> > > > using
>> > > > > -t 360000, and changed the connection timeout with the patch
>> shown below.
>> > > > > But I'm starting to get this error well before 360000 millis past
>> the
>> > > > test
>> > > > > startup. Therefore, I there is no possibility that the connection
>> attempt
>> > > > > hangs for 360000 millis before printing this exception. I was
>> thinking
>> > > > > whether there are upper bounds for these limits that cause my
>> settings to
>> > > > > be neglected.
>> > > > >
>> > > >
>> > > > You see. If java.net.ConnectException: Connection timed out is
>> thrown by
>> > > > java.net.PlainSocketImpl.socketConnect before 360000 millis elapse,
>> > > > there must be something wrong with the client side.
>> > > >
>> > >
>> > > Could this be a possible bug in httpcore-ab code? I'm quite sure that
>> it
>> > > doesn't wait 360000 millis before throwing this exception.
>> > >
>> >
>>
>> Hi Sadeep
>>
>> It did turn out to be an issue with the way HttpCore AB initialized
>> connections. It was basically using an old (Java 1.0) socket constructor
>> whose behavior can differ between OS platforms and possibly between JRE
>> major releases with regards to connect timeout handling. The connect
>> timeout value set in the HttpParams simply had no effect and therefore
>> connections in your tests were timing out before 360000 milliseconds
>> elapsed (probably using some sort of a system default).
>>
>> I committed a fix for the problem
>>
>> http://svn.apache.org/viewvc?rev=1238566&view=rev
>>
>> Please review
>>
>
> Many thanks for looking into this. I will review this at my earliest and
> let you know the outcome.
>
> Thanks,
> Sadeep
>
>
>>
>> Oleg
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>> For additional commands, e-mail: dev-help@hc.apache.org
>>
>>
>
>
> --
>
> Sadeep Jayasumana****
>
> Email: gayansadeep@gmail.com****
>
> Phone: +94-77-2266507
>
>
>
>
--
Sadeep Jayasumana****
Email: gayansadeep@gmail.com****
Phone: +94-77-2266507
Re: HTTPCore-NIO: Server rejecting socket connections?
Posted by Sadeep Jayasumana <ga...@gmail.com>.
Hi Oleg,
On Tue, Jan 31, 2012 at 7:48 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Mon, 2012-01-30 at 14:56 +0100, Oleg Kalnichevski wrote:
> > On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote:
> > > Hi,
> > >
> > > On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <ol...@apache.org>
> wrote:
> > >
> > > > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote:
> > > > > Hi,
> > > > >
> > > > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <
> olegk@apache.org>
> > > > wrote:
> > > > >
> > > > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have written an HTTP server using HTTPCore-NIO and currently
> > > > testing it
> > > > > > > with httpcore-ab [1]. I have been seeing very good TPS values
> so
> > > > far. But
> > > > > > > when I try to send 5KB HTTP messages with 2500 and concurrency
> > > > level, 50
> > > > > > > messages per thread, I'm getting the following exception
> several
> > > > times
> > > > > > from
> > > > > > > httpcore-ab and it results in "Failed requests" in the
> httpcore-ab
> > > > > > output.
> > > > > > > No exceptions are displayed from the server side. Hence, I
> think the
> > > > > > server
> > > > > > > is rejecting some socket connections because it's already fully
> > > > loaded.
> > > > > > >
> > > > > > > I'm running the server and load generator on two server
> machines
> > > > that are
> > > > > > > connected with Gigabit Ethernet. Each server has 16 CPU cores,
> 16 GB
> > > > > > memory
> > > > > > > etc.
> > > > > > >
> > > > > > > I have increased the socket timeout in both httpcore-ab and
> server
> > > > side
> > > > > > > ListeningIOReactor to 360000. 16 threads are allocated for the
> > > > reactor.
> > > > > > > Messages received by the server are processed in a separate
> worker
> > > > pool
> > > > > > > which has ~500 threads. I have also increased the open file
> limit of
> > > > my
> > > > > > OS
> > > > > > > (Debian) to 65535.
> > > > > > >
> > > > > > > Is there any tuning parameter of HTTPCore that I can used to
> get rid
> > > > of
> > > > > > > this exception? Perhaps, something similar to
> "acceptQueueSize" in
> > > > Jetty
> > > > > > > [2]?
> > > > > > >
> > > > > >
> > > > > > Hi Sadeep
> > > > > >
> > > > > > A couple of points in no particular logical order.
> > > > > >
> > > > > > (1) HttpCore does not depend on a logging toolkit and does not
> do any
> > > > > > logging per default. If it ever encounters an abnormal situation,
> > > > > > instead of logging such an event it throws an exception and
> terminates.
> > > > > > Quite often this is not desirable, so one can provide an
> exception
> > > > > > handler that can log using a a logging toolkit of choice or
> handle a
> > > > > > particular type of exception in an application specific way.
> > > > > >
> > > > > > If the server did not terminate under with an exception, I kind
> of
> > > > > > suspect it was still trying to chew through the load without
> > > > > > encountering anything abnormal. Unless you have added a custom
> > > > exception
> > > > > > handler.
> > > > > >
> > > > > > (2) HttpCore does not have a concept of accepted connection
> queue. As
> > > > > > soon as the listener accepts an incoming connection its channel
> gets
> > > > > > immediately added to the underlying i/o selector. No intermediate
> > > > > > queuing takes place.
> > > > >
> > > > >
> > > > > > (3) Given that the connection times out on the client, this
> sounds more
> > > > > > like a client side issue. Was connection really suck in
> > > > > > Socket#socketConnect for 360000?
> > > > > >
> > > > >
> > > > > I'm using httpcore-ab [1] as the client. I have set socket timeout
> of
> > > > using
> > > > > -t 360000, and changed the connection timeout with the patch shown
> below.
> > > > > But I'm starting to get this error well before 360000 millis past
> the
> > > > test
> > > > > startup. Therefore, I there is no possibility that the connection
> attempt
> > > > > hangs for 360000 millis before printing this exception. I was
> thinking
> > > > > whether there are upper bounds for these limits that cause my
> settings to
> > > > > be neglected.
> > > > >
> > > >
> > > > You see. If java.net.ConnectException: Connection timed out is
> thrown by
> > > > java.net.PlainSocketImpl.socketConnect before 360000 millis elapse,
> > > > there must be something wrong with the client side.
> > > >
> > >
> > > Could this be a possible bug in httpcore-ab code? I'm quite sure that
> it
> > > doesn't wait 360000 millis before throwing this exception.
> > >
> >
>
> Hi Sadeep
>
> It did turn out to be an issue with the way HttpCore AB initialized
> connections. It was basically using an old (Java 1.0) socket constructor
> whose behavior can differ between OS platforms and possibly between JRE
> major releases with regards to connect timeout handling. The connect
> timeout value set in the HttpParams simply had no effect and therefore
> connections in your tests were timing out before 360000 milliseconds
> elapsed (probably using some sort of a system default).
>
> I committed a fix for the problem
>
> http://svn.apache.org/viewvc?rev=1238566&view=rev
>
> Please review
>
Many thanks for looking into this. I will review this at my earliest and
let you know the outcome.
Thanks,
Sadeep
>
> Oleg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
--
Sadeep Jayasumana****
Email: gayansadeep@gmail.com****
Phone: +94-77-2266507
Re: HTTPCore-NIO: Server rejecting socket connections?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2012-01-30 at 14:56 +0100, Oleg Kalnichevski wrote:
> On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote:
> > Hi,
> >
> > On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> >
> > > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote:
> > > > Hi,
> > > >
> > > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <ol...@apache.org>
> > > wrote:
> > > >
> > > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I have written an HTTP server using HTTPCore-NIO and currently
> > > testing it
> > > > > > with httpcore-ab [1]. I have been seeing very good TPS values so
> > > far. But
> > > > > > when I try to send 5KB HTTP messages with 2500 and concurrency
> > > level, 50
> > > > > > messages per thread, I'm getting the following exception several
> > > times
> > > > > from
> > > > > > httpcore-ab and it results in "Failed requests" in the httpcore-ab
> > > > > output.
> > > > > > No exceptions are displayed from the server side. Hence, I think the
> > > > > server
> > > > > > is rejecting some socket connections because it's already fully
> > > loaded.
> > > > > >
> > > > > > I'm running the server and load generator on two server machines
> > > that are
> > > > > > connected with Gigabit Ethernet. Each server has 16 CPU cores, 16 GB
> > > > > memory
> > > > > > etc.
> > > > > >
> > > > > > I have increased the socket timeout in both httpcore-ab and server
> > > side
> > > > > > ListeningIOReactor to 360000. 16 threads are allocated for the
> > > reactor.
> > > > > > Messages received by the server are processed in a separate worker
> > > pool
> > > > > > which has ~500 threads. I have also increased the open file limit of
> > > my
> > > > > OS
> > > > > > (Debian) to 65535.
> > > > > >
> > > > > > Is there any tuning parameter of HTTPCore that I can used to get rid
> > > of
> > > > > > this exception? Perhaps, something similar to "acceptQueueSize" in
> > > Jetty
> > > > > > [2]?
> > > > > >
> > > > >
> > > > > Hi Sadeep
> > > > >
> > > > > A couple of points in no particular logical order.
> > > > >
> > > > > (1) HttpCore does not depend on a logging toolkit and does not do any
> > > > > logging per default. If it ever encounters an abnormal situation,
> > > > > instead of logging such an event it throws an exception and terminates.
> > > > > Quite often this is not desirable, so one can provide an exception
> > > > > handler that can log using a a logging toolkit of choice or handle a
> > > > > particular type of exception in an application specific way.
> > > > >
> > > > > If the server did not terminate under with an exception, I kind of
> > > > > suspect it was still trying to chew through the load without
> > > > > encountering anything abnormal. Unless you have added a custom
> > > exception
> > > > > handler.
> > > > >
> > > > > (2) HttpCore does not have a concept of accepted connection queue. As
> > > > > soon as the listener accepts an incoming connection its channel gets
> > > > > immediately added to the underlying i/o selector. No intermediate
> > > > > queuing takes place.
> > > >
> > > >
> > > > > (3) Given that the connection times out on the client, this sounds more
> > > > > like a client side issue. Was connection really suck in
> > > > > Socket#socketConnect for 360000?
> > > > >
> > > >
> > > > I'm using httpcore-ab [1] as the client. I have set socket timeout of
> > > using
> > > > -t 360000, and changed the connection timeout with the patch shown below.
> > > > But I'm starting to get this error well before 360000 millis past the
> > > test
> > > > startup. Therefore, I there is no possibility that the connection attempt
> > > > hangs for 360000 millis before printing this exception. I was thinking
> > > > whether there are upper bounds for these limits that cause my settings to
> > > > be neglected.
> > > >
> > >
> > > You see. If java.net.ConnectException: Connection timed out is thrown by
> > > java.net.PlainSocketImpl.socketConnect before 360000 millis elapse,
> > > there must be something wrong with the client side.
> > >
> >
> > Could this be a possible bug in httpcore-ab code? I'm quite sure that it
> > doesn't wait 360000 millis before throwing this exception.
> >
>
Hi Sadeep
It did turn out to be an issue with the way HttpCore AB initialized
connections. It was basically using an old (Java 1.0) socket constructor
whose behavior can differ between OS platforms and possibly between JRE
major releases with regards to connect timeout handling. The connect
timeout value set in the HttpParams simply had no effect and therefore
connections in your tests were timing out before 360000 milliseconds
elapsed (probably using some sort of a system default).
I committed a fix for the problem
http://svn.apache.org/viewvc?rev=1238566&view=rev
Please review
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: HTTPCore-NIO: Server rejecting socket connections?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote:
> Hi,
>
> On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote:
> > > Hi,
> > >
> > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <ol...@apache.org>
> > wrote:
> > >
> > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
> > > > > Hi,
> > > > >
> > > > > I have written an HTTP server using HTTPCore-NIO and currently
> > testing it
> > > > > with httpcore-ab [1]. I have been seeing very good TPS values so
> > far. But
> > > > > when I try to send 5KB HTTP messages with 2500 and concurrency
> > level, 50
> > > > > messages per thread, I'm getting the following exception several
> > times
> > > > from
> > > > > httpcore-ab and it results in "Failed requests" in the httpcore-ab
> > > > output.
> > > > > No exceptions are displayed from the server side. Hence, I think the
> > > > server
> > > > > is rejecting some socket connections because it's already fully
> > loaded.
> > > > >
> > > > > I'm running the server and load generator on two server machines
> > that are
> > > > > connected with Gigabit Ethernet. Each server has 16 CPU cores, 16 GB
> > > > memory
> > > > > etc.
> > > > >
> > > > > I have increased the socket timeout in both httpcore-ab and server
> > side
> > > > > ListeningIOReactor to 360000. 16 threads are allocated for the
> > reactor.
> > > > > Messages received by the server are processed in a separate worker
> > pool
> > > > > which has ~500 threads. I have also increased the open file limit of
> > my
> > > > OS
> > > > > (Debian) to 65535.
> > > > >
> > > > > Is there any tuning parameter of HTTPCore that I can used to get rid
> > of
> > > > > this exception? Perhaps, something similar to "acceptQueueSize" in
> > Jetty
> > > > > [2]?
> > > > >
> > > >
> > > > Hi Sadeep
> > > >
> > > > A couple of points in no particular logical order.
> > > >
> > > > (1) HttpCore does not depend on a logging toolkit and does not do any
> > > > logging per default. If it ever encounters an abnormal situation,
> > > > instead of logging such an event it throws an exception and terminates.
> > > > Quite often this is not desirable, so one can provide an exception
> > > > handler that can log using a a logging toolkit of choice or handle a
> > > > particular type of exception in an application specific way.
> > > >
> > > > If the server did not terminate under with an exception, I kind of
> > > > suspect it was still trying to chew through the load without
> > > > encountering anything abnormal. Unless you have added a custom
> > exception
> > > > handler.
> > > >
> > > > (2) HttpCore does not have a concept of accepted connection queue. As
> > > > soon as the listener accepts an incoming connection its channel gets
> > > > immediately added to the underlying i/o selector. No intermediate
> > > > queuing takes place.
> > >
> > >
> > > > (3) Given that the connection times out on the client, this sounds more
> > > > like a client side issue. Was connection really suck in
> > > > Socket#socketConnect for 360000?
> > > >
> > >
> > > I'm using httpcore-ab [1] as the client. I have set socket timeout of
> > using
> > > -t 360000, and changed the connection timeout with the patch shown below.
> > > But I'm starting to get this error well before 360000 millis past the
> > test
> > > startup. Therefore, I there is no possibility that the connection attempt
> > > hangs for 360000 millis before printing this exception. I was thinking
> > > whether there are upper bounds for these limits that cause my settings to
> > > be neglected.
> > >
> >
> > You see. If java.net.ConnectException: Connection timed out is thrown by
> > java.net.PlainSocketImpl.socketConnect before 360000 millis elapse,
> > there must be something wrong with the client side.
> >
>
> Could this be a possible bug in httpcore-ab code? I'm quite sure that it
> doesn't wait 360000 millis before throwing this exception.
>
Of course, it is not impossible. However, if you take look at the
exception stack trace, the exception originates from native part of JRE
code, which is responsible for low level TCP/IP functionality.
Have you tried using an infinite timeout (no timeout)?
Another thing can think of is trying a different client side JRE.
Oleg
> Thanks,
> Sadeep
>
>
> >
> > Here's the contract of the method:
> > --------------------------------
> > Connects this socket to the server with a specified timeout value. A
> > timeout of zero is interpreted as an infinite timeout. The connection
> > will then block until established or an error occurs.
> >
> > Parameters:
> > endpoint - the SocketAddress
> > timeout - the timeout value to be used in milliseconds.
> > Throws:
> > IOException - if an error occurs during the connection
> > SocketTimeoutException - if timeout expires before connecting
> > IllegalBlockingModeException - if this socket has an associated
> > channel, and the channel is in non-blocking mode
> > IllegalArgumentException - if endpoint is null or is a
> > SocketAddress subclass not supported by this socket
> > --------------------------------
> >
> > ConnectException is a subclass of SocketTimeoutException. My
> > understanding of the contract is that the exception cannot be thrown
> > unless the timeout in milliseconds fully elapses.
> >
> > Oleg
> >
> > > --- src/main/java/org/apache/http/benchmark/HttpBenchmark.java (revision
> > > 1205292)
> > > +++ src/main/java/org/apache/http/benchmark/HttpBenchmark.java (working
> > > copy)
> > > @@ -227,7 +227,8 @@
> > > .setParameter(HttpProtocolParams.USER_AGENT,
> > "HttpCore-AB/1.1")
> > > .setBooleanParameter(HttpProtocolParams.USE_EXPECT_CONTINUE,
> > > useExpectContinue)
> > >
> > > .setBooleanParameter(HttpConnectionParams.STALE_CONNECTION_CHECK, false)
> > > - .setIntParameter(HttpConnectionParams.SO_TIMEOUT,
> > > socketTimeout);
> > > + .setIntParameter(HttpConnectionParams.SO_TIMEOUT,
> > > socketTimeout)
> > > + .setIntParameter(HttpConnectionParams.CONNECTION_TIMEOUT,
> > 360
> > > * 1000);
> > > return params;
> > > }
> > > }
> > >
> > >
> > > >
> > > > (4) Quite often it is useful to see what it really going on on the I/O
> > > > reactor level. You might want to add I/O event and wire logging by
> > > > copying logging components from the contrib package [1]
> > > >
> > >
> > > I could do this. But will wire logging really help at this concurrency
> > > level and messages/thread?
> > >
> > > [1]
> > >
> > https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
> > >
> > > Thanks,
> > > Sadeep
> > >
> > > >
> > > > I hope this helps somewhat. In any way I can say I am quite sure
> > > > HttpCore does not reject incoming connections no matter what.
> > >
> > >
> > > > Oleg
> > > >
> > > > [1]
> > > >
> > > >
> > https://svn.apache.org/repos/asf/httpcomponents/httpcore/branches/4.1.x/httpcore-contrib/src/main/java/org/apache/http/contrib/logging/
> > > >
> > > > > [1]
> > > > >
> > > >
> > https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
> > > > > [2] http://docs.codehaus.org/display/JETTY/Configuring+Connectors
> > > > > [3]
> > > > > java.net.ConnectException: Connection timed out
> > > > > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> > > > > at
> > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> > > > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> > > > > at java.net.Socket.connect(Socket.java:529)
> > > > > at java.net.Socket.connect(Socket.java:478)
> > > > > at java.net.Socket.<init>(Socket.java:375)
> > > > > at java.net.Socket.<init>(Socket.java:189)
> > > > > at
> > > > org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> > > > > at
> > > > >
> > > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > at
> > > > >
> > > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > java.net.ConnectException: Connection timed out
> > > > > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> > > > > at
> > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> > > > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> > > > > at java.net.Socket.connect(Socket.java:529)
> > > > > at java.net.Socket.connect(Socket.java:478)
> > > > > at java.net.Socket.<init>(Socket.java:375)
> > > > > at java.net.Socket.<init>(Socket.java:189)
> > > > > at
> > > > org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> > > > > at
> > > > >
> > > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > at
> > > > >
> > > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > at java.lang.Thread.run(Thread.java:662)
> > > > >
> > > > > Thanks & Regards,
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > >
> > > >
> > >
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > For additional commands, e-mail: dev-help@hc.apache.org
> >
> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: HTTPCore-NIO: Server rejecting socket connections?
Posted by Sadeep Jayasumana <ga...@gmail.com>.
Hi,
On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote:
> > Hi,
> >
> > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <ol...@apache.org>
> wrote:
> >
> > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
> > > > Hi,
> > > >
> > > > I have written an HTTP server using HTTPCore-NIO and currently
> testing it
> > > > with httpcore-ab [1]. I have been seeing very good TPS values so
> far. But
> > > > when I try to send 5KB HTTP messages with 2500 and concurrency
> level, 50
> > > > messages per thread, I'm getting the following exception several
> times
> > > from
> > > > httpcore-ab and it results in "Failed requests" in the httpcore-ab
> > > output.
> > > > No exceptions are displayed from the server side. Hence, I think the
> > > server
> > > > is rejecting some socket connections because it's already fully
> loaded.
> > > >
> > > > I'm running the server and load generator on two server machines
> that are
> > > > connected with Gigabit Ethernet. Each server has 16 CPU cores, 16 GB
> > > memory
> > > > etc.
> > > >
> > > > I have increased the socket timeout in both httpcore-ab and server
> side
> > > > ListeningIOReactor to 360000. 16 threads are allocated for the
> reactor.
> > > > Messages received by the server are processed in a separate worker
> pool
> > > > which has ~500 threads. I have also increased the open file limit of
> my
> > > OS
> > > > (Debian) to 65535.
> > > >
> > > > Is there any tuning parameter of HTTPCore that I can used to get rid
> of
> > > > this exception? Perhaps, something similar to "acceptQueueSize" in
> Jetty
> > > > [2]?
> > > >
> > >
> > > Hi Sadeep
> > >
> > > A couple of points in no particular logical order.
> > >
> > > (1) HttpCore does not depend on a logging toolkit and does not do any
> > > logging per default. If it ever encounters an abnormal situation,
> > > instead of logging such an event it throws an exception and terminates.
> > > Quite often this is not desirable, so one can provide an exception
> > > handler that can log using a a logging toolkit of choice or handle a
> > > particular type of exception in an application specific way.
> > >
> > > If the server did not terminate under with an exception, I kind of
> > > suspect it was still trying to chew through the load without
> > > encountering anything abnormal. Unless you have added a custom
> exception
> > > handler.
> > >
> > > (2) HttpCore does not have a concept of accepted connection queue. As
> > > soon as the listener accepts an incoming connection its channel gets
> > > immediately added to the underlying i/o selector. No intermediate
> > > queuing takes place.
> >
> >
> > > (3) Given that the connection times out on the client, this sounds more
> > > like a client side issue. Was connection really suck in
> > > Socket#socketConnect for 360000?
> > >
> >
> > I'm using httpcore-ab [1] as the client. I have set socket timeout of
> using
> > -t 360000, and changed the connection timeout with the patch shown below.
> > But I'm starting to get this error well before 360000 millis past the
> test
> > startup. Therefore, I there is no possibility that the connection attempt
> > hangs for 360000 millis before printing this exception. I was thinking
> > whether there are upper bounds for these limits that cause my settings to
> > be neglected.
> >
>
> You see. If java.net.ConnectException: Connection timed out is thrown by
> java.net.PlainSocketImpl.socketConnect before 360000 millis elapse,
> there must be something wrong with the client side.
>
Could this be a possible bug in httpcore-ab code? I'm quite sure that it
doesn't wait 360000 millis before throwing this exception.
Thanks,
Sadeep
>
> Here's the contract of the method:
> --------------------------------
> Connects this socket to the server with a specified timeout value. A
> timeout of zero is interpreted as an infinite timeout. The connection
> will then block until established or an error occurs.
>
> Parameters:
> endpoint - the SocketAddress
> timeout - the timeout value to be used in milliseconds.
> Throws:
> IOException - if an error occurs during the connection
> SocketTimeoutException - if timeout expires before connecting
> IllegalBlockingModeException - if this socket has an associated
> channel, and the channel is in non-blocking mode
> IllegalArgumentException - if endpoint is null or is a
> SocketAddress subclass not supported by this socket
> --------------------------------
>
> ConnectException is a subclass of SocketTimeoutException. My
> understanding of the contract is that the exception cannot be thrown
> unless the timeout in milliseconds fully elapses.
>
> Oleg
>
> > --- src/main/java/org/apache/http/benchmark/HttpBenchmark.java (revision
> > 1205292)
> > +++ src/main/java/org/apache/http/benchmark/HttpBenchmark.java (working
> > copy)
> > @@ -227,7 +227,8 @@
> > .setParameter(HttpProtocolParams.USER_AGENT,
> "HttpCore-AB/1.1")
> > .setBooleanParameter(HttpProtocolParams.USE_EXPECT_CONTINUE,
> > useExpectContinue)
> >
> > .setBooleanParameter(HttpConnectionParams.STALE_CONNECTION_CHECK, false)
> > - .setIntParameter(HttpConnectionParams.SO_TIMEOUT,
> > socketTimeout);
> > + .setIntParameter(HttpConnectionParams.SO_TIMEOUT,
> > socketTimeout)
> > + .setIntParameter(HttpConnectionParams.CONNECTION_TIMEOUT,
> 360
> > * 1000);
> > return params;
> > }
> > }
> >
> >
> > >
> > > (4) Quite often it is useful to see what it really going on on the I/O
> > > reactor level. You might want to add I/O event and wire logging by
> > > copying logging components from the contrib package [1]
> > >
> >
> > I could do this. But will wire logging really help at this concurrency
> > level and messages/thread?
> >
> > [1]
> >
> https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
> >
> > Thanks,
> > Sadeep
> >
> > >
> > > I hope this helps somewhat. In any way I can say I am quite sure
> > > HttpCore does not reject incoming connections no matter what.
> >
> >
> > > Oleg
> > >
> > > [1]
> > >
> > >
> https://svn.apache.org/repos/asf/httpcomponents/httpcore/branches/4.1.x/httpcore-contrib/src/main/java/org/apache/http/contrib/logging/
> > >
> > > > [1]
> > > >
> > >
> https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
> > > > [2] http://docs.codehaus.org/display/JETTY/Configuring+Connectors
> > > > [3]
> > > > java.net.ConnectException: Connection timed out
> > > > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> > > > at
> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> > > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> > > > at java.net.Socket.connect(Socket.java:529)
> > > > at java.net.Socket.connect(Socket.java:478)
> > > > at java.net.Socket.<init>(Socket.java:375)
> > > > at java.net.Socket.<init>(Socket.java:189)
> > > > at
> > > org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> > > > at
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > at
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > at java.lang.Thread.run(Thread.java:662)
> > > > java.net.ConnectException: Connection timed out
> > > > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> > > > at
> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> > > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> > > > at java.net.Socket.connect(Socket.java:529)
> > > > at java.net.Socket.connect(Socket.java:478)
> > > > at java.net.Socket.<init>(Socket.java:375)
> > > > at java.net.Socket.<init>(Socket.java:189)
> > > > at
> > > org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> > > > at
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > at
> > > >
> > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > at java.lang.Thread.run(Thread.java:662)
> > > >
> > > > Thanks & Regards,
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: dev-help@hc.apache.org
> > >
> > >
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
--
Sadeep Jayasumana****
Email: gayansadeep@gmail.com****
Phone: +94-77-2266507
Re: HTTPCore-NIO: Server rejecting socket connections?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote:
> Hi,
>
> On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
> > > Hi,
> > >
> > > I have written an HTTP server using HTTPCore-NIO and currently testing it
> > > with httpcore-ab [1]. I have been seeing very good TPS values so far. But
> > > when I try to send 5KB HTTP messages with 2500 and concurrency level, 50
> > > messages per thread, I'm getting the following exception several times
> > from
> > > httpcore-ab and it results in "Failed requests" in the httpcore-ab
> > output.
> > > No exceptions are displayed from the server side. Hence, I think the
> > server
> > > is rejecting some socket connections because it's already fully loaded.
> > >
> > > I'm running the server and load generator on two server machines that are
> > > connected with Gigabit Ethernet. Each server has 16 CPU cores, 16 GB
> > memory
> > > etc.
> > >
> > > I have increased the socket timeout in both httpcore-ab and server side
> > > ListeningIOReactor to 360000. 16 threads are allocated for the reactor.
> > > Messages received by the server are processed in a separate worker pool
> > > which has ~500 threads. I have also increased the open file limit of my
> > OS
> > > (Debian) to 65535.
> > >
> > > Is there any tuning parameter of HTTPCore that I can used to get rid of
> > > this exception? Perhaps, something similar to "acceptQueueSize" in Jetty
> > > [2]?
> > >
> >
> > Hi Sadeep
> >
> > A couple of points in no particular logical order.
> >
> > (1) HttpCore does not depend on a logging toolkit and does not do any
> > logging per default. If it ever encounters an abnormal situation,
> > instead of logging such an event it throws an exception and terminates.
> > Quite often this is not desirable, so one can provide an exception
> > handler that can log using a a logging toolkit of choice or handle a
> > particular type of exception in an application specific way.
> >
> > If the server did not terminate under with an exception, I kind of
> > suspect it was still trying to chew through the load without
> > encountering anything abnormal. Unless you have added a custom exception
> > handler.
> >
> > (2) HttpCore does not have a concept of accepted connection queue. As
> > soon as the listener accepts an incoming connection its channel gets
> > immediately added to the underlying i/o selector. No intermediate
> > queuing takes place.
>
>
> > (3) Given that the connection times out on the client, this sounds more
> > like a client side issue. Was connection really suck in
> > Socket#socketConnect for 360000?
> >
>
> I'm using httpcore-ab [1] as the client. I have set socket timeout of using
> -t 360000, and changed the connection timeout with the patch shown below.
> But I'm starting to get this error well before 360000 millis past the test
> startup. Therefore, I there is no possibility that the connection attempt
> hangs for 360000 millis before printing this exception. I was thinking
> whether there are upper bounds for these limits that cause my settings to
> be neglected.
>
You see. If java.net.ConnectException: Connection timed out is thrown by
java.net.PlainSocketImpl.socketConnect before 360000 millis elapse,
there must be something wrong with the client side.
Here's the contract of the method:
--------------------------------
Connects this socket to the server with a specified timeout value. A
timeout of zero is interpreted as an infinite timeout. The connection
will then block until established or an error occurs.
Parameters:
endpoint - the SocketAddress
timeout - the timeout value to be used in milliseconds.
Throws:
IOException - if an error occurs during the connection
SocketTimeoutException - if timeout expires before connecting
IllegalBlockingModeException - if this socket has an associated
channel, and the channel is in non-blocking mode
IllegalArgumentException - if endpoint is null or is a
SocketAddress subclass not supported by this socket
--------------------------------
ConnectException is a subclass of SocketTimeoutException. My
understanding of the contract is that the exception cannot be thrown
unless the timeout in milliseconds fully elapses.
Oleg
> --- src/main/java/org/apache/http/benchmark/HttpBenchmark.java (revision
> 1205292)
> +++ src/main/java/org/apache/http/benchmark/HttpBenchmark.java (working
> copy)
> @@ -227,7 +227,8 @@
> .setParameter(HttpProtocolParams.USER_AGENT, "HttpCore-AB/1.1")
> .setBooleanParameter(HttpProtocolParams.USE_EXPECT_CONTINUE,
> useExpectContinue)
>
> .setBooleanParameter(HttpConnectionParams.STALE_CONNECTION_CHECK, false)
> - .setIntParameter(HttpConnectionParams.SO_TIMEOUT,
> socketTimeout);
> + .setIntParameter(HttpConnectionParams.SO_TIMEOUT,
> socketTimeout)
> + .setIntParameter(HttpConnectionParams.CONNECTION_TIMEOUT, 360
> * 1000);
> return params;
> }
> }
>
>
> >
> > (4) Quite often it is useful to see what it really going on on the I/O
> > reactor level. You might want to add I/O event and wire logging by
> > copying logging components from the contrib package [1]
> >
>
> I could do this. But will wire logging really help at this concurrency
> level and messages/thread?
>
> [1]
> https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
>
> Thanks,
> Sadeep
>
> >
> > I hope this helps somewhat. In any way I can say I am quite sure
> > HttpCore does not reject incoming connections no matter what.
>
>
> > Oleg
> >
> > [1]
> >
> > https://svn.apache.org/repos/asf/httpcomponents/httpcore/branches/4.1.x/httpcore-contrib/src/main/java/org/apache/http/contrib/logging/
> >
> > > [1]
> > >
> > https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
> > > [2] http://docs.codehaus.org/display/JETTY/Configuring+Connectors
> > > [3]
> > > java.net.ConnectException: Connection timed out
> > > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> > > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> > > at java.net.Socket.connect(Socket.java:529)
> > > at java.net.Socket.connect(Socket.java:478)
> > > at java.net.Socket.<init>(Socket.java:375)
> > > at java.net.Socket.<init>(Socket.java:189)
> > > at
> > org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > java.net.ConnectException: Connection timed out
> > > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> > > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> > > at java.net.Socket.connect(Socket.java:529)
> > > at java.net.Socket.connect(Socket.java:478)
> > > at java.net.Socket.<init>(Socket.java:375)
> > > at java.net.Socket.<init>(Socket.java:189)
> > > at
> > org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > >
> > > Thanks & Regards,
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > For additional commands, e-mail: dev-help@hc.apache.org
> >
> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: HTTPCore-NIO: Server rejecting socket connections?
Posted by Sadeep Jayasumana <ga...@gmail.com>.
Hi,
On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
> > Hi,
> >
> > I have written an HTTP server using HTTPCore-NIO and currently testing it
> > with httpcore-ab [1]. I have been seeing very good TPS values so far. But
> > when I try to send 5KB HTTP messages with 2500 and concurrency level, 50
> > messages per thread, I'm getting the following exception several times
> from
> > httpcore-ab and it results in "Failed requests" in the httpcore-ab
> output.
> > No exceptions are displayed from the server side. Hence, I think the
> server
> > is rejecting some socket connections because it's already fully loaded.
> >
> > I'm running the server and load generator on two server machines that are
> > connected with Gigabit Ethernet. Each server has 16 CPU cores, 16 GB
> memory
> > etc.
> >
> > I have increased the socket timeout in both httpcore-ab and server side
> > ListeningIOReactor to 360000. 16 threads are allocated for the reactor.
> > Messages received by the server are processed in a separate worker pool
> > which has ~500 threads. I have also increased the open file limit of my
> OS
> > (Debian) to 65535.
> >
> > Is there any tuning parameter of HTTPCore that I can used to get rid of
> > this exception? Perhaps, something similar to "acceptQueueSize" in Jetty
> > [2]?
> >
>
> Hi Sadeep
>
> A couple of points in no particular logical order.
>
> (1) HttpCore does not depend on a logging toolkit and does not do any
> logging per default. If it ever encounters an abnormal situation,
> instead of logging such an event it throws an exception and terminates.
> Quite often this is not desirable, so one can provide an exception
> handler that can log using a a logging toolkit of choice or handle a
> particular type of exception in an application specific way.
>
> If the server did not terminate under with an exception, I kind of
> suspect it was still trying to chew through the load without
> encountering anything abnormal. Unless you have added a custom exception
> handler.
>
> (2) HttpCore does not have a concept of accepted connection queue. As
> soon as the listener accepts an incoming connection its channel gets
> immediately added to the underlying i/o selector. No intermediate
> queuing takes place.
> (3) Given that the connection times out on the client, this sounds more
> like a client side issue. Was connection really suck in
> Socket#socketConnect for 360000?
>
I'm using httpcore-ab [1] as the client. I have set socket timeout of using
-t 360000, and changed the connection timeout with the patch shown below.
But I'm starting to get this error well before 360000 millis past the test
startup. Therefore, I there is no possibility that the connection attempt
hangs for 360000 millis before printing this exception. I was thinking
whether there are upper bounds for these limits that cause my settings to
be neglected.
--- src/main/java/org/apache/http/benchmark/HttpBenchmark.java (revision
1205292)
+++ src/main/java/org/apache/http/benchmark/HttpBenchmark.java (working
copy)
@@ -227,7 +227,8 @@
.setParameter(HttpProtocolParams.USER_AGENT, "HttpCore-AB/1.1")
.setBooleanParameter(HttpProtocolParams.USE_EXPECT_CONTINUE,
useExpectContinue)
.setBooleanParameter(HttpConnectionParams.STALE_CONNECTION_CHECK, false)
- .setIntParameter(HttpConnectionParams.SO_TIMEOUT,
socketTimeout);
+ .setIntParameter(HttpConnectionParams.SO_TIMEOUT,
socketTimeout)
+ .setIntParameter(HttpConnectionParams.CONNECTION_TIMEOUT, 360
* 1000);
return params;
}
}
>
> (4) Quite often it is useful to see what it really going on on the I/O
> reactor level. You might want to add I/O event and wire logging by
> copying logging components from the contrib package [1]
>
I could do this. But will wire logging really help at this concurrency
level and messages/thread?
[1]
https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
Thanks,
Sadeep
>
> I hope this helps somewhat. In any way I can say I am quite sure
> HttpCore does not reject incoming connections no matter what.
> Oleg
>
> [1]
>
> https://svn.apache.org/repos/asf/httpcomponents/httpcore/branches/4.1.x/httpcore-contrib/src/main/java/org/apache/http/contrib/logging/
>
> > [1]
> >
> https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
> > [2] http://docs.codehaus.org/display/JETTY/Configuring+Connectors
> > [3]
> > java.net.ConnectException: Connection timed out
> > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> > at java.net.Socket.connect(Socket.java:529)
> > at java.net.Socket.connect(Socket.java:478)
> > at java.net.Socket.<init>(Socket.java:375)
> > at java.net.Socket.<init>(Socket.java:189)
> > at
> org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > java.net.ConnectException: Connection timed out
> > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> > at java.net.Socket.connect(Socket.java:529)
> > at java.net.Socket.connect(Socket.java:478)
> > at java.net.Socket.<init>(Socket.java:375)
> > at java.net.Socket.<init>(Socket.java:189)
> > at
> org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> >
> > Thanks & Regards,
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
--
Sadeep Jayasumana****
Email: gayansadeep@gmail.com****
Phone: +94-77-2266507
Re: HTTPCore-NIO: Server rejecting socket connections?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
> Hi,
>
> I have written an HTTP server using HTTPCore-NIO and currently testing it
> with httpcore-ab [1]. I have been seeing very good TPS values so far. But
> when I try to send 5KB HTTP messages with 2500 and concurrency level, 50
> messages per thread, I'm getting the following exception several times from
> httpcore-ab and it results in "Failed requests" in the httpcore-ab output.
> No exceptions are displayed from the server side. Hence, I think the server
> is rejecting some socket connections because it's already fully loaded.
>
> I'm running the server and load generator on two server machines that are
> connected with Gigabit Ethernet. Each server has 16 CPU cores, 16 GB memory
> etc.
>
> I have increased the socket timeout in both httpcore-ab and server side
> ListeningIOReactor to 360000. 16 threads are allocated for the reactor.
> Messages received by the server are processed in a separate worker pool
> which has ~500 threads. I have also increased the open file limit of my OS
> (Debian) to 65535.
>
> Is there any tuning parameter of HTTPCore that I can used to get rid of
> this exception? Perhaps, something similar to "acceptQueueSize" in Jetty
> [2]?
>
Hi Sadeep
A couple of points in no particular logical order.
(1) HttpCore does not depend on a logging toolkit and does not do any
logging per default. If it ever encounters an abnormal situation,
instead of logging such an event it throws an exception and terminates.
Quite often this is not desirable, so one can provide an exception
handler that can log using a a logging toolkit of choice or handle a
particular type of exception in an application specific way.
If the server did not terminate under with an exception, I kind of
suspect it was still trying to chew through the load without
encountering anything abnormal. Unless you have added a custom exception
handler.
(2) HttpCore does not have a concept of accepted connection queue. As
soon as the listener accepts an incoming connection its channel gets
immediately added to the underlying i/o selector. No intermediate
queuing takes place.
(3) Given that the connection times out on the client, this sounds more
like a client side issue. Was connection really suck in
Socket#socketConnect for 360000?
(4) Quite often it is useful to see what it really going on on the I/O
reactor level. You might want to add I/O event and wire logging by
copying logging components from the contrib package [1]
I hope this helps somewhat. In any way I can say I am quite sure
HttpCore does not reject incoming connections no matter what.
Oleg
[1]
https://svn.apache.org/repos/asf/httpcomponents/httpcore/branches/4.1.x/httpcore-contrib/src/main/java/org/apache/http/contrib/logging/
> [1]
> https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
> [2] http://docs.codehaus.org/display/JETTY/Configuring+Connectors
> [3]
> java.net.ConnectException: Connection timed out
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:529)
> at java.net.Socket.connect(Socket.java:478)
> at java.net.Socket.<init>(Socket.java:375)
> at java.net.Socket.<init>(Socket.java:189)
> at org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> java.net.ConnectException: Connection timed out
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:529)
> at java.net.Socket.connect(Socket.java:478)
> at java.net.Socket.<init>(Socket.java:375)
> at java.net.Socket.<init>(Socket.java:189)
> at org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Thanks & Regards,
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org