Load Balancing Socket connections

[Stephane Rainville <st...@hotmail.com>]

Question for the community.
 
In Ip4 we are limited to 2exp16 ports. 
Which is VERY high for HTTP GET because they are very short lived connections.
 
Has anybody designed an application using long lived SOCKETS? That limit could be reached fairly quickly and you would to load balance to multiple servers (Or maybe multiple interfaces on the server could be another solution).
 
My understanding is that most load balancers are HTTP based (Pound etc..) and was wondering if anybody here has more information on that subject or designed such an application.
 
Thanks
 
Stéphane
_________________________________________________________________

Re: Load Balancing Socket connections

[Emmanuel Lecharny <el...@gmail.com>]

Hi Wenrui,

Wenrui Guo wrote:
> As you mentioned, the limitation of amount of ports one process can open is really exist. Beside that, please also  note that not all process can allow up to 50,000 connections simultaneously, because in linux environment, one session(connection) established also occupied a file descriptor of the server process, so usually when about 2000 connections is established, a Too many open files exception must be thrown, unless you modify the ulimit value of such process.
>   
Modifying this parameter is absolutely mandatory, and I think that every 
person who is running a server in production knows that :) But it won't 
be a solution to your load balancing problem, anyway !
> Obviously, as Emmanuel said, for the application itself, we usually start several application instances on the same physical machine, but that's not the best solution since too many connections(even the event handling of http requests is based on asynchronous way) can cause excessive context switch, particularly when the OS is not good at task management and not enough CPU is present. 
The apache server (Httpd) itself have a limited number of threads to 
deal with incoming requests. But handling a request is a different thing 
than managing the connections, as we may have hundred of thousands 
opened connections, but only a few hundreds of request being processed 
simultaneously. If you don't have a high ration of processed request/ 
number of connections, that should be ok.
> In the case, please follow the rule said by Emmmanuel, make sure your application can process incoming request as quickly as possible or consider to reject flood http requests from huge clients. Please remember, excessive contention can reduce tps of server.
>   
The initial questions was about long lived sockets, not about handling a 
high number of incoming requests. If your server is supposed to receive 
thousands of requests per second, with a pretty expensive processing to 
handle each requests, then you better define a very good architecture, 
and MINA won't help you in this area ...
> However, Solutions mentioned above definitly a scalable solution, because application must be spread to multiple physicial machines to support more requests. So usually in telecom area, load balancing hardwares are so common because it can spread all http requests to many business servers which sits behind the LB. e.g F5 device. Once workload is above processing capability limitation, just need to add more machines next to LB to support more requests as long as LB can distribute flood flows. 
Assuming that you manage the sessions (all the LB box have a kind of 
sticky session capability), because hoping from one server to another 
during a session is really painful to manage from the application POV ! 
Generally speaking, unless you are delivering static content, 
scalability is not only a matter of dropping new boxes in the rack ... :)

But i'm afraid that this thread is going too far away from your initial 
point, and if we continue, it will become a complete review of all the 
possible architectures ;)

Thanks !

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

RE: Load Balancing Socket connections

[Wenrui Guo <we...@ericsson.com>]

Very sophicated question. :P

If you only focus on HTTP protocol, I guess maybe the following experience can help you.

As you mentioned, the limitation of amount of ports one process can open is really exist. Beside that, please also  note that not all process can allow up to 50,000 connections simultaneously, because in linux environment, one session(connection) established also occupied a file descriptor of the server process, so usually when about 2000 connections is established, a Too many open files exception must be thrown, unless you modify the ulimit value of such process.

Obviously, as Emmanuel said, for the application itself, we usually start several application instances on the same physical machine, but that's not the best solution since too many connections(even the event handling of http requests is based on asynchronous way) can cause excessive context switch, particularly when the OS is not good at task management and not enough CPU is present. In the case, please follow the rule said by Emmmanuel, make sure your application can process incoming request as quickly as possible or consider to reject flood http requests from huge clients. Please remember, excessive contention can reduce tps of server.

However, Solutions mentioned above definitly a scalable solution, because application must be spread to multiple physicial machines to support more requests. So usually in telecom area, load balancing hardwares are so common because it can spread all http requests to many business servers which sits behind the LB. e.g F5 device. Once workload is above processing capability limitation, just need to add more machines next to LB to support more requests as long as LB can distribute flood flows. Diagram shown below describe the network structure:

Client --->    ---> application server
	  LB 
Client --->   ----> application server

In extreme cases, maybe you don't believe LB can handle 200k~ requests, please remember to use DNS to spead http requests. Please also take a look at below picture:

Client --->                     -----> application server
                DNS ----> LB
Client --->		-----> application server
                DNS -----------------------------------------------------------> LB ---> application servers
							 
							 LB  ----> application servers
Client ---> 

Best regards

anderson

-----Original Message-----
From: Emmanuel Lecharny [mailto:elecharny@gmail.com] 
Sent: Sunday, September 14, 2008 3:41 PM
To: users@mina.apache.org
Subject: Re: Load Balancing Socket connections

Stephane Rainville wrote:
> I dont' think so.
>  
> Altough you are right a Socket can listen and open multiple connection on one port (Like a web server on port 80) when the  server responds it responds on a RANDOM port between 1000 and 65000.
>   
It responds using a random port associated to the client IP address. If a specific client tries to open more then 64536 connections to a server (very unlikely, if even possible !), then you will reach the limit. But as you will have many clients, everything is fine. Remember that a socket is defined by a couple of an IP address and a port.
>  
> So my question remains using MINA and long lived socket connections am i limited to 65000 long lived socket connections and how can I load balance that.
>   
You are not limited to 65000 long lived session with more than _one_ client.
>  
> Although the question is mostly for comprehension sake's I'm curious of the experience of People using MINA under HEAVY load and how they reacted.
>   
Some guys have tested MINA with 200K connections established, but the load was null (they just established the session and not sending any byte). The main problem will be with the data processing. If you have a multi-processor server with a lot of memory, that could scale, up to a point.

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

RE: Load Balancing Socket connections

[Stephane Rainville <st...@hotmail.com>]

Excellent answer!
 
Thanks a lot.> Date: Sun, 14 Sep 2008 09:40:51 +0200> To: users@mina.apache.org> Subject: Re: Load Balancing Socket connections> From: elecharny@gmail.com> > Stephane Rainville wrote:> > I dont' think so.> > > > Altough you are right a Socket can listen and open multiple connection on one port (Like a web server on port 80) when the server responds it responds on a RANDOM port between 1000 and 65000.> > > It responds using a random port associated to the client IP address. If > a specific client tries to open more then 64536 connections to a server > (very unlikely, if even possible !), then you will reach the limit. But > as you will have many clients, everything is fine. Remember that a > socket is defined by a couple of an IP address and a port.> > > > So my question remains using MINA and long lived socket connections am i limited to 65000 long lived socket connections and how can I load balance that.> > > You are not limited to 65000 long lived session with more than _one_ > client.> > > > Although the question is mostly for comprehension sake's I'm curious of the experience of People using MINA under HEAVY load and how they reacted.> > > Some guys have tested MINA with 200K connections established, but the > load was null (they just established the session and not sending any > byte). The main problem will be with the data processing. If you have a > multi-processor server with a lot of memory, that could scale, up to a > point.> > -- > --> cordialement, regards,> Emmanuel Lécharny> www.iktek.com> directory.apache.org> > 
_________________________________________________________________

Re: Load Balancing Socket connections

[Emmanuel Lecharny <el...@gmail.com>]

Stephane Rainville wrote:
> I dont' think so.
>  
> Altough you are right a Socket can listen and open multiple connection on one port (Like a web server on port 80) when the  server responds it responds on a RANDOM port between 1000 and 65000.
>   
It responds using a random port associated to the client IP address. If 
a specific client tries to open more then 64536 connections to a server 
(very unlikely, if even possible !), then you will reach the limit. But 
as you will have many clients, everything is fine. Remember that a 
socket is defined by a couple of an IP address and a port.
>  
> So my question remains using MINA and long lived socket connections am i limited to 65000 long lived socket connections and how can I load balance that.
>   
You are not limited to 65000 long lived session with more than _one_ 
client.
>  
> Although the question is mostly for comprehension sake's I'm curious of the experience of People using MINA under HEAVY load and how they reacted.
>   
Some guys have tested MINA with 200K connections established, but the 
load was null (they just established the session and not sending any 
byte). The main problem will be with the data processing. If you have a 
multi-processor server with a lot of memory, that could scale, up to a 
point.

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

RE: Load Balancing Socket connections

[Stephane Rainville <st...@hotmail.com>]

I dont' think so.
 
Altough you are right a Socket can listen and open multiple connection on one port (Like a web server on port 80) when the  server responds it responds on a RANDOM port between 1000 and 65000.
 
You can test this using netstat -a.
 
So my question remains using MINA and long lived socket connections am i limited to 65000 long lived socket connections and how can I load balance that.
 
Although the question is mostly for comprehension sake's I'm curious of the experience of People using MINA under HEAVY load and how they reacted.
 
Thanks
 
Stéphane> Date: Sat, 13 Sep 2008 16:19:07 +0200> To: users@mina.apache.org> Subject: Re: Load Balancing Socket connections> From: elecharny@gmail.com> > Stephane Rainville wrote:> > Question for the community.> > > > In Ip4 we are limited to 2exp16 ports. > > Which is VERY high for HTTP GET because they are very short lived connections.> > > I think you are confusing the number of available ports with the number > of connection a port can handle. A web server accept multiple requests > on one single port (80 usually).> > > > Has anybody designed an application using long lived SOCKETS? That limit could be reached fairly quickly and you would to load balance to multiple servers (Or maybe multiple interfaces on the server could be another solution).> > > You usually rely on semi-permanet sockets, even on HTTP (HTTP 1.1 + > Keepalive), otherwise the cost of opening a connection will be overkilling.> You then should consider that even on HTTP, connections are long lived.> > -- > --> cordialement, regards,> Emmanuel Lécharny> www.iktek.com> directory.apache.org> > 
_________________________________________________________________

Re: Load Balancing Socket connections

[Emmanuel Lecharny <el...@gmail.com>]

Stephane Rainville wrote:
> Question for the community.
>  
> In Ip4 we are limited to 2exp16 ports. 
> Which is VERY high for HTTP GET because they are very short lived connections.
>   
I think you are confusing the number of available ports with the number 
of connection a port can handle. A web server accept multiple requests 
on one single port (80 usually).
>  
> Has anybody designed an application using long lived SOCKETS? That limit could be reached fairly quickly and you would to load balance to multiple servers (Or maybe multiple interfaces on the server could be another solution).
>   
You usually rely on semi-permanet sockets, even on HTTP (HTTP 1.1 + 
Keepalive), otherwise the cost of opening a connection will be overkilling.
You then should consider that even on HTTP, connections are long lived.

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org