You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by David Latorre <dv...@gmail.com> on 2010/03/30 10:29:29 UTC

FTPServer handling of multiple concurrent connections.

Hello,

When inspecting http://issues.apache.org/jira/browse/FTPSERVER-359 I
noticed that we might have an issue with threading in Ftpserver.

I think we  are using a 'fixed size thread pool' with max-threads=16
for the Executor threadpool in which commands are executed. This would
mean that if we had 16 data transfers currently running , FTPServer
would be blocked until some of these data transfers ended- Is my
understanding correct?

Is this behaviour desirable for us? I don't think that 16 data
transfers are that many, and some users might need to send huge files.

RE: FTPServer handling of multiple concurrent connections.

Posted by Peter van der Velde <Pe...@anachron.com>.
Great, I will fetch the trunk then.

Thanks!

Greetings,

Peter.

-----Original Message-----
From: Sai Pullabhotla [mailto:sai.pullabhotla@jmethods.com] 
Sent: vrijdag 23 april 2010 15:24
To: dev@mina.apache.org
Subject: Re: FTPServer handling of multiple concurrent connections.

I'm pretty sure it is checked into the trunk.

Regards,
Sai Pullabhotla




On Fri, Apr 23, 2010 at 7:35 AM, Peter van der Velde <
Peter.vanderVelde@anachron.com> wrote:

> L.S.
>
> Is there an ETA for this change?
>
> Greetings,
>
> Peter van der Velde
>
> -----Original Message-----
> From: David Latorre [mailto:dvlato@gmail.com]
> Sent: donderdag 1 april 2010 12:57
> To: dev@mina.apache.org
> Subject: Re: FTPServer handling of multiple concurrent connections.
>
> 2010/3/31 Niklas Gustavsson <ni...@protocol7.com>:
> > On Wed, Mar 31, 2010 at 6:32 PM, Sai Pullabhotla 
> > <sa...@jmethods.com> wrote:
> >> Since I did not hear back anything on this, I will ask again :).
> >>
> >> Are you guys okay with the proposed short term solution?
> >
> > +1 to option 1 and to defaulting to max users.
>
> Sounds good :-)
>
> > /niklas
> >
>

Re: FTPServer handling of multiple concurrent connections.

Posted by Sai Pullabhotla <sa...@jmethods.com>.
I'm pretty sure it is checked into the trunk.

Regards,
Sai Pullabhotla




On Fri, Apr 23, 2010 at 7:35 AM, Peter van der Velde <
Peter.vanderVelde@anachron.com> wrote:

> L.S.
>
> Is there an ETA for this change?
>
> Greetings,
>
> Peter van der Velde
>
> -----Original Message-----
> From: David Latorre [mailto:dvlato@gmail.com]
> Sent: donderdag 1 april 2010 12:57
> To: dev@mina.apache.org
> Subject: Re: FTPServer handling of multiple concurrent connections.
>
> 2010/3/31 Niklas Gustavsson <ni...@protocol7.com>:
> > On Wed, Mar 31, 2010 at 6:32 PM, Sai Pullabhotla
> > <sa...@jmethods.com> wrote:
> >> Since I did not hear back anything on this, I will ask again :).
> >>
> >> Are you guys okay with the proposed short term solution?
> >
> > +1 to option 1 and to defaulting to max users.
>
> Sounds good :-)
>
> > /niklas
> >
>

RE: FTPServer handling of multiple concurrent connections.

Posted by Peter van der Velde <Pe...@anachron.com>.
L.S.

Is there an ETA for this change? 

Greetings,

Peter van der Velde

-----Original Message-----
From: David Latorre [mailto:dvlato@gmail.com] 
Sent: donderdag 1 april 2010 12:57
To: dev@mina.apache.org
Subject: Re: FTPServer handling of multiple concurrent connections.

2010/3/31 Niklas Gustavsson <ni...@protocol7.com>:
> On Wed, Mar 31, 2010 at 6:32 PM, Sai Pullabhotla 
> <sa...@jmethods.com> wrote:
>> Since I did not hear back anything on this, I will ask again :).
>>
>> Are you guys okay with the proposed short term solution?
>
> +1 to option 1 and to defaulting to max users.

Sounds good :-)

> /niklas
>

Re: FTPServer handling of multiple concurrent connections.

Posted by David Latorre <dv...@gmail.com>.
2010/3/31 Niklas Gustavsson <ni...@protocol7.com>:
> On Wed, Mar 31, 2010 at 6:32 PM, Sai Pullabhotla
> <sa...@jmethods.com> wrote:
>> Since I did not hear back anything on this, I will ask again :).
>>
>> Are you guys okay with the proposed short term solution?
>
> +1 to option 1 and to defaulting to max users.

Sounds good :-)

> /niklas
>

Re: FTPServer handling of multiple concurrent connections.

Posted by Niklas Gustavsson <ni...@protocol7.com>.
On Wed, Mar 31, 2010 at 6:32 PM, Sai Pullabhotla
<sa...@jmethods.com> wrote:
> Since I did not hear back anything on this, I will ask again :).
>
> Are you guys okay with the proposed short term solution?

+1 to option 1 and to defaulting to max users.

/niklas

Re: FTPServer handling of multiple concurrent connections.

Posted by Sai Pullabhotla <sa...@jmethods.com>.
Since I did not hear back anything on this, I will ask again :).

Are you guys okay with the proposed short term solution?

Regards,
Sai Pullabhotla





On Tue, Mar 30, 2010 at 10:10 AM, Sai Pullabhotla
<sa...@jmethods.com> wrote:
> Since changing everything over to MINA could be quite a bit of work,
> and the issue we have is somewhat serious, we should come up with a
> short term solution first and release a patch. Perhaps this patch
> would do the following:
>
> Option 1: Have Max Threads as a configurable option at the server
> level. Each listener would share the same thread pool.
>
> Option 2: Have Max Threads as a configurable option at the listener
> (acceptor) level.
>
> I prefer option 1 as most people would look at the server level rather
> than listener level. In other words, my server should be able to
> handle 200 concurrent users. If Max Threads is not specified, may be
> we should default it to maxUsers/connections that we have. Not sure
> what the current keep-alive time for the threads is, but perhaps
> having a shorter keep-alive time may help in some cases, or make it as
> a configurable option.
>
> What do you think?
>
> Regards,
> Sai Pullabhotla
>
>
>
>
>
> On Tue, Mar 30, 2010 at 8:33 AM, Niklas Gustavsson <ni...@protocol7.com> wrote:
>> On Tue, Mar 30, 2010 at 2:22 PM, David Latorre <dv...@gmail.com> wrote:
>>> I would rather go for a solution that make it impossible to block
>>> FTPServer rather than making it 'more difficult'.
>>> For this, we might limit the total number of data connections which
>>> wouldn't be perfect but might help... or maybe we can enforce a rule
>>> that MaxUsers < MaxThreads.
>>
>> Depending on your user profiles, I think there might be different
>> configurations that suites you best. So, I think we should supply both
>> the max concurrent users option (that we do today) and a max threads
>> option. Then, we need to provide reasonable defaults and documentation
>> on how to best use them.
>>
>>> - We could have a filter that limited bandwidth usage, although I
>>> don't think there is anything like that in FTPServer.
>>
>> We already do bandwidth limitation in our blocking data connections.
>>
>> /niklas
>>
>

Re: FTPServer handling of multiple concurrent connections.

Posted by Sai Pullabhotla <sa...@jmethods.com>.
Since changing everything over to MINA could be quite a bit of work,
and the issue we have is somewhat serious, we should come up with a
short term solution first and release a patch. Perhaps this patch
would do the following:

Option 1: Have Max Threads as a configurable option at the server
level. Each listener would share the same thread pool.

Option 2: Have Max Threads as a configurable option at the listener
(acceptor) level.

I prefer option 1 as most people would look at the server level rather
than listener level. In other words, my server should be able to
handle 200 concurrent users. If Max Threads is not specified, may be
we should default it to maxUsers/connections that we have. Not sure
what the current keep-alive time for the threads is, but perhaps
having a shorter keep-alive time may help in some cases, or make it as
a configurable option.

What do you think?

Regards,
Sai Pullabhotla





On Tue, Mar 30, 2010 at 8:33 AM, Niklas Gustavsson <ni...@protocol7.com> wrote:
> On Tue, Mar 30, 2010 at 2:22 PM, David Latorre <dv...@gmail.com> wrote:
>> I would rather go for a solution that make it impossible to block
>> FTPServer rather than making it 'more difficult'.
>> For this, we might limit the total number of data connections which
>> wouldn't be perfect but might help... or maybe we can enforce a rule
>> that MaxUsers < MaxThreads.
>
> Depending on your user profiles, I think there might be different
> configurations that suites you best. So, I think we should supply both
> the max concurrent users option (that we do today) and a max threads
> option. Then, we need to provide reasonable defaults and documentation
> on how to best use them.
>
>> - We could have a filter that limited bandwidth usage, although I
>> don't think there is anything like that in FTPServer.
>
> We already do bandwidth limitation in our blocking data connections.
>
> /niklas
>

Re: FTPServer handling of multiple concurrent connections.

Posted by Niklas Gustavsson <ni...@protocol7.com>.
On Tue, Mar 30, 2010 at 2:22 PM, David Latorre <dv...@gmail.com> wrote:
> I would rather go for a solution that make it impossible to block
> FTPServer rather than making it 'more difficult'.
> For this, we might limit the total number of data connections which
> wouldn't be perfect but might help... or maybe we can enforce a rule
> that MaxUsers < MaxThreads.

Depending on your user profiles, I think there might be different
configurations that suites you best. So, I think we should supply both
the max concurrent users option (that we do today) and a max threads
option. Then, we need to provide reasonable defaults and documentation
on how to best use them.

> - We could have a filter that limited bandwidth usage, although I
> don't think there is anything like that in FTPServer.

We already do bandwidth limitation in our blocking data connections.

/niklas

Re: FTPServer handling of multiple concurrent connections.

Posted by David Latorre <dv...@gmail.com>.
2010/3/30 Niklas Gustavsson <ni...@protocol7.com>:
> On Tue, Mar 30, 2010 at 10:33 AM, Sai Pullabhotla
> <sa...@jmethods.com> wrote:
>> I don't think that was intended. If this is in fact an issue, we
>> should probably consider adding a configuration option such as
>> maxThreads as the default max we choose may not be the best in all
>> cases.
>
> I think David is correct, and I agree with your proposed solution
> (defaulting the value higher than 16 thou). We should also look into
> moving the data connection over to using MINA, if someone got the
> time.


I would rather go for a solution that make it impossible to block
FTPServer rather than making it 'more difficult'.
For this, we might limit the total number of data connections which
wouldn't be perfect but might help... or maybe we can enforce a rule
that MaxUsers < MaxThreads.

Regarding the move of the data connection to MINA, I understand Sai's
position that is using blocking-IO here is probably lighter ( It would
be nice if we could have the same 'MINA pipeline' accepting on several
ports...). Still I see an advantage  if we used MINA -  filters:

- We could have different filters for ASCII mode, binary mode, 'Z'
mode or other modes of encoding we or the users might think of
(Although it is easy to have custom streams that do this for us with
the current implementation).

- We can include a fine grained monitoring of current transfers, a
filter might inform via JMX  or other means of e.g., the amount of
data transferred

- A filter could limit the quota for a user, for a session, for a
week/month for this user  in 'real time'. There are other ways to do
this as well...

- We could have a filter that limited bandwidth usage, although I
don't think there is anything like that in FTPServer.


Still, several of the possibilities we can think of seem not that
useful since we should have multiple 'chains' for data connections.









> /niklas
>

Re: FTPServer handling of multiple concurrent connections.

Posted by Niklas Gustavsson <ni...@protocol7.com>.
On Tue, Mar 30, 2010 at 1:39 PM, Sai Pullabhotla
<sa...@jmethods.com> wrote:
> What would be the benefit of using MINA compared to the traditional
> sockets, especially for passive connections?

That we don't get one thread for each one socket. Perhaps this is not
such a big issue, at least not as far as your not planning for
thousands of concurrent data connections.

/niklas

Re: FTPServer handling of multiple concurrent connections.

Posted by Sai Pullabhotla <sa...@jmethods.com>.
What would be the benefit of using MINA compared to the traditional
sockets, especially for passive connections?

Regards,
Sai Pullabhotla





On Tue, Mar 30, 2010 at 7:33 AM, Niklas Gustavsson <ni...@protocol7.com> wrote:
> On Tue, Mar 30, 2010 at 10:33 AM, Sai Pullabhotla
> <sa...@jmethods.com> wrote:
>> I don't think that was intended. If this is in fact an issue, we
>> should probably consider adding a configuration option such as
>> maxThreads as the default max we choose may not be the best in all
>> cases.
>
> I think David is correct, and I agree with your proposed solution
> (defaulting the value higher than 16 thou). We should also look into
> moving the data connection over to using MINA, if someone got the
> time.
>
> /niklas
>

Re: FTPServer handling of multiple concurrent connections.

Posted by Niklas Gustavsson <ni...@protocol7.com>.
On Tue, Mar 30, 2010 at 10:33 AM, Sai Pullabhotla
<sa...@jmethods.com> wrote:
> I don't think that was intended. If this is in fact an issue, we
> should probably consider adding a configuration option such as
> maxThreads as the default max we choose may not be the best in all
> cases.

I think David is correct, and I agree with your proposed solution
(defaulting the value higher than 16 thou). We should also look into
moving the data connection over to using MINA, if someone got the
time.

/niklas

Re: FTPServer handling of multiple concurrent connections.

Posted by Sai Pullabhotla <sa...@jmethods.com>.
I don't think that was intended. If this is in fact an issue, we
should probably consider adding a configuration option such as
maxThreads as the default max we choose may not be the best in all
cases.

Regards,
Sai Pullabhotla





On Tue, Mar 30, 2010 at 3:29 AM, David Latorre <dv...@gmail.com> wrote:
> Hello,
>
> When inspecting http://issues.apache.org/jira/browse/FTPSERVER-359 I
> noticed that we might have an issue with threading in Ftpserver.
>
> I think we  are using a 'fixed size thread pool' with max-threads=16
> for the Executor threadpool in which commands are executed. This would
> mean that if we had 16 data transfers currently running , FTPServer
> would be blocked until some of these data transfers ended- Is my
> understanding correct?
>
> Is this behaviour desirable for us? I don't think that 16 data
> transfers are that many, and some users might need to send huge files.
>