Creating new connections in integration tests

[Stefan Seelmann <se...@apache.org>]

Hi guys,

one of our Hudson builds currently fails [1], the cause is an
IOException (Too many open files). The reason this happens is that in
our integration test (server-integ and ldap-api-test) we open real
network connections. In many cases a new connection is opened in each
test method. I also found a case where we open new connections within
a loop.

Most times the connections is closed after usage. However, as you
certainly know, there is still a problem with allocating many new
connections: sockets remain in TIME_WAIT state for some time after
bein closed, see [2]. On the Hudson servers we can of course increase
the ulimit -n and reduce the TIME_WAIT value. But I think that won't
scale when connections are opened in a loop.

I'd like to discuss possible options to improve the situation:

- Fist I think in general we should avoid to create new connections
within a loop in integration tests. We have other techniques to run
such test [3]. In general such test patterns are stress or performance
tests, but not integration tests.

- Second we should avoid to create a new connection for each test
method. One option to achieve this is to use a single connection per
test class without opening/closing them @Before/@After each test.
Maybe the better option is to use a connection pool. Of course when
testing connect/disconnect/bind/unbind it may be required to use a
fresh connection objects.

Thoughts?

Kind Regards,
Stefan


[1] https://hudson.apache.org/hudson/job/dir-apacheds-jdk15-ubuntu-deploy/
[2] http://mina.apache.org/faq.html#FAQ-Myserverfailswithjava.net.SocketException%253AToomanyfilesopen
[3] http://s.apache.org/mtmi

Re: Creating new connections in integration tests

[Emmanuel Lecharny <el...@gmail.com>]

On 2/5/11 12:51 PM, Stefan Seelmann wrote:
> On Sat, Feb 5, 2011 at 8:35 AM, Emmanuel Lecharny wrote:
>> On 2/4/11 10:58 PM, Stefan Seelmann wrote:
>>> Hi guys,
>>>
>>> one of our Hudson builds currently fails [1], the cause is an
>>> IOException (Too many open files). The reason this happens is that in
>>> our integration test (server-integ and ldap-api-test) we open real
>>> network connections. In many cases a new connection is opened in each
>>> test method. I also found a case where we open new connections within
>>> a loop.
>> I added some performance tests for each operation, but those tests are
>> @Ignored.
>>
>> What is the tests containing a loop ?
> SimpleBindRequestTest.testSimpleBindAnonymous() in ldap-api-test.

AFAIR, this test was used last year when we had a problem with multiple 
bind occuring. At some point in the test, the Bind failed (it was after 
a few hundred failures). Right now, I'm nt sure it makes sense anymore 
to loop, even if it's only for 5 attempts...


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Creating new connections in integration tests

[Stefan Seelmann <se...@apache.org>]

On Sat, Feb 5, 2011 at 8:35 AM, Emmanuel Lecharny wrote:
> On 2/4/11 10:58 PM, Stefan Seelmann wrote:
>>
>> Hi guys,
>>
>> one of our Hudson builds currently fails [1], the cause is an
>> IOException (Too many open files). The reason this happens is that in
>> our integration test (server-integ and ldap-api-test) we open real
>> network connections. In many cases a new connection is opened in each
>> test method. I also found a case where we open new connections within
>> a loop.
>
> I added some performance tests for each operation, but those tests are
> @Ignored.
>
> What is the tests containing a loop ?

SimpleBindRequestTest.testSimpleBindAnonymous() in ldap-api-test.

>> Most times the connections is closed after usage. However, as you
>> certainly know, there is still a problem with allocating many new
>> connections: sockets remain in TIME_WAIT state for some time after
>> bein closed, see [2]. On the Hudson servers we can of course increase
>> the ulimit -n and reduce the TIME_WAIT value. But I think that won't
>> scale when connections are opened in a loop.
>
> Sure. We can also ask to have  short lived socket, like 5 secs. Usually, a
> socket remain in TIME_WAIT state for 30 secs on a standard linux box.
>>
>> I'd like to discuss possible options to improve the situation:
>>
>> - Fist I think in general we should avoid to create new connections
>> within a loop in integration tests. We have other techniques to run
>> such test [3]. In general such test patterns are stress or performance
>> tests, but not integration tests.
>
> +1
>>
>> - Second we should avoid to create a new connection for each test
>> method. One option to achieve this is to use a single connection per
>> test class without opening/closing them @Before/@After each test.
>> Maybe the better option is to use a connection pool. Of course when
>> testing connect/disconnect/bind/unbind it may be required to use a
>> fresh connection objects.
>
> If we use our client API, we have to add a pool into the API. In the tests
> where we use JNDI, we can use a pool, that's for sure.

I just saw that we already have a LdapConnectionPool class in the API,
I'll try to use that.

Kind Regards,
Stefan

Re: Creating new connections in integration tests

[Emmanuel Lecharny <el...@gmail.com>]

On 2/4/11 10:58 PM, Stefan Seelmann wrote:
> Hi guys,
>
> one of our Hudson builds currently fails [1], the cause is an
> IOException (Too many open files). The reason this happens is that in
> our integration test (server-integ and ldap-api-test) we open real
> network connections. In many cases a new connection is opened in each
> test method. I also found a case where we open new connections within
> a loop.
I added some performance tests for each operation, but those tests are 
@Ignored.

What is the tests containing a loop ?
> Most times the connections is closed after usage. However, as you
> certainly know, there is still a problem with allocating many new
> connections: sockets remain in TIME_WAIT state for some time after
> bein closed, see [2]. On the Hudson servers we can of course increase
> the ulimit -n and reduce the TIME_WAIT value. But I think that won't
> scale when connections are opened in a loop.
Sure. We can also ask to have  short lived socket, like 5 secs. Usually, 
a socket remain in TIME_WAIT state for 30 secs on a standard linux box.
> I'd like to discuss possible options to improve the situation:
>
> - Fist I think in general we should avoid to create new connections
> within a loop in integration tests. We have other techniques to run
> such test [3]. In general such test patterns are stress or performance
> tests, but not integration tests.
+1
> - Second we should avoid to create a new connection for each test
> method. One option to achieve this is to use a single connection per
> test class without opening/closing them @Before/@After each test.
> Maybe the better option is to use a connection pool. Of course when
> testing connect/disconnect/bind/unbind it may be required to use a
> fresh connection objects.
If we use our client API, we have to add a pool into the API. In the 
tests where we use JNDI, we can use a pool, that's for sure.


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Creating new connections in integration tests

[Alex Karasulu <ak...@apache.org>]

On Fri, Feb 4, 2011 at 11:58 PM, Stefan Seelmann <se...@apache.org>wrote:

> Hi guys,
>
> one of our Hudson builds currently fails [1], the cause is an
> IOException (Too many open files). The reason this happens is that in
> our integration test (server-integ and ldap-api-test) we open real
> network connections. In many cases a new connection is opened in each
> test method. I also found a case where we open new connections within
> a loop.
>
>
Yes the tests all need to be revamped to make integration tests run smooth.
Some tests are performance tests like this one that was recently ignored:

     http://goo.gl/Cws6O

We also have network tests happening as you can see in the core-integ area
which should not be testing services. I'll take a pass at trying to organize
the tests so we at least localize all the network tests under server-integ.


> Most times the connections is closed after usage. However, as you
> certainly know, there is still a problem with allocating many new
> connections: sockets remain in TIME_WAIT state for some time after
> bein closed, see [2]. On the Hudson servers we can of course increase
> the ulimit -n and reduce the TIME_WAIT value. But I think that won't
> scale when connections are opened in a loop.
>

Agreed.


> I'd like to discuss possible options to improve the situation:
>
> - Fist I think in general we should avoid to create new connections
> within a loop in integration tests. We have other techniques to run
> such test [3]. In general such test patterns are stress or performance
> tests, but not integration tests.
>

You're right, we need to be more disciplined with making sure only
integration tests are present. We pay for these heavy long durations with
builds taking 15+ minutes on laptops.

It's a good thing we're refreshing our build config on hudson. It's showing
us we need to clean up a little.


> - Second we should avoid to create a new connection for each test
> method. One option to achieve this is to use a single connection per
> test class without opening/closing them @Before/@After each test.
> Maybe the better option is to use a connection pool. Of course when
> testing connect/disconnect/bind/unbind it may be required to use a
> fresh connection objects.
>
> Thoughts?
>

+1

If the test permits the reuse then we definitely should be reusing the
connection for each test class.

Thanks,
Alex