DistributedSearch$Client.updateSegments() blocking other threads

[Andrzej Bialecki <ab...@getopt.org>]

Hi,

I was doing performance testing of a distributed search setup, with 
JMeter, using the code from trunk/.

Whenever one of the backend Servers goes down, there is a hiccup on the 
frontend, because all ParallelCalls started by the Client, which still 
use that dead address, need to timeout. This is expected, and acceptable.

New calls being made in the meantime (before updateSegments() discovers 
that the host is down) will also need to timeout - which is so so, I 
think it could be improved by removing the offending address at the 
first sign of trouble, i.e. not to wait for updateSegments() but 
immediately remove the dead host from liveAddresses. Anyway, read on...

What was curious was that the same hiccup would then occur every 10 
seconds, which is the hardcoded interval for calling 
Client.updateSegments(). It was as if the call to updateSegments() was 
synchronized on the whole class, so that all other calls are blocked 
until updateSegments() completes. I modified the code, so that instead 
of using DistributedSearch$Client itself as a Thread instance, a new 
independent Thread instance is created.

The hiccups are gone now - the list of liveAddresses is still being 
updated as it should whenever Servers go down/up, but now 
updateSegments() doesn't interfere with other calls. I attach the patch 
- but to be honest I'm still not quite sure what was happening...

-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

Re: DistributedSearch$Client.updateSegments() blocking other threads

[Piotr Kosiorowski <pk...@gmail.com>]

Hello Andrzej,
You can also try http://issues.apache.org/jira/browse/NUTCH-79
- I think it should also help here - it is a bit complicated as it 
contain additional functionality but if you have any problems I am 
willing to help. I am going to perform some test of it again and maybe 
commit it in some time if others think it is worth it.
Regrads
Piotr

Andrzej Bialecki wrote:
> Hi,
> 
> I was doing performance testing of a distributed search setup, with 
> JMeter, using the code from trunk/.
> 
> Whenever one of the backend Servers goes down, there is a hiccup on the 
> frontend, because all ParallelCalls started by the Client, which still 
> use that dead address, need to timeout. This is expected, and acceptable.
> 
> New calls being made in the meantime (before updateSegments() discovers 
> that the host is down) will also need to timeout - which is so so, I 
> think it could be improved by removing the offending address at the 
> first sign of trouble, i.e. not to wait for updateSegments() but 
> immediately remove the dead host from liveAddresses. Anyway, read on...
> 
> What was curious was that the same hiccup would then occur every 10 
> seconds, which is the hardcoded interval for calling 
> Client.updateSegments(). It was as if the call to updateSegments() was 
> synchronized on the whole class, so that all other calls are blocked 
> until updateSegments() completes. I modified the code, so that instead 
> of using DistributedSearch$Client itself as a Thread instance, a new 
> independent Thread instance is created.
> 
> The hiccups are gone now - the list of liveAddresses is still being 
> updated as it should whenever Servers go down/up, but now 
> updateSegments() doesn't interfere with other calls. I attach the patch 
> - but to be honest I'm still not quite sure what was happening...
> 
> 
> ------------------------------------------------------------------------
> 
> Index: DistributedSearch.java
> ===================================================================
> --- DistributedSearch.java	(revision 280515)
> +++ DistributedSearch.java	(working copy)
> @@ -112,8 +112,9 @@
>      public Client(InetSocketAddress[] addresses) throws IOException {
>        this.defaultAddresses = addresses;
>        updateSegments();
> -      setDaemon(true);
> -      start();
> +      Thread t = new Thread(this);
> +      t.setDaemon(true);
> +      t.start();
>      }
>      
>      private static final Method GET_SEGMENTS;
> @@ -168,8 +169,10 @@
>          liveSegments+=segments.length;
>        }
>  
> -      this.liveAddresses = (InetSocketAddress[]) // update liveAddresses
> -        liveAddresses.toArray(new InetSocketAddress[liveAddresses.size()]);
> +      synchronized(this.liveAddresses) {
> +        this.liveAddresses = (InetSocketAddress[]) // update liveAddresses
> +          liveAddresses.toArray(new InetSocketAddress[liveAddresses.size()]);
> +      }
>  
>        LOG.info("STATS: "+liveServers+" servers, "+liveSegments+" segments.");
>      }