You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Vincent Barat <vi...@gmail.com> on 2011/08/26 14:30:13 UTC

PIG 0.8.1 leaks Zookeeper connections when using HBaseStorage

Hi,

I run PIG jobs from a Java process (using PigServer). Most of which 
use HBaseStorage to load data from HBase.
Each job is run using a new PigServer object, and I correctly call 
pigServer.shutdown() when my pig server is no longer used.

Nevertheless, after a few hours of run, I notice that the number of 
connections to my Zookeeper servers reach the limit (300 in my case).
It appears that each job leak 4 or 5 Zookeeper connections.

It was not the case with PIG 0.6.1

To solve this issue (temporarily) by killing the process running PIG 
after a few set of jobs have been run : connections are correctly 
closed.
My process don't use HBase by itself, only HBaseStorage, so I guess 
the leak is in the code of HBaseStorage: maybe to cnx to HBase are 
not closed.

Should I report this a an issue ?

Thanks for your help

Re: PIG 0.8.1 leaks Zookeeper connections when using HBaseStorage

Posted by Bill Graham <bi...@gmail.com>.
>
> Should I report this a an issue ?
>

Yes, please. I've found other resource leaks when using PigServer this way,
so this seems like a likely bug. Also, seeing that HTables are never closed
by HBaseStorage is not a good sign.


On Fri, Aug 26, 2011 at 5:30 AM, Vincent Barat <vi...@gmail.com>wrote:

> Hi,
>
> I run PIG jobs from a Java process (using PigServer). Most of which use
> HBaseStorage to load data from HBase.
> Each job is run using a new PigServer object, and I correctly call
> pigServer.shutdown() when my pig server is no longer used.
>
> Nevertheless, after a few hours of run, I notice that the number of
> connections to my Zookeeper servers reach the limit (300 in my case).
> It appears that each job leak 4 or 5 Zookeeper connections.
>
> It was not the case with PIG 0.6.1
>
> To solve this issue (temporarily) by killing the process running PIG after
> a few set of jobs have been run : connections are correctly closed.
> My process don't use HBase by itself, only HBaseStorage, so I guess the
> leak is in the code of HBaseStorage: maybe to cnx to HBase are not closed.
>
> Should I report this a an issue ?
>
> Thanks for your help
>

Re: PIG 0.8.1 leaks Zookeeper connections when using HBaseStorage

Posted by Vincent Barat <vi...@gmail.com>.
I'm affraid it will be difficult. Nevertheles, I have cut & paste 
some of the code used.

The issue is https://issues.apache.org/jira/browse/PIG-2251

By the way, I use PIG 0.8.1 + 
https://issues.apache.org/jira/browse/PIG-2193

Le 26/08/11 18:19, Ashutosh Chauhan a écrit :
> Hey Vincent,
>
> Will it be easy for you to isolate this in a test code. That will help to
> debug the issue and also fixing it.
>
> Ashutosh
> On Fri, Aug 26, 2011 at 05:30, Vincent Barat<vi...@gmail.com>wrote:
>
>> Hi,
>>
>> I run PIG jobs from a Java process (using PigServer). Most of which use
>> HBaseStorage to load data from HBase.
>> Each job is run using a new PigServer object, and I correctly call
>> pigServer.shutdown() when my pig server is no longer used.
>>
>> Nevertheless, after a few hours of run, I notice that the number of
>> connections to my Zookeeper servers reach the limit (300 in my case).
>> It appears that each job leak 4 or 5 Zookeeper connections.
>>
>> It was not the case with PIG 0.6.1
>>
>> To solve this issue (temporarily) by killing the process running PIG after
>> a few set of jobs have been run : connections are correctly closed.
>> My process don't use HBase by itself, only HBaseStorage, so I guess the
>> leak is in the code of HBaseStorage: maybe to cnx to HBase are not closed.
>>
>> Should I report this a an issue ?
>>
>> Thanks for your help
>>

Re: PIG 0.8.1 leaks Zookeeper connections when using HBaseStorage

Posted by Ashutosh Chauhan <ha...@apache.org>.
Hey Vincent,

Will it be easy for you to isolate this in a test code. That will help to
debug the issue and also fixing it.

Ashutosh
On Fri, Aug 26, 2011 at 05:30, Vincent Barat <vi...@gmail.com>wrote:

> Hi,
>
> I run PIG jobs from a Java process (using PigServer). Most of which use
> HBaseStorage to load data from HBase.
> Each job is run using a new PigServer object, and I correctly call
> pigServer.shutdown() when my pig server is no longer used.
>
> Nevertheless, after a few hours of run, I notice that the number of
> connections to my Zookeeper servers reach the limit (300 in my case).
> It appears that each job leak 4 or 5 Zookeeper connections.
>
> It was not the case with PIG 0.6.1
>
> To solve this issue (temporarily) by killing the process running PIG after
> a few set of jobs have been run : connections are correctly closed.
> My process don't use HBase by itself, only HBaseStorage, so I guess the
> leak is in the code of HBaseStorage: maybe to cnx to HBase are not closed.
>
> Should I report this a an issue ?
>
> Thanks for your help
>