You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Vincent Barat <vi...@gmail.com> on 2011/10/25 11:49:20 UTC
PIG 0.9 leaks HBaseStorage instances
Hi,
I try to figure out why PIG is using so many zookeeper connections
(from the frontend machine) when using HBaseStorage().
I added a trace in the constructor of HBaseStorage()
I wrote a simple script loading an HBase table:
sessions = LOAD 'hbase://mytable' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid
meta:timestamp') AS (sid:chararray, start:long);
dump sessions;
When I run the script:
vbarat@lancelot:~$ pig -x local -f
/Users/vbarat/ermin/pig/script/test.pig
2011-10-25 11:32:41,482 [main] INFO org.apache.pig.Main - Logging
error messages to: /Users/vbarat/pig_1319535161481.log
2011-10-25 11:32:41,563 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to hadoop file system at: file:///
2011-10-25 11:32:41,884 [main] INFO
org.apache.pig.backend.hadoop.hbase.HBaseStorage - ***********
HBASESTORAGE *********************
2011-10-25 11:32:41,970 [main] INFO
org.apache.pig.backend.hadoop.hbase.HBaseStorage - ***********
HBASESTORAGE *********************
2011-10-25 11:32:42,035 [main] INFO
org.apache.pig.backend.hadoop.hbase.HBaseStorage - ***********
HBASESTORAGE *********************
2011-10-25 11:32:42,073 [main] INFO
org.apache.pig.backend.hadoop.hbase.HBaseStorage - ***********
HBASESTORAGE *********************
2011-10-25 11:32:42,184 [main] INFO
org.apache.pig.backend.hadoop.hbase.HBaseStorage - ***********
HBASESTORAGE *********************
2011-10-25 11:32:42,207 [main] INFO
org.apache.pig.backend.hadoop.hbase.HBaseStorage - ***********
HBASESTORAGE *********************
2011-10-25 11:32:42,233 [main] INFO
org.apache.pig.backend.hadoop.hbase.HBaseStorage - ***********
HBASESTORAGE *********************
2011-10-25 11:32:42,256 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2011-10-25 11:32:42,317 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics
with processName=JobTracker, sessionId=
2011-10-25 11:32:42,374 [main] INFO
org.apache.pig.backend.hadoop.hbase.HBaseStorage - ***********
HBASESTORAGE *********************
2011-10-25 11:32:42,391 [main] INFO
org.apache.pig.backend.hadoop.hbase.HBaseStorage - ***********
HBASESTORAGE *********************
2011-10-25 11:32:42,425 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler
- File concatenation threshold: 100 optimistic? false
2011-10-25 11:32:42,449 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
So HBaseStorage is create 10 times (and so the table is opened 9 times).
I'd like to konw why so many creation ?
Also, when I change my script to load 2 tables and join them, the
HBaseStorage object is created 40 times !
Can someone give me some insight to help me investigating the issue ?
Thanks a lot
Re: PIG 0.9 leaks HBaseStorage instances
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
We need to connect to HBase to figure out input slices, etc. This
involves connecting to ZK (via HBase innards -- we don't do this
explicitly).
On Fri, Oct 28, 2011 at 9:30 AM, Alan Gates <ga...@hortonworks.com> wrote:
> Which method is HBaseStorage connecting to Zookeeper in? There are methods that are guaranteed to only be called on the front or back end, so you can avoid these kinds of traps. HCatalog in particular had to work around the same thing to avoid connecting to the metastore server in places it should not. Do you need to connect to ZooKeeper on the front end at all?
>
> Alan.
>
> On Oct 28, 2011, at 9:19 AM, Vincent Barat wrote:
>
>> :-(
>>
>> ok I understand. I don't see why the optimizer need to instantiate the loaders, but I'm not a PIG specialist.
>>
>> Concerning HBaseStorage, this is a big issue, as the number of cnx to zookeeper is fairly limited (30 in the default configuration).
>> So a simple join of 2 tables would break this limit.
>>
>> The finalize() method is of course not a solution, as it is called only when the GC want to call it.
>>
>> Anyway I'll try it just to get some insight.
>>
>> Cheers,
>>
>> Le 26/10/11 21:12, Dmitriy Ryaboy a écrit :
>>> unfortunately Pig creates UDFs willy-nilly to do various checks during
>>> script compilation.
>>>
>>> Maybe we should override the finalize() method and close the
>>> connection there, that would help.
>>>
>>> D
>>>
>>> On Tue, Oct 25, 2011 at 2:49 AM, Vincent Barat<vi...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> I try to figure out why PIG is using so many zookeeper connections (from the
>>>> frontend machine) when using HBaseStorage().
>>>>
>>>> I added a trace in the constructor of HBaseStorage()
>>>>
>>>> I wrote a simple script loading an HBase table:
>>>>
>>>> sessions = LOAD 'hbase://mytable' USING
>>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid meta:timestamp')
>>>> AS (sid:chararray, start:long);
>>>> dump sessions;
>>>>
>>>> When I run the script:
>>>>
>>>> vbarat@lancelot:~$ pig -x local -f /Users/vbarat/ermin/pig/script/test.pig
>>>> 2011-10-25 11:32:41,482 [main] INFO org.apache.pig.Main - Logging error
>>>> messages to: /Users/vbarat/pig_1319535161481.log
>>>> 2011-10-25 11:32:41,563 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
>>>> to hadoop file system at: file:///
>>>> 2011-10-25 11:32:41,884 [main] INFO
>>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>>> *********************
>>>> 2011-10-25 11:32:41,970 [main] INFO
>>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>>> *********************
>>>> 2011-10-25 11:32:42,035 [main] INFO
>>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>>> *********************
>>>> 2011-10-25 11:32:42,073 [main] INFO
>>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>>> *********************
>>>> 2011-10-25 11:32:42,184 [main] INFO
>>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>>> *********************
>>>> 2011-10-25 11:32:42,207 [main] INFO
>>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>>> *********************
>>>> 2011-10-25 11:32:42,233 [main] INFO
>>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>>> *********************
>>>> 2011-10-25 11:32:42,256 [main] INFO
>>>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
>>>> script: UNKNOWN
>>>> 2011-10-25 11:32:42,317 [main] INFO
>>>> org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
>>>> processName=JobTracker, sessionId=
>>>> 2011-10-25 11:32:42,374 [main] INFO
>>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>>> *********************
>>>> 2011-10-25 11:32:42,391 [main] INFO
>>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>>> *********************
>>>> 2011-10-25 11:32:42,425 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
>>>> File concatenation threshold: 100 optimistic? false
>>>> 2011-10-25 11:32:42,449 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>>>> - MR plan size before optimization: 1
>>>>
>>>> So HBaseStorage is create 10 times (and so the table is opened 9 times).
>>>>
>>>> I'd like to konw why so many creation ?
>>>>
>>>> Also, when I change my script to load 2 tables and join them, the
>>>> HBaseStorage object is created 40 times !
>>>>
>>>> Can someone give me some insight to help me investigating the issue ?
>>>>
>>>> Thanks a lot
>>>>
>
>
Re: PIG 0.9 leaks HBaseStorage instances
Posted by Alan Gates <ga...@hortonworks.com>.
Which method is HBaseStorage connecting to Zookeeper in? There are methods that are guaranteed to only be called on the front or back end, so you can avoid these kinds of traps. HCatalog in particular had to work around the same thing to avoid connecting to the metastore server in places it should not. Do you need to connect to ZooKeeper on the front end at all?
Alan.
On Oct 28, 2011, at 9:19 AM, Vincent Barat wrote:
> :-(
>
> ok I understand. I don't see why the optimizer need to instantiate the loaders, but I'm not a PIG specialist.
>
> Concerning HBaseStorage, this is a big issue, as the number of cnx to zookeeper is fairly limited (30 in the default configuration).
> So a simple join of 2 tables would break this limit.
>
> The finalize() method is of course not a solution, as it is called only when the GC want to call it.
>
> Anyway I'll try it just to get some insight.
>
> Cheers,
>
> Le 26/10/11 21:12, Dmitriy Ryaboy a écrit :
>> unfortunately Pig creates UDFs willy-nilly to do various checks during
>> script compilation.
>>
>> Maybe we should override the finalize() method and close the
>> connection there, that would help.
>>
>> D
>>
>> On Tue, Oct 25, 2011 at 2:49 AM, Vincent Barat<vi...@gmail.com> wrote:
>>> Hi,
>>>
>>> I try to figure out why PIG is using so many zookeeper connections (from the
>>> frontend machine) when using HBaseStorage().
>>>
>>> I added a trace in the constructor of HBaseStorage()
>>>
>>> I wrote a simple script loading an HBase table:
>>>
>>> sessions = LOAD 'hbase://mytable' USING
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid meta:timestamp')
>>> AS (sid:chararray, start:long);
>>> dump sessions;
>>>
>>> When I run the script:
>>>
>>> vbarat@lancelot:~$ pig -x local -f /Users/vbarat/ermin/pig/script/test.pig
>>> 2011-10-25 11:32:41,482 [main] INFO org.apache.pig.Main - Logging error
>>> messages to: /Users/vbarat/pig_1319535161481.log
>>> 2011-10-25 11:32:41,563 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
>>> to hadoop file system at: file:///
>>> 2011-10-25 11:32:41,884 [main] INFO
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>> *********************
>>> 2011-10-25 11:32:41,970 [main] INFO
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>> *********************
>>> 2011-10-25 11:32:42,035 [main] INFO
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>> *********************
>>> 2011-10-25 11:32:42,073 [main] INFO
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>> *********************
>>> 2011-10-25 11:32:42,184 [main] INFO
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>> *********************
>>> 2011-10-25 11:32:42,207 [main] INFO
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>> *********************
>>> 2011-10-25 11:32:42,233 [main] INFO
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>> *********************
>>> 2011-10-25 11:32:42,256 [main] INFO
>>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
>>> script: UNKNOWN
>>> 2011-10-25 11:32:42,317 [main] INFO
>>> org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
>>> processName=JobTracker, sessionId=
>>> 2011-10-25 11:32:42,374 [main] INFO
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>> *********************
>>> 2011-10-25 11:32:42,391 [main] INFO
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>>> *********************
>>> 2011-10-25 11:32:42,425 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
>>> File concatenation threshold: 100 optimistic? false
>>> 2011-10-25 11:32:42,449 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>>> - MR plan size before optimization: 1
>>>
>>> So HBaseStorage is create 10 times (and so the table is opened 9 times).
>>>
>>> I'd like to konw why so many creation ?
>>>
>>> Also, when I change my script to load 2 tables and join them, the
>>> HBaseStorage object is created 40 times !
>>>
>>> Can someone give me some insight to help me investigating the issue ?
>>>
>>> Thanks a lot
>>>
Re: PIG 0.9 leaks HBaseStorage instances
Posted by Vincent Barat <vi...@gmail.com>.
:-(
ok I understand. I don't see why the optimizer need to instantiate
the loaders, but I'm not a PIG specialist.
Concerning HBaseStorage, this is a big issue, as the number of cnx
to zookeeper is fairly limited (30 in the default configuration).
So a simple join of 2 tables would break this limit.
The finalize() method is of course not a solution, as it is called
only when the GC want to call it.
Anyway I'll try it just to get some insight.
Cheers,
Le 26/10/11 21:12, Dmitriy Ryaboy a écrit :
> unfortunately Pig creates UDFs willy-nilly to do various checks during
> script compilation.
>
> Maybe we should override the finalize() method and close the
> connection there, that would help.
>
> D
>
> On Tue, Oct 25, 2011 at 2:49 AM, Vincent Barat<vi...@gmail.com> wrote:
>> Hi,
>>
>> I try to figure out why PIG is using so many zookeeper connections (from the
>> frontend machine) when using HBaseStorage().
>>
>> I added a trace in the constructor of HBaseStorage()
>>
>> I wrote a simple script loading an HBase table:
>>
>> sessions = LOAD 'hbase://mytable' USING
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid meta:timestamp')
>> AS (sid:chararray, start:long);
>> dump sessions;
>>
>> When I run the script:
>>
>> vbarat@lancelot:~$ pig -x local -f /Users/vbarat/ermin/pig/script/test.pig
>> 2011-10-25 11:32:41,482 [main] INFO org.apache.pig.Main - Logging error
>> messages to: /Users/vbarat/pig_1319535161481.log
>> 2011-10-25 11:32:41,563 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
>> to hadoop file system at: file:///
>> 2011-10-25 11:32:41,884 [main] INFO
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>> *********************
>> 2011-10-25 11:32:41,970 [main] INFO
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>> *********************
>> 2011-10-25 11:32:42,035 [main] INFO
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>> *********************
>> 2011-10-25 11:32:42,073 [main] INFO
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>> *********************
>> 2011-10-25 11:32:42,184 [main] INFO
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>> *********************
>> 2011-10-25 11:32:42,207 [main] INFO
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>> *********************
>> 2011-10-25 11:32:42,233 [main] INFO
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>> *********************
>> 2011-10-25 11:32:42,256 [main] INFO
>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
>> script: UNKNOWN
>> 2011-10-25 11:32:42,317 [main] INFO
>> org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
>> processName=JobTracker, sessionId=
>> 2011-10-25 11:32:42,374 [main] INFO
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>> *********************
>> 2011-10-25 11:32:42,391 [main] INFO
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
>> *********************
>> 2011-10-25 11:32:42,425 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
>> File concatenation threshold: 100 optimistic? false
>> 2011-10-25 11:32:42,449 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>> - MR plan size before optimization: 1
>>
>> So HBaseStorage is create 10 times (and so the table is opened 9 times).
>>
>> I'd like to konw why so many creation ?
>>
>> Also, when I change my script to load 2 tables and join them, the
>> HBaseStorage object is created 40 times !
>>
>> Can someone give me some insight to help me investigating the issue ?
>>
>> Thanks a lot
>>
Re: PIG 0.9 leaks HBaseStorage instances
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
unfortunately Pig creates UDFs willy-nilly to do various checks during
script compilation.
Maybe we should override the finalize() method and close the
connection there, that would help.
D
On Tue, Oct 25, 2011 at 2:49 AM, Vincent Barat <vi...@gmail.com> wrote:
> Hi,
>
> I try to figure out why PIG is using so many zookeeper connections (from the
> frontend machine) when using HBaseStorage().
>
> I added a trace in the constructor of HBaseStorage()
>
> I wrote a simple script loading an HBase table:
>
> sessions = LOAD 'hbase://mytable' USING
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid meta:timestamp')
> AS (sid:chararray, start:long);
> dump sessions;
>
> When I run the script:
>
> vbarat@lancelot:~$ pig -x local -f /Users/vbarat/ermin/pig/script/test.pig
> 2011-10-25 11:32:41,482 [main] INFO org.apache.pig.Main - Logging error
> messages to: /Users/vbarat/pig_1319535161481.log
> 2011-10-25 11:32:41,563 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to hadoop file system at: file:///
> 2011-10-25 11:32:41,884 [main] INFO
> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
> *********************
> 2011-10-25 11:32:41,970 [main] INFO
> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
> *********************
> 2011-10-25 11:32:42,035 [main] INFO
> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
> *********************
> 2011-10-25 11:32:42,073 [main] INFO
> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
> *********************
> 2011-10-25 11:32:42,184 [main] INFO
> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
> *********************
> 2011-10-25 11:32:42,207 [main] INFO
> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
> *********************
> 2011-10-25 11:32:42,233 [main] INFO
> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
> *********************
> 2011-10-25 11:32:42,256 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2011-10-25 11:32:42,317 [main] INFO
> org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 2011-10-25 11:32:42,374 [main] INFO
> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
> *********************
> 2011-10-25 11:32:42,391 [main] INFO
> org.apache.pig.backend.hadoop.hbase.HBaseStorage - *********** HBASESTORAGE
> *********************
> 2011-10-25 11:32:42,425 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
> 2011-10-25 11:32:42,449 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
>
> So HBaseStorage is create 10 times (and so the table is opened 9 times).
>
> I'd like to konw why so many creation ?
>
> Also, when I change my script to load 2 tables and join them, the
> HBaseStorage object is created 40 times !
>
> Can someone give me some insight to help me investigating the issue ?
>
> Thanks a lot
>