You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Hari Prasanna <ha...@slideshare.com> on 2012/08/06 09:20:16 UTC
HBaseStorage looks for local zookeeper instance
Hi -
I've been having problems with running Pig(0.9.2) scripts with
HBase(0.92.1) as source and target.
I'm running a 3 node cluster. Node 1 has the NameNode, JobTracker,
Zookeeper server and HBase Master
Node 2 & 3 has the DataNode, TaskTracker and HBase region servers
Pig Script:
A = LOAD 'hbase://tb1' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf1:ssId_20120730
cf1:arId_20120730','-limit 10') AS (ss_id:int, ar_id:int);
B = GROUP A BY ss_id;
C = FOREACH B GENERATE group, COUNT(A) as hits;
STORE C INTO 'hbase://tbl2' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf1:20120730');
Env variables:
HADOOP_CLASSPATH=.:/usr/lib/hbase/hbase.jar:/etc/hbase/conf
PIG_CLASSPATH=/usr/lib/hbase/hbase.jar:/usr/lib/hbase
HBASE_HOME=/usr/lib/hbase
The Pig script is able to load the data from HBase(I tried 'dump A'
which worked fine). However, when trying to store in HBase, it is
trying to look for a local zookeeper instance at port 2181 and since
there are no zookeeper instances in the corresponding nodes, the map
tasks are being killed. I assumed that this is because pig does not
take the hbase.zookeeper.quorum setting in the hbase config; so I
tried setting the value in the grunt shell manually (set
hbase.zookeeper.quorum 'node1'). But, the script still looks for a
local zookeeper instance.
2012-08-06 01:44:02,367 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server localhost.localdomain/127.0.0.1:2181
2012-08-06 01:44:02,367 WARN
org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException:
java.lang.SecurityException: Unable to locate a login configuration
occurred when trying to find JAAS configuration.
2012-08-06 01:44:02,367 INFO
org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not
SASL-authenticate because the default JAAS configuration section
'Client' could not be found. If you are not using SASL, you may ignore
this. On the other hand, if you expected SASL to work, please fix your
JAAS configuration.
2012-08-06 01:44:02,368 WARN org.apache.zookeeper.ClientCnxn: Session
0x0 for server null, unexpected error, closing socket connection and
attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1039)
2012-08-06 01:44:02,470 WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
transient ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
2012-08-06 01:44:02,470 ERROR
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper
exists failed after 3 retries
2012-08-06 01:44:02,470 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
hconnection Unable to set watcher on znode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:580)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:569)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:186)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:194)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:195)
at org.apache.pig.backend.hadoop.hbase.HBaseStorage.getOutputFormat(HBaseStorage.java:521)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:91)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:70)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
at org.apache.hadoop.mapred.Task.initialize(Task.java:514)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:308)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
I also tried this -
http://mail-archives.apache.org/mod_mbox/pig-user/201205.mbox/%3C011b01cd2a11$43073990$c915acb0$@gmail.com%3E.
Doesn't work either. In the thread above, the LOAD operation did not
work. In my case, the LOAD operation works fine(when I dump it in the
grunt shell) but STORE doesn't.
Thanks,
Hari Prasanna
Re: HBaseStorage looks for local zookeeper instance
Posted by Hari Prasanna <ha...@slideshare.com>.
Hi Cheolsoo,
I am running CDH4. Please do send me that jar.
Thanks for your help
Hari
On Mon, Aug 6, 2012 at 10:25 PM, Cheolsoo Park <ch...@cloudera.com> wrote:
> Hi Hari,
>
> This is a known issue with Pig 0.9.2. Here is how you can fix it:
> 1) Rebuild your pig-withouthadoop.jar with the patch for PIG-2115.
> 2) Set "hbase.zookeeper.quorum" to "node1:2181" in
> /usr/lib/pig/conf/pig.properties.
>
> Please let me know if you're running CDH4. I can send you
> the pig-withouthadoop.jar that includes PIG-2115.
>
> Thanks!
> Cheolsoo
>
> On Mon, Aug 6, 2012 at 12:20 AM, Hari Prasanna <ha...@slideshare.com> wrote:
>
>> Hi -
>>
>> I've been having problems with running Pig(0.9.2) scripts with
>> HBase(0.92.1) as source and target.
>>
>> I'm running a 3 node cluster. Node 1 has the NameNode, JobTracker,
>> Zookeeper server and HBase Master
>> Node 2 & 3 has the DataNode, TaskTracker and HBase region servers
>>
>> Pig Script:
>>
>> A = LOAD 'hbase://tb1' USING
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf1:ssId_20120730
>> cf1:arId_20120730','-limit 10') AS (ss_id:int, ar_id:int);
>> B = GROUP A BY ss_id;
>> C = FOREACH B GENERATE group, COUNT(A) as hits;
>> STORE C INTO 'hbase://tbl2' USING
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf1:20120730');
>>
>> Env variables:
>>
>> HADOOP_CLASSPATH=.:/usr/lib/hbase/hbase.jar:/etc/hbase/conf
>> PIG_CLASSPATH=/usr/lib/hbase/hbase.jar:/usr/lib/hbase
>> HBASE_HOME=/usr/lib/hbase
>>
>>
>> The Pig script is able to load the data from HBase(I tried 'dump A'
>> which worked fine). However, when trying to store in HBase, it is
>> trying to look for a local zookeeper instance at port 2181 and since
>> there are no zookeeper instances in the corresponding nodes, the map
>> tasks are being killed. I assumed that this is because pig does not
>> take the hbase.zookeeper.quorum setting in the hbase config; so I
>> tried setting the value in the grunt shell manually (set
>> hbase.zookeeper.quorum 'node1'). But, the script still looks for a
>> local zookeeper instance.
>>
>>
>> 2012-08-06 01:44:02,367 INFO org.apache.zookeeper.ClientCnxn: Opening
>> socket connection to server localhost.localdomain/127.0.0.1:2181
>> 2012-08-06 01:44:02,367 WARN
>> org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException:
>> java.lang.SecurityException: Unable to locate a login configuration
>> occurred when trying to find JAAS configuration.
>> 2012-08-06 01:44:02,367 INFO
>> org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not
>> SASL-authenticate because the default JAAS configuration section
>> 'Client' could not be found. If you are not using SASL, you may ignore
>> this. On the other hand, if you expected SASL to work, please fix your
>> JAAS configuration.
>> 2012-08-06 01:44:02,368 WARN org.apache.zookeeper.ClientCnxn: Session
>> 0x0 for server null, unexpected error, closing socket connection and
>> attempting reconnect
>> java.net.ConnectException: Connection refused
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>> at
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
>> at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1039)
>> 2012-08-06 01:44:02,470 WARN
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
>> transient ZooKeeper exception:
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /hbase/master
>> 2012-08-06 01:44:02,470 ERROR
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper
>> exists failed after 3 retries
>> 2012-08-06 01:44:02,470 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
>> hconnection Unable to set watcher on znode /hbase/master
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /hbase/master
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
>> at
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:580)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:569)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:186)
>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:194)
>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
>> at
>> org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:195)
>> at
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage.getOutputFormat(HBaseStorage.java:521)
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:91)
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:70)
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
>> at org.apache.hadoop.mapred.Task.initialize(Task.java:514)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:308)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>>
>> I also tried this -
>>
>> http://mail-archives.apache.org/mod_mbox/pig-user/201205.mbox/%3C011b01cd2a11$43073990$c915acb0$@gmail.com%3E
>> .
>> Doesn't work either. In the thread above, the LOAD operation did not
>> work. In my case, the LOAD operation works fine(when I dump it in the
>> grunt shell) but STORE doesn't.
>>
>>
>> Thanks,
>> Hari Prasanna
>>
--
Hari
Re: HBaseStorage looks for local zookeeper instance
Posted by Cheolsoo Park <ch...@cloudera.com>.
Hi Hari,
This is a known issue with Pig 0.9.2. Here is how you can fix it:
1) Rebuild your pig-withouthadoop.jar with the patch for PIG-2115.
2) Set "hbase.zookeeper.quorum" to "node1:2181" in
/usr/lib/pig/conf/pig.properties.
Please let me know if you're running CDH4. I can send you
the pig-withouthadoop.jar that includes PIG-2115.
Thanks!
Cheolsoo
On Mon, Aug 6, 2012 at 12:20 AM, Hari Prasanna <ha...@slideshare.com> wrote:
> Hi -
>
> I've been having problems with running Pig(0.9.2) scripts with
> HBase(0.92.1) as source and target.
>
> I'm running a 3 node cluster. Node 1 has the NameNode, JobTracker,
> Zookeeper server and HBase Master
> Node 2 & 3 has the DataNode, TaskTracker and HBase region servers
>
> Pig Script:
>
> A = LOAD 'hbase://tb1' USING
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf1:ssId_20120730
> cf1:arId_20120730','-limit 10') AS (ss_id:int, ar_id:int);
> B = GROUP A BY ss_id;
> C = FOREACH B GENERATE group, COUNT(A) as hits;
> STORE C INTO 'hbase://tbl2' USING
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf1:20120730');
>
> Env variables:
>
> HADOOP_CLASSPATH=.:/usr/lib/hbase/hbase.jar:/etc/hbase/conf
> PIG_CLASSPATH=/usr/lib/hbase/hbase.jar:/usr/lib/hbase
> HBASE_HOME=/usr/lib/hbase
>
>
> The Pig script is able to load the data from HBase(I tried 'dump A'
> which worked fine). However, when trying to store in HBase, it is
> trying to look for a local zookeeper instance at port 2181 and since
> there are no zookeeper instances in the corresponding nodes, the map
> tasks are being killed. I assumed that this is because pig does not
> take the hbase.zookeeper.quorum setting in the hbase config; so I
> tried setting the value in the grunt shell manually (set
> hbase.zookeeper.quorum 'node1'). But, the script still looks for a
> local zookeeper instance.
>
>
> 2012-08-06 01:44:02,367 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server localhost.localdomain/127.0.0.1:2181
> 2012-08-06 01:44:02,367 WARN
> org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException:
> java.lang.SecurityException: Unable to locate a login configuration
> occurred when trying to find JAAS configuration.
> 2012-08-06 01:44:02,367 INFO
> org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not
> SASL-authenticate because the default JAAS configuration section
> 'Client' could not be found. If you are not using SASL, you may ignore
> this. On the other hand, if you expected SASL to work, please fix your
> JAAS configuration.
> 2012-08-06 01:44:02,368 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x0 for server null, unexpected error, closing socket connection and
> attempting reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1039)
> 2012-08-06 01:44:02,470 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly
> transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> 2012-08-06 01:44:02,470 ERROR
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper
> exists failed after 3 retries
> 2012-08-06 01:44:02,470 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> hconnection Unable to set watcher on znode /hbase/master
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:580)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:569)
> at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:186)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:194)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
> at
> org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:195)
> at
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.getOutputFormat(HBaseStorage.java:521)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:91)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:70)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
> at org.apache.hadoop.mapred.Task.initialize(Task.java:514)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:308)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>
> I also tried this -
>
> http://mail-archives.apache.org/mod_mbox/pig-user/201205.mbox/%3C011b01cd2a11$43073990$c915acb0$@gmail.com%3E
> .
> Doesn't work either. In the thread above, the LOAD operation did not
> work. In my case, the LOAD operation works fine(when I dump it in the
> grunt shell) but STORE doesn't.
>
>
> Thanks,
> Hari Prasanna
>