You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by byambajargal <by...@gmail.com> on 2011/04/24 19:40:16 UTC
What is the recommended way to get pig 0.8 to talk with CDH3u0 HBase
Hello guys
I am running cloudere distribution cdh3u0 on my cluster and i am trying
to connect pig with Hbase. I have 11 nodes on my cluster so i have
configured one machine as HBaseMaster and rest are Hbase Region Servers.
On my Master manshine:
1. Output of the command $*sudo jps is :*
6907 HMaster
14108 Jps
31470 QuorumPeerMain
22240 JobTracker
21602 SecondaryNameNode
21155 NameNode
2. Output of the command*$status 'detailed'*
-bash-3.2$ hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.90.1-cdh3u0, r, Fri Mar 25 16:10:51 PDT 2011
hbase(main):001:0> *status 'detailed'*
version 0.90.1-cdh3u0
0 regionsInTransition
10 live servers
haisen2.ux.uis.no:60020 1303287066114
requests=0, regions=1, usedHeap=34, maxHeap=987
-ROOT-,,0
stores=1, storefiles=1, storefileSizeMB=0,
memstoreSizeMB=0,
storefi
leIndexSizeMB=0
haisen10.ux.uis.no:60020 1303287733515
requests=0, regions=1, usedHeap=35, maxHeap=987
task1,,1303326469811.418e9f137f5b1d33604419ef6a8acb6e.
stores=1, storefiles=0, storefileSizeMB=0,
memstoreSizeMB=0,
storefi
leIndexSizeMB=0
haisen4.ux.uis.no:60020 1303287334273
requests=0, regions=0, usedHeap=33, maxHeap=987
haisen1.ux.uis.no:60020 1303643944066
requests=0, regions=0, usedHeap=35, maxHeap=987
haisen7.ux.uis.no:60020 1303287560071
requests=0, regions=1, usedHeap=32, maxHeap=987
test,,1303644907708.63ec7397e09d3335c8471aa90bcc5549.
stores=1, storefiles=0, storefileSizeMB=0,
memstoreSizeMB=0,
storefi
leIndexSizeMB=0
haisen9.ux.uis.no:60020 1303287672340
requests=0, regions=1, usedHeap=24, maxHeap=987
.META.,,1
stores=1, storefiles=1, storefileSizeMB=0,
memstoreSizeMB=0,
storefi
leIndexSizeMB=0
haisen5.ux.uis.no:60020 1303287407405
requests=0, regions=0, usedHeap=28, maxHeap=987
haisen6.ux.uis.no:60020 1303287480842
requests=0, regions=0, usedHeap=23, maxHeap=987
haisen3.ux.uis.no:60020 1303287189985
requests=0, regions=0, usedHeap=27, maxHeap=987
haisen8.ux.uis.no:60020 1303287626373
requests=0, regions=0, usedHeap=30, maxHeap=987
0 dead servers
I can put data into HBase and pull it out just fine by hbase shell
command and i have a problem when i load data from hbase by using the
following Pig query:
*my_data = LOAD 'hbase://test' using org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as (a:chararray);dump my_data;*
i got a error message and my job was failure. If someone could help me to connect hbase with pig that would be great.
the problem I am having now is the following exception on the pig log file:
Backend error message
---------------------
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:293)
at
org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
at
org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:302)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:989)
... 15 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
... 16 more
Pig Stack Trace
---------------
ERROR 2997: Unable to recreate exception from backed error:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
to open iterator for alias my_data. Backend error : Unable to recreate
exception from backed error:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.pig.PigServer.openIterator(PigServer.java:742)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
Thanks for helping
Byambajargal
Re: What is the recommended way to get pig 0.8 to talk with CDH3u0
HBase
Posted by byambajargal <by...@gmail.com>.
Thank you guys and Bill Graham
I have solved the problem.
I just add the following lines of shell to conf/hadoop-env.sh
# if using HBase, likely want to include HBase config
HBASE_CONF_DIR=${HBASE_CONF_DIR:-/etc/hbase/conf}
if [ -n "$HBASE_CONF_DIR" ] && [ -d "$HBASE_CONF_DIR" ]; then
export HADOOP_CLASSPATH=$HBASE_CONF_DIR:$HADOOP_CLASSPATH
fi
it works fine
thanks
Byambajargal
On 4/24/11 23:45, Bill Graham wrote:
> I had this issue and had to add the HBase conf dir to HADOOP_CLASSPATH
> in conf/hadoop-env.sh on each of the nodes in the cluster so they
> could find Zookeeper.
>
>
> On Sun, Apr 24, 2011 at 1:04 PM, Dmitriy Ryaboy<dv...@gmail.com> wrote:
>> I suspect the problem here is that you don't have your hbase config
>> directory on Pig's classpath. You can add it by modifying the PIG_CLASSPATH
>> environment variable.
>>
>> D
>>
>> On Sun, Apr 24, 2011 at 12:41 PM, sulabh choudhury<su...@gmail.com>wrote:
>>
>>> Have you tried loading like :-
>>>
>>> *my_data = LOAD 'hbase://test' using
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
>>> *(a:bytearray)*;dump my_data;*
>>> or
>>> *my_data = LOAD 'hbase://test' using
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
>>> (a);dump my_data;*
>>>
>>> I am just trying to see if it works if you supply a different datatype ,
>>> bytearray instead of chararray, or without any datatype at all.
>>>
>>> On Sun, Apr 24, 2011 at 10:40 AM, byambajargal<byambaa.0111@gmail.com
>>>> wrote:
>>>> Hello guys
>>>>
>>>> I am running cloudere distribution cdh3u0 on my cluster and i am trying
>>> to
>>>> connect pig with Hbase. I have 11 nodes on my cluster so i have
>>> configured
>>>> one machine as HBaseMaster and rest are Hbase Region Servers.
>>>> On my Master manshine:
>>>>
>>>> 1. Output of the command $*sudo jps is :*
>>>>
>>>> 6907 HMaster
>>>> 14108 Jps
>>>> 31470 QuorumPeerMain
>>>> 22240 JobTracker
>>>> 21602 SecondaryNameNode
>>>> 21155 NameNode
>>>>
>>>> 2. Output of the command*$status 'detailed'*
>>>>
>>>> -bash-3.2$ hbase shell
>>>> HBase Shell; enter 'help<RETURN>' for list of supported commands.
>>>> Type "exit<RETURN>" to leave the HBase Shell
>>>> Version 0.90.1-cdh3u0, r, Fri Mar 25 16:10:51 PDT 2011
>>>>
>>>> hbase(main):001:0> *status 'detailed'*
>>>> version 0.90.1-cdh3u0
>>>> 0 regionsInTransition
>>>> 10 live servers
>>>> haisen2.ux.uis.no:60020 1303287066114
>>>> requests=0, regions=1, usedHeap=34, maxHeap=987
>>>> -ROOT-,,0
>>>> stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
>>>> storefi
>>>> leIndexSizeMB=0
>>>> haisen10.ux.uis.no:60020 1303287733515
>>>> requests=0, regions=1, usedHeap=35, maxHeap=987
>>>> task1,,1303326469811.418e9f137f5b1d33604419ef6a8acb6e.
>>>> stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
>>>> storefi
>>>> leIndexSizeMB=0
>>>> haisen4.ux.uis.no:60020 1303287334273
>>>> requests=0, regions=0, usedHeap=33, maxHeap=987
>>>> haisen1.ux.uis.no:60020 1303643944066
>>>> requests=0, regions=0, usedHeap=35, maxHeap=987
>>>> haisen7.ux.uis.no:60020 1303287560071
>>>> requests=0, regions=1, usedHeap=32, maxHeap=987
>>>> test,,1303644907708.63ec7397e09d3335c8471aa90bcc5549.
>>>> stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
>>>> storefi
>>>> leIndexSizeMB=0
>>>> haisen9.ux.uis.no:60020 1303287672340
>>>> requests=0, regions=1, usedHeap=24, maxHeap=987
>>>> .META.,,1
>>>> stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
>>>> storefi
>>>> leIndexSizeMB=0
>>>> haisen5.ux.uis.no:60020 1303287407405
>>>> requests=0, regions=0, usedHeap=28, maxHeap=987
>>>> haisen6.ux.uis.no:60020 1303287480842
>>>> requests=0, regions=0, usedHeap=23, maxHeap=987
>>>> haisen3.ux.uis.no:60020 1303287189985
>>>> requests=0, regions=0, usedHeap=27, maxHeap=987
>>>> haisen8.ux.uis.no:60020 1303287626373
>>>> requests=0, regions=0, usedHeap=30, maxHeap=987
>>>> 0 dead servers
>>>>
>>>>
>>>> I can put data into HBase and pull it out just fine by hbase shell
>>> command
>>>> and i have a problem when i load data from hbase by using the following
>>> Pig
>>>> query:
>>>>
>>>> *my_data = LOAD 'hbase://test' using
>>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100')
>>> as
>>>> (a:chararray);dump my_data;*
>>>>
>>>> i got a error message and my job was failure. If someone could help me to
>>>> connect hbase with pig that would be great.
>>>>
>>>>
>>>> the problem I am having now is the following exception on the pig log
>>> file:
>>>> Backend error message
>>>> ---------------------
>>>> org.apache.hadoop.hbase.ZooKeeperConnectionException:
>>>> org.apache.hadoop.hbase.ZooKeeperConnectionException:
>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> KeeperErrorCode = ConnectionLoss for /hbase
>>>> at
>>>>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991)
>>>> at
>>>>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302)
>>>> at
>>>>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:293)
>>>> at
>>>>
>>> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
>>>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
>>>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
>>>> at
>>>>
>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:302)
>>>> at
>>>>
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
>>>> at
>>>>
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
>>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>>
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
>>>> at org.apache.hadoop.mapred.Child.main(Child.java:262)
>>>> Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> KeeperErrorCode = ConnectionLoss for /hbase
>>>> at
>>>>
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147)
>>>> at
>>>>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:989)
>>>> ... 15 more
>>>> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> KeeperErrorCode = ConnectionLoss for /hbase
>>>> at
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>>>> at
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>>>> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
>>>> at
>>>>
>>> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
>>>> at
>>>>
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
>>>> ... 16 more
>>>>
>>>> Pig Stack Trace
>>>> ---------------
>>>> ERROR 2997: Unable to recreate exception from backed error:
>>>> org.apache.hadoop.hbase.ZooKeeperConnectionException:
>>>> org.apache.hadoop.hbase.ZooKeeperConnectionException:
>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> KeeperErrorCode = ConnectionLoss for /hbase
>>>>
>>>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
>>>> open iterator for alias my_data. Backend error : Unable to recreate
>>>> exception from backed error:
>>>> org.apache.hadoop.hbase.ZooKeeperConnectionException:
>>>> org.apache.hadoop.hbase.ZooKeeperConnectionException:
>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> KeeperErrorCode = ConnectionLoss for /hbase
>>>> at org.apache.pig.PigServer.openIterator(PigServer.java:742)
>>>> at
>>>> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
>>>> at
>>>>
>>> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
>>>> at
>>>>
>>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
>>>>
>>>>
>>>>
>>>> Thanks for helping
>>>>
>>>>
>>>> Byambajargal
>>>>
>>>>
>>>
>>> --
>>>
>>> --
>>> Thanks and Regards,
>>> Sulabh Choudhury
>>>
Re: What is the recommended way to get pig 0.8 to talk with CDH3u0 HBase
Posted by Bill Graham <bi...@gmail.com>.
I had this issue and had to add the HBase conf dir to HADOOP_CLASSPATH
in conf/hadoop-env.sh on each of the nodes in the cluster so they
could find Zookeeper.
On Sun, Apr 24, 2011 at 1:04 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> I suspect the problem here is that you don't have your hbase config
> directory on Pig's classpath. You can add it by modifying the PIG_CLASSPATH
> environment variable.
>
> D
>
> On Sun, Apr 24, 2011 at 12:41 PM, sulabh choudhury <su...@gmail.com>wrote:
>
>> Have you tried loading like :-
>>
>> *my_data = LOAD 'hbase://test' using
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
>> *(a:bytearray)*;dump my_data;*
>> or
>> *my_data = LOAD 'hbase://test' using
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
>> (a);dump my_data;*
>>
>> I am just trying to see if it works if you supply a different datatype ,
>> bytearray instead of chararray, or without any datatype at all.
>>
>> On Sun, Apr 24, 2011 at 10:40 AM, byambajargal <byambaa.0111@gmail.com
>> >wrote:
>>
>> > Hello guys
>> >
>> > I am running cloudere distribution cdh3u0 on my cluster and i am trying
>> to
>> > connect pig with Hbase. I have 11 nodes on my cluster so i have
>> configured
>> > one machine as HBaseMaster and rest are Hbase Region Servers.
>> > On my Master manshine:
>> >
>> > 1. Output of the command $*sudo jps is :*
>> >
>> > 6907 HMaster
>> > 14108 Jps
>> > 31470 QuorumPeerMain
>> > 22240 JobTracker
>> > 21602 SecondaryNameNode
>> > 21155 NameNode
>> >
>> > 2. Output of the command*$status 'detailed'*
>> >
>> > -bash-3.2$ hbase shell
>> > HBase Shell; enter 'help<RETURN>' for list of supported commands.
>> > Type "exit<RETURN>" to leave the HBase Shell
>> > Version 0.90.1-cdh3u0, r, Fri Mar 25 16:10:51 PDT 2011
>> >
>> > hbase(main):001:0> *status 'detailed'*
>> > version 0.90.1-cdh3u0
>> > 0 regionsInTransition
>> > 10 live servers
>> > haisen2.ux.uis.no:60020 1303287066114
>> > requests=0, regions=1, usedHeap=34, maxHeap=987
>> > -ROOT-,,0
>> > stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
>> > storefi
>> > leIndexSizeMB=0
>> > haisen10.ux.uis.no:60020 1303287733515
>> > requests=0, regions=1, usedHeap=35, maxHeap=987
>> > task1,,1303326469811.418e9f137f5b1d33604419ef6a8acb6e.
>> > stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
>> > storefi
>> > leIndexSizeMB=0
>> > haisen4.ux.uis.no:60020 1303287334273
>> > requests=0, regions=0, usedHeap=33, maxHeap=987
>> > haisen1.ux.uis.no:60020 1303643944066
>> > requests=0, regions=0, usedHeap=35, maxHeap=987
>> > haisen7.ux.uis.no:60020 1303287560071
>> > requests=0, regions=1, usedHeap=32, maxHeap=987
>> > test,,1303644907708.63ec7397e09d3335c8471aa90bcc5549.
>> > stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
>> > storefi
>> > leIndexSizeMB=0
>> > haisen9.ux.uis.no:60020 1303287672340
>> > requests=0, regions=1, usedHeap=24, maxHeap=987
>> > .META.,,1
>> > stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
>> > storefi
>> > leIndexSizeMB=0
>> > haisen5.ux.uis.no:60020 1303287407405
>> > requests=0, regions=0, usedHeap=28, maxHeap=987
>> > haisen6.ux.uis.no:60020 1303287480842
>> > requests=0, regions=0, usedHeap=23, maxHeap=987
>> > haisen3.ux.uis.no:60020 1303287189985
>> > requests=0, regions=0, usedHeap=27, maxHeap=987
>> > haisen8.ux.uis.no:60020 1303287626373
>> > requests=0, regions=0, usedHeap=30, maxHeap=987
>> > 0 dead servers
>> >
>> >
>> > I can put data into HBase and pull it out just fine by hbase shell
>> command
>> > and i have a problem when i load data from hbase by using the following
>> Pig
>> > query:
>> >
>> > *my_data = LOAD 'hbase://test' using
>> > org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100')
>> as
>> > (a:chararray);dump my_data;*
>> >
>> > i got a error message and my job was failure. If someone could help me to
>> > connect hbase with pig that would be great.
>> >
>> >
>> > the problem I am having now is the following exception on the pig log
>> file:
>> >
>> > Backend error message
>> > ---------------------
>> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /hbase
>> > at
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991)
>> > at
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302)
>> > at
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:293)
>> > at
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
>> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
>> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
>> > at
>> >
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:302)
>> > at
>> >
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
>> > at
>> >
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
>> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
>> > at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
>> > at org.apache.hadoop.mapred.Child.main(Child.java:262)
>> > Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /hbase
>> > at
>> >
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147)
>> > at
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:989)
>> > ... 15 more
>> > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /hbase
>> > at
>> > org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>> > at
>> > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>> > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
>> > at
>> >
>> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
>> > at
>> >
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
>> > ... 16 more
>> >
>> > Pig Stack Trace
>> > ---------------
>> > ERROR 2997: Unable to recreate exception from backed error:
>> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /hbase
>> >
>> > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
>> > open iterator for alias my_data. Backend error : Unable to recreate
>> > exception from backed error:
>> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> > org.apache.zookeeper.KeeperException$ConnectionLossException:
>> > KeeperErrorCode = ConnectionLoss for /hbase
>> > at org.apache.pig.PigServer.openIterator(PigServer.java:742)
>> > at
>> > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
>> > at
>> >
>> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
>> > at
>> >
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
>> >
>> >
>> >
>> >
>> > Thanks for helping
>> >
>> >
>> > Byambajargal
>> >
>> >
>>
>>
>> --
>>
>> --
>> Thanks and Regards,
>> Sulabh Choudhury
>>
>
Re: What is the recommended way to get pig 0.8 to talk with CDH3u0 HBase
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
I suspect the problem here is that you don't have your hbase config
directory on Pig's classpath. You can add it by modifying the PIG_CLASSPATH
environment variable.
D
On Sun, Apr 24, 2011 at 12:41 PM, sulabh choudhury <su...@gmail.com>wrote:
> Have you tried loading like :-
>
> *my_data = LOAD 'hbase://test' using
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
> *(a:bytearray)*;dump my_data;*
> or
> *my_data = LOAD 'hbase://test' using
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
> (a);dump my_data;*
>
> I am just trying to see if it works if you supply a different datatype ,
> bytearray instead of chararray, or without any datatype at all.
>
> On Sun, Apr 24, 2011 at 10:40 AM, byambajargal <byambaa.0111@gmail.com
> >wrote:
>
> > Hello guys
> >
> > I am running cloudere distribution cdh3u0 on my cluster and i am trying
> to
> > connect pig with Hbase. I have 11 nodes on my cluster so i have
> configured
> > one machine as HBaseMaster and rest are Hbase Region Servers.
> > On my Master manshine:
> >
> > 1. Output of the command $*sudo jps is :*
> >
> > 6907 HMaster
> > 14108 Jps
> > 31470 QuorumPeerMain
> > 22240 JobTracker
> > 21602 SecondaryNameNode
> > 21155 NameNode
> >
> > 2. Output of the command*$status 'detailed'*
> >
> > -bash-3.2$ hbase shell
> > HBase Shell; enter 'help<RETURN>' for list of supported commands.
> > Type "exit<RETURN>" to leave the HBase Shell
> > Version 0.90.1-cdh3u0, r, Fri Mar 25 16:10:51 PDT 2011
> >
> > hbase(main):001:0> *status 'detailed'*
> > version 0.90.1-cdh3u0
> > 0 regionsInTransition
> > 10 live servers
> > haisen2.ux.uis.no:60020 1303287066114
> > requests=0, regions=1, usedHeap=34, maxHeap=987
> > -ROOT-,,0
> > stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
> > storefi
> > leIndexSizeMB=0
> > haisen10.ux.uis.no:60020 1303287733515
> > requests=0, regions=1, usedHeap=35, maxHeap=987
> > task1,,1303326469811.418e9f137f5b1d33604419ef6a8acb6e.
> > stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
> > storefi
> > leIndexSizeMB=0
> > haisen4.ux.uis.no:60020 1303287334273
> > requests=0, regions=0, usedHeap=33, maxHeap=987
> > haisen1.ux.uis.no:60020 1303643944066
> > requests=0, regions=0, usedHeap=35, maxHeap=987
> > haisen7.ux.uis.no:60020 1303287560071
> > requests=0, regions=1, usedHeap=32, maxHeap=987
> > test,,1303644907708.63ec7397e09d3335c8471aa90bcc5549.
> > stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
> > storefi
> > leIndexSizeMB=0
> > haisen9.ux.uis.no:60020 1303287672340
> > requests=0, regions=1, usedHeap=24, maxHeap=987
> > .META.,,1
> > stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
> > storefi
> > leIndexSizeMB=0
> > haisen5.ux.uis.no:60020 1303287407405
> > requests=0, regions=0, usedHeap=28, maxHeap=987
> > haisen6.ux.uis.no:60020 1303287480842
> > requests=0, regions=0, usedHeap=23, maxHeap=987
> > haisen3.ux.uis.no:60020 1303287189985
> > requests=0, regions=0, usedHeap=27, maxHeap=987
> > haisen8.ux.uis.no:60020 1303287626373
> > requests=0, regions=0, usedHeap=30, maxHeap=987
> > 0 dead servers
> >
> >
> > I can put data into HBase and pull it out just fine by hbase shell
> command
> > and i have a problem when i load data from hbase by using the following
> Pig
> > query:
> >
> > *my_data = LOAD 'hbase://test' using
> > org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100')
> as
> > (a:chararray);dump my_data;*
> >
> > i got a error message and my job was failure. If someone could help me to
> > connect hbase with pig that would be great.
> >
> >
> > the problem I am having now is the following exception on the pig log
> file:
> >
> > Backend error message
> > ---------------------
> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:293)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
> > at
> >
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:302)
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
> > at org.apache.hadoop.mapred.Child.main(Child.java:262)
> > Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:989)
> > ... 15 more
> > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
> > ... 16 more
> >
> > Pig Stack Trace
> > ---------------
> > ERROR 2997: Unable to recreate exception from backed error:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase
> >
> > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> > open iterator for alias my_data. Backend error : Unable to recreate
> > exception from backed error:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase
> > at org.apache.pig.PigServer.openIterator(PigServer.java:742)
> > at
> > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
> > at
> >
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> > at
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> >
> >
> >
> >
> > Thanks for helping
> >
> >
> > Byambajargal
> >
> >
>
>
> --
>
> --
> Thanks and Regards,
> Sulabh Choudhury
>
Re: What is the recommended way to get pig 0.8 to talk with CDH3u0 HBase
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
I suspect the problem here is that you don't have your hbase config
directory on Pig's classpath. You can add it by modifying the PIG_CLASSPATH
environment variable.
D
On Sun, Apr 24, 2011 at 12:41 PM, sulabh choudhury <su...@gmail.com>wrote:
> Have you tried loading like :-
>
> *my_data = LOAD 'hbase://test' using
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
> *(a:bytearray)*;dump my_data;*
> or
> *my_data = LOAD 'hbase://test' using
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
> (a);dump my_data;*
>
> I am just trying to see if it works if you supply a different datatype ,
> bytearray instead of chararray, or without any datatype at all.
>
> On Sun, Apr 24, 2011 at 10:40 AM, byambajargal <byambaa.0111@gmail.com
> >wrote:
>
> > Hello guys
> >
> > I am running cloudere distribution cdh3u0 on my cluster and i am trying
> to
> > connect pig with Hbase. I have 11 nodes on my cluster so i have
> configured
> > one machine as HBaseMaster and rest are Hbase Region Servers.
> > On my Master manshine:
> >
> > 1. Output of the command $*sudo jps is :*
> >
> > 6907 HMaster
> > 14108 Jps
> > 31470 QuorumPeerMain
> > 22240 JobTracker
> > 21602 SecondaryNameNode
> > 21155 NameNode
> >
> > 2. Output of the command*$status 'detailed'*
> >
> > -bash-3.2$ hbase shell
> > HBase Shell; enter 'help<RETURN>' for list of supported commands.
> > Type "exit<RETURN>" to leave the HBase Shell
> > Version 0.90.1-cdh3u0, r, Fri Mar 25 16:10:51 PDT 2011
> >
> > hbase(main):001:0> *status 'detailed'*
> > version 0.90.1-cdh3u0
> > 0 regionsInTransition
> > 10 live servers
> > haisen2.ux.uis.no:60020 1303287066114
> > requests=0, regions=1, usedHeap=34, maxHeap=987
> > -ROOT-,,0
> > stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
> > storefi
> > leIndexSizeMB=0
> > haisen10.ux.uis.no:60020 1303287733515
> > requests=0, regions=1, usedHeap=35, maxHeap=987
> > task1,,1303326469811.418e9f137f5b1d33604419ef6a8acb6e.
> > stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
> > storefi
> > leIndexSizeMB=0
> > haisen4.ux.uis.no:60020 1303287334273
> > requests=0, regions=0, usedHeap=33, maxHeap=987
> > haisen1.ux.uis.no:60020 1303643944066
> > requests=0, regions=0, usedHeap=35, maxHeap=987
> > haisen7.ux.uis.no:60020 1303287560071
> > requests=0, regions=1, usedHeap=32, maxHeap=987
> > test,,1303644907708.63ec7397e09d3335c8471aa90bcc5549.
> > stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
> > storefi
> > leIndexSizeMB=0
> > haisen9.ux.uis.no:60020 1303287672340
> > requests=0, regions=1, usedHeap=24, maxHeap=987
> > .META.,,1
> > stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
> > storefi
> > leIndexSizeMB=0
> > haisen5.ux.uis.no:60020 1303287407405
> > requests=0, regions=0, usedHeap=28, maxHeap=987
> > haisen6.ux.uis.no:60020 1303287480842
> > requests=0, regions=0, usedHeap=23, maxHeap=987
> > haisen3.ux.uis.no:60020 1303287189985
> > requests=0, regions=0, usedHeap=27, maxHeap=987
> > haisen8.ux.uis.no:60020 1303287626373
> > requests=0, regions=0, usedHeap=30, maxHeap=987
> > 0 dead servers
> >
> >
> > I can put data into HBase and pull it out just fine by hbase shell
> command
> > and i have a problem when i load data from hbase by using the following
> Pig
> > query:
> >
> > *my_data = LOAD 'hbase://test' using
> > org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100')
> as
> > (a:chararray);dump my_data;*
> >
> > i got a error message and my job was failure. If someone could help me to
> > connect hbase with pig that would be great.
> >
> >
> > the problem I am having now is the following exception on the pig log
> file:
> >
> > Backend error message
> > ---------------------
> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:293)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
> > at
> >
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:302)
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
> > at org.apache.hadoop.mapred.Child.main(Child.java:262)
> > Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:989)
> > ... 15 more
> > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
> > ... 16 more
> >
> > Pig Stack Trace
> > ---------------
> > ERROR 2997: Unable to recreate exception from backed error:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase
> >
> > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> > open iterator for alias my_data. Backend error : Unable to recreate
> > exception from backed error:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase
> > at org.apache.pig.PigServer.openIterator(PigServer.java:742)
> > at
> > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
> > at
> >
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> > at
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> >
> >
> >
> >
> > Thanks for helping
> >
> >
> > Byambajargal
> >
> >
>
>
> --
>
> --
> Thanks and Regards,
> Sulabh Choudhury
>
Re: What is the recommended way to get pig 0.8 to talk with CDH3u0 HBase
Posted by sulabh choudhury <su...@gmail.com>.
Have you tried loading like :-
*my_data = LOAD 'hbase://test' using
org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
*(a:bytearray)*;dump my_data;*
or
*my_data = LOAD 'hbase://test' using
org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
(a);dump my_data;*
I am just trying to see if it works if you supply a different datatype ,
bytearray instead of chararray, or without any datatype at all.
On Sun, Apr 24, 2011 at 10:40 AM, byambajargal <by...@gmail.com>wrote:
> Hello guys
>
> I am running cloudere distribution cdh3u0 on my cluster and i am trying to
> connect pig with Hbase. I have 11 nodes on my cluster so i have configured
> one machine as HBaseMaster and rest are Hbase Region Servers.
> On my Master manshine:
>
> 1. Output of the command $*sudo jps is :*
>
> 6907 HMaster
> 14108 Jps
> 31470 QuorumPeerMain
> 22240 JobTracker
> 21602 SecondaryNameNode
> 21155 NameNode
>
> 2. Output of the command*$status 'detailed'*
>
> -bash-3.2$ hbase shell
> HBase Shell; enter 'help<RETURN>' for list of supported commands.
> Type "exit<RETURN>" to leave the HBase Shell
> Version 0.90.1-cdh3u0, r, Fri Mar 25 16:10:51 PDT 2011
>
> hbase(main):001:0> *status 'detailed'*
> version 0.90.1-cdh3u0
> 0 regionsInTransition
> 10 live servers
> haisen2.ux.uis.no:60020 1303287066114
> requests=0, regions=1, usedHeap=34, maxHeap=987
> -ROOT-,,0
> stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
> storefi
> leIndexSizeMB=0
> haisen10.ux.uis.no:60020 1303287733515
> requests=0, regions=1, usedHeap=35, maxHeap=987
> task1,,1303326469811.418e9f137f5b1d33604419ef6a8acb6e.
> stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefi
> leIndexSizeMB=0
> haisen4.ux.uis.no:60020 1303287334273
> requests=0, regions=0, usedHeap=33, maxHeap=987
> haisen1.ux.uis.no:60020 1303643944066
> requests=0, regions=0, usedHeap=35, maxHeap=987
> haisen7.ux.uis.no:60020 1303287560071
> requests=0, regions=1, usedHeap=32, maxHeap=987
> test,,1303644907708.63ec7397e09d3335c8471aa90bcc5549.
> stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefi
> leIndexSizeMB=0
> haisen9.ux.uis.no:60020 1303287672340
> requests=0, regions=1, usedHeap=24, maxHeap=987
> .META.,,1
> stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
> storefi
> leIndexSizeMB=0
> haisen5.ux.uis.no:60020 1303287407405
> requests=0, regions=0, usedHeap=28, maxHeap=987
> haisen6.ux.uis.no:60020 1303287480842
> requests=0, regions=0, usedHeap=23, maxHeap=987
> haisen3.ux.uis.no:60020 1303287189985
> requests=0, regions=0, usedHeap=27, maxHeap=987
> haisen8.ux.uis.no:60020 1303287626373
> requests=0, regions=0, usedHeap=30, maxHeap=987
> 0 dead servers
>
>
> I can put data into HBase and pull it out just fine by hbase shell command
> and i have a problem when i load data from hbase by using the following Pig
> query:
>
> *my_data = LOAD 'hbase://test' using
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
> (a:chararray);dump my_data;*
>
> i got a error message and my job was failure. If someone could help me to
> connect hbase with pig that would be great.
>
>
> the problem I am having now is the following exception on the pig log file:
>
> Backend error message
> ---------------------
> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:293)
> at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
> at
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:302)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:989)
> ... 15 more
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
> ... 16 more
>
> Pig Stack Trace
> ---------------
> ERROR 2997: Unable to recreate exception from backed error:
> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> open iterator for alias my_data. Backend error : Unable to recreate
> exception from backed error:
> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at org.apache.pig.PigServer.openIterator(PigServer.java:742)
> at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
>
>
>
>
> Thanks for helping
>
>
> Byambajargal
>
>
--
--
Thanks and Regards,
Sulabh Choudhury
Re: What is the recommended way to get pig 0.8 to talk with CDH3u0 HBase
Posted by sulabh choudhury <su...@gmail.com>.
Have you tried loading like :-
*my_data = LOAD 'hbase://test' using
org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
*(a:bytearray)*;dump my_data;*
or
*my_data = LOAD 'hbase://test' using
org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
(a);dump my_data;*
I am just trying to see if it works if you supply a different datatype ,
bytearray instead of chararray, or without any datatype at all.
On Sun, Apr 24, 2011 at 10:40 AM, byambajargal <by...@gmail.com>wrote:
> Hello guys
>
> I am running cloudere distribution cdh3u0 on my cluster and i am trying to
> connect pig with Hbase. I have 11 nodes on my cluster so i have configured
> one machine as HBaseMaster and rest are Hbase Region Servers.
> On my Master manshine:
>
> 1. Output of the command $*sudo jps is :*
>
> 6907 HMaster
> 14108 Jps
> 31470 QuorumPeerMain
> 22240 JobTracker
> 21602 SecondaryNameNode
> 21155 NameNode
>
> 2. Output of the command*$status 'detailed'*
>
> -bash-3.2$ hbase shell
> HBase Shell; enter 'help<RETURN>' for list of supported commands.
> Type "exit<RETURN>" to leave the HBase Shell
> Version 0.90.1-cdh3u0, r, Fri Mar 25 16:10:51 PDT 2011
>
> hbase(main):001:0> *status 'detailed'*
> version 0.90.1-cdh3u0
> 0 regionsInTransition
> 10 live servers
> haisen2.ux.uis.no:60020 1303287066114
> requests=0, regions=1, usedHeap=34, maxHeap=987
> -ROOT-,,0
> stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
> storefi
> leIndexSizeMB=0
> haisen10.ux.uis.no:60020 1303287733515
> requests=0, regions=1, usedHeap=35, maxHeap=987
> task1,,1303326469811.418e9f137f5b1d33604419ef6a8acb6e.
> stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefi
> leIndexSizeMB=0
> haisen4.ux.uis.no:60020 1303287334273
> requests=0, regions=0, usedHeap=33, maxHeap=987
> haisen1.ux.uis.no:60020 1303643944066
> requests=0, regions=0, usedHeap=35, maxHeap=987
> haisen7.ux.uis.no:60020 1303287560071
> requests=0, regions=1, usedHeap=32, maxHeap=987
> test,,1303644907708.63ec7397e09d3335c8471aa90bcc5549.
> stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
> storefi
> leIndexSizeMB=0
> haisen9.ux.uis.no:60020 1303287672340
> requests=0, regions=1, usedHeap=24, maxHeap=987
> .META.,,1
> stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
> storefi
> leIndexSizeMB=0
> haisen5.ux.uis.no:60020 1303287407405
> requests=0, regions=0, usedHeap=28, maxHeap=987
> haisen6.ux.uis.no:60020 1303287480842
> requests=0, regions=0, usedHeap=23, maxHeap=987
> haisen3.ux.uis.no:60020 1303287189985
> requests=0, regions=0, usedHeap=27, maxHeap=987
> haisen8.ux.uis.no:60020 1303287626373
> requests=0, regions=0, usedHeap=30, maxHeap=987
> 0 dead servers
>
>
> I can put data into HBase and pull it out just fine by hbase shell command
> and i have a problem when i load data from hbase by using the following Pig
> query:
>
> *my_data = LOAD 'hbase://test' using
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
> (a:chararray);dump my_data;*
>
> i got a error message and my job was failure. If someone could help me to
> connect hbase with pig that would be great.
>
>
> the problem I am having now is the following exception on the pig log file:
>
> Backend error message
> ---------------------
> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:293)
> at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
> at
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:302)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:989)
> ... 15 more
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
> ... 16 more
>
> Pig Stack Trace
> ---------------
> ERROR 2997: Unable to recreate exception from backed error:
> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> open iterator for alias my_data. Backend error : Unable to recreate
> exception from backed error:
> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at org.apache.pig.PigServer.openIterator(PigServer.java:742)
> at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
>
>
>
>
> Thanks for helping
>
>
> Byambajargal
>
>
--
--
Thanks and Regards,
Sulabh Choudhury