You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by praveenesh kumar <pr...@gmail.com> on 2011/07/05 11:39:18 UTC
how to check whether zookeeper quorum is working fine or not ?
Hi,
I have a 12-node hbase cluster setup with 3 nodes as a part of zookeeper
quorum.
I am able to run hbase shell, create tables.. and able to access tables in
the shell.
Now I am configuring pig to use hbase. While accessing records using pig, it
is giving some Zookeeper exception and saying to check the zookeeper logs.
I am sending the latest zookeeper logs of the 3 nodes along with
hbase-site.xml
Can anyone help me figuring out whether everything is okay with my zookeeper
and if not, what is the issue ?
Hbase-site.xml ---> http://pastebin.com/8aJ7D54T
Zookeeper log on ub11 ---> http://pastebin.com/HMuL9aCJ
Zookeeper log on ub12 ---> http://pastebin.com/8XdmVmDW
Zookeeper log on ub13 ---> http://pastebin.com/2373Rrat
Thanks,
Praveenesh
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by Stack <st...@duboce.net>.
hmmm... I just took a look and I don't see zk dumping it out anywhere.
Could run a script that puts up 2k -1 clients and see how far you
get? (Maybe someone else has better idea).
St.Ack
On Fri, Jul 15, 2011 at 3:22 AM, praveenesh kumar <pr...@gmail.com> wrote:
> I am not able to see the total number of connections as 2k.
> Here is my zookeeper output :
>
> HBase is rooted at /hbase
> Master address: ub13:60000
> Region server holding ROOT: ub5:60020
> Region servers:
> ub8:60020
> ub12:60020
> ub11:60020
> ub4:60020
> ub2:60020
> ub13:60020
> ub7:60020
> ub3:60020
> ub10:60020
> ub1:60020
> ub9:60020
> ub5:60020
> Quorum Server Statistics:
> ub11:2181
> Zookeeper version: 3.3.2-1031432, built on 11/05/2010 05:32 GMT
> Clients:
> /162.192.100.53:51589[1](queued=0,recved=24,sent=24)
> /162.192.100.53:51591[0](queued=0,recved=1,sent=0)
>
> Latency min/avg/max: 0/1/22
> Received: 25
> Sent: 24
> Outstanding: 0
> Zxid: 0x60000026e
> Mode: leader
> Node count: 23
> ub12:2181
> Zookeeper version: 3.3.2-1031432, built on 11/05/2010 05:32 GMT
> Clients:
> /162.192.100.53:34213[0](queued=0,recved=1,sent=0)
>
> Latency min/avg/max: 0/0/0
> Received: 1
> Sent: 0
> Outstanding: 0
> Zxid: 0x60000026e
> Mode: follower
> Node count: 23
> ub13:2181
> Zookeeper version: 3.3.2-1031432, built on 11/05/2010 05:32 GMT
> Clients:
> /162.192.100.41:49999[1](queued=0,recved=141,sent=180)
> /162.192.100.53:57877[1](queued=0,recved=38,sent=41)
> /162.192.100.45:48814[1](queued=0,recved=148,sent=185)
> /162.192.100.53:57884[1](queued=0,recved=38,sent=40)
> /162.192.100.42:60452[1](queued=0,recved=38,sent=40)
> /162.192.100.44:57546[1](queued=0,recved=146,sent=186)
> /162.192.100.50:55070[1](queued=0,recved=33,sent=37)
> /162.192.100.45:48813[1](queued=0,recved=38,sent=40)
> /162.192.100.47:51598[1](queued=0,recved=127,sent=161)
> /162.192.100.53:57878[1](queued=0,recved=127,sent=163)
> /162.192.100.49:42398[1](queued=0,recved=38,sent=40)
> /162.192.100.51:52655[1](queued=0,recved=38,sent=40)
> /162.192.100.50:55068[1](queued=0,recved=137,sent=177)
> /162.192.100.52:60928[1](queued=0,recved=38,sent=40)
> /162.192.100.44:57544[1](queued=0,recved=38,sent=40)
> /162.192.100.53:57879[1](queued=0,recved=868,sent=1202)
> /162.192.100.51:52656[1](queued=0,recved=127,sent=162)
> /162.192.100.47:51596[1](queued=0,recved=38,sent=40)
> /162.192.100.53:48015[0](queued=0,recved=1,sent=0)
> /162.192.100.42:60455[1](queued=0,recved=146,sent=185)
> /162.192.100.48:60603[1](queued=0,recved=118,sent=150)
> /162.192.100.41:49997[1](queued=0,recved=38,sent=41)
> /162.192.100.43:49647[1](queued=0,recved=33,sent=37)
> /162.192.100.52:60930[1](queued=0,recved=118,sent=150)
> /162.192.100.43:49645[1](queued=0,recved=123,sent=159)
> /162.192.100.49:42399[1](queued=0,recved=127,sent=162)
> /162.192.100.48:60602[1](queued=0,recved=38,sent=40)
>
> Latency min/avg/max: 0/24/208
> Received: 2967
> Sent: 3767
> Outstanding: 0
> Zxid: 0x60000026e
> Mode: follower
> Node count: 23
>
>
>
> On Fri, Jul 8, 2011 at 9:02 PM, Stack <st...@duboce.net> wrote:
>
>> When you looked at zk with zkdump or via the zkcli did it show max
>> connections as 2k?
>> St.Ack
>>
>> On Fri, Jul 8, 2011 at 8:14 AM, praveenesh kumar <pr...@gmail.com>
>> wrote:
>> > yeah..!!
>> >
>> > On Fri, Jul 8, 2011 at 8:43 PM, Stack <st...@duboce.net> wrote:
>> >
>> >> Is hbase managing zk? Did you see zk restart when you restarted hbase?
>> >> St.Ack
>> >>
>> >> On Fri, Jul 8, 2011 at 8:09 AM, praveenesh kumar <pr...@gmail.com>
>> >> wrote:
>> >> > I restarted Hbase.. Is there any specific way to do that ?
>> >> >
>> >> > On Fri, Jul 8, 2011 at 8:35 PM, Stack <st...@duboce.net> wrote:
>> >> >
>> >> >> Did you restart your zk ensemble after making the change?
>> >> >> St.Ack
>> >> >>
>> >> >> On Fri, Jul 8, 2011 at 2:31 AM, praveenesh kumar <
>> praveenesh@gmail.com>
>> >> >> wrote:
>> >> >> > Guys.. I have increased the zookeeper connections to 2000, still I
>> am
>> >> >> > getting the same zookeeper exception while running the pig scripts.
>> >> >> > Can anyone tell me what can be the reason ?
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Praveenesh
>> >> >> >
>> >> >> > On Fri, Jul 8, 2011 at 12:16 AM, Barney Frank <
>> >> barneyfranks1@gmail.com
>> >> >> >wrote:
>> >> >> >
>> >> >> >> Why not just set the maxClientCnxcns to 0? That would make it
>> >> >> unlimited,
>> >> >> >> right?
>> >> >> >>
>> >> >> >> On Thu, Jul 7, 2011 at 12:55 PM, Ted Yu <yu...@gmail.com>
>> wrote:
>> >> >> >>
>> >> >> >> > To 1000.
>> >> >> >> >
>> >> >> >> > Click on 'zk dump' link from Master UI.
>> >> >> >> > You would know roughly how many connections were established.
>> >> >> >> > If 1000 is exhausted, increase more.
>> >> >> >> >
>> >> >> >> > See
>> >> >> >> >
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
>> >> >> >> >
>> >> >> >> > On Thu, Jul 7, 2011 at 10:52 AM, praveenesh kumar <
>> >> >> praveenesh@gmail.com
>> >> >> >> > >wrote:
>> >> >> >> >
>> >> >> >> > > I did that.. I increased it to 200.. still facing the same
>> >> error..
>> >> >> what
>> >> >> >> > > should be the ideal value ?
>> >> >> >> > >
>> >> >> >> > > On Thu, Jul 7, 2011 at 11:20 PM, Ted Yu <yu...@gmail.com>
>> >> >> wrote:
>> >> >> >> > >
>> >> >> >> > > > Increase hbase.zookeeper.property.maxClientCnxns to much
>> bigger
>> >> >> >> value.
>> >> >> >> > > >
>> >> >> >> > > >
>> >> >> >> > > > On Thu, Jul 7, 2011 at 10:46 AM, praveenesh kumar <
>> >> >> >> > praveenesh@gmail.com
>> >> >> >> > > > >wrote:
>> >> >> >> > > >
>> >> >> >> > > > > My HBase and pig both are running fine for me..
>> >> >> >> > > > > Only problem is coming whenever I am trying to access big
>> >> hbase
>> >> >> >> table
>> >> >> >> > > > > containing too many columns and rows..
>> >> >> >> > > > > whenever I am trying to access hbase table from pig... I
>> am
>> >> >> getting
>> >> >> >> > > this
>> >> >> >> > > > > following error in the middle of the map process:
>> >> >> >> > > > >
>> >> >> >> > > > > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997:
>> >> >> Unable
>> >> >> >> to
>> >> >> >> > > > > recreate exception from backed error:
>> >> >> >> > > > > org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> HBase
>> >> is
>> >> >> able
>> >> >> >> > to
>> >> >> >> > > > > connect to ZooKeeper but the connection closes
>> immediately.
>> >> This
>> >> >> >> > could
>> >> >> >> > > be
>> >> >> >> > > > a
>> >> >> >> > > > > sign that the server has too many connections (30 is the
>> >> >> default).
>> >> >> >> > > > Consider
>> >> >> >> > > > > inspecting your ZK server logs for that error and then
>> make
>> >> sure
>> >> >> >> you
>> >> >> >> > > are
>> >> >> >> > > > > reusing HBaseConfiguration as often as you can. See
>> HTable's
>> >> >> >> javadoc
>> >> >> >> > > for
>> >> >> >> > > > > more information.
>> >> >> >> > > > > But if I access a small HBase table from pig.. its working
>> >> >> fine..
>> >> >> >> no
>> >> >> >> > > > > zookeeper exception is coming. I am able to do all other
>> >> things
>> >> >> >> > > perfectly
>> >> >> >> > > > > fine.
>> >> >> >> > > > > This error made me think that there is some problem in my
>> >> >> zookeeper
>> >> >> >> > > that
>> >> >> >> > > > is
>> >> >> >> > > > > why I was asking is there anyway to check whether the
>> >> zookeeper
>> >> >> are
>> >> >> >> > > > working
>> >> >> >> > > > > fine or not ?
>> >> >> >> > > > >
>> >> >> >> > > > > My zookeeper log -- http://pastebin.com/RRdzZYui
>> >> >> >> > > > > My hbase-site.xml
>> >> >> >> > > > >
>> >> >> >> > > > > <configuration>
>> >> >> >> > > > > <property>
>> >> >> >> > > > > <name>hbase.rootdir</name>
>> >> >> >> > > > > <value>hdfs://ub13:54310/hbase</value>
>> >> >> >> > > > > <final>true</final>
>> >> >> >> > > > > </property>
>> >> >> >> > > > >
>> >> >> >> > > > >
>> >> >> >> > > > > <property>
>> >> >> >> > > > > <name>hbase.zookeeper.quorum</name>
>> >> >> >> > > > > <value>ub13,ub12,ub11</value>
>> >> >> >> > > > > <final>true</final>
>> >> >> >> > > > > </property>
>> >> >> >> > > > >
>> >> >> >> > > > > <property>
>> >> >> >> > > > > <name>hbase.zookeeper.property.maxClientCnxns</name>
>> >> >> >> > > > > <value>200</value>
>> >> >> >> > > > > <final>true</final>
>> >> >> >> > > > > </property>
>> >> >> >> > > > >
>> >> >> >> > > > > <property>
>> >> >> >> > > > > <name>hbase.cluster.distributed</name>
>> >> >> >> > > > > <value>true</value>
>> >> >> >> > > > > <description>The mode the cluster will be in. Possible
>> values
>> >> >> are
>> >> >> >> > > > > false: standalone and pseudo-distributed setups with
>> managed
>> >> >> >> > Zookeeper
>> >> >> >> > > > > true: fully-distributed with unmanaged Zookeeper Quorum
>> >> >> >> > > > > </description>
>> >> >> >> > > > > </property>
>> >> >> >> > > > >
>> >> >> >> > > > >
>> >> >> >> > > > >
>> >> >> >> > > > > On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net>
>> >> >> wrote:
>> >> >> >> > > > >
>> >> >> >> > > > > > On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <
>> >> >> >> > > praveenesh@gmail.com
>> >> >> >> > > > >
>> >> >> >> > > > > > wrote:
>> >> >> >> > > > > > > Can you please go through this and tell me what can be
>> >> the
>> >> >> >> issue
>> >> >> >> > ?
>> >> >> >> > > > > > >
>> >> >> >> > > > > >
>> >> >> >> > > > > > Can you summarize? (Life is short)
>> >> >> >> > > > > >
>> >> >> >> > > > > > If stuff works generally -- shell? And can you do the
>> >> hbase
>> >> >> >> > > > > > rowcounter mapreduce job successfully -- but then when
>> you
>> >> put
>> >> >> >> PIG
>> >> >> >> > in
>> >> >> >> > > > > > the picture it does not, it would seem to be an issue
>> with
>> >> >> your
>> >> >> >> PIG
>> >> >> >> > > > > > setup?
>> >> >> >> > > > > >
>> >> >> >> > > > > > St.Ack
>> >> >> >> > > > > >
>> >> >> >> > > > >
>> >> >> >> > > >
>> >> >> >> > >
>> >> >> >> >
>> >> >> >>
>> >> >> >
>> >> >>
>> >> >
>> >>
>> >
>>
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by praveenesh kumar <pr...@gmail.com>.
I am not able to see the total number of connections as 2k.
Here is my zookeeper output :
HBase is rooted at /hbase
Master address: ub13:60000
Region server holding ROOT: ub5:60020
Region servers:
ub8:60020
ub12:60020
ub11:60020
ub4:60020
ub2:60020
ub13:60020
ub7:60020
ub3:60020
ub10:60020
ub1:60020
ub9:60020
ub5:60020
Quorum Server Statistics:
ub11:2181
Zookeeper version: 3.3.2-1031432, built on 11/05/2010 05:32 GMT
Clients:
/162.192.100.53:51589[1](queued=0,recved=24,sent=24)
/162.192.100.53:51591[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/1/22
Received: 25
Sent: 24
Outstanding: 0
Zxid: 0x60000026e
Mode: leader
Node count: 23
ub12:2181
Zookeeper version: 3.3.2-1031432, built on 11/05/2010 05:32 GMT
Clients:
/162.192.100.53:34213[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 1
Sent: 0
Outstanding: 0
Zxid: 0x60000026e
Mode: follower
Node count: 23
ub13:2181
Zookeeper version: 3.3.2-1031432, built on 11/05/2010 05:32 GMT
Clients:
/162.192.100.41:49999[1](queued=0,recved=141,sent=180)
/162.192.100.53:57877[1](queued=0,recved=38,sent=41)
/162.192.100.45:48814[1](queued=0,recved=148,sent=185)
/162.192.100.53:57884[1](queued=0,recved=38,sent=40)
/162.192.100.42:60452[1](queued=0,recved=38,sent=40)
/162.192.100.44:57546[1](queued=0,recved=146,sent=186)
/162.192.100.50:55070[1](queued=0,recved=33,sent=37)
/162.192.100.45:48813[1](queued=0,recved=38,sent=40)
/162.192.100.47:51598[1](queued=0,recved=127,sent=161)
/162.192.100.53:57878[1](queued=0,recved=127,sent=163)
/162.192.100.49:42398[1](queued=0,recved=38,sent=40)
/162.192.100.51:52655[1](queued=0,recved=38,sent=40)
/162.192.100.50:55068[1](queued=0,recved=137,sent=177)
/162.192.100.52:60928[1](queued=0,recved=38,sent=40)
/162.192.100.44:57544[1](queued=0,recved=38,sent=40)
/162.192.100.53:57879[1](queued=0,recved=868,sent=1202)
/162.192.100.51:52656[1](queued=0,recved=127,sent=162)
/162.192.100.47:51596[1](queued=0,recved=38,sent=40)
/162.192.100.53:48015[0](queued=0,recved=1,sent=0)
/162.192.100.42:60455[1](queued=0,recved=146,sent=185)
/162.192.100.48:60603[1](queued=0,recved=118,sent=150)
/162.192.100.41:49997[1](queued=0,recved=38,sent=41)
/162.192.100.43:49647[1](queued=0,recved=33,sent=37)
/162.192.100.52:60930[1](queued=0,recved=118,sent=150)
/162.192.100.43:49645[1](queued=0,recved=123,sent=159)
/162.192.100.49:42399[1](queued=0,recved=127,sent=162)
/162.192.100.48:60602[1](queued=0,recved=38,sent=40)
Latency min/avg/max: 0/24/208
Received: 2967
Sent: 3767
Outstanding: 0
Zxid: 0x60000026e
Mode: follower
Node count: 23
On Fri, Jul 8, 2011 at 9:02 PM, Stack <st...@duboce.net> wrote:
> When you looked at zk with zkdump or via the zkcli did it show max
> connections as 2k?
> St.Ack
>
> On Fri, Jul 8, 2011 at 8:14 AM, praveenesh kumar <pr...@gmail.com>
> wrote:
> > yeah..!!
> >
> > On Fri, Jul 8, 2011 at 8:43 PM, Stack <st...@duboce.net> wrote:
> >
> >> Is hbase managing zk? Did you see zk restart when you restarted hbase?
> >> St.Ack
> >>
> >> On Fri, Jul 8, 2011 at 8:09 AM, praveenesh kumar <pr...@gmail.com>
> >> wrote:
> >> > I restarted Hbase.. Is there any specific way to do that ?
> >> >
> >> > On Fri, Jul 8, 2011 at 8:35 PM, Stack <st...@duboce.net> wrote:
> >> >
> >> >> Did you restart your zk ensemble after making the change?
> >> >> St.Ack
> >> >>
> >> >> On Fri, Jul 8, 2011 at 2:31 AM, praveenesh kumar <
> praveenesh@gmail.com>
> >> >> wrote:
> >> >> > Guys.. I have increased the zookeeper connections to 2000, still I
> am
> >> >> > getting the same zookeeper exception while running the pig scripts.
> >> >> > Can anyone tell me what can be the reason ?
> >> >> >
> >> >> > Thanks,
> >> >> > Praveenesh
> >> >> >
> >> >> > On Fri, Jul 8, 2011 at 12:16 AM, Barney Frank <
> >> barneyfranks1@gmail.com
> >> >> >wrote:
> >> >> >
> >> >> >> Why not just set the maxClientCnxcns to 0? That would make it
> >> >> unlimited,
> >> >> >> right?
> >> >> >>
> >> >> >> On Thu, Jul 7, 2011 at 12:55 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >> >> >>
> >> >> >> > To 1000.
> >> >> >> >
> >> >> >> > Click on 'zk dump' link from Master UI.
> >> >> >> > You would know roughly how many connections were established.
> >> >> >> > If 1000 is exhausted, increase more.
> >> >> >> >
> >> >> >> > See
> >> >> >> >
> >> >> >> >
> >> >> >>
> >> >>
> >>
> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
> >> >> >> >
> >> >> >> > On Thu, Jul 7, 2011 at 10:52 AM, praveenesh kumar <
> >> >> praveenesh@gmail.com
> >> >> >> > >wrote:
> >> >> >> >
> >> >> >> > > I did that.. I increased it to 200.. still facing the same
> >> error..
> >> >> what
> >> >> >> > > should be the ideal value ?
> >> >> >> > >
> >> >> >> > > On Thu, Jul 7, 2011 at 11:20 PM, Ted Yu <yu...@gmail.com>
> >> >> wrote:
> >> >> >> > >
> >> >> >> > > > Increase hbase.zookeeper.property.maxClientCnxns to much
> bigger
> >> >> >> value.
> >> >> >> > > >
> >> >> >> > > >
> >> >> >> > > > On Thu, Jul 7, 2011 at 10:46 AM, praveenesh kumar <
> >> >> >> > praveenesh@gmail.com
> >> >> >> > > > >wrote:
> >> >> >> > > >
> >> >> >> > > > > My HBase and pig both are running fine for me..
> >> >> >> > > > > Only problem is coming whenever I am trying to access big
> >> hbase
> >> >> >> table
> >> >> >> > > > > containing too many columns and rows..
> >> >> >> > > > > whenever I am trying to access hbase table from pig... I
> am
> >> >> getting
> >> >> >> > > this
> >> >> >> > > > > following error in the middle of the map process:
> >> >> >> > > > >
> >> >> >> > > > > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997:
> >> >> Unable
> >> >> >> to
> >> >> >> > > > > recreate exception from backed error:
> >> >> >> > > > > org.apache.hadoop.hbase.ZooKeeperConnectionException:
> HBase
> >> is
> >> >> able
> >> >> >> > to
> >> >> >> > > > > connect to ZooKeeper but the connection closes
> immediately.
> >> This
> >> >> >> > could
> >> >> >> > > be
> >> >> >> > > > a
> >> >> >> > > > > sign that the server has too many connections (30 is the
> >> >> default).
> >> >> >> > > > Consider
> >> >> >> > > > > inspecting your ZK server logs for that error and then
> make
> >> sure
> >> >> >> you
> >> >> >> > > are
> >> >> >> > > > > reusing HBaseConfiguration as often as you can. See
> HTable's
> >> >> >> javadoc
> >> >> >> > > for
> >> >> >> > > > > more information.
> >> >> >> > > > > But if I access a small HBase table from pig.. its working
> >> >> fine..
> >> >> >> no
> >> >> >> > > > > zookeeper exception is coming. I am able to do all other
> >> things
> >> >> >> > > perfectly
> >> >> >> > > > > fine.
> >> >> >> > > > > This error made me think that there is some problem in my
> >> >> zookeeper
> >> >> >> > > that
> >> >> >> > > > is
> >> >> >> > > > > why I was asking is there anyway to check whether the
> >> zookeeper
> >> >> are
> >> >> >> > > > working
> >> >> >> > > > > fine or not ?
> >> >> >> > > > >
> >> >> >> > > > > My zookeeper log -- http://pastebin.com/RRdzZYui
> >> >> >> > > > > My hbase-site.xml
> >> >> >> > > > >
> >> >> >> > > > > <configuration>
> >> >> >> > > > > <property>
> >> >> >> > > > > <name>hbase.rootdir</name>
> >> >> >> > > > > <value>hdfs://ub13:54310/hbase</value>
> >> >> >> > > > > <final>true</final>
> >> >> >> > > > > </property>
> >> >> >> > > > >
> >> >> >> > > > >
> >> >> >> > > > > <property>
> >> >> >> > > > > <name>hbase.zookeeper.quorum</name>
> >> >> >> > > > > <value>ub13,ub12,ub11</value>
> >> >> >> > > > > <final>true</final>
> >> >> >> > > > > </property>
> >> >> >> > > > >
> >> >> >> > > > > <property>
> >> >> >> > > > > <name>hbase.zookeeper.property.maxClientCnxns</name>
> >> >> >> > > > > <value>200</value>
> >> >> >> > > > > <final>true</final>
> >> >> >> > > > > </property>
> >> >> >> > > > >
> >> >> >> > > > > <property>
> >> >> >> > > > > <name>hbase.cluster.distributed</name>
> >> >> >> > > > > <value>true</value>
> >> >> >> > > > > <description>The mode the cluster will be in. Possible
> values
> >> >> are
> >> >> >> > > > > false: standalone and pseudo-distributed setups with
> managed
> >> >> >> > Zookeeper
> >> >> >> > > > > true: fully-distributed with unmanaged Zookeeper Quorum
> >> >> >> > > > > </description>
> >> >> >> > > > > </property>
> >> >> >> > > > >
> >> >> >> > > > >
> >> >> >> > > > >
> >> >> >> > > > > On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net>
> >> >> wrote:
> >> >> >> > > > >
> >> >> >> > > > > > On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <
> >> >> >> > > praveenesh@gmail.com
> >> >> >> > > > >
> >> >> >> > > > > > wrote:
> >> >> >> > > > > > > Can you please go through this and tell me what can be
> >> the
> >> >> >> issue
> >> >> >> > ?
> >> >> >> > > > > > >
> >> >> >> > > > > >
> >> >> >> > > > > > Can you summarize? (Life is short)
> >> >> >> > > > > >
> >> >> >> > > > > > If stuff works generally -- shell? And can you do the
> >> hbase
> >> >> >> > > > > > rowcounter mapreduce job successfully -- but then when
> you
> >> put
> >> >> >> PIG
> >> >> >> > in
> >> >> >> > > > > > the picture it does not, it would seem to be an issue
> with
> >> >> your
> >> >> >> PIG
> >> >> >> > > > > > setup?
> >> >> >> > > > > >
> >> >> >> > > > > > St.Ack
> >> >> >> > > > > >
> >> >> >> > > > >
> >> >> >> > > >
> >> >> >> > >
> >> >> >> >
> >> >> >>
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by Stack <st...@duboce.net>.
When you looked at zk with zkdump or via the zkcli did it show max
connections as 2k?
St.Ack
On Fri, Jul 8, 2011 at 8:14 AM, praveenesh kumar <pr...@gmail.com> wrote:
> yeah..!!
>
> On Fri, Jul 8, 2011 at 8:43 PM, Stack <st...@duboce.net> wrote:
>
>> Is hbase managing zk? Did you see zk restart when you restarted hbase?
>> St.Ack
>>
>> On Fri, Jul 8, 2011 at 8:09 AM, praveenesh kumar <pr...@gmail.com>
>> wrote:
>> > I restarted Hbase.. Is there any specific way to do that ?
>> >
>> > On Fri, Jul 8, 2011 at 8:35 PM, Stack <st...@duboce.net> wrote:
>> >
>> >> Did you restart your zk ensemble after making the change?
>> >> St.Ack
>> >>
>> >> On Fri, Jul 8, 2011 at 2:31 AM, praveenesh kumar <pr...@gmail.com>
>> >> wrote:
>> >> > Guys.. I have increased the zookeeper connections to 2000, still I am
>> >> > getting the same zookeeper exception while running the pig scripts.
>> >> > Can anyone tell me what can be the reason ?
>> >> >
>> >> > Thanks,
>> >> > Praveenesh
>> >> >
>> >> > On Fri, Jul 8, 2011 at 12:16 AM, Barney Frank <
>> barneyfranks1@gmail.com
>> >> >wrote:
>> >> >
>> >> >> Why not just set the maxClientCnxcns to 0? That would make it
>> >> unlimited,
>> >> >> right?
>> >> >>
>> >> >> On Thu, Jul 7, 2011 at 12:55 PM, Ted Yu <yu...@gmail.com> wrote:
>> >> >>
>> >> >> > To 1000.
>> >> >> >
>> >> >> > Click on 'zk dump' link from Master UI.
>> >> >> > You would know roughly how many connections were established.
>> >> >> > If 1000 is exhausted, increase more.
>> >> >> >
>> >> >> > See
>> >> >> >
>> >> >> >
>> >> >>
>> >>
>> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
>> >> >> >
>> >> >> > On Thu, Jul 7, 2011 at 10:52 AM, praveenesh kumar <
>> >> praveenesh@gmail.com
>> >> >> > >wrote:
>> >> >> >
>> >> >> > > I did that.. I increased it to 200.. still facing the same
>> error..
>> >> what
>> >> >> > > should be the ideal value ?
>> >> >> > >
>> >> >> > > On Thu, Jul 7, 2011 at 11:20 PM, Ted Yu <yu...@gmail.com>
>> >> wrote:
>> >> >> > >
>> >> >> > > > Increase hbase.zookeeper.property.maxClientCnxns to much bigger
>> >> >> value.
>> >> >> > > >
>> >> >> > > >
>> >> >> > > > On Thu, Jul 7, 2011 at 10:46 AM, praveenesh kumar <
>> >> >> > praveenesh@gmail.com
>> >> >> > > > >wrote:
>> >> >> > > >
>> >> >> > > > > My HBase and pig both are running fine for me..
>> >> >> > > > > Only problem is coming whenever I am trying to access big
>> hbase
>> >> >> table
>> >> >> > > > > containing too many columns and rows..
>> >> >> > > > > whenever I am trying to access hbase table from pig... I am
>> >> getting
>> >> >> > > this
>> >> >> > > > > following error in the middle of the map process:
>> >> >> > > > >
>> >> >> > > > > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997:
>> >> Unable
>> >> >> to
>> >> >> > > > > recreate exception from backed error:
>> >> >> > > > > org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase
>> is
>> >> able
>> >> >> > to
>> >> >> > > > > connect to ZooKeeper but the connection closes immediately.
>> This
>> >> >> > could
>> >> >> > > be
>> >> >> > > > a
>> >> >> > > > > sign that the server has too many connections (30 is the
>> >> default).
>> >> >> > > > Consider
>> >> >> > > > > inspecting your ZK server logs for that error and then make
>> sure
>> >> >> you
>> >> >> > > are
>> >> >> > > > > reusing HBaseConfiguration as often as you can. See HTable's
>> >> >> javadoc
>> >> >> > > for
>> >> >> > > > > more information.
>> >> >> > > > > But if I access a small HBase table from pig.. its working
>> >> fine..
>> >> >> no
>> >> >> > > > > zookeeper exception is coming. I am able to do all other
>> things
>> >> >> > > perfectly
>> >> >> > > > > fine.
>> >> >> > > > > This error made me think that there is some problem in my
>> >> zookeeper
>> >> >> > > that
>> >> >> > > > is
>> >> >> > > > > why I was asking is there anyway to check whether the
>> zookeeper
>> >> are
>> >> >> > > > working
>> >> >> > > > > fine or not ?
>> >> >> > > > >
>> >> >> > > > > My zookeeper log -- http://pastebin.com/RRdzZYui
>> >> >> > > > > My hbase-site.xml
>> >> >> > > > >
>> >> >> > > > > <configuration>
>> >> >> > > > > <property>
>> >> >> > > > > <name>hbase.rootdir</name>
>> >> >> > > > > <value>hdfs://ub13:54310/hbase</value>
>> >> >> > > > > <final>true</final>
>> >> >> > > > > </property>
>> >> >> > > > >
>> >> >> > > > >
>> >> >> > > > > <property>
>> >> >> > > > > <name>hbase.zookeeper.quorum</name>
>> >> >> > > > > <value>ub13,ub12,ub11</value>
>> >> >> > > > > <final>true</final>
>> >> >> > > > > </property>
>> >> >> > > > >
>> >> >> > > > > <property>
>> >> >> > > > > <name>hbase.zookeeper.property.maxClientCnxns</name>
>> >> >> > > > > <value>200</value>
>> >> >> > > > > <final>true</final>
>> >> >> > > > > </property>
>> >> >> > > > >
>> >> >> > > > > <property>
>> >> >> > > > > <name>hbase.cluster.distributed</name>
>> >> >> > > > > <value>true</value>
>> >> >> > > > > <description>The mode the cluster will be in. Possible values
>> >> are
>> >> >> > > > > false: standalone and pseudo-distributed setups with managed
>> >> >> > Zookeeper
>> >> >> > > > > true: fully-distributed with unmanaged Zookeeper Quorum
>> >> >> > > > > </description>
>> >> >> > > > > </property>
>> >> >> > > > >
>> >> >> > > > >
>> >> >> > > > >
>> >> >> > > > > On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net>
>> >> wrote:
>> >> >> > > > >
>> >> >> > > > > > On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <
>> >> >> > > praveenesh@gmail.com
>> >> >> > > > >
>> >> >> > > > > > wrote:
>> >> >> > > > > > > Can you please go through this and tell me what can be
>> the
>> >> >> issue
>> >> >> > ?
>> >> >> > > > > > >
>> >> >> > > > > >
>> >> >> > > > > > Can you summarize? (Life is short)
>> >> >> > > > > >
>> >> >> > > > > > If stuff works generally -- shell? And can you do the
>> hbase
>> >> >> > > > > > rowcounter mapreduce job successfully -- but then when you
>> put
>> >> >> PIG
>> >> >> > in
>> >> >> > > > > > the picture it does not, it would seem to be an issue with
>> >> your
>> >> >> PIG
>> >> >> > > > > > setup?
>> >> >> > > > > >
>> >> >> > > > > > St.Ack
>> >> >> > > > > >
>> >> >> > > > >
>> >> >> > > >
>> >> >> > >
>> >> >> >
>> >> >>
>> >> >
>> >>
>> >
>>
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by praveenesh kumar <pr...@gmail.com>.
yeah..!!
On Fri, Jul 8, 2011 at 8:43 PM, Stack <st...@duboce.net> wrote:
> Is hbase managing zk? Did you see zk restart when you restarted hbase?
> St.Ack
>
> On Fri, Jul 8, 2011 at 8:09 AM, praveenesh kumar <pr...@gmail.com>
> wrote:
> > I restarted Hbase.. Is there any specific way to do that ?
> >
> > On Fri, Jul 8, 2011 at 8:35 PM, Stack <st...@duboce.net> wrote:
> >
> >> Did you restart your zk ensemble after making the change?
> >> St.Ack
> >>
> >> On Fri, Jul 8, 2011 at 2:31 AM, praveenesh kumar <pr...@gmail.com>
> >> wrote:
> >> > Guys.. I have increased the zookeeper connections to 2000, still I am
> >> > getting the same zookeeper exception while running the pig scripts.
> >> > Can anyone tell me what can be the reason ?
> >> >
> >> > Thanks,
> >> > Praveenesh
> >> >
> >> > On Fri, Jul 8, 2011 at 12:16 AM, Barney Frank <
> barneyfranks1@gmail.com
> >> >wrote:
> >> >
> >> >> Why not just set the maxClientCnxcns to 0? That would make it
> >> unlimited,
> >> >> right?
> >> >>
> >> >> On Thu, Jul 7, 2011 at 12:55 PM, Ted Yu <yu...@gmail.com> wrote:
> >> >>
> >> >> > To 1000.
> >> >> >
> >> >> > Click on 'zk dump' link from Master UI.
> >> >> > You would know roughly how many connections were established.
> >> >> > If 1000 is exhausted, increase more.
> >> >> >
> >> >> > See
> >> >> >
> >> >> >
> >> >>
> >>
> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
> >> >> >
> >> >> > On Thu, Jul 7, 2011 at 10:52 AM, praveenesh kumar <
> >> praveenesh@gmail.com
> >> >> > >wrote:
> >> >> >
> >> >> > > I did that.. I increased it to 200.. still facing the same
> error..
> >> what
> >> >> > > should be the ideal value ?
> >> >> > >
> >> >> > > On Thu, Jul 7, 2011 at 11:20 PM, Ted Yu <yu...@gmail.com>
> >> wrote:
> >> >> > >
> >> >> > > > Increase hbase.zookeeper.property.maxClientCnxns to much bigger
> >> >> value.
> >> >> > > >
> >> >> > > >
> >> >> > > > On Thu, Jul 7, 2011 at 10:46 AM, praveenesh kumar <
> >> >> > praveenesh@gmail.com
> >> >> > > > >wrote:
> >> >> > > >
> >> >> > > > > My HBase and pig both are running fine for me..
> >> >> > > > > Only problem is coming whenever I am trying to access big
> hbase
> >> >> table
> >> >> > > > > containing too many columns and rows..
> >> >> > > > > whenever I am trying to access hbase table from pig... I am
> >> getting
> >> >> > > this
> >> >> > > > > following error in the middle of the map process:
> >> >> > > > >
> >> >> > > > > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997:
> >> Unable
> >> >> to
> >> >> > > > > recreate exception from backed error:
> >> >> > > > > org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase
> is
> >> able
> >> >> > to
> >> >> > > > > connect to ZooKeeper but the connection closes immediately.
> This
> >> >> > could
> >> >> > > be
> >> >> > > > a
> >> >> > > > > sign that the server has too many connections (30 is the
> >> default).
> >> >> > > > Consider
> >> >> > > > > inspecting your ZK server logs for that error and then make
> sure
> >> >> you
> >> >> > > are
> >> >> > > > > reusing HBaseConfiguration as often as you can. See HTable's
> >> >> javadoc
> >> >> > > for
> >> >> > > > > more information.
> >> >> > > > > But if I access a small HBase table from pig.. its working
> >> fine..
> >> >> no
> >> >> > > > > zookeeper exception is coming. I am able to do all other
> things
> >> >> > > perfectly
> >> >> > > > > fine.
> >> >> > > > > This error made me think that there is some problem in my
> >> zookeeper
> >> >> > > that
> >> >> > > > is
> >> >> > > > > why I was asking is there anyway to check whether the
> zookeeper
> >> are
> >> >> > > > working
> >> >> > > > > fine or not ?
> >> >> > > > >
> >> >> > > > > My zookeeper log -- http://pastebin.com/RRdzZYui
> >> >> > > > > My hbase-site.xml
> >> >> > > > >
> >> >> > > > > <configuration>
> >> >> > > > > <property>
> >> >> > > > > <name>hbase.rootdir</name>
> >> >> > > > > <value>hdfs://ub13:54310/hbase</value>
> >> >> > > > > <final>true</final>
> >> >> > > > > </property>
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > <property>
> >> >> > > > > <name>hbase.zookeeper.quorum</name>
> >> >> > > > > <value>ub13,ub12,ub11</value>
> >> >> > > > > <final>true</final>
> >> >> > > > > </property>
> >> >> > > > >
> >> >> > > > > <property>
> >> >> > > > > <name>hbase.zookeeper.property.maxClientCnxns</name>
> >> >> > > > > <value>200</value>
> >> >> > > > > <final>true</final>
> >> >> > > > > </property>
> >> >> > > > >
> >> >> > > > > <property>
> >> >> > > > > <name>hbase.cluster.distributed</name>
> >> >> > > > > <value>true</value>
> >> >> > > > > <description>The mode the cluster will be in. Possible values
> >> are
> >> >> > > > > false: standalone and pseudo-distributed setups with managed
> >> >> > Zookeeper
> >> >> > > > > true: fully-distributed with unmanaged Zookeeper Quorum
> >> >> > > > > </description>
> >> >> > > > > </property>
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net>
> >> wrote:
> >> >> > > > >
> >> >> > > > > > On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <
> >> >> > > praveenesh@gmail.com
> >> >> > > > >
> >> >> > > > > > wrote:
> >> >> > > > > > > Can you please go through this and tell me what can be
> the
> >> >> issue
> >> >> > ?
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > > > Can you summarize? (Life is short)
> >> >> > > > > >
> >> >> > > > > > If stuff works generally -- shell? And can you do the
> hbase
> >> >> > > > > > rowcounter mapreduce job successfully -- but then when you
> put
> >> >> PIG
> >> >> > in
> >> >> > > > > > the picture it does not, it would seem to be an issue with
> >> your
> >> >> PIG
> >> >> > > > > > setup?
> >> >> > > > > >
> >> >> > > > > > St.Ack
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by Stack <st...@duboce.net>.
Is hbase managing zk? Did you see zk restart when you restarted hbase?
St.Ack
On Fri, Jul 8, 2011 at 8:09 AM, praveenesh kumar <pr...@gmail.com> wrote:
> I restarted Hbase.. Is there any specific way to do that ?
>
> On Fri, Jul 8, 2011 at 8:35 PM, Stack <st...@duboce.net> wrote:
>
>> Did you restart your zk ensemble after making the change?
>> St.Ack
>>
>> On Fri, Jul 8, 2011 at 2:31 AM, praveenesh kumar <pr...@gmail.com>
>> wrote:
>> > Guys.. I have increased the zookeeper connections to 2000, still I am
>> > getting the same zookeeper exception while running the pig scripts.
>> > Can anyone tell me what can be the reason ?
>> >
>> > Thanks,
>> > Praveenesh
>> >
>> > On Fri, Jul 8, 2011 at 12:16 AM, Barney Frank <barneyfranks1@gmail.com
>> >wrote:
>> >
>> >> Why not just set the maxClientCnxcns to 0? That would make it
>> unlimited,
>> >> right?
>> >>
>> >> On Thu, Jul 7, 2011 at 12:55 PM, Ted Yu <yu...@gmail.com> wrote:
>> >>
>> >> > To 1000.
>> >> >
>> >> > Click on 'zk dump' link from Master UI.
>> >> > You would know roughly how many connections were established.
>> >> > If 1000 is exhausted, increase more.
>> >> >
>> >> > See
>> >> >
>> >> >
>> >>
>> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
>> >> >
>> >> > On Thu, Jul 7, 2011 at 10:52 AM, praveenesh kumar <
>> praveenesh@gmail.com
>> >> > >wrote:
>> >> >
>> >> > > I did that.. I increased it to 200.. still facing the same error..
>> what
>> >> > > should be the ideal value ?
>> >> > >
>> >> > > On Thu, Jul 7, 2011 at 11:20 PM, Ted Yu <yu...@gmail.com>
>> wrote:
>> >> > >
>> >> > > > Increase hbase.zookeeper.property.maxClientCnxns to much bigger
>> >> value.
>> >> > > >
>> >> > > >
>> >> > > > On Thu, Jul 7, 2011 at 10:46 AM, praveenesh kumar <
>> >> > praveenesh@gmail.com
>> >> > > > >wrote:
>> >> > > >
>> >> > > > > My HBase and pig both are running fine for me..
>> >> > > > > Only problem is coming whenever I am trying to access big hbase
>> >> table
>> >> > > > > containing too many columns and rows..
>> >> > > > > whenever I am trying to access hbase table from pig... I am
>> getting
>> >> > > this
>> >> > > > > following error in the middle of the map process:
>> >> > > > >
>> >> > > > > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997:
>> Unable
>> >> to
>> >> > > > > recreate exception from backed error:
>> >> > > > > org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is
>> able
>> >> > to
>> >> > > > > connect to ZooKeeper but the connection closes immediately. This
>> >> > could
>> >> > > be
>> >> > > > a
>> >> > > > > sign that the server has too many connections (30 is the
>> default).
>> >> > > > Consider
>> >> > > > > inspecting your ZK server logs for that error and then make sure
>> >> you
>> >> > > are
>> >> > > > > reusing HBaseConfiguration as often as you can. See HTable's
>> >> javadoc
>> >> > > for
>> >> > > > > more information.
>> >> > > > > But if I access a small HBase table from pig.. its working
>> fine..
>> >> no
>> >> > > > > zookeeper exception is coming. I am able to do all other things
>> >> > > perfectly
>> >> > > > > fine.
>> >> > > > > This error made me think that there is some problem in my
>> zookeeper
>> >> > > that
>> >> > > > is
>> >> > > > > why I was asking is there anyway to check whether the zookeeper
>> are
>> >> > > > working
>> >> > > > > fine or not ?
>> >> > > > >
>> >> > > > > My zookeeper log -- http://pastebin.com/RRdzZYui
>> >> > > > > My hbase-site.xml
>> >> > > > >
>> >> > > > > <configuration>
>> >> > > > > <property>
>> >> > > > > <name>hbase.rootdir</name>
>> >> > > > > <value>hdfs://ub13:54310/hbase</value>
>> >> > > > > <final>true</final>
>> >> > > > > </property>
>> >> > > > >
>> >> > > > >
>> >> > > > > <property>
>> >> > > > > <name>hbase.zookeeper.quorum</name>
>> >> > > > > <value>ub13,ub12,ub11</value>
>> >> > > > > <final>true</final>
>> >> > > > > </property>
>> >> > > > >
>> >> > > > > <property>
>> >> > > > > <name>hbase.zookeeper.property.maxClientCnxns</name>
>> >> > > > > <value>200</value>
>> >> > > > > <final>true</final>
>> >> > > > > </property>
>> >> > > > >
>> >> > > > > <property>
>> >> > > > > <name>hbase.cluster.distributed</name>
>> >> > > > > <value>true</value>
>> >> > > > > <description>The mode the cluster will be in. Possible values
>> are
>> >> > > > > false: standalone and pseudo-distributed setups with managed
>> >> > Zookeeper
>> >> > > > > true: fully-distributed with unmanaged Zookeeper Quorum
>> >> > > > > </description>
>> >> > > > > </property>
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net>
>> wrote:
>> >> > > > >
>> >> > > > > > On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <
>> >> > > praveenesh@gmail.com
>> >> > > > >
>> >> > > > > > wrote:
>> >> > > > > > > Can you please go through this and tell me what can be the
>> >> issue
>> >> > ?
>> >> > > > > > >
>> >> > > > > >
>> >> > > > > > Can you summarize? (Life is short)
>> >> > > > > >
>> >> > > > > > If stuff works generally -- shell? And can you do the hbase
>> >> > > > > > rowcounter mapreduce job successfully -- but then when you put
>> >> PIG
>> >> > in
>> >> > > > > > the picture it does not, it would seem to be an issue with
>> your
>> >> PIG
>> >> > > > > > setup?
>> >> > > > > >
>> >> > > > > > St.Ack
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>>
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by praveenesh kumar <pr...@gmail.com>.
I restarted Hbase.. Is there any specific way to do that ?
On Fri, Jul 8, 2011 at 8:35 PM, Stack <st...@duboce.net> wrote:
> Did you restart your zk ensemble after making the change?
> St.Ack
>
> On Fri, Jul 8, 2011 at 2:31 AM, praveenesh kumar <pr...@gmail.com>
> wrote:
> > Guys.. I have increased the zookeeper connections to 2000, still I am
> > getting the same zookeeper exception while running the pig scripts.
> > Can anyone tell me what can be the reason ?
> >
> > Thanks,
> > Praveenesh
> >
> > On Fri, Jul 8, 2011 at 12:16 AM, Barney Frank <barneyfranks1@gmail.com
> >wrote:
> >
> >> Why not just set the maxClientCnxcns to 0? That would make it
> unlimited,
> >> right?
> >>
> >> On Thu, Jul 7, 2011 at 12:55 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> > To 1000.
> >> >
> >> > Click on 'zk dump' link from Master UI.
> >> > You would know roughly how many connections were established.
> >> > If 1000 is exhausted, increase more.
> >> >
> >> > See
> >> >
> >> >
> >>
> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
> >> >
> >> > On Thu, Jul 7, 2011 at 10:52 AM, praveenesh kumar <
> praveenesh@gmail.com
> >> > >wrote:
> >> >
> >> > > I did that.. I increased it to 200.. still facing the same error..
> what
> >> > > should be the ideal value ?
> >> > >
> >> > > On Thu, Jul 7, 2011 at 11:20 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >> > >
> >> > > > Increase hbase.zookeeper.property.maxClientCnxns to much bigger
> >> value.
> >> > > >
> >> > > >
> >> > > > On Thu, Jul 7, 2011 at 10:46 AM, praveenesh kumar <
> >> > praveenesh@gmail.com
> >> > > > >wrote:
> >> > > >
> >> > > > > My HBase and pig both are running fine for me..
> >> > > > > Only problem is coming whenever I am trying to access big hbase
> >> table
> >> > > > > containing too many columns and rows..
> >> > > > > whenever I am trying to access hbase table from pig... I am
> getting
> >> > > this
> >> > > > > following error in the middle of the map process:
> >> > > > >
> >> > > > > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997:
> Unable
> >> to
> >> > > > > recreate exception from backed error:
> >> > > > > org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is
> able
> >> > to
> >> > > > > connect to ZooKeeper but the connection closes immediately. This
> >> > could
> >> > > be
> >> > > > a
> >> > > > > sign that the server has too many connections (30 is the
> default).
> >> > > > Consider
> >> > > > > inspecting your ZK server logs for that error and then make sure
> >> you
> >> > > are
> >> > > > > reusing HBaseConfiguration as often as you can. See HTable's
> >> javadoc
> >> > > for
> >> > > > > more information.
> >> > > > > But if I access a small HBase table from pig.. its working
> fine..
> >> no
> >> > > > > zookeeper exception is coming. I am able to do all other things
> >> > > perfectly
> >> > > > > fine.
> >> > > > > This error made me think that there is some problem in my
> zookeeper
> >> > > that
> >> > > > is
> >> > > > > why I was asking is there anyway to check whether the zookeeper
> are
> >> > > > working
> >> > > > > fine or not ?
> >> > > > >
> >> > > > > My zookeeper log -- http://pastebin.com/RRdzZYui
> >> > > > > My hbase-site.xml
> >> > > > >
> >> > > > > <configuration>
> >> > > > > <property>
> >> > > > > <name>hbase.rootdir</name>
> >> > > > > <value>hdfs://ub13:54310/hbase</value>
> >> > > > > <final>true</final>
> >> > > > > </property>
> >> > > > >
> >> > > > >
> >> > > > > <property>
> >> > > > > <name>hbase.zookeeper.quorum</name>
> >> > > > > <value>ub13,ub12,ub11</value>
> >> > > > > <final>true</final>
> >> > > > > </property>
> >> > > > >
> >> > > > > <property>
> >> > > > > <name>hbase.zookeeper.property.maxClientCnxns</name>
> >> > > > > <value>200</value>
> >> > > > > <final>true</final>
> >> > > > > </property>
> >> > > > >
> >> > > > > <property>
> >> > > > > <name>hbase.cluster.distributed</name>
> >> > > > > <value>true</value>
> >> > > > > <description>The mode the cluster will be in. Possible values
> are
> >> > > > > false: standalone and pseudo-distributed setups with managed
> >> > Zookeeper
> >> > > > > true: fully-distributed with unmanaged Zookeeper Quorum
> >> > > > > </description>
> >> > > > > </property>
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net>
> wrote:
> >> > > > >
> >> > > > > > On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <
> >> > > praveenesh@gmail.com
> >> > > > >
> >> > > > > > wrote:
> >> > > > > > > Can you please go through this and tell me what can be the
> >> issue
> >> > ?
> >> > > > > > >
> >> > > > > >
> >> > > > > > Can you summarize? (Life is short)
> >> > > > > >
> >> > > > > > If stuff works generally -- shell? And can you do the hbase
> >> > > > > > rowcounter mapreduce job successfully -- but then when you put
> >> PIG
> >> > in
> >> > > > > > the picture it does not, it would seem to be an issue with
> your
> >> PIG
> >> > > > > > setup?
> >> > > > > >
> >> > > > > > St.Ack
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by Stack <st...@duboce.net>.
Did you restart your zk ensemble after making the change?
St.Ack
On Fri, Jul 8, 2011 at 2:31 AM, praveenesh kumar <pr...@gmail.com> wrote:
> Guys.. I have increased the zookeeper connections to 2000, still I am
> getting the same zookeeper exception while running the pig scripts.
> Can anyone tell me what can be the reason ?
>
> Thanks,
> Praveenesh
>
> On Fri, Jul 8, 2011 at 12:16 AM, Barney Frank <ba...@gmail.com>wrote:
>
>> Why not just set the maxClientCnxcns to 0? That would make it unlimited,
>> right?
>>
>> On Thu, Jul 7, 2011 at 12:55 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > To 1000.
>> >
>> > Click on 'zk dump' link from Master UI.
>> > You would know roughly how many connections were established.
>> > If 1000 is exhausted, increase more.
>> >
>> > See
>> >
>> >
>> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
>> >
>> > On Thu, Jul 7, 2011 at 10:52 AM, praveenesh kumar <praveenesh@gmail.com
>> > >wrote:
>> >
>> > > I did that.. I increased it to 200.. still facing the same error.. what
>> > > should be the ideal value ?
>> > >
>> > > On Thu, Jul 7, 2011 at 11:20 PM, Ted Yu <yu...@gmail.com> wrote:
>> > >
>> > > > Increase hbase.zookeeper.property.maxClientCnxns to much bigger
>> value.
>> > > >
>> > > >
>> > > > On Thu, Jul 7, 2011 at 10:46 AM, praveenesh kumar <
>> > praveenesh@gmail.com
>> > > > >wrote:
>> > > >
>> > > > > My HBase and pig both are running fine for me..
>> > > > > Only problem is coming whenever I am trying to access big hbase
>> table
>> > > > > containing too many columns and rows..
>> > > > > whenever I am trying to access hbase table from pig... I am getting
>> > > this
>> > > > > following error in the middle of the map process:
>> > > > >
>> > > > > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable
>> to
>> > > > > recreate exception from backed error:
>> > > > > org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
>> > to
>> > > > > connect to ZooKeeper but the connection closes immediately. This
>> > could
>> > > be
>> > > > a
>> > > > > sign that the server has too many connections (30 is the default).
>> > > > Consider
>> > > > > inspecting your ZK server logs for that error and then make sure
>> you
>> > > are
>> > > > > reusing HBaseConfiguration as often as you can. See HTable's
>> javadoc
>> > > for
>> > > > > more information.
>> > > > > But if I access a small HBase table from pig.. its working fine..
>> no
>> > > > > zookeeper exception is coming. I am able to do all other things
>> > > perfectly
>> > > > > fine.
>> > > > > This error made me think that there is some problem in my zookeeper
>> > > that
>> > > > is
>> > > > > why I was asking is there anyway to check whether the zookeeper are
>> > > > working
>> > > > > fine or not ?
>> > > > >
>> > > > > My zookeeper log -- http://pastebin.com/RRdzZYui
>> > > > > My hbase-site.xml
>> > > > >
>> > > > > <configuration>
>> > > > > <property>
>> > > > > <name>hbase.rootdir</name>
>> > > > > <value>hdfs://ub13:54310/hbase</value>
>> > > > > <final>true</final>
>> > > > > </property>
>> > > > >
>> > > > >
>> > > > > <property>
>> > > > > <name>hbase.zookeeper.quorum</name>
>> > > > > <value>ub13,ub12,ub11</value>
>> > > > > <final>true</final>
>> > > > > </property>
>> > > > >
>> > > > > <property>
>> > > > > <name>hbase.zookeeper.property.maxClientCnxns</name>
>> > > > > <value>200</value>
>> > > > > <final>true</final>
>> > > > > </property>
>> > > > >
>> > > > > <property>
>> > > > > <name>hbase.cluster.distributed</name>
>> > > > > <value>true</value>
>> > > > > <description>The mode the cluster will be in. Possible values are
>> > > > > false: standalone and pseudo-distributed setups with managed
>> > Zookeeper
>> > > > > true: fully-distributed with unmanaged Zookeeper Quorum
>> > > > > </description>
>> > > > > </property>
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net> wrote:
>> > > > >
>> > > > > > On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <
>> > > praveenesh@gmail.com
>> > > > >
>> > > > > > wrote:
>> > > > > > > Can you please go through this and tell me what can be the
>> issue
>> > ?
>> > > > > > >
>> > > > > >
>> > > > > > Can you summarize? (Life is short)
>> > > > > >
>> > > > > > If stuff works generally -- shell? And can you do the hbase
>> > > > > > rowcounter mapreduce job successfully -- but then when you put
>> PIG
>> > in
>> > > > > > the picture it does not, it would seem to be an issue with your
>> PIG
>> > > > > > setup?
>> > > > > >
>> > > > > > St.Ack
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by praveenesh kumar <pr...@gmail.com>.
Hi,
I have 3 machines configured as my zookeepr quorum. I am pasting the latest
zookeeper logs of these 3 machines..
Can anyone have a look and tell me whether everything is going fine or not ?
http://pastebin.com/EC2tgEN9 --------- on Machine ub13
http://pastebin.com/JccPy5Ry --------- on Machine ub12
http://pastebin.com/2KwYzBX7 --------- on Machine ub11
*******hbase-site.xml*******
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://ub13:54310/hbase</value>
<final>true</final>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>ub13,ub12,ub11</value>
<final>true</final>
</property>
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>200</value>
<final>true</final>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum
</description>
</property>
Thanks,
Praveenesh
On Fri, Jul 8, 2011 at 3:27 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you check zookeeper log ?
>
> Cheers
>
>
>
> On Jul 8, 2011, at 2:31 AM, praveenesh kumar <pr...@gmail.com> wrote:
>
> > Guys.. I have increased the zookeeper connections to 2000, still I am
> > getting the same zookeeper exception while running the pig scripts.
> > Can anyone tell me what can be the reason ?
> >
> > Thanks,
> > Praveenesh
> >
> > On Fri, Jul 8, 2011 at 12:16 AM, Barney Frank <barneyfranks1@gmail.com
> >wrote:
> >
> >> Why not just set the maxClientCnxcns to 0? That would make it
> unlimited,
> >> right?
> >>
> >> On Thu, Jul 7, 2011 at 12:55 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >>> To 1000.
> >>>
> >>> Click on 'zk dump' link from Master UI.
> >>> You would know roughly how many connections were established.
> >>> If 1000 is exhausted, increase more.
> >>>
> >>> See
> >>>
> >>>
> >>
> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
> >>>
> >>> On Thu, Jul 7, 2011 at 10:52 AM, praveenesh kumar <
> praveenesh@gmail.com
> >>>> wrote:
> >>>
> >>>> I did that.. I increased it to 200.. still facing the same error..
> what
> >>>> should be the ideal value ?
> >>>>
> >>>> On Thu, Jul 7, 2011 at 11:20 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>
> >>>>> Increase hbase.zookeeper.property.maxClientCnxns to much bigger
> >> value.
> >>>>>
> >>>>>
> >>>>> On Thu, Jul 7, 2011 at 10:46 AM, praveenesh kumar <
> >>> praveenesh@gmail.com
> >>>>>> wrote:
> >>>>>
> >>>>>> My HBase and pig both are running fine for me..
> >>>>>> Only problem is coming whenever I am trying to access big hbase
> >> table
> >>>>>> containing too many columns and rows..
> >>>>>> whenever I am trying to access hbase table from pig... I am getting
> >>>> this
> >>>>>> following error in the middle of the map process:
> >>>>>>
> >>>>>> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable
> >> to
> >>>>>> recreate exception from backed error:
> >>>>>> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
> >>> to
> >>>>>> connect to ZooKeeper but the connection closes immediately. This
> >>> could
> >>>> be
> >>>>> a
> >>>>>> sign that the server has too many connections (30 is the default).
> >>>>> Consider
> >>>>>> inspecting your ZK server logs for that error and then make sure
> >> you
> >>>> are
> >>>>>> reusing HBaseConfiguration as often as you can. See HTable's
> >> javadoc
> >>>> for
> >>>>>> more information.
> >>>>>> But if I access a small HBase table from pig.. its working fine..
> >> no
> >>>>>> zookeeper exception is coming. I am able to do all other things
> >>>> perfectly
> >>>>>> fine.
> >>>>>> This error made me think that there is some problem in my zookeeper
> >>>> that
> >>>>> is
> >>>>>> why I was asking is there anyway to check whether the zookeeper are
> >>>>> working
> >>>>>> fine or not ?
> >>>>>>
> >>>>>> My zookeeper log -- http://pastebin.com/RRdzZYui
> >>>>>> My hbase-site.xml
> >>>>>>
> >>>>>> <configuration>
> >>>>>> <property>
> >>>>>> <name>hbase.rootdir</name>
> >>>>>> <value>hdfs://ub13:54310/hbase</value>
> >>>>>> <final>true</final>
> >>>>>> </property>
> >>>>>>
> >>>>>>
> >>>>>> <property>
> >>>>>> <name>hbase.zookeeper.quorum</name>
> >>>>>> <value>ub13,ub12,ub11</value>
> >>>>>> <final>true</final>
> >>>>>> </property>
> >>>>>>
> >>>>>> <property>
> >>>>>> <name>hbase.zookeeper.property.maxClientCnxns</name>
> >>>>>> <value>200</value>
> >>>>>> <final>true</final>
> >>>>>> </property>
> >>>>>>
> >>>>>> <property>
> >>>>>> <name>hbase.cluster.distributed</name>
> >>>>>> <value>true</value>
> >>>>>> <description>The mode the cluster will be in. Possible values are
> >>>>>> false: standalone and pseudo-distributed setups with managed
> >>> Zookeeper
> >>>>>> true: fully-distributed with unmanaged Zookeeper Quorum
> >>>>>> </description>
> >>>>>> </property>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net> wrote:
> >>>>>>
> >>>>>>> On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <
> >>>> praveenesh@gmail.com
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>> Can you please go through this and tell me what can be the
> >> issue
> >>> ?
> >>>>>>>>
> >>>>>>>
> >>>>>>> Can you summarize? (Life is short)
> >>>>>>>
> >>>>>>> If stuff works generally -- shell? And can you do the hbase
> >>>>>>> rowcounter mapreduce job successfully -- but then when you put
> >> PIG
> >>> in
> >>>>>>> the picture it does not, it would seem to be an issue with your
> >> PIG
> >>>>>>> setup?
> >>>>>>>
> >>>>>>> St.Ack
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by Ted Yu <yu...@gmail.com>.
Can you check zookeeper log ?
Cheers
On Jul 8, 2011, at 2:31 AM, praveenesh kumar <pr...@gmail.com> wrote:
> Guys.. I have increased the zookeeper connections to 2000, still I am
> getting the same zookeeper exception while running the pig scripts.
> Can anyone tell me what can be the reason ?
>
> Thanks,
> Praveenesh
>
> On Fri, Jul 8, 2011 at 12:16 AM, Barney Frank <ba...@gmail.com>wrote:
>
>> Why not just set the maxClientCnxcns to 0? That would make it unlimited,
>> right?
>>
>> On Thu, Jul 7, 2011 at 12:55 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> To 1000.
>>>
>>> Click on 'zk dump' link from Master UI.
>>> You would know roughly how many connections were established.
>>> If 1000 is exhausted, increase more.
>>>
>>> See
>>>
>>>
>> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
>>>
>>> On Thu, Jul 7, 2011 at 10:52 AM, praveenesh kumar <praveenesh@gmail.com
>>>> wrote:
>>>
>>>> I did that.. I increased it to 200.. still facing the same error.. what
>>>> should be the ideal value ?
>>>>
>>>> On Thu, Jul 7, 2011 at 11:20 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> Increase hbase.zookeeper.property.maxClientCnxns to much bigger
>> value.
>>>>>
>>>>>
>>>>> On Thu, Jul 7, 2011 at 10:46 AM, praveenesh kumar <
>>> praveenesh@gmail.com
>>>>>> wrote:
>>>>>
>>>>>> My HBase and pig both are running fine for me..
>>>>>> Only problem is coming whenever I am trying to access big hbase
>> table
>>>>>> containing too many columns and rows..
>>>>>> whenever I am trying to access hbase table from pig... I am getting
>>>> this
>>>>>> following error in the middle of the map process:
>>>>>>
>>>>>> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable
>> to
>>>>>> recreate exception from backed error:
>>>>>> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
>>> to
>>>>>> connect to ZooKeeper but the connection closes immediately. This
>>> could
>>>> be
>>>>> a
>>>>>> sign that the server has too many connections (30 is the default).
>>>>> Consider
>>>>>> inspecting your ZK server logs for that error and then make sure
>> you
>>>> are
>>>>>> reusing HBaseConfiguration as often as you can. See HTable's
>> javadoc
>>>> for
>>>>>> more information.
>>>>>> But if I access a small HBase table from pig.. its working fine..
>> no
>>>>>> zookeeper exception is coming. I am able to do all other things
>>>> perfectly
>>>>>> fine.
>>>>>> This error made me think that there is some problem in my zookeeper
>>>> that
>>>>> is
>>>>>> why I was asking is there anyway to check whether the zookeeper are
>>>>> working
>>>>>> fine or not ?
>>>>>>
>>>>>> My zookeeper log -- http://pastebin.com/RRdzZYui
>>>>>> My hbase-site.xml
>>>>>>
>>>>>> <configuration>
>>>>>> <property>
>>>>>> <name>hbase.rootdir</name>
>>>>>> <value>hdfs://ub13:54310/hbase</value>
>>>>>> <final>true</final>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>> <property>
>>>>>> <name>hbase.zookeeper.quorum</name>
>>>>>> <value>ub13,ub12,ub11</value>
>>>>>> <final>true</final>
>>>>>> </property>
>>>>>>
>>>>>> <property>
>>>>>> <name>hbase.zookeeper.property.maxClientCnxns</name>
>>>>>> <value>200</value>
>>>>>> <final>true</final>
>>>>>> </property>
>>>>>>
>>>>>> <property>
>>>>>> <name>hbase.cluster.distributed</name>
>>>>>> <value>true</value>
>>>>>> <description>The mode the cluster will be in. Possible values are
>>>>>> false: standalone and pseudo-distributed setups with managed
>>> Zookeeper
>>>>>> true: fully-distributed with unmanaged Zookeeper Quorum
>>>>>> </description>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net> wrote:
>>>>>>
>>>>>>> On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <
>>>> praveenesh@gmail.com
>>>>>>
>>>>>>> wrote:
>>>>>>>> Can you please go through this and tell me what can be the
>> issue
>>> ?
>>>>>>>>
>>>>>>>
>>>>>>> Can you summarize? (Life is short)
>>>>>>>
>>>>>>> If stuff works generally -- shell? And can you do the hbase
>>>>>>> rowcounter mapreduce job successfully -- but then when you put
>> PIG
>>> in
>>>>>>> the picture it does not, it would seem to be an issue with your
>> PIG
>>>>>>> setup?
>>>>>>>
>>>>>>> St.Ack
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by praveenesh kumar <pr...@gmail.com>.
Guys.. I have increased the zookeeper connections to 2000, still I am
getting the same zookeeper exception while running the pig scripts.
Can anyone tell me what can be the reason ?
Thanks,
Praveenesh
On Fri, Jul 8, 2011 at 12:16 AM, Barney Frank <ba...@gmail.com>wrote:
> Why not just set the maxClientCnxcns to 0? That would make it unlimited,
> right?
>
> On Thu, Jul 7, 2011 at 12:55 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > To 1000.
> >
> > Click on 'zk dump' link from Master UI.
> > You would know roughly how many connections were established.
> > If 1000 is exhausted, increase more.
> >
> > See
> >
> >
> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
> >
> > On Thu, Jul 7, 2011 at 10:52 AM, praveenesh kumar <praveenesh@gmail.com
> > >wrote:
> >
> > > I did that.. I increased it to 200.. still facing the same error.. what
> > > should be the ideal value ?
> > >
> > > On Thu, Jul 7, 2011 at 11:20 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Increase hbase.zookeeper.property.maxClientCnxns to much bigger
> value.
> > > >
> > > >
> > > > On Thu, Jul 7, 2011 at 10:46 AM, praveenesh kumar <
> > praveenesh@gmail.com
> > > > >wrote:
> > > >
> > > > > My HBase and pig both are running fine for me..
> > > > > Only problem is coming whenever I am trying to access big hbase
> table
> > > > > containing too many columns and rows..
> > > > > whenever I am trying to access hbase table from pig... I am getting
> > > this
> > > > > following error in the middle of the map process:
> > > > >
> > > > > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable
> to
> > > > > recreate exception from backed error:
> > > > > org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
> > to
> > > > > connect to ZooKeeper but the connection closes immediately. This
> > could
> > > be
> > > > a
> > > > > sign that the server has too many connections (30 is the default).
> > > > Consider
> > > > > inspecting your ZK server logs for that error and then make sure
> you
> > > are
> > > > > reusing HBaseConfiguration as often as you can. See HTable's
> javadoc
> > > for
> > > > > more information.
> > > > > But if I access a small HBase table from pig.. its working fine..
> no
> > > > > zookeeper exception is coming. I am able to do all other things
> > > perfectly
> > > > > fine.
> > > > > This error made me think that there is some problem in my zookeeper
> > > that
> > > > is
> > > > > why I was asking is there anyway to check whether the zookeeper are
> > > > working
> > > > > fine or not ?
> > > > >
> > > > > My zookeeper log -- http://pastebin.com/RRdzZYui
> > > > > My hbase-site.xml
> > > > >
> > > > > <configuration>
> > > > > <property>
> > > > > <name>hbase.rootdir</name>
> > > > > <value>hdfs://ub13:54310/hbase</value>
> > > > > <final>true</final>
> > > > > </property>
> > > > >
> > > > >
> > > > > <property>
> > > > > <name>hbase.zookeeper.quorum</name>
> > > > > <value>ub13,ub12,ub11</value>
> > > > > <final>true</final>
> > > > > </property>
> > > > >
> > > > > <property>
> > > > > <name>hbase.zookeeper.property.maxClientCnxns</name>
> > > > > <value>200</value>
> > > > > <final>true</final>
> > > > > </property>
> > > > >
> > > > > <property>
> > > > > <name>hbase.cluster.distributed</name>
> > > > > <value>true</value>
> > > > > <description>The mode the cluster will be in. Possible values are
> > > > > false: standalone and pseudo-distributed setups with managed
> > Zookeeper
> > > > > true: fully-distributed with unmanaged Zookeeper Quorum
> > > > > </description>
> > > > > </property>
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net> wrote:
> > > > >
> > > > > > On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <
> > > praveenesh@gmail.com
> > > > >
> > > > > > wrote:
> > > > > > > Can you please go through this and tell me what can be the
> issue
> > ?
> > > > > > >
> > > > > >
> > > > > > Can you summarize? (Life is short)
> > > > > >
> > > > > > If stuff works generally -- shell? And can you do the hbase
> > > > > > rowcounter mapreduce job successfully -- but then when you put
> PIG
> > in
> > > > > > the picture it does not, it would seem to be an issue with your
> PIG
> > > > > > setup?
> > > > > >
> > > > > > St.Ack
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by Barney Frank <ba...@gmail.com>.
Why not just set the maxClientCnxcns to 0? That would make it unlimited,
right?
On Thu, Jul 7, 2011 at 12:55 PM, Ted Yu <yu...@gmail.com> wrote:
> To 1000.
>
> Click on 'zk dump' link from Master UI.
> You would know roughly how many connections were established.
> If 1000 is exhausted, increase more.
>
> See
>
> http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
>
> On Thu, Jul 7, 2011 at 10:52 AM, praveenesh kumar <praveenesh@gmail.com
> >wrote:
>
> > I did that.. I increased it to 200.. still facing the same error.. what
> > should be the ideal value ?
> >
> > On Thu, Jul 7, 2011 at 11:20 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Increase hbase.zookeeper.property.maxClientCnxns to much bigger value.
> > >
> > >
> > > On Thu, Jul 7, 2011 at 10:46 AM, praveenesh kumar <
> praveenesh@gmail.com
> > > >wrote:
> > >
> > > > My HBase and pig both are running fine for me..
> > > > Only problem is coming whenever I am trying to access big hbase table
> > > > containing too many columns and rows..
> > > > whenever I am trying to access hbase table from pig... I am getting
> > this
> > > > following error in the middle of the map process:
> > > >
> > > > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
> > > > recreate exception from backed error:
> > > > org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able
> to
> > > > connect to ZooKeeper but the connection closes immediately. This
> could
> > be
> > > a
> > > > sign that the server has too many connections (30 is the default).
> > > Consider
> > > > inspecting your ZK server logs for that error and then make sure you
> > are
> > > > reusing HBaseConfiguration as often as you can. See HTable's javadoc
> > for
> > > > more information.
> > > > But if I access a small HBase table from pig.. its working fine.. no
> > > > zookeeper exception is coming. I am able to do all other things
> > perfectly
> > > > fine.
> > > > This error made me think that there is some problem in my zookeeper
> > that
> > > is
> > > > why I was asking is there anyway to check whether the zookeeper are
> > > working
> > > > fine or not ?
> > > >
> > > > My zookeeper log -- http://pastebin.com/RRdzZYui
> > > > My hbase-site.xml
> > > >
> > > > <configuration>
> > > > <property>
> > > > <name>hbase.rootdir</name>
> > > > <value>hdfs://ub13:54310/hbase</value>
> > > > <final>true</final>
> > > > </property>
> > > >
> > > >
> > > > <property>
> > > > <name>hbase.zookeeper.quorum</name>
> > > > <value>ub13,ub12,ub11</value>
> > > > <final>true</final>
> > > > </property>
> > > >
> > > > <property>
> > > > <name>hbase.zookeeper.property.maxClientCnxns</name>
> > > > <value>200</value>
> > > > <final>true</final>
> > > > </property>
> > > >
> > > > <property>
> > > > <name>hbase.cluster.distributed</name>
> > > > <value>true</value>
> > > > <description>The mode the cluster will be in. Possible values are
> > > > false: standalone and pseudo-distributed setups with managed
> Zookeeper
> > > > true: fully-distributed with unmanaged Zookeeper Quorum
> > > > </description>
> > > > </property>
> > > >
> > > >
> > > >
> > > > On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <
> > praveenesh@gmail.com
> > > >
> > > > > wrote:
> > > > > > Can you please go through this and tell me what can be the issue
> ?
> > > > > >
> > > > >
> > > > > Can you summarize? (Life is short)
> > > > >
> > > > > If stuff works generally -- shell? And can you do the hbase
> > > > > rowcounter mapreduce job successfully -- but then when you put PIG
> in
> > > > > the picture it does not, it would seem to be an issue with your PIG
> > > > > setup?
> > > > >
> > > > > St.Ack
> > > > >
> > > >
> > >
> >
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by Ted Yu <yu...@gmail.com>.
To 1000.
Click on 'zk dump' link from Master UI.
You would know roughly how many connections were established.
If 1000 is exhausted, increase more.
See
http://zhihongyu.blogspot.com/2011/04/managing-connections-in-hbase-090-and.html
On Thu, Jul 7, 2011 at 10:52 AM, praveenesh kumar <pr...@gmail.com>wrote:
> I did that.. I increased it to 200.. still facing the same error.. what
> should be the ideal value ?
>
> On Thu, Jul 7, 2011 at 11:20 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Increase hbase.zookeeper.property.maxClientCnxns to much bigger value.
> >
> >
> > On Thu, Jul 7, 2011 at 10:46 AM, praveenesh kumar <praveenesh@gmail.com
> > >wrote:
> >
> > > My HBase and pig both are running fine for me..
> > > Only problem is coming whenever I am trying to access big hbase table
> > > containing too many columns and rows..
> > > whenever I am trying to access hbase table from pig... I am getting
> this
> > > following error in the middle of the map process:
> > >
> > > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
> > > recreate exception from backed error:
> > > org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> > > connect to ZooKeeper but the connection closes immediately. This could
> be
> > a
> > > sign that the server has too many connections (30 is the default).
> > Consider
> > > inspecting your ZK server logs for that error and then make sure you
> are
> > > reusing HBaseConfiguration as often as you can. See HTable's javadoc
> for
> > > more information.
> > > But if I access a small HBase table from pig.. its working fine.. no
> > > zookeeper exception is coming. I am able to do all other things
> perfectly
> > > fine.
> > > This error made me think that there is some problem in my zookeeper
> that
> > is
> > > why I was asking is there anyway to check whether the zookeeper are
> > working
> > > fine or not ?
> > >
> > > My zookeeper log -- http://pastebin.com/RRdzZYui
> > > My hbase-site.xml
> > >
> > > <configuration>
> > > <property>
> > > <name>hbase.rootdir</name>
> > > <value>hdfs://ub13:54310/hbase</value>
> > > <final>true</final>
> > > </property>
> > >
> > >
> > > <property>
> > > <name>hbase.zookeeper.quorum</name>
> > > <value>ub13,ub12,ub11</value>
> > > <final>true</final>
> > > </property>
> > >
> > > <property>
> > > <name>hbase.zookeeper.property.maxClientCnxns</name>
> > > <value>200</value>
> > > <final>true</final>
> > > </property>
> > >
> > > <property>
> > > <name>hbase.cluster.distributed</name>
> > > <value>true</value>
> > > <description>The mode the cluster will be in. Possible values are
> > > false: standalone and pseudo-distributed setups with managed Zookeeper
> > > true: fully-distributed with unmanaged Zookeeper Quorum
> > > </description>
> > > </property>
> > >
> > >
> > >
> > > On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <
> praveenesh@gmail.com
> > >
> > > > wrote:
> > > > > Can you please go through this and tell me what can be the issue ?
> > > > >
> > > >
> > > > Can you summarize? (Life is short)
> > > >
> > > > If stuff works generally -- shell? And can you do the hbase
> > > > rowcounter mapreduce job successfully -- but then when you put PIG in
> > > > the picture it does not, it would seem to be an issue with your PIG
> > > > setup?
> > > >
> > > > St.Ack
> > > >
> > >
> >
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by praveenesh kumar <pr...@gmail.com>.
I did that.. I increased it to 200.. still facing the same error.. what
should be the ideal value ?
On Thu, Jul 7, 2011 at 11:20 PM, Ted Yu <yu...@gmail.com> wrote:
> Increase hbase.zookeeper.property.maxClientCnxns to much bigger value.
>
>
> On Thu, Jul 7, 2011 at 10:46 AM, praveenesh kumar <praveenesh@gmail.com
> >wrote:
>
> > My HBase and pig both are running fine for me..
> > Only problem is coming whenever I am trying to access big hbase table
> > containing too many columns and rows..
> > whenever I am trying to access hbase table from pig... I am getting this
> > following error in the middle of the map process:
> >
> > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
> > recreate exception from backed error:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> > connect to ZooKeeper but the connection closes immediately. This could be
> a
> > sign that the server has too many connections (30 is the default).
> Consider
> > inspecting your ZK server logs for that error and then make sure you are
> > reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> > more information.
> > But if I access a small HBase table from pig.. its working fine.. no
> > zookeeper exception is coming. I am able to do all other things perfectly
> > fine.
> > This error made me think that there is some problem in my zookeeper that
> is
> > why I was asking is there anyway to check whether the zookeeper are
> working
> > fine or not ?
> >
> > My zookeeper log -- http://pastebin.com/RRdzZYui
> > My hbase-site.xml
> >
> > <configuration>
> > <property>
> > <name>hbase.rootdir</name>
> > <value>hdfs://ub13:54310/hbase</value>
> > <final>true</final>
> > </property>
> >
> >
> > <property>
> > <name>hbase.zookeeper.quorum</name>
> > <value>ub13,ub12,ub11</value>
> > <final>true</final>
> > </property>
> >
> > <property>
> > <name>hbase.zookeeper.property.maxClientCnxns</name>
> > <value>200</value>
> > <final>true</final>
> > </property>
> >
> > <property>
> > <name>hbase.cluster.distributed</name>
> > <value>true</value>
> > <description>The mode the cluster will be in. Possible values are
> > false: standalone and pseudo-distributed setups with managed Zookeeper
> > true: fully-distributed with unmanaged Zookeeper Quorum
> > </description>
> > </property>
> >
> >
> >
> > On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net> wrote:
> >
> > > On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <praveenesh@gmail.com
> >
> > > wrote:
> > > > Can you please go through this and tell me what can be the issue ?
> > > >
> > >
> > > Can you summarize? (Life is short)
> > >
> > > If stuff works generally -- shell? And can you do the hbase
> > > rowcounter mapreduce job successfully -- but then when you put PIG in
> > > the picture it does not, it would seem to be an issue with your PIG
> > > setup?
> > >
> > > St.Ack
> > >
> >
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by Ted Yu <yu...@gmail.com>.
Increase hbase.zookeeper.property.maxClientCnxns to much bigger value.
On Thu, Jul 7, 2011 at 10:46 AM, praveenesh kumar <pr...@gmail.com>wrote:
> My HBase and pig both are running fine for me..
> Only problem is coming whenever I am trying to access big hbase table
> containing too many columns and rows..
> whenever I am trying to access hbase table from pig... I am getting this
> following error in the middle of the map process:
>
> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
> recreate exception from backed error:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
> But if I access a small HBase table from pig.. its working fine.. no
> zookeeper exception is coming. I am able to do all other things perfectly
> fine.
> This error made me think that there is some problem in my zookeeper that is
> why I was asking is there anyway to check whether the zookeeper are working
> fine or not ?
>
> My zookeeper log -- http://pastebin.com/RRdzZYui
> My hbase-site.xml
>
> <configuration>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://ub13:54310/hbase</value>
> <final>true</final>
> </property>
>
>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>ub13,ub12,ub11</value>
> <final>true</final>
> </property>
>
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>200</value>
> <final>true</final>
> </property>
>
> <property>
> <name>hbase.cluster.distributed</name>
> <value>true</value>
> <description>The mode the cluster will be in. Possible values are
> false: standalone and pseudo-distributed setups with managed Zookeeper
> true: fully-distributed with unmanaged Zookeeper Quorum
> </description>
> </property>
>
>
>
> On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net> wrote:
>
> > On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <pr...@gmail.com>
> > wrote:
> > > Can you please go through this and tell me what can be the issue ?
> > >
> >
> > Can you summarize? (Life is short)
> >
> > If stuff works generally -- shell? And can you do the hbase
> > rowcounter mapreduce job successfully -- but then when you put PIG in
> > the picture it does not, it would seem to be an issue with your PIG
> > setup?
> >
> > St.Ack
> >
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by praveenesh kumar <pr...@gmail.com>.
My HBase and pig both are running fine for me..
Only problem is coming whenever I am trying to access big hbase table
containing too many columns and rows..
whenever I am trying to access hbase table from pig... I am getting this
following error in the middle of the map process:
org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
recreate exception from backed error:
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could be a
sign that the server has too many connections (30 is the default). Consider
inspecting your ZK server logs for that error and then make sure you are
reusing HBaseConfiguration as often as you can. See HTable's javadoc for
more information.
But if I access a small HBase table from pig.. its working fine.. no
zookeeper exception is coming. I am able to do all other things perfectly
fine.
This error made me think that there is some problem in my zookeeper that is
why I was asking is there anyway to check whether the zookeeper are working
fine or not ?
My zookeeper log -- http://pastebin.com/RRdzZYui
My hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://ub13:54310/hbase</value>
<final>true</final>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>ub13,ub12,ub11</value>
<final>true</final>
</property>
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>200</value>
<final>true</final>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum
</description>
</property>
On Thu, Jul 7, 2011 at 10:47 PM, Stack <st...@duboce.net> wrote:
> On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <pr...@gmail.com>
> wrote:
> > Can you please go through this and tell me what can be the issue ?
> >
>
> Can you summarize? (Life is short)
>
> If stuff works generally -- shell? And can you do the hbase
> rowcounter mapreduce job successfully -- but then when you put PIG in
> the picture it does not, it would seem to be an issue with your PIG
> setup?
>
> St.Ack
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by Stack <st...@duboce.net>.
On Thu, Jul 7, 2011 at 8:54 AM, praveenesh kumar <pr...@gmail.com> wrote:
> Can you please go through this and tell me what can be the issue ?
>
Can you summarize? (Life is short)
If stuff works generally -- shell? And can you do the hbase
rowcounter mapreduce job successfully -- but then when you put PIG in
the picture it does not, it would seem to be an issue with your PIG
setup?
St.Ack
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by praveenesh kumar <pr...@gmail.com>.
yeah.. exactly.,,my pig jobs are failing.
I have send the issue in user@pig mailing list.. but still couldn't get good
response from anyone.
Here is the link of my post
http://mail-archives.apache.org/mod_mbox/pig-user/201107.mbox/%3cCAMbwnT__EMFG_C5spH+1mz=eSu3TDWsEhAdKvAjTdGSrT+qZ4g@mail.gmail.com%3e
I have written my issue there.. along with the exceptions and also post the
links of my zookeeper log.
Can you please go through this and tell me what can be the issue ?
Thanks,
Praveenesh
On Thu, Jul 7, 2011 at 2:22 AM, Stack <st...@duboce.net> wrote:
> What is the issue that you are seeing (I could look in your logs but
> I'd like to know first what it is I'm looking for). Are the PIG jobs
> failing? If so, what does fail look like?
> St.Ack
>
> On Tue, Jul 5, 2011 at 2:39 AM, praveenesh kumar <pr...@gmail.com>
> wrote:
> > Hi,
> > I have a 12-node hbase cluster setup with 3 nodes as a part of zookeeper
> > quorum.
> > I am able to run hbase shell, create tables.. and able to access tables
> in
> > the shell.
> >
> > Now I am configuring pig to use hbase. While accessing records using pig,
> it
> > is giving some Zookeeper exception and saying to check the zookeeper
> logs.
> >
> > I am sending the latest zookeeper logs of the 3 nodes along with
> > hbase-site.xml
> > Can anyone help me figuring out whether everything is okay with my
> zookeeper
> > and if not, what is the issue ?
> >
> > Hbase-site.xml ---> http://pastebin.com/8aJ7D54T
> >
> > Zookeeper log on ub11 ---> http://pastebin.com/HMuL9aCJ
> > Zookeeper log on ub12 ---> http://pastebin.com/8XdmVmDW
> > Zookeeper log on ub13 ---> http://pastebin.com/2373Rrat
> >
> > Thanks,
> > Praveenesh
> >
>
Re: how to check whether zookeeper quorum is working fine or not ?
Posted by Stack <st...@duboce.net>.
What is the issue that you are seeing (I could look in your logs but
I'd like to know first what it is I'm looking for). Are the PIG jobs
failing? If so, what does fail look like?
St.Ack
On Tue, Jul 5, 2011 at 2:39 AM, praveenesh kumar <pr...@gmail.com> wrote:
> Hi,
> I have a 12-node hbase cluster setup with 3 nodes as a part of zookeeper
> quorum.
> I am able to run hbase shell, create tables.. and able to access tables in
> the shell.
>
> Now I am configuring pig to use hbase. While accessing records using pig, it
> is giving some Zookeeper exception and saying to check the zookeeper logs.
>
> I am sending the latest zookeeper logs of the 3 nodes along with
> hbase-site.xml
> Can anyone help me figuring out whether everything is okay with my zookeeper
> and if not, what is the issue ?
>
> Hbase-site.xml ---> http://pastebin.com/8aJ7D54T
>
> Zookeeper log on ub11 ---> http://pastebin.com/HMuL9aCJ
> Zookeeper log on ub12 ---> http://pastebin.com/8XdmVmDW
> Zookeeper log on ub13 ---> http://pastebin.com/2373Rrat
>
> Thanks,
> Praveenesh
>