You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by kaveh minooie <ka...@plutoz.com> on 2013/02/20 23:46:04 UTC
HBase newbie
Hello everyone
I was wondering if any one has tried to run hbase 0.90.x on a hadoop
1.1.1 cluster? I have replaced a few of the jar files with the ones that
came with the packages that I am using in my cluster:
commons-configuration-1.6.jar
hadoop-core-1.1.1.jar
slf4j-api-1.6.1.jar
slf4j-log4j12-1.6.1.jar
zookeeper-3.4.5.jar (tried with both 3.3.2 and 3.4.5. same result)
after these changes I am able to run the region servers but the master
node still crashes when I try to start it:
> 2013-02-20 14:25:59,554 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to zk1/10.0.0.66:2181, initiating session
> 2013-02-20 14:25:59,556 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server zk1/10.0.0.66:2181, sessionid = 0x13cc3f3bcdc0095, negotiated timeout = 40000
> 2013-02-20 14:25:59,569 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@151205ab
> 2013-02-20 14:25:59,587 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
> java.lang.IllegalArgumentException: �^@^@^@^\31321@d1r2n2.prod.plutoz.comENABLED
> at org.apache.hadoop.hbase.zookeeper.ZKTable.getTableState(ZKTable.java:119)
> at org.apache.hadoop.hbase.zookeeper.ZKTable.populateTableStates(ZKTable.java:94)
> at org.apache.hadoop.hbase.zookeeper.ZKTable.<init>(ZKTable.java:79)
> at org.apache.hadoop.hbase.master.AssignmentManager.<init>(AssignmentManager.java:171)
> at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:353)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:281)
> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
> 2013-02-20 14:25:59,590 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000
> 2013-02-20 14:25:59,591 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: exiting
does anyone have any idea or suggestion about what I can do to be able
to run a hbase 0.90.x on hadoop 1.1.1?
thanks,
Kaveh Minooie
www.plutoz.com
Re: does hbase master need to be a hadoop datanode as well?
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Harish,
HBase still run ZooKeeper even in standalone mode:
http://hbase.apache.org/book/standalone_dist.html
So you still can face same kind of issues. But since it's not using Hadoop,
there is more memory available for you HBase, which might reduce GCs
"issues".
JM
2013/2/22 Harish Krishnan <ha...@gmail.com>
> So in case of a stand alone set up, how come we are not experiencing those
> issues?
> Is it because of the limited data set that we are operating on?
>
> Thanks & Regards,
> Harish.T.K
>
>
> On Fri, Feb 22, 2013 at 1:19 PM, Harsh J <ha...@cloudera.com> wrote:
>
> > While this general guideline of not requiring a DN on HM is correct, I
> > just wanted to point out that the HM is hardly high on memory
> > usage/requirements (as possibly stated below), as all it does is
> > maintain the cluster and not serve nor hold any client-useful data or
> > frequent requests.
> >
> > On Sat, Feb 23, 2013 at 1:13 AM, Mohammad Tariq <do...@gmail.com>
> > wrote:
> > > Hello Kaven,
> > >
> > > It's not all necessary to run HM on server running DN. Actually
> > it's
> > > not advisable to do so, as HM has high memory requirements and running
> > > other daemons on the same machine may lead you to problems. The basic
> > > cluster setup, which people normally use looks somewhat like this :
> > >
> > > NN+JT(Hadoop master)
> > > SNN
> > > HM(Hbase master)
> > > DN+TT+RS
> > >
> > >
> > > Warm Regards,
> > > Tariq
> > > https://mtariq.jux.com/
> > > cloudfront.blogspot.com
> > >
> > >
> > > On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com>
> > wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> I had a quick question. do I need to be running datanode on the server
> > >> that is running hbase master or datanode should be running only on
> > servers
> > >> that are running as regionserver?
> > >>
> > >> thanks,
> > >> --
> > >> Kaveh Minooie
> > >>
> >
> >
> >
> > --
> > Harsh J
> >
>
Re: does hbase master need to be a hadoop datanode as well?
Posted by Harish Krishnan <ha...@gmail.com>.
So in case of a stand alone set up, how come we are not experiencing those
issues?
Is it because of the limited data set that we are operating on?
Thanks & Regards,
Harish.T.K
On Fri, Feb 22, 2013 at 1:19 PM, Harsh J <ha...@cloudera.com> wrote:
> While this general guideline of not requiring a DN on HM is correct, I
> just wanted to point out that the HM is hardly high on memory
> usage/requirements (as possibly stated below), as all it does is
> maintain the cluster and not serve nor hold any client-useful data or
> frequent requests.
>
> On Sat, Feb 23, 2013 at 1:13 AM, Mohammad Tariq <do...@gmail.com>
> wrote:
> > Hello Kaven,
> >
> > It's not all necessary to run HM on server running DN. Actually
> it's
> > not advisable to do so, as HM has high memory requirements and running
> > other daemons on the same machine may lead you to problems. The basic
> > cluster setup, which people normally use looks somewhat like this :
> >
> > NN+JT(Hadoop master)
> > SNN
> > HM(Hbase master)
> > DN+TT+RS
> >
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com>
> wrote:
> >
> >> Hi everyone,
> >>
> >> I had a quick question. do I need to be running datanode on the server
> >> that is running hbase master or datanode should be running only on
> servers
> >> that are running as regionserver?
> >>
> >> thanks,
> >> --
> >> Kaveh Minooie
> >>
>
>
>
> --
> Harsh J
>
Re: does hbase master need to be a hadoop datanode as well?
Posted by Harsh J <ha...@cloudera.com>.
While this general guideline of not requiring a DN on HM is correct, I
just wanted to point out that the HM is hardly high on memory
usage/requirements (as possibly stated below), as all it does is
maintain the cluster and not serve nor hold any client-useful data or
frequent requests.
On Sat, Feb 23, 2013 at 1:13 AM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Kaven,
>
> It's not all necessary to run HM on server running DN. Actually it's
> not advisable to do so, as HM has high memory requirements and running
> other daemons on the same machine may lead you to problems. The basic
> cluster setup, which people normally use looks somewhat like this :
>
> NN+JT(Hadoop master)
> SNN
> HM(Hbase master)
> DN+TT+RS
>
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com> wrote:
>
>> Hi everyone,
>>
>> I had a quick question. do I need to be running datanode on the server
>> that is running hbase master or datanode should be running only on servers
>> that are running as regionserver?
>>
>> thanks,
>> --
>> Kaveh Minooie
>>
--
Harsh J
Re: does hbase master need to be a hadoop datanode as well?
Posted by Mohammad Tariq <do...@gmail.com>.
Yeah..Good point JM.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Sat, Feb 23, 2013 at 1:19 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Just to add to Mohammad's advices, you should avoid to run ZK on the same
> servers as you are running HBase.
>
> Reason is, if you are running in long GCs, ZK might miss the heartbeats and
> thinks servers are down. So safer to run same separately if you can.
>
> JM
>
> 2013/2/22 Mohammad Tariq <do...@gmail.com>
>
> > Hello Kaven,
> >
> > It's not all necessary to run HM on server running DN. Actually
> it's
> > not advisable to do so, as HM has high memory requirements and running
> > other daemons on the same machine may lead you to problems. The basic
> > cluster setup, which people normally use looks somewhat like this :
> >
> > NN+JT(Hadoop master)
> > SNN
> > HM(Hbase master)
> > DN+TT+RS
> >
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com>
> wrote:
> >
> > > Hi everyone,
> > >
> > > I had a quick question. do I need to be running datanode on the server
> > > that is running hbase master or datanode should be running only on
> > servers
> > > that are running as regionserver?
> > >
> > > thanks,
> > > --
> > > Kaveh Minooie
> > >
> >
>
Re: does hbase master need to be a hadoop datanode as well?
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
"as long as machine is powerful"
That's the key ;)
If you run you standalone HBase in a virtual machine, it might not be
powerful enought and you might miss the heartbeats. Same if you run on a
computer without enought memory and it's starting to swap.
JM
2013/2/23 anil gupta <an...@gmail.com>
> On Feb 22, 2013 11:50 AM, "Jean-Marc Spaggiari" <je...@spaggiari.org>
> wrote:
> >
> > Just to add to Mohammad's advices, you should avoid to run ZK on the same
> > servers as you are running HBase.
> >
> > Reason is, if you are running in long GCs, ZK might miss the heartbeats
> and
> > thinks servers are down. So safer to run same separately if you can.
> >
> Zk and HM run in seperate JVM; so how would running them on the same
> machine will cause ZK missing hearbeats?
> As far as I know as long as machine is powerful enough to handle the load
> of ZK and HM, then it should be fine. One thing to note is that Zk will
> work well if you have dedicated disk for it.
> > JM
> >
> > 2013/2/22 Mohammad Tariq <do...@gmail.com>
> >
> > > Hello Kaven,
> > >
> > > It's not all necessary to run HM on server running DN. Actually
> it's
> > > not advisable to do so, as HM has high memory requirements and running
> > > other daemons on the same machine may lead you to problems. The basic
> > > cluster setup, which people normally use looks somewhat like this :
> > >
> > > NN+JT(Hadoop master)
> > > SNN
> > > HM(Hbase master)
> > > DN+TT+RS
> > >
> > >
> > > Warm Regards,
> > > Tariq
> > > https://mtariq.jux.com/
> > > cloudfront.blogspot.com
> > >
> > >
> > > On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com>
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I had a quick question. do I need to be running datanode on the
> server
> > > > that is running hbase master or datanode should be running only on
> > > servers
> > > > that are running as regionserver?
> > > >
> > > > thanks,
> > > > --
> > > > Kaveh Minooie
> > > >
> > >
>
Re: does hbase master need to be a hadoop datanode as well?
Posted by anil gupta <an...@gmail.com>.
On Feb 22, 2013 11:50 AM, "Jean-Marc Spaggiari" <je...@spaggiari.org>
wrote:
>
> Just to add to Mohammad's advices, you should avoid to run ZK on the same
> servers as you are running HBase.
>
> Reason is, if you are running in long GCs, ZK might miss the heartbeats
and
> thinks servers are down. So safer to run same separately if you can.
>
Zk and HM run in seperate JVM; so how would running them on the same
machine will cause ZK missing hearbeats?
As far as I know as long as machine is powerful enough to handle the load
of ZK and HM, then it should be fine. One thing to note is that Zk will
work well if you have dedicated disk for it.
> JM
>
> 2013/2/22 Mohammad Tariq <do...@gmail.com>
>
> > Hello Kaven,
> >
> > It's not all necessary to run HM on server running DN. Actually
it's
> > not advisable to do so, as HM has high memory requirements and running
> > other daemons on the same machine may lead you to problems. The basic
> > cluster setup, which people normally use looks somewhat like this :
> >
> > NN+JT(Hadoop master)
> > SNN
> > HM(Hbase master)
> > DN+TT+RS
> >
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com>
wrote:
> >
> > > Hi everyone,
> > >
> > > I had a quick question. do I need to be running datanode on the server
> > > that is running hbase master or datanode should be running only on
> > servers
> > > that are running as regionserver?
> > >
> > > thanks,
> > > --
> > > Kaveh Minooie
> > >
> >
Re: does hbase master need to be a hadoop datanode as well?
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Just to add to Mohammad's advices, you should avoid to run ZK on the same
servers as you are running HBase.
Reason is, if you are running in long GCs, ZK might miss the heartbeats and
thinks servers are down. So safer to run same separately if you can.
JM
2013/2/22 Mohammad Tariq <do...@gmail.com>
> Hello Kaven,
>
> It's not all necessary to run HM on server running DN. Actually it's
> not advisable to do so, as HM has high memory requirements and running
> other daemons on the same machine may lead you to problems. The basic
> cluster setup, which people normally use looks somewhat like this :
>
> NN+JT(Hadoop master)
> SNN
> HM(Hbase master)
> DN+TT+RS
>
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com> wrote:
>
> > Hi everyone,
> >
> > I had a quick question. do I need to be running datanode on the server
> > that is running hbase master or datanode should be running only on
> servers
> > that are running as regionserver?
> >
> > thanks,
> > --
> > Kaveh Minooie
> >
>
Re: does hbase master need to be a hadoop datanode as well?
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Kaven,
It's not all necessary to run HM on server running DN. Actually it's
not advisable to do so, as HM has high memory requirements and running
other daemons on the same machine may lead you to problems. The basic
cluster setup, which people normally use looks somewhat like this :
NN+JT(Hadoop master)
SNN
HM(Hbase master)
DN+TT+RS
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com> wrote:
> Hi everyone,
>
> I had a quick question. do I need to be running datanode on the server
> that is running hbase master or datanode should be running only on servers
> that are running as regionserver?
>
> thanks,
> --
> Kaveh Minooie
>
does hbase master need to be a hadoop datanode as well?
Posted by kaveh minooie <ka...@plutoz.com>.
Hi everyone,
I had a quick question. do I need to be running datanode on the server
that is running hbase master or datanode should be running only on
servers that are running as regionserver?
thanks,
--
Kaveh Minooie
Re: issue with nutch-gora+hbase+zookeeper
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Kaveh,
This is great news.
Thanks for persisting. I will update the Nutch wiki accordingly.
Lewis
On Fri, Feb 22, 2013 at 8:57 PM, kaveh minooie <ka...@plutoz.com> wrote:
> In case anyone was wondering, the issue was resolved by copying the
> zoo.cfg in the hadoop conf directory (on classpath really) on the entire
> cluster. thanks
>
>
> On 02/22/2013 12:31 PM, kaveh minooie wrote:
>
>> Hi everyone
>>
>> I am having this problem for couple of days now and would appreciate any
>> idea or suggestion that any one might have. I am using nutch 2.x with
>> hbase. due to nutch requirement I need to use older version of hbase (I am
>> using 0.90.6 over hadoop 1.1.1 with 10 nodes with zookeepr 3.5.0 [trunk] )
>>
>> hbase seems to be running fine, thou I appreciate if someone can show me
>> how I can actually test it systematically. but I seem to be able to create
>> and read data from hbase, but when I run any nutch command something very
>> similar to this happens as soon as the job starts running. this is for
>> example, the output of nutch inject
>>
>> 13/02/22 12:07:30 INFO mapred.JobClient: map 0% reduce 0%
>> 13/02/22 12:07:52 INFO mapred.JobClient: Task Id :
>> attempt_201302191325_0013_m_**000000_0, Status : FAILED
>> org.apache.gora.util.**GoraException: org.apache.hadoop.hbase.**ZooKeeperConnectionException:
>> HBase is able to connect to ZooKeeper but the connection closes
>> immediately. This could be a sign that the server has too many connections
>> (30 is the default). Consider inspecting your ZK server logs for that error
>> and then make sure you are reusing HBaseConfiguration as often as you can.
>> See HTable's javadoc for more information.
>> at org.apache.gora.store.**DataStoreFactory.**createDataStore(**
>> DataStoreFactory.java:167)
>> at org.apache.gora.store.**DataStoreFactory.**createDataStore(**
>> DataStoreFactory.java:118)
>> at org.apache.gora.mapreduce.**GoraOutputFormat.**getRecordWriter(**
>> GoraOutputFormat.java:88)
>> at org.apache.hadoop.mapred.**MapTask$**NewDirectOutputCollector.<**
>> init>(MapTask.java:628)
>> at org.apache.hadoop.mapred.**MapTask.runNewMapper(MapTask.**
>> java:753)
>> at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.**Child$4.run(Child.java:255)
>> at java.security.**AccessController.doPrivileged(**Native Method)
>> at javax.security.auth.Subject.**doAs(Unknown Source)
>> at org.apache.hadoop.security.**UserGroupInformation.doAs(**
>> UserGroupInformation.java:**1136)
>> at org.apache.hadoop.mapred.**Child.main(Child.java:249)
>> Caused by: org.apache.hadoop.hbase.**ZooKeeperConnectionException: HBase
>> is able to connect to ZooKeeper but the connection closes immediately. This
>> could be a sign that the server has too many connections (30 is the
>> default). Consider inspecting your ZK server logs for that error and then
>> make sure you are reusing HBaseConfiguration as often as you can. See
>> HTable's javadoc for more information.
>> at org.apache.hadoop.hbase.**zookeeper.ZooKeeperWatcher.<**
>> init>(ZooKeeperWatcher.java:**156)
>> at org.apache.hadoop.hbase.**client.HConnectionManager$**
>> HConnectionImplementation.**getZooKeeperWatcher(**
>> HConnectionManager.java:1265)
>> at org.apache.hadoop.hbase.**client.HConnectionManager$**
>> HConnectionImplementation.**setupZookeeperTrackers(**
>> HConnectionManager.java:526)
>> at org.apache.hadoop.hbase.**client.HConnectionManager$**
>> HConnectionImplementation.<**init>(HConnectionManager.java:**516)
>> at org.apache.hadoop.hbase.**client.HConnectionManager.**
>> getConnection(**HConnectionManager.java:173)
>> at org.apache.hadoop.hbase.**client.HBaseAdmin.<init>(**
>> HBaseAdmin.java:93)
>> at org.apache.gora.hbase.store.**HBaseStore.initialize(**
>> HBaseStore.java:108)
>> at org.apache.gora.store.**DataStoreFactory.**initializeDataStore(**
>> DataStoreFactory.java:102)
>> at org.apache.gora.store.**DataStoreFactory.**createDataStore(**
>> DataStoreFactory.java:161)
>> ... 10 more
>> Caused by: org.apache.zookeeper.**KeeperException$**ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /hbase
>> at org.apache.zookeeper.**KeeperException.create(**
>> KeeperException.java:99)
>> at org.apache.zookeeper.**KeeperException.create(**
>> KeeperException.java:51)
>> at org.apache.zookeeper.**ZooKeeper.exists(ZooKeeper.**java:1237)
>> at org.apache.zookeeper.**ZooKeeper.exists(ZooKeeper.**java:1265)
>> at org.apache.hadoop.hbase.**zookeeper.ZKUtil.**
>> createAndFailSilent(ZKUtil.**java:931)
>> at org.apache.hadoop.hbase.**zookeeper.ZooKeeperWatcher.<**
>> init>(ZooKeeperWatcher.java:**134)
>> ... 18 more
>>
>>
>>
>> Now I know that I am not running out of connection. for one thing I have
>> increased the number of connection to 200 in zoo.cfg, and also here is what
>> is in the zookeeper log file around that time:
>>
>> 2013-02-22 12:07:27,704 [myid:] - INFO [NIOServerCxnFactory.**
>> AcceptThread:0.0.0.0/0.0.0.0:**2181:NIOServerCnxnFactory$**
>> AcceptThread@289<ht...@289>]
>> - Accepted socket connection from /127.0.0.1:55073
>> 2013-02-22 12:07:27,707 [myid:] - INFO [NIOWorkerThread-3:**
>> ZooKeeperServer@810] - Client attempting to establish new session at /
>> 127.0.0.1:55073
>> 2013-02-22 12:07:27,720 [myid:] - INFO [SyncThread:0:ZooKeeperServer@**566]
>> - Established session 0x13d037b8e6b0016 with negotiated timeout 40000 for
>> client /127.0.0.1:55073
>> 2013-02-22 12:07:27,945 [myid:] - INFO [NIOServerCxnFactory.**
>> AcceptThread:0.0.0.0/0.0.0.0:**2181:NIOServerCnxnFactory$**
>> AcceptThread@289<ht...@289>]
>> - Accepted socket connection from /127.0.0.1:55075
>> 2013-02-22 12:07:27,946 [myid:] - INFO [NIOWorkerThread-2:**
>> ZooKeeperServer@810] - Client attempting to establish new session at /
>> 127.0.0.1:55075
>> 2013-02-22 12:07:27,953 [myid:] - INFO [SyncThread:0:ZooKeeperServer@**566]
>> - Established session 0x13d037b8e6b0017 with negotiated timeout 40000 for
>> client /127.0.0.1:55075
>> 2013-02-22 12:07:28,010 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::*
>> *PrepRequestProcessor@533] - Processed session termination for
>> sessionid: 0x13d037b8e6b0017
>> 2013-02-22 12:07:28,011 [myid:] - INFO [NIOWorkerThread-6:**
>> NIOServerCnxn@1000] - Closed socket connection for client /
>> 127.0.0.1:55075 which had sessionid 0x13d037b8e6b0017
>> 2013-02-22 12:08:14,005 [myid:] - WARN [NIOWorkerThread-7:**
>> NIOServerCnxn@362] - Unable to read additional data from client
>> sessionid 0x13d037b8e6b0016, likely client has closed socket
>> 2013-02-22 12:08:14,005 [myid:] - INFO [NIOWorkerThread-7:**
>> NIOServerCnxn@1000] - Closed socket connection for client /
>> 127.0.0.1:55073 which had sessionid 0x13d037b8e6b0016
>> 2013-02-22 12:08:48,000 [myid:] - INFO [SessionTracker:**
>> ZooKeeperServer@304] - Expiring session 0x13d037b8e6b0016, timeout of
>> 40000ms exceeded
>> 2013-02-22 12:08:48,001 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::*
>> *PrepRequestProcessor@533] - Processed session termination for
>> sessionid: 0x13d037b8e6b0016
>>
>>
>> I also don't think that it is a heartbeat or GC related issue since there
>> is really no load at all on these servers right now. I know this is a
>> hybrid problem involving three separate product (nutch, hbase, zookeeper)
>> but I am asking this in all the mailing list. Also I am gonna say it agian
>> in order to avoid confusion with the similar problems in older versions
>> that are supposedly solved. I am using hbase 0.90.6 and zookeeper 3.5.0
>> (commit 46b565e6) with nutch 2.x (commit f02dcf625 ) both are either the
>> latest or very recent updates.
>>
>> if anyone has any idea what is happening here I very much like to hear
>> that.
>>
>> thanks,
>>
>>
>>
>
--
*Lewis*
Re: issue with nutch-gora+hbase+zookeeper
Posted by kaveh minooie <ka...@plutoz.com>.
In case anyone was wondering, the issue was resolved by copying the
zoo.cfg in the hadoop conf directory (on classpath really) on the entire
cluster. thanks
On 02/22/2013 12:31 PM, kaveh minooie wrote:
> Hi everyone
>
> I am having this problem for couple of days now and would appreciate
> any idea or suggestion that any one might have. I am using nutch 2.x
> with hbase. due to nutch requirement I need to use older version of
> hbase (I am using 0.90.6 over hadoop 1.1.1 with 10 nodes with
> zookeepr 3.5.0 [trunk] )
>
> hbase seems to be running fine, thou I appreciate if someone can show
> me how I can actually test it systematically. but I seem to be able to
> create and read data from hbase, but when I run any nutch command
> something very similar to this happens as soon as the job starts
> running. this is for example, the output of nutch inject
>
> 13/02/22 12:07:30 INFO mapred.JobClient: map 0% reduce 0%
> 13/02/22 12:07:52 INFO mapred.JobClient: Task Id :
> attempt_201302191325_0013_m_000000_0, Status : FAILED
> org.apache.gora.util.GoraException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could
> be a sign that the server has too many connections (30 is the
> default). Consider inspecting your ZK server logs for that error and
> then make sure you are reusing HBaseConfiguration as often as you can.
> See HTable's javadoc for more information.
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)
> at
> org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:628)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase
> is able to connect to ZooKeeper but the connection closes immediately.
> This could be a sign that the server has too many connections (30 is
> the default). Consider inspecting your ZK server logs for that error
> and then make sure you are reusing HBaseConfiguration as often as you
> can. See HTable's javadoc for more information.
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:156)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1265)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:526)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:516)
> at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:173)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:93)
> at
> org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:108)
> at
> org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
> ... 10 more
> Caused by:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1237)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1265)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:931)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
> ... 18 more
>
>
>
> Now I know that I am not running out of connection. for one thing I
> have increased the number of connection to 200 in zoo.cfg, and also
> here is what is in the zookeeper log file around that time:
>
> 2013-02-22 12:07:27,704 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
> - Accepted socket connection from /127.0.0.1:55073
> 2013-02-22 12:07:27,707 [myid:] - INFO
> [NIOWorkerThread-3:ZooKeeperServer@810] - Client attempting to
> establish new session at /127.0.0.1:55073
> 2013-02-22 12:07:27,720 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@566] - Established session
> 0x13d037b8e6b0016 with negotiated timeout 40000 for client
> /127.0.0.1:55073
> 2013-02-22 12:07:27,945 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
> - Accepted socket connection from /127.0.0.1:55075
> 2013-02-22 12:07:27,946 [myid:] - INFO
> [NIOWorkerThread-2:ZooKeeperServer@810] - Client attempting to
> establish new session at /127.0.0.1:55075
> 2013-02-22 12:07:27,953 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@566] - Established session
> 0x13d037b8e6b0017 with negotiated timeout 40000 for client
> /127.0.0.1:55075
> 2013-02-22 12:07:28,010 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@533] - Processed session termination
> for sessionid: 0x13d037b8e6b0017
> 2013-02-22 12:07:28,011 [myid:] - INFO
> [NIOWorkerThread-6:NIOServerCnxn@1000] - Closed socket connection for
> client /127.0.0.1:55075 which had sessionid 0x13d037b8e6b0017
> 2013-02-22 12:08:14,005 [myid:] - WARN
> [NIOWorkerThread-7:NIOServerCnxn@362] - Unable to read additional data
> from client sessionid 0x13d037b8e6b0016, likely client has closed socket
> 2013-02-22 12:08:14,005 [myid:] - INFO
> [NIOWorkerThread-7:NIOServerCnxn@1000] - Closed socket connection for
> client /127.0.0.1:55073 which had sessionid 0x13d037b8e6b0016
> 2013-02-22 12:08:48,000 [myid:] - INFO
> [SessionTracker:ZooKeeperServer@304] - Expiring session
> 0x13d037b8e6b0016, timeout of 40000ms exceeded
> 2013-02-22 12:08:48,001 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@533] - Processed session termination
> for sessionid: 0x13d037b8e6b0016
>
>
> I also don't think that it is a heartbeat or GC related issue since
> there is really no load at all on these servers right now. I know this
> is a hybrid problem involving three separate product (nutch, hbase,
> zookeeper) but I am asking this in all the mailing list. Also I am
> gonna say it agian in order to avoid confusion with the similar
> problems in older versions that are supposedly solved. I am using
> hbase 0.90.6 and zookeeper 3.5.0 (commit 46b565e6) with nutch 2.x
> (commit f02dcf625 ) both are either the latest or very recent updates.
>
> if anyone has any idea what is happening here I very much like to hear
> that.
>
> thanks,
>
>
Re: issue with nutch-gora+hbase+zookeeper
Posted by kaveh minooie <ka...@plutoz.com>.
In case anyone was wondering, the issue was resolved by copying the
zoo.cfg in the hadoop conf directory (on classpath really) on the entire
cluster. thanks
On 02/22/2013 12:31 PM, kaveh minooie wrote:
> Hi everyone
>
> I am having this problem for couple of days now and would appreciate
> any idea or suggestion that any one might have. I am using nutch 2.x
> with hbase. due to nutch requirement I need to use older version of
> hbase (I am using 0.90.6 over hadoop 1.1.1 with 10 nodes with
> zookeepr 3.5.0 [trunk] )
>
> hbase seems to be running fine, thou I appreciate if someone can show
> me how I can actually test it systematically. but I seem to be able to
> create and read data from hbase, but when I run any nutch command
> something very similar to this happens as soon as the job starts
> running. this is for example, the output of nutch inject
>
> 13/02/22 12:07:30 INFO mapred.JobClient: map 0% reduce 0%
> 13/02/22 12:07:52 INFO mapred.JobClient: Task Id :
> attempt_201302191325_0013_m_000000_0, Status : FAILED
> org.apache.gora.util.GoraException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could
> be a sign that the server has too many connections (30 is the
> default). Consider inspecting your ZK server logs for that error and
> then make sure you are reusing HBaseConfiguration as often as you can.
> See HTable's javadoc for more information.
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)
> at
> org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:628)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase
> is able to connect to ZooKeeper but the connection closes immediately.
> This could be a sign that the server has too many connections (30 is
> the default). Consider inspecting your ZK server logs for that error
> and then make sure you are reusing HBaseConfiguration as often as you
> can. See HTable's javadoc for more information.
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:156)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1265)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:526)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:516)
> at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:173)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:93)
> at
> org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:108)
> at
> org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
> ... 10 more
> Caused by:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1237)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1265)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:931)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
> ... 18 more
>
>
>
> Now I know that I am not running out of connection. for one thing I
> have increased the number of connection to 200 in zoo.cfg, and also
> here is what is in the zookeeper log file around that time:
>
> 2013-02-22 12:07:27,704 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
> - Accepted socket connection from /127.0.0.1:55073
> 2013-02-22 12:07:27,707 [myid:] - INFO
> [NIOWorkerThread-3:ZooKeeperServer@810] - Client attempting to
> establish new session at /127.0.0.1:55073
> 2013-02-22 12:07:27,720 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@566] - Established session
> 0x13d037b8e6b0016 with negotiated timeout 40000 for client
> /127.0.0.1:55073
> 2013-02-22 12:07:27,945 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
> - Accepted socket connection from /127.0.0.1:55075
> 2013-02-22 12:07:27,946 [myid:] - INFO
> [NIOWorkerThread-2:ZooKeeperServer@810] - Client attempting to
> establish new session at /127.0.0.1:55075
> 2013-02-22 12:07:27,953 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@566] - Established session
> 0x13d037b8e6b0017 with negotiated timeout 40000 for client
> /127.0.0.1:55075
> 2013-02-22 12:07:28,010 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@533] - Processed session termination
> for sessionid: 0x13d037b8e6b0017
> 2013-02-22 12:07:28,011 [myid:] - INFO
> [NIOWorkerThread-6:NIOServerCnxn@1000] - Closed socket connection for
> client /127.0.0.1:55075 which had sessionid 0x13d037b8e6b0017
> 2013-02-22 12:08:14,005 [myid:] - WARN
> [NIOWorkerThread-7:NIOServerCnxn@362] - Unable to read additional data
> from client sessionid 0x13d037b8e6b0016, likely client has closed socket
> 2013-02-22 12:08:14,005 [myid:] - INFO
> [NIOWorkerThread-7:NIOServerCnxn@1000] - Closed socket connection for
> client /127.0.0.1:55073 which had sessionid 0x13d037b8e6b0016
> 2013-02-22 12:08:48,000 [myid:] - INFO
> [SessionTracker:ZooKeeperServer@304] - Expiring session
> 0x13d037b8e6b0016, timeout of 40000ms exceeded
> 2013-02-22 12:08:48,001 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@533] - Processed session termination
> for sessionid: 0x13d037b8e6b0016
>
>
> I also don't think that it is a heartbeat or GC related issue since
> there is really no load at all on these servers right now. I know this
> is a hybrid problem involving three separate product (nutch, hbase,
> zookeeper) but I am asking this in all the mailing list. Also I am
> gonna say it agian in order to avoid confusion with the similar
> problems in older versions that are supposedly solved. I am using
> hbase 0.90.6 and zookeeper 3.5.0 (commit 46b565e6) with nutch 2.x
> (commit f02dcf625 ) both are either the latest or very recent updates.
>
> if anyone has any idea what is happening here I very much like to hear
> that.
>
> thanks,
>
>
Re: issue with nutch-gora+hbase+zookeeper
Posted by ka...@plutoz.com.
all the ports are open. there is no iptable.
-----Original Message-----
From: alxsss@aim.com
Sent: Friday, February 22, 2013 1:02pm
To: user@hbase.apache.org
Subject: Re: issue with nutch-gora+hbase+zookeeper
Turn off iptables and see if it resolves errors. You need to open all ports that hbase and zookeeper need to connect each node.
Alex.
-----Original Message-----
From: kaveh minooie <ka...@plutoz.com>
To: user <us...@hbase.apache.org>
Sent: Fri, Feb 22, 2013 12:56 pm
Subject: Re: issue with nutch-gora+hbase+zookeeper
no here are the logs, master:
2013-02-22 12:03:21,136 INFO
org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
servers=10 regions=3 average=0.3 mostloaded=1 leastloaded=0
2013-02-22 12:03:21,141 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 1 catalog row(s)
and gc'd 0 unreferenced parent region(s)
2013-02-22 12:08:21,135 DEBUG
org.apache.hadoop.hbase.master.LoadBalancer: Server information:
d1r1n12.prod.plutoz.com,60020,1361563095874=1,
d1r1n16.prod.plutoz.com,60020,1361563095878
=0, d1r1n17.prod.plutoz.com,60020,1361563095900=0,
d1r1n13.prod.plutoz.com,60020,1361563095880=0,
d1r1n14.prod.plutoz.com,60020,1361563095886=0,
d1r1n10.prod.plutoz.com,60020,1361563095853=1,
d1r1n20.prod.plutoz.com,60020,1361563095890=1,
d1r1n15.prod.plutoz.com,60020,1361563095870=0,
d1r1n21.prod.plutoz.com,60020,1361563095888=0,
d1r1n19.prod.plutoz.com,60020,1361563095879=0
2013-02-22 12:08:21,135 INFO
org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
servers=10 regions=3 average=0.3 mostloaded=1 leastloaded=0
2013-02-22 12:08:21,142 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 1 catalog row(s)
and gc'd 0 unreferenced parent region(s)
one of the regionservers:
2013-02-22 12:03:17,199 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=?%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=?%,
evictions=0, evicted=0, evictedPerRun=NaN
2013-02-22 12:08:17,198 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=?%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=?%,
evictions=0, evicted=0, evictedPerRun=NaN
2013-02-22 12:13:17,198 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=?%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=?%,
evictions=0, evicted=0, evictedPerRun=NaN
another regionserver:
2013-02-22 12:03:17,279 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=�%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%,
evictions=0, evicted=0, evictedPerRun=NaN
2013-02-22 12:08:17,278 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=�%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%,
evictions=0, evicted=0, evictedPerRun=NaN
2013-02-22 12:13:17,278 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=�%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%,
evictions=0, evicted=0, evictedPerRun=NaN
On 02/22/2013 12:49 PM, alxsss@aim.com wrote:
> Hi,
>
> Do you see any errors in HMaster and Regionserver logs?
>
> Alex.
>
>
>
>
>
>
>
> -----Original Message-----
> From: kaveh minooie <ka...@plutoz.com>
> To: user <us...@hbase.apache.org>; user <us...@nutch.apache.org>; user
<us...@zookeeper.apache.org>
> Sent: Fri, Feb 22, 2013 12:31 pm
> Subject: issue with nutch-gora+hbase+zookeeper
>
>
> Hi everyone
>
> I am having this problem for couple of days now and would appreciate any
> idea or suggestion that any one might have. I am using nutch 2.x with
> hbase. due to nutch requirement I need to use older version of hbase (I
> am using 0.90.6 over hadoop 1.1.1 with 10 nodes with zookeepr 3.5.0
> [trunk] )
>
> hbase seems to be running fine, thou I appreciate if someone can show me
> how I can actually test it systematically. but I seem to be able to
> create and read data from hbase, but when I run any nutch command
> something very similar to this happens as soon as the job starts
> running. this is for example, the output of nutch inject
>
> 13/02/22 12:07:30 INFO mapred.JobClient: map 0% reduce 0%
> 13/02/22 12:07:52 INFO mapred.JobClient: Task Id :
> attempt_201302191325_0013_m_000000_0, Status : FAILED
> org.apache.gora.util.GoraException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could
> be a sign that the server has too many connections (30 is the default).
> Consider inspecting your ZK server logs for that error and then make
> sure you are reusing HBaseConfiguration as often as you can. See
> HTable's javadoc for more information.
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)
> at
> org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:628)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase
> is able to connect to ZooKeeper but the connection closes immediately.
> This could be a sign that the server has too many connections (30 is the
> default). Consider inspecting your ZK server logs for that error and
> then make sure you are reusing HBaseConfiguration as often as you can.
> See HTable's javadoc for more information.
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:156)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1265)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:526)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:516)
> at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:173)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:93)
> at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:108)
> at
> org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
> ... 10 more
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1237)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1265)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:931)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
> ... 18 more
>
>
>
> Now I know that I am not running out of connection. for one thing I have
> increased the number of connection to 200 in zoo.cfg, and also here is
> what is in the zookeeper log file around that time:
>
> 2013-02-22 12:07:27,704 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
>
> - Accepted socket connection from /127.0.0.1:55073
> 2013-02-22 12:07:27,707 [myid:] - INFO
> [NIOWorkerThread-3:ZooKeeperServer@810] - Client attempting to establish
> new session at /127.0.0.1:55073
> 2013-02-22 12:07:27,720 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@566] - Established session
> 0x13d037b8e6b0016 with negotiated timeout 40000 for client /127.0.0.1:55073
> 2013-02-22 12:07:27,945 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
>
> - Accepted socket connection from /127.0.0.1:55075
> 2013-02-22 12:07:27,946 [myid:] - INFO
> [NIOWorkerThread-2:ZooKeeperServer@810] - Client attempting to establish
> new session at /127.0.0.1:55075
> 2013-02-22 12:07:27,953 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@566] - Established session
> 0x13d037b8e6b0017 with negotiated timeout 40000 for client /127.0.0.1:55075
> 2013-02-22 12:07:28,010 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@533] - Processed session termination for
> sessionid: 0x13d037b8e6b0017
> 2013-02-22 12:07:28,011 [myid:] - INFO
> [NIOWorkerThread-6:NIOServerCnxn@1000] - Closed socket connection for
> client /127.0.0.1:55075 which had sessionid 0x13d037b8e6b0017
> 2013-02-22 12:08:14,005 [myid:] - WARN
> [NIOWorkerThread-7:NIOServerCnxn@362] - Unable to read additional data
> from client sessionid 0x13d037b8e6b0016, likely client has closed socket
> 2013-02-22 12:08:14,005 [myid:] - INFO
> [NIOWorkerThread-7:NIOServerCnxn@1000] - Closed socket connection for
> client /127.0.0.1:55073 which had sessionid 0x13d037b8e6b0016
> 2013-02-22 12:08:48,000 [myid:] - INFO
> [SessionTracker:ZooKeeperServer@304] - Expiring session
> 0x13d037b8e6b0016, timeout of 40000ms exceeded
> 2013-02-22 12:08:48,001 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@533] - Processed session termination for
> sessionid: 0x13d037b8e6b0016
>
>
> I also don't think that it is a heartbeat or GC related issue since
> there is really no load at all on these servers right now. I know this
> is a hybrid problem involving three separate product (nutch, hbase,
> zookeeper) but I am asking this in all the mailing list. Also I am gonna
> say it agian in order to avoid confusion with the similar problems in
> older versions that are supposedly solved. I am using hbase 0.90.6 and
> zookeeper 3.5.0 (commit 46b565e6) with nutch 2.x (commit f02dcf625 )
> both are either the latest or very recent updates.
>
> if anyone has any idea what is happening here I very much like to hear
> that.
>
> thanks,
>
>
--
Kaveh Minooie
Re: issue with nutch-gora+hbase+zookeeper
Posted by al...@aim.com.
Turn off iptables and see if it resolves errors. You need to open all ports that hbase and zookeeper need to connect each node.
Alex.
-----Original Message-----
From: kaveh minooie <ka...@plutoz.com>
To: user <us...@hbase.apache.org>
Sent: Fri, Feb 22, 2013 12:56 pm
Subject: Re: issue with nutch-gora+hbase+zookeeper
no here are the logs, master:
2013-02-22 12:03:21,136 INFO
org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
servers=10 regions=3 average=0.3 mostloaded=1 leastloaded=0
2013-02-22 12:03:21,141 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 1 catalog row(s)
and gc'd 0 unreferenced parent region(s)
2013-02-22 12:08:21,135 DEBUG
org.apache.hadoop.hbase.master.LoadBalancer: Server information:
d1r1n12.prod.plutoz.com,60020,1361563095874=1,
d1r1n16.prod.plutoz.com,60020,1361563095878
=0, d1r1n17.prod.plutoz.com,60020,1361563095900=0,
d1r1n13.prod.plutoz.com,60020,1361563095880=0,
d1r1n14.prod.plutoz.com,60020,1361563095886=0,
d1r1n10.prod.plutoz.com,60020,1361563095853=1,
d1r1n20.prod.plutoz.com,60020,1361563095890=1,
d1r1n15.prod.plutoz.com,60020,1361563095870=0,
d1r1n21.prod.plutoz.com,60020,1361563095888=0,
d1r1n19.prod.plutoz.com,60020,1361563095879=0
2013-02-22 12:08:21,135 INFO
org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
servers=10 regions=3 average=0.3 mostloaded=1 leastloaded=0
2013-02-22 12:08:21,142 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 1 catalog row(s)
and gc'd 0 unreferenced parent region(s)
one of the regionservers:
2013-02-22 12:03:17,199 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=?%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=?%,
evictions=0, evicted=0, evictedPerRun=NaN
2013-02-22 12:08:17,198 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=?%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=?%,
evictions=0, evicted=0, evictedPerRun=NaN
2013-02-22 12:13:17,198 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=?%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=?%,
evictions=0, evicted=0, evictedPerRun=NaN
another regionserver:
2013-02-22 12:03:17,279 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=�%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%,
evictions=0, evicted=0, evictedPerRun=NaN
2013-02-22 12:08:17,278 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=�%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%,
evictions=0, evicted=0, evictedPerRun=NaN
2013-02-22 12:13:17,278 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=�%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%,
evictions=0, evicted=0, evictedPerRun=NaN
On 02/22/2013 12:49 PM, alxsss@aim.com wrote:
> Hi,
>
> Do you see any errors in HMaster and Regionserver logs?
>
> Alex.
>
>
>
>
>
>
>
> -----Original Message-----
> From: kaveh minooie <ka...@plutoz.com>
> To: user <us...@hbase.apache.org>; user <us...@nutch.apache.org>; user
<us...@zookeeper.apache.org>
> Sent: Fri, Feb 22, 2013 12:31 pm
> Subject: issue with nutch-gora+hbase+zookeeper
>
>
> Hi everyone
>
> I am having this problem for couple of days now and would appreciate any
> idea or suggestion that any one might have. I am using nutch 2.x with
> hbase. due to nutch requirement I need to use older version of hbase (I
> am using 0.90.6 over hadoop 1.1.1 with 10 nodes with zookeepr 3.5.0
> [trunk] )
>
> hbase seems to be running fine, thou I appreciate if someone can show me
> how I can actually test it systematically. but I seem to be able to
> create and read data from hbase, but when I run any nutch command
> something very similar to this happens as soon as the job starts
> running. this is for example, the output of nutch inject
>
> 13/02/22 12:07:30 INFO mapred.JobClient: map 0% reduce 0%
> 13/02/22 12:07:52 INFO mapred.JobClient: Task Id :
> attempt_201302191325_0013_m_000000_0, Status : FAILED
> org.apache.gora.util.GoraException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could
> be a sign that the server has too many connections (30 is the default).
> Consider inspecting your ZK server logs for that error and then make
> sure you are reusing HBaseConfiguration as often as you can. See
> HTable's javadoc for more information.
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)
> at
> org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:628)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase
> is able to connect to ZooKeeper but the connection closes immediately.
> This could be a sign that the server has too many connections (30 is the
> default). Consider inspecting your ZK server logs for that error and
> then make sure you are reusing HBaseConfiguration as often as you can.
> See HTable's javadoc for more information.
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:156)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1265)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:526)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:516)
> at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:173)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:93)
> at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:108)
> at
> org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
> ... 10 more
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1237)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1265)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:931)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
> ... 18 more
>
>
>
> Now I know that I am not running out of connection. for one thing I have
> increased the number of connection to 200 in zoo.cfg, and also here is
> what is in the zookeeper log file around that time:
>
> 2013-02-22 12:07:27,704 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
>
> - Accepted socket connection from /127.0.0.1:55073
> 2013-02-22 12:07:27,707 [myid:] - INFO
> [NIOWorkerThread-3:ZooKeeperServer@810] - Client attempting to establish
> new session at /127.0.0.1:55073
> 2013-02-22 12:07:27,720 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@566] - Established session
> 0x13d037b8e6b0016 with negotiated timeout 40000 for client /127.0.0.1:55073
> 2013-02-22 12:07:27,945 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
>
> - Accepted socket connection from /127.0.0.1:55075
> 2013-02-22 12:07:27,946 [myid:] - INFO
> [NIOWorkerThread-2:ZooKeeperServer@810] - Client attempting to establish
> new session at /127.0.0.1:55075
> 2013-02-22 12:07:27,953 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@566] - Established session
> 0x13d037b8e6b0017 with negotiated timeout 40000 for client /127.0.0.1:55075
> 2013-02-22 12:07:28,010 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@533] - Processed session termination for
> sessionid: 0x13d037b8e6b0017
> 2013-02-22 12:07:28,011 [myid:] - INFO
> [NIOWorkerThread-6:NIOServerCnxn@1000] - Closed socket connection for
> client /127.0.0.1:55075 which had sessionid 0x13d037b8e6b0017
> 2013-02-22 12:08:14,005 [myid:] - WARN
> [NIOWorkerThread-7:NIOServerCnxn@362] - Unable to read additional data
> from client sessionid 0x13d037b8e6b0016, likely client has closed socket
> 2013-02-22 12:08:14,005 [myid:] - INFO
> [NIOWorkerThread-7:NIOServerCnxn@1000] - Closed socket connection for
> client /127.0.0.1:55073 which had sessionid 0x13d037b8e6b0016
> 2013-02-22 12:08:48,000 [myid:] - INFO
> [SessionTracker:ZooKeeperServer@304] - Expiring session
> 0x13d037b8e6b0016, timeout of 40000ms exceeded
> 2013-02-22 12:08:48,001 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@533] - Processed session termination for
> sessionid: 0x13d037b8e6b0016
>
>
> I also don't think that it is a heartbeat or GC related issue since
> there is really no load at all on these servers right now. I know this
> is a hybrid problem involving three separate product (nutch, hbase,
> zookeeper) but I am asking this in all the mailing list. Also I am gonna
> say it agian in order to avoid confusion with the similar problems in
> older versions that are supposedly solved. I am using hbase 0.90.6 and
> zookeeper 3.5.0 (commit 46b565e6) with nutch 2.x (commit f02dcf625 )
> both are either the latest or very recent updates.
>
> if anyone has any idea what is happening here I very much like to hear
> that.
>
> thanks,
>
>
--
Kaveh Minooie
Re: issue with nutch-gora+hbase+zookeeper
Posted by kaveh minooie <ka...@plutoz.com>.
no here are the logs, master:
2013-02-22 12:03:21,136 INFO
org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
servers=10 regions=3 average=0.3 mostloaded=1 leastloaded=0
2013-02-22 12:03:21,141 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 1 catalog row(s)
and gc'd 0 unreferenced parent region(s)
2013-02-22 12:08:21,135 DEBUG
org.apache.hadoop.hbase.master.LoadBalancer: Server information:
d1r1n12.prod.plutoz.com,60020,1361563095874=1,
d1r1n16.prod.plutoz.com,60020,1361563095878
=0, d1r1n17.prod.plutoz.com,60020,1361563095900=0,
d1r1n13.prod.plutoz.com,60020,1361563095880=0,
d1r1n14.prod.plutoz.com,60020,1361563095886=0,
d1r1n10.prod.plutoz.com,60020,1361563095853=1,
d1r1n20.prod.plutoz.com,60020,1361563095890=1,
d1r1n15.prod.plutoz.com,60020,1361563095870=0,
d1r1n21.prod.plutoz.com,60020,1361563095888=0,
d1r1n19.prod.plutoz.com,60020,1361563095879=0
2013-02-22 12:08:21,135 INFO
org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
servers=10 regions=3 average=0.3 mostloaded=1 leastloaded=0
2013-02-22 12:08:21,142 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 1 catalog row(s)
and gc'd 0 unreferenced parent region(s)
one of the regionservers:
2013-02-22 12:03:17,199 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=?%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=?%,
evictions=0, evicted=0, evictedPerRun=NaN
2013-02-22 12:08:17,198 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=?%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=?%,
evictions=0, evicted=0, evictedPerRun=NaN
2013-02-22 12:13:17,198 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=?%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=?%,
evictions=0, evicted=0, evictedPerRun=NaN
another regionserver:
2013-02-22 12:03:17,279 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=�%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%,
evictions=0, evicted=0, evictedPerRun=NaN
2013-02-22 12:08:17,278 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=�%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%,
evictions=0, evicted=0, evictedPerRun=NaN
2013-02-22 12:13:17,278 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.51
MB, free=786.84 MB, max=793.35 MB, blocks=0, accesses=0, hits=0,
hitRatio=�%, cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%,
evictions=0, evicted=0, evictedPerRun=NaN
On 02/22/2013 12:49 PM, alxsss@aim.com wrote:
> Hi,
>
> Do you see any errors in HMaster and Regionserver logs?
>
> Alex.
>
>
>
>
>
>
>
> -----Original Message-----
> From: kaveh minooie <ka...@plutoz.com>
> To: user <us...@hbase.apache.org>; user <us...@nutch.apache.org>; user <us...@zookeeper.apache.org>
> Sent: Fri, Feb 22, 2013 12:31 pm
> Subject: issue with nutch-gora+hbase+zookeeper
>
>
> Hi everyone
>
> I am having this problem for couple of days now and would appreciate any
> idea or suggestion that any one might have. I am using nutch 2.x with
> hbase. due to nutch requirement I need to use older version of hbase (I
> am using 0.90.6 over hadoop 1.1.1 with 10 nodes with zookeepr 3.5.0
> [trunk] )
>
> hbase seems to be running fine, thou I appreciate if someone can show me
> how I can actually test it systematically. but I seem to be able to
> create and read data from hbase, but when I run any nutch command
> something very similar to this happens as soon as the job starts
> running. this is for example, the output of nutch inject
>
> 13/02/22 12:07:30 INFO mapred.JobClient: map 0% reduce 0%
> 13/02/22 12:07:52 INFO mapred.JobClient: Task Id :
> attempt_201302191325_0013_m_000000_0, Status : FAILED
> org.apache.gora.util.GoraException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could
> be a sign that the server has too many connections (30 is the default).
> Consider inspecting your ZK server logs for that error and then make
> sure you are reusing HBaseConfiguration as often as you can. See
> HTable's javadoc for more information.
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)
> at
> org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:628)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase
> is able to connect to ZooKeeper but the connection closes immediately.
> This could be a sign that the server has too many connections (30 is the
> default). Consider inspecting your ZK server logs for that error and
> then make sure you are reusing HBaseConfiguration as often as you can.
> See HTable's javadoc for more information.
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:156)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1265)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:526)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:516)
> at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:173)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:93)
> at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:108)
> at
> org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
> ... 10 more
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1237)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1265)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:931)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
> ... 18 more
>
>
>
> Now I know that I am not running out of connection. for one thing I have
> increased the number of connection to 200 in zoo.cfg, and also here is
> what is in the zookeeper log file around that time:
>
> 2013-02-22 12:07:27,704 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
>
> - Accepted socket connection from /127.0.0.1:55073
> 2013-02-22 12:07:27,707 [myid:] - INFO
> [NIOWorkerThread-3:ZooKeeperServer@810] - Client attempting to establish
> new session at /127.0.0.1:55073
> 2013-02-22 12:07:27,720 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@566] - Established session
> 0x13d037b8e6b0016 with negotiated timeout 40000 for client /127.0.0.1:55073
> 2013-02-22 12:07:27,945 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
>
> - Accepted socket connection from /127.0.0.1:55075
> 2013-02-22 12:07:27,946 [myid:] - INFO
> [NIOWorkerThread-2:ZooKeeperServer@810] - Client attempting to establish
> new session at /127.0.0.1:55075
> 2013-02-22 12:07:27,953 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@566] - Established session
> 0x13d037b8e6b0017 with negotiated timeout 40000 for client /127.0.0.1:55075
> 2013-02-22 12:07:28,010 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@533] - Processed session termination for
> sessionid: 0x13d037b8e6b0017
> 2013-02-22 12:07:28,011 [myid:] - INFO
> [NIOWorkerThread-6:NIOServerCnxn@1000] - Closed socket connection for
> client /127.0.0.1:55075 which had sessionid 0x13d037b8e6b0017
> 2013-02-22 12:08:14,005 [myid:] - WARN
> [NIOWorkerThread-7:NIOServerCnxn@362] - Unable to read additional data
> from client sessionid 0x13d037b8e6b0016, likely client has closed socket
> 2013-02-22 12:08:14,005 [myid:] - INFO
> [NIOWorkerThread-7:NIOServerCnxn@1000] - Closed socket connection for
> client /127.0.0.1:55073 which had sessionid 0x13d037b8e6b0016
> 2013-02-22 12:08:48,000 [myid:] - INFO
> [SessionTracker:ZooKeeperServer@304] - Expiring session
> 0x13d037b8e6b0016, timeout of 40000ms exceeded
> 2013-02-22 12:08:48,001 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@533] - Processed session termination for
> sessionid: 0x13d037b8e6b0016
>
>
> I also don't think that it is a heartbeat or GC related issue since
> there is really no load at all on these servers right now. I know this
> is a hybrid problem involving three separate product (nutch, hbase,
> zookeeper) but I am asking this in all the mailing list. Also I am gonna
> say it agian in order to avoid confusion with the similar problems in
> older versions that are supposedly solved. I am using hbase 0.90.6 and
> zookeeper 3.5.0 (commit 46b565e6) with nutch 2.x (commit f02dcf625 )
> both are either the latest or very recent updates.
>
> if anyone has any idea what is happening here I very much like to hear
> that.
>
> thanks,
>
>
--
Kaveh Minooie
Re: issue with nutch-gora+hbase+zookeeper
Posted by al...@aim.com.
Hi,
Do you see any errors in HMaster and Regionserver logs?
Alex.
-----Original Message-----
From: kaveh minooie <ka...@plutoz.com>
To: user <us...@hbase.apache.org>; user <us...@nutch.apache.org>; user <us...@zookeeper.apache.org>
Sent: Fri, Feb 22, 2013 12:31 pm
Subject: issue with nutch-gora+hbase+zookeeper
Hi everyone
I am having this problem for couple of days now and would appreciate any
idea or suggestion that any one might have. I am using nutch 2.x with
hbase. due to nutch requirement I need to use older version of hbase (I
am using 0.90.6 over hadoop 1.1.1 with 10 nodes with zookeepr 3.5.0
[trunk] )
hbase seems to be running fine, thou I appreciate if someone can show me
how I can actually test it systematically. but I seem to be able to
create and read data from hbase, but when I run any nutch command
something very similar to this happens as soon as the job starts
running. this is for example, the output of nutch inject
13/02/22 12:07:30 INFO mapred.JobClient: map 0% reduce 0%
13/02/22 12:07:52 INFO mapred.JobClient: Task Id :
attempt_201302191325_0013_m_000000_0, Status : FAILED
org.apache.gora.util.GoraException:
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could
be a sign that the server has too many connections (30 is the default).
Consider inspecting your ZK server logs for that error and then make
sure you are reusing HBaseConfiguration as often as you can. See
HTable's javadoc for more information.
at
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
at
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)
at
org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:628)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase
is able to connect to ZooKeeper but the connection closes immediately.
This could be a sign that the server has too many connections (30 is the
default). Consider inspecting your ZK server logs for that error and
then make sure you are reusing HBaseConfiguration as often as you can.
See HTable's javadoc for more information.
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:156)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1265)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:526)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:516)
at
org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:173)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:93)
at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:108)
at
org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
at
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
... 10 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1237)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1265)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:931)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
... 18 more
Now I know that I am not running out of connection. for one thing I have
increased the number of connection to 200 in zoo.cfg, and also here is
what is in the zookeeper log file around that time:
2013-02-22 12:07:27,704 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
- Accepted socket connection from /127.0.0.1:55073
2013-02-22 12:07:27,707 [myid:] - INFO
[NIOWorkerThread-3:ZooKeeperServer@810] - Client attempting to establish
new session at /127.0.0.1:55073
2013-02-22 12:07:27,720 [myid:] - INFO
[SyncThread:0:ZooKeeperServer@566] - Established session
0x13d037b8e6b0016 with negotiated timeout 40000 for client /127.0.0.1:55073
2013-02-22 12:07:27,945 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
- Accepted socket connection from /127.0.0.1:55075
2013-02-22 12:07:27,946 [myid:] - INFO
[NIOWorkerThread-2:ZooKeeperServer@810] - Client attempting to establish
new session at /127.0.0.1:55075
2013-02-22 12:07:27,953 [myid:] - INFO
[SyncThread:0:ZooKeeperServer@566] - Established session
0x13d037b8e6b0017 with negotiated timeout 40000 for client /127.0.0.1:55075
2013-02-22 12:07:28,010 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequestProcessor@533] - Processed session termination for
sessionid: 0x13d037b8e6b0017
2013-02-22 12:07:28,011 [myid:] - INFO
[NIOWorkerThread-6:NIOServerCnxn@1000] - Closed socket connection for
client /127.0.0.1:55075 which had sessionid 0x13d037b8e6b0017
2013-02-22 12:08:14,005 [myid:] - WARN
[NIOWorkerThread-7:NIOServerCnxn@362] - Unable to read additional data
from client sessionid 0x13d037b8e6b0016, likely client has closed socket
2013-02-22 12:08:14,005 [myid:] - INFO
[NIOWorkerThread-7:NIOServerCnxn@1000] - Closed socket connection for
client /127.0.0.1:55073 which had sessionid 0x13d037b8e6b0016
2013-02-22 12:08:48,000 [myid:] - INFO
[SessionTracker:ZooKeeperServer@304] - Expiring session
0x13d037b8e6b0016, timeout of 40000ms exceeded
2013-02-22 12:08:48,001 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequestProcessor@533] - Processed session termination for
sessionid: 0x13d037b8e6b0016
I also don't think that it is a heartbeat or GC related issue since
there is really no load at all on these servers right now. I know this
is a hybrid problem involving three separate product (nutch, hbase,
zookeeper) but I am asking this in all the mailing list. Also I am gonna
say it agian in order to avoid confusion with the similar problems in
older versions that are supposedly solved. I am using hbase 0.90.6 and
zookeeper 3.5.0 (commit 46b565e6) with nutch 2.x (commit f02dcf625 )
both are either the latest or very recent updates.
if anyone has any idea what is happening here I very much like to hear
that.
thanks,
--
Kaveh Minooie
www.plutoz.com
Re: issue with nutch-gora+hbase+zookeeper
Posted by kaveh minooie <ka...@plutoz.com>.
In case anyone was wondering, the issue was resolved by copying the
zoo.cfg in the hadoop conf directory (on classpath really) on the entire
cluster. thanks
On 02/22/2013 12:31 PM, kaveh minooie wrote:
> Hi everyone
>
> I am having this problem for couple of days now and would appreciate
> any idea or suggestion that any one might have. I am using nutch 2.x
> with hbase. due to nutch requirement I need to use older version of
> hbase (I am using 0.90.6 over hadoop 1.1.1 with 10 nodes with
> zookeepr 3.5.0 [trunk] )
>
> hbase seems to be running fine, thou I appreciate if someone can show
> me how I can actually test it systematically. but I seem to be able to
> create and read data from hbase, but when I run any nutch command
> something very similar to this happens as soon as the job starts
> running. this is for example, the output of nutch inject
>
> 13/02/22 12:07:30 INFO mapred.JobClient: map 0% reduce 0%
> 13/02/22 12:07:52 INFO mapred.JobClient: Task Id :
> attempt_201302191325_0013_m_000000_0, Status : FAILED
> org.apache.gora.util.GoraException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could
> be a sign that the server has too many connections (30 is the
> default). Consider inspecting your ZK server logs for that error and
> then make sure you are reusing HBaseConfiguration as often as you can.
> See HTable's javadoc for more information.
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)
> at
> org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:628)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase
> is able to connect to ZooKeeper but the connection closes immediately.
> This could be a sign that the server has too many connections (30 is
> the default). Consider inspecting your ZK server logs for that error
> and then make sure you are reusing HBaseConfiguration as often as you
> can. See HTable's javadoc for more information.
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:156)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1265)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:526)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:516)
> at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:173)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:93)
> at
> org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:108)
> at
> org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
> at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
> ... 10 more
> Caused by:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1237)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1265)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:931)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
> ... 18 more
>
>
>
> Now I know that I am not running out of connection. for one thing I
> have increased the number of connection to 200 in zoo.cfg, and also
> here is what is in the zookeeper log file around that time:
>
> 2013-02-22 12:07:27,704 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
> - Accepted socket connection from /127.0.0.1:55073
> 2013-02-22 12:07:27,707 [myid:] - INFO
> [NIOWorkerThread-3:ZooKeeperServer@810] - Client attempting to
> establish new session at /127.0.0.1:55073
> 2013-02-22 12:07:27,720 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@566] - Established session
> 0x13d037b8e6b0016 with negotiated timeout 40000 for client
> /127.0.0.1:55073
> 2013-02-22 12:07:27,945 [myid:] - INFO
> [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
> - Accepted socket connection from /127.0.0.1:55075
> 2013-02-22 12:07:27,946 [myid:] - INFO
> [NIOWorkerThread-2:ZooKeeperServer@810] - Client attempting to
> establish new session at /127.0.0.1:55075
> 2013-02-22 12:07:27,953 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@566] - Established session
> 0x13d037b8e6b0017 with negotiated timeout 40000 for client
> /127.0.0.1:55075
> 2013-02-22 12:07:28,010 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@533] - Processed session termination
> for sessionid: 0x13d037b8e6b0017
> 2013-02-22 12:07:28,011 [myid:] - INFO
> [NIOWorkerThread-6:NIOServerCnxn@1000] - Closed socket connection for
> client /127.0.0.1:55075 which had sessionid 0x13d037b8e6b0017
> 2013-02-22 12:08:14,005 [myid:] - WARN
> [NIOWorkerThread-7:NIOServerCnxn@362] - Unable to read additional data
> from client sessionid 0x13d037b8e6b0016, likely client has closed socket
> 2013-02-22 12:08:14,005 [myid:] - INFO
> [NIOWorkerThread-7:NIOServerCnxn@1000] - Closed socket connection for
> client /127.0.0.1:55073 which had sessionid 0x13d037b8e6b0016
> 2013-02-22 12:08:48,000 [myid:] - INFO
> [SessionTracker:ZooKeeperServer@304] - Expiring session
> 0x13d037b8e6b0016, timeout of 40000ms exceeded
> 2013-02-22 12:08:48,001 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@533] - Processed session termination
> for sessionid: 0x13d037b8e6b0016
>
>
> I also don't think that it is a heartbeat or GC related issue since
> there is really no load at all on these servers right now. I know this
> is a hybrid problem involving three separate product (nutch, hbase,
> zookeeper) but I am asking this in all the mailing list. Also I am
> gonna say it agian in order to avoid confusion with the similar
> problems in older versions that are supposedly solved. I am using
> hbase 0.90.6 and zookeeper 3.5.0 (commit 46b565e6) with nutch 2.x
> (commit f02dcf625 ) both are either the latest or very recent updates.
>
> if anyone has any idea what is happening here I very much like to hear
> that.
>
> thanks,
>
>
issue with nutch-gora+hbase+zookeeper
Posted by kaveh minooie <ka...@plutoz.com>.
Hi everyone
I am having this problem for couple of days now and would appreciate any
idea or suggestion that any one might have. I am using nutch 2.x with
hbase. due to nutch requirement I need to use older version of hbase (I
am using 0.90.6 over hadoop 1.1.1 with 10 nodes with zookeepr 3.5.0
[trunk] )
hbase seems to be running fine, thou I appreciate if someone can show me
how I can actually test it systematically. but I seem to be able to
create and read data from hbase, but when I run any nutch command
something very similar to this happens as soon as the job starts
running. this is for example, the output of nutch inject
13/02/22 12:07:30 INFO mapred.JobClient: map 0% reduce 0%
13/02/22 12:07:52 INFO mapred.JobClient: Task Id :
attempt_201302191325_0013_m_000000_0, Status : FAILED
org.apache.gora.util.GoraException:
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could
be a sign that the server has too many connections (30 is the default).
Consider inspecting your ZK server logs for that error and then make
sure you are reusing HBaseConfiguration as often as you can. See
HTable's javadoc for more information.
at
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
at
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)
at
org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:628)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase
is able to connect to ZooKeeper but the connection closes immediately.
This could be a sign that the server has too many connections (30 is the
default). Consider inspecting your ZK server logs for that error and
then make sure you are reusing HBaseConfiguration as often as you can.
See HTable's javadoc for more information.
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:156)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1265)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:526)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:516)
at
org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:173)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:93)
at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:108)
at
org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
at
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
... 10 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1237)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1265)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:931)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
... 18 more
Now I know that I am not running out of connection. for one thing I have
increased the number of connection to 200 in zoo.cfg, and also here is
what is in the zookeeper log file around that time:
2013-02-22 12:07:27,704 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
- Accepted socket connection from /127.0.0.1:55073
2013-02-22 12:07:27,707 [myid:] - INFO
[NIOWorkerThread-3:ZooKeeperServer@810] - Client attempting to establish
new session at /127.0.0.1:55073
2013-02-22 12:07:27,720 [myid:] - INFO
[SyncThread:0:ZooKeeperServer@566] - Established session
0x13d037b8e6b0016 with negotiated timeout 40000 for client /127.0.0.1:55073
2013-02-22 12:07:27,945 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
- Accepted socket connection from /127.0.0.1:55075
2013-02-22 12:07:27,946 [myid:] - INFO
[NIOWorkerThread-2:ZooKeeperServer@810] - Client attempting to establish
new session at /127.0.0.1:55075
2013-02-22 12:07:27,953 [myid:] - INFO
[SyncThread:0:ZooKeeperServer@566] - Established session
0x13d037b8e6b0017 with negotiated timeout 40000 for client /127.0.0.1:55075
2013-02-22 12:07:28,010 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequestProcessor@533] - Processed session termination for
sessionid: 0x13d037b8e6b0017
2013-02-22 12:07:28,011 [myid:] - INFO
[NIOWorkerThread-6:NIOServerCnxn@1000] - Closed socket connection for
client /127.0.0.1:55075 which had sessionid 0x13d037b8e6b0017
2013-02-22 12:08:14,005 [myid:] - WARN
[NIOWorkerThread-7:NIOServerCnxn@362] - Unable to read additional data
from client sessionid 0x13d037b8e6b0016, likely client has closed socket
2013-02-22 12:08:14,005 [myid:] - INFO
[NIOWorkerThread-7:NIOServerCnxn@1000] - Closed socket connection for
client /127.0.0.1:55073 which had sessionid 0x13d037b8e6b0016
2013-02-22 12:08:48,000 [myid:] - INFO
[SessionTracker:ZooKeeperServer@304] - Expiring session
0x13d037b8e6b0016, timeout of 40000ms exceeded
2013-02-22 12:08:48,001 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequestProcessor@533] - Processed session termination for
sessionid: 0x13d037b8e6b0016
I also don't think that it is a heartbeat or GC related issue since
there is really no load at all on these servers right now. I know this
is a hybrid problem involving three separate product (nutch, hbase,
zookeeper) but I am asking this in all the mailing list. Also I am gonna
say it agian in order to avoid confusion with the similar problems in
older versions that are supposedly solved. I am using hbase 0.90.6 and
zookeeper 3.5.0 (commit 46b565e6) with nutch 2.x (commit f02dcf625 )
both are either the latest or very recent updates.
if anyone has any idea what is happening here I very much like to hear
that.
thanks,
--
Kaveh Minooie
www.plutoz.com
issue with nutch-gora+hbase+zookeeper
Posted by kaveh minooie <ka...@plutoz.com>.
Hi everyone
I am having this problem for couple of days now and would appreciate any
idea or suggestion that any one might have. I am using nutch 2.x with
hbase. due to nutch requirement I need to use older version of hbase (I
am using 0.90.6 over hadoop 1.1.1 with 10 nodes with zookeepr 3.5.0
[trunk] )
hbase seems to be running fine, thou I appreciate if someone can show me
how I can actually test it systematically. but I seem to be able to
create and read data from hbase, but when I run any nutch command
something very similar to this happens as soon as the job starts
running. this is for example, the output of nutch inject
13/02/22 12:07:30 INFO mapred.JobClient: map 0% reduce 0%
13/02/22 12:07:52 INFO mapred.JobClient: Task Id :
attempt_201302191325_0013_m_000000_0, Status : FAILED
org.apache.gora.util.GoraException:
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could
be a sign that the server has too many connections (30 is the default).
Consider inspecting your ZK server logs for that error and then make
sure you are reusing HBaseConfiguration as often as you can. See
HTable's javadoc for more information.
at
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
at
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)
at
org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:628)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase
is able to connect to ZooKeeper but the connection closes immediately.
This could be a sign that the server has too many connections (30 is the
default). Consider inspecting your ZK server logs for that error and
then make sure you are reusing HBaseConfiguration as often as you can.
See HTable's javadoc for more information.
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:156)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1265)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:526)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:516)
at
org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:173)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:93)
at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:108)
at
org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
at
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
... 10 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1237)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1265)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:931)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
... 18 more
Now I know that I am not running out of connection. for one thing I have
increased the number of connection to 200 in zoo.cfg, and also here is
what is in the zookeeper log file around that time:
2013-02-22 12:07:27,704 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
- Accepted socket connection from /127.0.0.1:55073
2013-02-22 12:07:27,707 [myid:] - INFO
[NIOWorkerThread-3:ZooKeeperServer@810] - Client attempting to establish
new session at /127.0.0.1:55073
2013-02-22 12:07:27,720 [myid:] - INFO
[SyncThread:0:ZooKeeperServer@566] - Established session
0x13d037b8e6b0016 with negotiated timeout 40000 for client /127.0.0.1:55073
2013-02-22 12:07:27,945 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
- Accepted socket connection from /127.0.0.1:55075
2013-02-22 12:07:27,946 [myid:] - INFO
[NIOWorkerThread-2:ZooKeeperServer@810] - Client attempting to establish
new session at /127.0.0.1:55075
2013-02-22 12:07:27,953 [myid:] - INFO
[SyncThread:0:ZooKeeperServer@566] - Established session
0x13d037b8e6b0017 with negotiated timeout 40000 for client /127.0.0.1:55075
2013-02-22 12:07:28,010 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequestProcessor@533] - Processed session termination for
sessionid: 0x13d037b8e6b0017
2013-02-22 12:07:28,011 [myid:] - INFO
[NIOWorkerThread-6:NIOServerCnxn@1000] - Closed socket connection for
client /127.0.0.1:55075 which had sessionid 0x13d037b8e6b0017
2013-02-22 12:08:14,005 [myid:] - WARN
[NIOWorkerThread-7:NIOServerCnxn@362] - Unable to read additional data
from client sessionid 0x13d037b8e6b0016, likely client has closed socket
2013-02-22 12:08:14,005 [myid:] - INFO
[NIOWorkerThread-7:NIOServerCnxn@1000] - Closed socket connection for
client /127.0.0.1:55073 which had sessionid 0x13d037b8e6b0016
2013-02-22 12:08:48,000 [myid:] - INFO
[SessionTracker:ZooKeeperServer@304] - Expiring session
0x13d037b8e6b0016, timeout of 40000ms exceeded
2013-02-22 12:08:48,001 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequestProcessor@533] - Processed session termination for
sessionid: 0x13d037b8e6b0016
I also don't think that it is a heartbeat or GC related issue since
there is really no load at all on these servers right now. I know this
is a hybrid problem involving three separate product (nutch, hbase,
zookeeper) but I am asking this in all the mailing list. Also I am gonna
say it agian in order to avoid confusion with the similar problems in
older versions that are supposedly solved. I am using hbase 0.90.6 and
zookeeper 3.5.0 (commit 46b565e6) with nutch 2.x (commit f02dcf625 )
both are either the latest or very recent updates.
if anyone has any idea what is happening here I very much like to hear
that.
thanks,
--
Kaveh Minooie
www.plutoz.com
issue with nutch-gora+hbase+zookeeper
Posted by kaveh minooie <ka...@plutoz.com>.
Hi everyone
I am having this problem for couple of days now and would appreciate any
idea or suggestion that any one might have. I am using nutch 2.x with
hbase. due to nutch requirement I need to use older version of hbase (I
am using 0.90.6 over hadoop 1.1.1 with 10 nodes with zookeepr 3.5.0
[trunk] )
hbase seems to be running fine, thou I appreciate if someone can show me
how I can actually test it systematically. but I seem to be able to
create and read data from hbase, but when I run any nutch command
something very similar to this happens as soon as the job starts
running. this is for example, the output of nutch inject
13/02/22 12:07:30 INFO mapred.JobClient: map 0% reduce 0%
13/02/22 12:07:52 INFO mapred.JobClient: Task Id :
attempt_201302191325_0013_m_000000_0, Status : FAILED
org.apache.gora.util.GoraException:
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could
be a sign that the server has too many connections (30 is the default).
Consider inspecting your ZK server logs for that error and then make
sure you are reusing HBaseConfiguration as often as you can. See
HTable's javadoc for more information.
at
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
at
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)
at
org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:628)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase
is able to connect to ZooKeeper but the connection closes immediately.
This could be a sign that the server has too many connections (30 is the
default). Consider inspecting your ZK server logs for that error and
then make sure you are reusing HBaseConfiguration as often as you can.
See HTable's javadoc for more information.
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:156)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1265)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:526)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:516)
at
org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:173)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:93)
at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:108)
at
org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
at
org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
... 10 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1237)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1265)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:931)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
... 18 more
Now I know that I am not running out of connection. for one thing I have
increased the number of connection to 200 in zoo.cfg, and also here is
what is in the zookeeper log file around that time:
2013-02-22 12:07:27,704 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
- Accepted socket connection from /127.0.0.1:55073
2013-02-22 12:07:27,707 [myid:] - INFO
[NIOWorkerThread-3:ZooKeeperServer@810] - Client attempting to establish
new session at /127.0.0.1:55073
2013-02-22 12:07:27,720 [myid:] - INFO
[SyncThread:0:ZooKeeperServer@566] - Established session
0x13d037b8e6b0016 with negotiated timeout 40000 for client /127.0.0.1:55073
2013-02-22 12:07:27,945 [myid:] - INFO
[NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory$AcceptThread@289]
- Accepted socket connection from /127.0.0.1:55075
2013-02-22 12:07:27,946 [myid:] - INFO
[NIOWorkerThread-2:ZooKeeperServer@810] - Client attempting to establish
new session at /127.0.0.1:55075
2013-02-22 12:07:27,953 [myid:] - INFO
[SyncThread:0:ZooKeeperServer@566] - Established session
0x13d037b8e6b0017 with negotiated timeout 40000 for client /127.0.0.1:55075
2013-02-22 12:07:28,010 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequestProcessor@533] - Processed session termination for
sessionid: 0x13d037b8e6b0017
2013-02-22 12:07:28,011 [myid:] - INFO
[NIOWorkerThread-6:NIOServerCnxn@1000] - Closed socket connection for
client /127.0.0.1:55075 which had sessionid 0x13d037b8e6b0017
2013-02-22 12:08:14,005 [myid:] - WARN
[NIOWorkerThread-7:NIOServerCnxn@362] - Unable to read additional data
from client sessionid 0x13d037b8e6b0016, likely client has closed socket
2013-02-22 12:08:14,005 [myid:] - INFO
[NIOWorkerThread-7:NIOServerCnxn@1000] - Closed socket connection for
client /127.0.0.1:55073 which had sessionid 0x13d037b8e6b0016
2013-02-22 12:08:48,000 [myid:] - INFO
[SessionTracker:ZooKeeperServer@304] - Expiring session
0x13d037b8e6b0016, timeout of 40000ms exceeded
2013-02-22 12:08:48,001 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequestProcessor@533] - Processed session termination for
sessionid: 0x13d037b8e6b0016
I also don't think that it is a heartbeat or GC related issue since
there is really no load at all on these servers right now. I know this
is a hybrid problem involving three separate product (nutch, hbase,
zookeeper) but I am asking this in all the mailing list. Also I am gonna
say it agian in order to avoid confusion with the similar problems in
older versions that are supposedly solved. I am using hbase 0.90.6 and
zookeeper 3.5.0 (commit 46b565e6) with nutch 2.x (commit f02dcf625 )
both are either the latest or very recent updates.
if anyone has any idea what is happening here I very much like to hear
that.
thanks,
--
Kaveh Minooie
www.plutoz.com
Re: HBase newbie
Posted by Ted Yu <yu...@gmail.com>.
In case you encounter further issue(s), take a look at
http://hbase.apache.org/book.html#trouble
Cheers
On Wed, Feb 20, 2013 at 4:39 PM, kaveh minooie <ka...@plutoz.com> wrote:
> thanks, this time it was the hbase directory in the hdfs :). everything
> seems to be working now.
>
>
> On 02/20/2013 04:13 PM, Ted Yu wrote:
>
>> Have you checked the region server log from d1r1n19.prod.plutoz.comaround
>> this time ?
>>
>> Cheers
>>
>> On Wed, Feb 20, 2013 at 4:10 PM, kaveh minooie <ka...@plutoz.com> wrote:
>>
>> ok so I used to run hbase 0.94.4 before i realized that i had to
>>> downgrade. my zookeeper hbase directory had the files from that version
>>> before. once I cleaned the zookeeper directory using zkCli, master node
>>> came up. and after that i checked and all the data directories on the
>>> cluster are empty (there was no data in hbase). but now when i
>>> start-hbase
>>> the master node log file just keeps saying this :
>>>
>>> 2013-02-20 15:45:39,518 INFO org.apache.hadoop.hbase.****
>>> master.AssignmentManager:
>>>
>>> Using pre-existing plan for region -ROOT-,,0.70236052;
>>> plan=hri=-ROOT-,,0.70236052, src=, dest=d1r1n19.prod.plutoz.com,****
>>> 60020,1361403838824
>>> 2013-02-20 15:45:39,518 INFO org.apache.hadoop.hbase.****
>>> master.AssignmentManager:
>>> Assigning region -ROOT-,,0.70236052 to d1r1n19.prod.plutoz.com,60020,**
>>> **
>>> 1361403838824
>>> 2013-02-20 15:45:39,524 DEBUG org.apache.hadoop.hbase.****
>>> master.AssignmentManager:
>>> Handling transition=RS_ZK_REGION_****OPENING, server=
>>> d1r1n19.prod.plutoz.com
>>> **,60020,1361403838824, region=70236052/-ROOT-
>>> 2013-02-20 15:45:39,532 DEBUG org.apache.hadoop.hbase.****
>>> master.AssignmentManager:
>>> Handling transition=RS_ZK_REGION_****FAILED_OPEN, server=
>>> d1r1n19.prod.plutoz.com**,**60020,1361403838824, region=70236052/-ROOT-
>>> 2013-02-20 15:45:39,532 DEBUG org.apache.hadoop.hbase.****
>>> master.handler.****ClosedRegionHandler:
>>>
>>> Handling CLOSED event for 70236052
>>> 2013-02-20 15:45:39,532 DEBUG org.apache.hadoop.hbase.****
>>> master.AssignmentManager:
>>>
>>> Forcing OFFLINE; was=-ROOT-,,0.70236052 state=CLOSED, ts=1361403939527
>>> 2013-02-20 15:45:39,532 DEBUG org.apache.hadoop.hbase.****
>>> zookeeper.ZKAssign:
>>>
>>> master:60000-0x13cc3f3bcdc00b9 Creating (or updating) unassigned node for
>>> 70236052 with OFFLINE state
>>>
>>>
>>> again and again until i kill the java process.
>>>
>>> any idea what this means ?
>>>
>>>
>>>
>>>
>>> On 02/20/2013 03:16 PM, kaveh minooie wrote:
>>>
>>> I can't upgrade. I need hbase for nutch and it currently only supports
>>>> 0.90.x ( i am using 0.90.6)
>>>>
>>>> the zookeeper log is:
>>>> 2013-02-20 14:25:59,035 [myid:] - INFO
>>>> [NIOServerCxn.Factory:0.0.0.0/****0.0.0.0:2181:****
>>>> NIOServerCnxnFactory@197<ht...@197>
>>>> <ht...@197>
>>>> >]
>>>>
>>>> -
>>>> Accepted socket connection from /10.0.0.66:49971
>>>> 2013-02-20 14:25:59,040 [myid:] - WARN
>>>> [NIOServerCxn.Factory:0.0.0.0/****0.0.0.0:2181:**ZooKeeperServer@**793<http://0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**793>
>>>> <ht...@793>
>>>> >]
>>>>
>>>> -
>>>> Connection request from old client /10.0.0.66:49971; will be dropped if
>>>> server
>>>> is in r-o mode
>>>> 2013-02-20 14:25:59,040 [myid:] - INFO
>>>> [NIOServerCxn.Factory:0.0.0.0/****0.0.0.0:2181:**ZooKeeperServer@**839<http://0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**839>
>>>> <ht...@839>
>>>> >]
>>>>
>>>> - Client
>>>> attempting to establish new session at /10.0.0.66:49971
>>>> 2013-02-20 14:25:59,054 [myid:] - INFO
>>>> [SyncThread:0:ZooKeeperServer@****595] - Established session
>>>>
>>>> 0x13cc3f3bcdc0094 with negotiated timeout 40000 for client /
>>>> 10.0.0.66:49971
>>>> 2013-02-20 14:25:59,554 [myid:] - INFO
>>>> [NIOServerCxn.Factory:0.0.0.0/****0.0.0.0:2181:****
>>>> NIOServerCnxnFactory@197<ht...@197>
>>>> <ht...@197>
>>>> >]
>>>>
>>>> -
>>>> Accepted socket connection from /10.0.0.66:49984
>>>> 2013-02-20 14:25:59,554 [myid:] - WARN
>>>> [NIOServerCxn.Factory:0.0.0.0/****0.0.0.0:2181:**ZooKeeperServer@**793<http://0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**793>
>>>> <ht...@793>
>>>> >]
>>>>
>>>> -
>>>> Connection request from old client /10.0.0.66:49984; will be dropped if
>>>> server is in r-o mode
>>>> 2013-02-20 14:25:59,554 [myid:] - INFO
>>>> [NIOServerCxn.Factory:0.0.0.0/****0.0.0.0:2181:**ZooKeeperServer@**839<http://0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**839>
>>>> <ht...@839>
>>>> >]
>>>>
>>>> - Client
>>>> attempting to establish new session at /10.0.0.66:49984
>>>> 2013-02-20 14:25:59,556 [myid:] - INFO
>>>> [SyncThread:0:ZooKeeperServer@****595] - Established session
>>>>
>>>> 0x13cc3f3bcdc0095 with negotiated timeout 40000 for client /
>>>> 10.0.0.66:49984
>>>> 2013-02-20 14:25:59,598 [myid:] - INFO [ProcessThread(sid:0
>>>> cport:-1)::****PrepRequestProcessor@476] - Processed session
>>>> termination
>>>>
>>>> for
>>>> sessionid: 0x13cc3f3bcdc0095
>>>> 2013-02-20 14:25:59,599 [myid:] - INFO
>>>> [NIOServerCxn.Factory:0.0.0.0/****0.0.0.0:2181:NIOServerCnxn@****1001<http://0.0.0.0/**0.0.0.0:2181:NIOServerCnxn@**1001>
>>>> <ht...@1001>
>>>> >]
>>>>
>>>> - Closed
>>>> socket connection for client /10.0.0.66:49984 which had sessionid
>>>> 0x13cc3f3bcdc0095
>>>> 2013-02-20 14:25:59,600 [myid:] - INFO [ProcessThread(sid:0
>>>> cport:-1)::****PrepRequestProcessor@476] - Processed session
>>>> termination
>>>>
>>>> for
>>>> sessionid: 0x13cc3f3bcdc0094
>>>> 2013-02-20 14:25:59,601 [myid:] - INFO
>>>> [NIOServerCxn.Factory:0.0.0.0/****0.0.0.0:2181:NIOServerCnxn@****1001<http://0.0.0.0/**0.0.0.0:2181:NIOServerCnxn@**1001>
>>>> <ht...@1001>
>>>> >]
>>>>
>>>> - Closed
>>>> socket connection for client /10.0.0.66:49971 which had sessionid
>>>> 0x13cc3f3bcdc0094
>>>> 2013-02-20 14:26:00,218 [myid:] - INFO
>>>> [NIOServerCxn.Factory:0.0.0.0/****0.0.0.0:2181:****
>>>> NIOServerCnxnFactory@197<ht...@197>
>>>> <ht...@197>
>>>> >]
>>>>
>>>> -
>>>> Accepted socket connection from /10.0.0.13:48286
>>>> 2013-02-20 14:26:00,220 [myid:] - WARN
>>>> [NIOServerCxn.Factory:0.0.0.0/****0.0.0.0:2181:**ZooKeeperServer@**793<http://0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**793>
>>>> <ht...@793>
>>>> >]
>>>>
>>>> -
>>>> Connection request from old client /10.0.0.13:48286; will be dropped if
>>>> server is in r-o mode
>>>> 2013-02-20 14:26:00,221 [myid:] - INFO
>>>> [NIOServerCxn.Factory:0.0.0.0/****0.0.0.0:2181:**ZooKeeperServer@**839<http://0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**839>
>>>> <ht...@839>
>>>> >]
>>>>
>>>> - Client
>>>> attempting to establish new session at /10.0.0.13:48286
>>>> 2013-02-20 14:26:00,222 [myid:] - INFO
>>>> [SyncThread:0:ZooKeeperServer@****595] - Established session
>>>>
>>>> 0x13cc3f3bcdc0096 with negotiated timeout 40000 for client /
>>>> 10.0.0.13:48286
>>>>
>>>>
>>>> 10.0.0.13 is one of the regionservers.
>>>>
>>>> thanks,
>>>>
>>>>
>>>>
>>>> On 02/20/2013 02:48 PM, Ted Yu wrote:
>>>>
>>>> The error seemed to be related to zookeeper.
>>>>>
>>>>> Can you post log from zk1 around 14:25:59 ?
>>>>>
>>>>> BTW stable release is 0.94.5, can you upgrade ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Wed, Feb 20, 2013 at 2:46 PM, kaveh minooie <ka...@plutoz.com>
>>>>> wrote:
>>>>>
>>>>> Hello everyone
>>>>>
>>>>>> I was wondering if any one has tried to run hbase 0.90.x on a
>>>>>> hadoop
>>>>>> 1.1.1 cluster? I have replaced a few of the jar files with the ones
>>>>>> that
>>>>>> came with the packages that I am using in my cluster:
>>>>>>
>>>>>> commons-configuration-1.6.jar
>>>>>> hadoop-core-1.1.1.jar
>>>>>> slf4j-api-1.6.1.jar
>>>>>> slf4j-log4j12-1.6.1.jar
>>>>>> zookeeper-3.4.5.jar (tried with both 3.3.2 and 3.4.5. same result)
>>>>>>
>>>>>> after these changes I am able to run the region servers but the master
>>>>>> node still crashes when I try to start it:
>>>>>>
>>>>>> 2013-02-20 14:25:59,554 INFO org.apache.zookeeper.******
>>>>>> ClientCnxn:
>>>>>> Socket
>>>>>>
>>>>>> connection established to zk1/10.0.0.66:2181, initiating session
>>>>>>> 2013-02-20 14:25:59,556 INFO org.apache.zookeeper.******ClientCnxn:
>>>>>>>
>>>>>>> Session
>>>>>>> establishment complete on server zk1/10.0.0.66:2181, sessionid =
>>>>>>> 0x13cc3f3bcdc0095, negotiated timeout = 40000
>>>>>>> 2013-02-20 14:25:59,569 DEBUG
>>>>>>> org.apache.hadoop.hbase.******catalog.CatalogTracker:
>>>>>>> Starting catalog tracker org.apache.hadoop.hbase.**
>>>>>>> catalog.CatalogTracker@******151205ab
>>>>>>> 2013-02-20 14:25:59,587 FATAL org.apache.hadoop.hbase.****
>>>>>>> master.HMaster:
>>>>>>> Unhandled exception. Starting shutdown.
>>>>>>> java.lang.******IllegalArgumentException:
>>>>>>> �^@^@^@^\31321@d1r2n2.prod.**
>>>>>>> plutoz.comENABLED
>>>>>>> at org.apache.hadoop.hbase.******zookeeper.ZKTable.**
>>>>>>> getTableState(ZKTable.java:******119)
>>>>>>> at org.apache.hadoop.hbase.******zookeeper.ZKTable.**
>>>>>>> populateTableStates(ZKTable.******java:94)
>>>>>>> at org.apache.hadoop.hbase.******
>>>>>>> zookeeper.ZKTable.<init>(**
>>>>>>> ZKTable.java:79)
>>>>>>> at org.apache.hadoop.hbase.******
>>>>>>> master.AssignmentManager.<**
>>>>>>> init>(AssignmentManager.java:******171)
>>>>>>> at org.apache.hadoop.hbase.******master.HMaster.**
>>>>>>> finishInitialization(HMaster.******java:353)
>>>>>>> at org.apache.hadoop.hbase.******
>>>>>>> master.HMaster.run(HMaster.**
>>>>>>> java:281)
>>>>>>> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.hbase.****
>>>>>>> master.HMaster:
>>>>>>> Aborting
>>>>>>> 2013-02-20 14:25:59,590 DEBUG org.apache.hadoop.hbase.****
>>>>>>> master.HMaster:
>>>>>>> Stopping service threads
>>>>>>> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.ipc.******
>>>>>>> HBaseServer:
>>>>>>> Stopping server on 60000
>>>>>>> 2013-02-20 14:25:59,591 INFO org.apache.hadoop.ipc.******
>>>>>>> HBaseServer:
>>>>>>>
>>>>>>> IPC
>>>>>>> Server handler 0 on 60000: exiting
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> does anyone have any idea or suggestion about what I can do to be
>>>>>> able to
>>>>>> run a hbase 0.90.x on hadoop 1.1.1?
>>>>>>
>>>>>> thanks,
>>>>>>
>>>>>> Kaveh Minooie
>>>>>>
>>>>>> www.plutoz.com
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>> --
>>> Kaveh Minooie
>>>
>>> www.plutoz.com
>>>
>>>
>>
> --
> Kaveh Minooie
>
> www.plutoz.com
>
Re: HBase newbie
Posted by kaveh minooie <ka...@plutoz.com>.
thanks, this time it was the hbase directory in the hdfs :). everything
seems to be working now.
On 02/20/2013 04:13 PM, Ted Yu wrote:
> Have you checked the region server log from d1r1n19.prod.plutoz.com around
> this time ?
>
> Cheers
>
> On Wed, Feb 20, 2013 at 4:10 PM, kaveh minooie <ka...@plutoz.com> wrote:
>
>> ok so I used to run hbase 0.94.4 before i realized that i had to
>> downgrade. my zookeeper hbase directory had the files from that version
>> before. once I cleaned the zookeeper directory using zkCli, master node
>> came up. and after that i checked and all the data directories on the
>> cluster are empty (there was no data in hbase). but now when i start-hbase
>> the master node log file just keeps saying this :
>>
>> 2013-02-20 15:45:39,518 INFO org.apache.hadoop.hbase.**master.AssignmentManager:
>> Using pre-existing plan for region -ROOT-,,0.70236052;
>> plan=hri=-ROOT-,,0.70236052, src=, dest=d1r1n19.prod.plutoz.com,**
>> 60020,1361403838824
>> 2013-02-20 15:45:39,518 INFO org.apache.hadoop.hbase.**master.AssignmentManager:
>> Assigning region -ROOT-,,0.70236052 to d1r1n19.prod.plutoz.com,60020,**
>> 1361403838824
>> 2013-02-20 15:45:39,524 DEBUG org.apache.hadoop.hbase.**master.AssignmentManager:
>> Handling transition=RS_ZK_REGION_**OPENING, server=d1r1n19.prod.plutoz.com
>> **,60020,1361403838824, region=70236052/-ROOT-
>> 2013-02-20 15:45:39,532 DEBUG org.apache.hadoop.hbase.**master.AssignmentManager:
>> Handling transition=RS_ZK_REGION_**FAILED_OPEN, server=
>> d1r1n19.prod.plutoz.com**,60020,1361403838824, region=70236052/-ROOT-
>> 2013-02-20 15:45:39,532 DEBUG org.apache.hadoop.hbase.**master.handler.**ClosedRegionHandler:
>> Handling CLOSED event for 70236052
>> 2013-02-20 15:45:39,532 DEBUG org.apache.hadoop.hbase.**master.AssignmentManager:
>> Forcing OFFLINE; was=-ROOT-,,0.70236052 state=CLOSED, ts=1361403939527
>> 2013-02-20 15:45:39,532 DEBUG org.apache.hadoop.hbase.**zookeeper.ZKAssign:
>> master:60000-0x13cc3f3bcdc00b9 Creating (or updating) unassigned node for
>> 70236052 with OFFLINE state
>>
>>
>> again and again until i kill the java process.
>>
>> any idea what this means ?
>>
>>
>>
>>
>> On 02/20/2013 03:16 PM, kaveh minooie wrote:
>>
>>> I can't upgrade. I need hbase for nutch and it currently only supports
>>> 0.90.x ( i am using 0.90.6)
>>>
>>> the zookeeper log is:
>>> 2013-02-20 14:25:59,035 [myid:] - INFO
>>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:**NIOServerCnxnFactory@197<ht...@197>]
>>> -
>>> Accepted socket connection from /10.0.0.66:49971
>>> 2013-02-20 14:25:59,040 [myid:] - WARN
>>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**793<ht...@793>]
>>> -
>>> Connection request from old client /10.0.0.66:49971; will be dropped if
>>> server
>>> is in r-o mode
>>> 2013-02-20 14:25:59,040 [myid:] - INFO
>>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**839<ht...@839>]
>>> - Client
>>> attempting to establish new session at /10.0.0.66:49971
>>> 2013-02-20 14:25:59,054 [myid:] - INFO
>>> [SyncThread:0:ZooKeeperServer@**595] - Established session
>>> 0x13cc3f3bcdc0094 with negotiated timeout 40000 for client /
>>> 10.0.0.66:49971
>>> 2013-02-20 14:25:59,554 [myid:] - INFO
>>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:**NIOServerCnxnFactory@197<ht...@197>]
>>> -
>>> Accepted socket connection from /10.0.0.66:49984
>>> 2013-02-20 14:25:59,554 [myid:] - WARN
>>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**793<ht...@793>]
>>> -
>>> Connection request from old client /10.0.0.66:49984; will be dropped if
>>> server is in r-o mode
>>> 2013-02-20 14:25:59,554 [myid:] - INFO
>>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**839<ht...@839>]
>>> - Client
>>> attempting to establish new session at /10.0.0.66:49984
>>> 2013-02-20 14:25:59,556 [myid:] - INFO
>>> [SyncThread:0:ZooKeeperServer@**595] - Established session
>>> 0x13cc3f3bcdc0095 with negotiated timeout 40000 for client /
>>> 10.0.0.66:49984
>>> 2013-02-20 14:25:59,598 [myid:] - INFO [ProcessThread(sid:0
>>> cport:-1)::**PrepRequestProcessor@476] - Processed session termination
>>> for
>>> sessionid: 0x13cc3f3bcdc0095
>>> 2013-02-20 14:25:59,599 [myid:] - INFO
>>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:NIOServerCnxn@**1001<ht...@1001>]
>>> - Closed
>>> socket connection for client /10.0.0.66:49984 which had sessionid
>>> 0x13cc3f3bcdc0095
>>> 2013-02-20 14:25:59,600 [myid:] - INFO [ProcessThread(sid:0
>>> cport:-1)::**PrepRequestProcessor@476] - Processed session termination
>>> for
>>> sessionid: 0x13cc3f3bcdc0094
>>> 2013-02-20 14:25:59,601 [myid:] - INFO
>>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:NIOServerCnxn@**1001<ht...@1001>]
>>> - Closed
>>> socket connection for client /10.0.0.66:49971 which had sessionid
>>> 0x13cc3f3bcdc0094
>>> 2013-02-20 14:26:00,218 [myid:] - INFO
>>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:**NIOServerCnxnFactory@197<ht...@197>]
>>> -
>>> Accepted socket connection from /10.0.0.13:48286
>>> 2013-02-20 14:26:00,220 [myid:] - WARN
>>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**793<ht...@793>]
>>> -
>>> Connection request from old client /10.0.0.13:48286; will be dropped if
>>> server is in r-o mode
>>> 2013-02-20 14:26:00,221 [myid:] - INFO
>>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**839<ht...@839>]
>>> - Client
>>> attempting to establish new session at /10.0.0.13:48286
>>> 2013-02-20 14:26:00,222 [myid:] - INFO
>>> [SyncThread:0:ZooKeeperServer@**595] - Established session
>>> 0x13cc3f3bcdc0096 with negotiated timeout 40000 for client /
>>> 10.0.0.13:48286
>>>
>>>
>>> 10.0.0.13 is one of the regionservers.
>>>
>>> thanks,
>>>
>>>
>>>
>>> On 02/20/2013 02:48 PM, Ted Yu wrote:
>>>
>>>> The error seemed to be related to zookeeper.
>>>>
>>>> Can you post log from zk1 around 14:25:59 ?
>>>>
>>>> BTW stable release is 0.94.5, can you upgrade ?
>>>>
>>>> Cheers
>>>>
>>>> On Wed, Feb 20, 2013 at 2:46 PM, kaveh minooie <ka...@plutoz.com> wrote:
>>>>
>>>> Hello everyone
>>>>> I was wondering if any one has tried to run hbase 0.90.x on a hadoop
>>>>> 1.1.1 cluster? I have replaced a few of the jar files with the ones that
>>>>> came with the packages that I am using in my cluster:
>>>>>
>>>>> commons-configuration-1.6.jar
>>>>> hadoop-core-1.1.1.jar
>>>>> slf4j-api-1.6.1.jar
>>>>> slf4j-log4j12-1.6.1.jar
>>>>> zookeeper-3.4.5.jar (tried with both 3.3.2 and 3.4.5. same result)
>>>>>
>>>>> after these changes I am able to run the region servers but the master
>>>>> node still crashes when I try to start it:
>>>>>
>>>>> 2013-02-20 14:25:59,554 INFO org.apache.zookeeper.****ClientCnxn:
>>>>> Socket
>>>>>
>>>>>> connection established to zk1/10.0.0.66:2181, initiating session
>>>>>> 2013-02-20 14:25:59,556 INFO org.apache.zookeeper.****ClientCnxn:
>>>>>> Session
>>>>>> establishment complete on server zk1/10.0.0.66:2181, sessionid =
>>>>>> 0x13cc3f3bcdc0095, negotiated timeout = 40000
>>>>>> 2013-02-20 14:25:59,569 DEBUG
>>>>>> org.apache.hadoop.hbase.****catalog.CatalogTracker:
>>>>>> Starting catalog tracker org.apache.hadoop.hbase.**
>>>>>> catalog.CatalogTracker@****151205ab
>>>>>> 2013-02-20 14:25:59,587 FATAL org.apache.hadoop.hbase.****
>>>>>> master.HMaster:
>>>>>> Unhandled exception. Starting shutdown.
>>>>>> java.lang.****IllegalArgumentException: �^@^@^@^\31321@d1r2n2.prod.**
>>>>>> plutoz.comENABLED
>>>>>> at org.apache.hadoop.hbase.****zookeeper.ZKTable.**
>>>>>> getTableState(ZKTable.java:****119)
>>>>>> at org.apache.hadoop.hbase.****zookeeper.ZKTable.**
>>>>>> populateTableStates(ZKTable.****java:94)
>>>>>> at org.apache.hadoop.hbase.****zookeeper.ZKTable.<init>(**
>>>>>> ZKTable.java:79)
>>>>>> at org.apache.hadoop.hbase.****master.AssignmentManager.<**
>>>>>> init>(AssignmentManager.java:****171)
>>>>>> at org.apache.hadoop.hbase.****master.HMaster.**
>>>>>> finishInitialization(HMaster.****java:353)
>>>>>> at org.apache.hadoop.hbase.****master.HMaster.run(HMaster.**
>>>>>> java:281)
>>>>>> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.hbase.****
>>>>>> master.HMaster:
>>>>>> Aborting
>>>>>> 2013-02-20 14:25:59,590 DEBUG org.apache.hadoop.hbase.****
>>>>>> master.HMaster:
>>>>>> Stopping service threads
>>>>>> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.ipc.****HBaseServer:
>>>>>> Stopping server on 60000
>>>>>> 2013-02-20 14:25:59,591 INFO org.apache.hadoop.ipc.****HBaseServer:
>>>>>> IPC
>>>>>> Server handler 0 on 60000: exiting
>>>>>>
>>>>>>
>>>>>
>>>>> does anyone have any idea or suggestion about what I can do to be
>>>>> able to
>>>>> run a hbase 0.90.x on hadoop 1.1.1?
>>>>>
>>>>> thanks,
>>>>>
>>>>> Kaveh Minooie
>>>>>
>>>>> www.plutoz.com
>>>>>
>>>>>
>>>>
>>>
>> --
>> Kaveh Minooie
>>
>> www.plutoz.com
>>
>
--
Kaveh Minooie
www.plutoz.com
Re: HBase newbie
Posted by Ted Yu <yu...@gmail.com>.
Have you checked the region server log from d1r1n19.prod.plutoz.com around
this time ?
Cheers
On Wed, Feb 20, 2013 at 4:10 PM, kaveh minooie <ka...@plutoz.com> wrote:
> ok so I used to run hbase 0.94.4 before i realized that i had to
> downgrade. my zookeeper hbase directory had the files from that version
> before. once I cleaned the zookeeper directory using zkCli, master node
> came up. and after that i checked and all the data directories on the
> cluster are empty (there was no data in hbase). but now when i start-hbase
> the master node log file just keeps saying this :
>
> 2013-02-20 15:45:39,518 INFO org.apache.hadoop.hbase.**master.AssignmentManager:
> Using pre-existing plan for region -ROOT-,,0.70236052;
> plan=hri=-ROOT-,,0.70236052, src=, dest=d1r1n19.prod.plutoz.com,**
> 60020,1361403838824
> 2013-02-20 15:45:39,518 INFO org.apache.hadoop.hbase.**master.AssignmentManager:
> Assigning region -ROOT-,,0.70236052 to d1r1n19.prod.plutoz.com,60020,**
> 1361403838824
> 2013-02-20 15:45:39,524 DEBUG org.apache.hadoop.hbase.**master.AssignmentManager:
> Handling transition=RS_ZK_REGION_**OPENING, server=d1r1n19.prod.plutoz.com
> **,60020,1361403838824, region=70236052/-ROOT-
> 2013-02-20 15:45:39,532 DEBUG org.apache.hadoop.hbase.**master.AssignmentManager:
> Handling transition=RS_ZK_REGION_**FAILED_OPEN, server=
> d1r1n19.prod.plutoz.com**,60020,1361403838824, region=70236052/-ROOT-
> 2013-02-20 15:45:39,532 DEBUG org.apache.hadoop.hbase.**master.handler.**ClosedRegionHandler:
> Handling CLOSED event for 70236052
> 2013-02-20 15:45:39,532 DEBUG org.apache.hadoop.hbase.**master.AssignmentManager:
> Forcing OFFLINE; was=-ROOT-,,0.70236052 state=CLOSED, ts=1361403939527
> 2013-02-20 15:45:39,532 DEBUG org.apache.hadoop.hbase.**zookeeper.ZKAssign:
> master:60000-0x13cc3f3bcdc00b9 Creating (or updating) unassigned node for
> 70236052 with OFFLINE state
>
>
> again and again until i kill the java process.
>
> any idea what this means ?
>
>
>
>
> On 02/20/2013 03:16 PM, kaveh minooie wrote:
>
>> I can't upgrade. I need hbase for nutch and it currently only supports
>> 0.90.x ( i am using 0.90.6)
>>
>> the zookeeper log is:
>> 2013-02-20 14:25:59,035 [myid:] - INFO
>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:**NIOServerCnxnFactory@197<ht...@197>]
>> -
>> Accepted socket connection from /10.0.0.66:49971
>> 2013-02-20 14:25:59,040 [myid:] - WARN
>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**793<ht...@793>]
>> -
>> Connection request from old client /10.0.0.66:49971; will be dropped if
>> server
>> is in r-o mode
>> 2013-02-20 14:25:59,040 [myid:] - INFO
>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**839<ht...@839>]
>> - Client
>> attempting to establish new session at /10.0.0.66:49971
>> 2013-02-20 14:25:59,054 [myid:] - INFO
>> [SyncThread:0:ZooKeeperServer@**595] - Established session
>> 0x13cc3f3bcdc0094 with negotiated timeout 40000 for client /
>> 10.0.0.66:49971
>> 2013-02-20 14:25:59,554 [myid:] - INFO
>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:**NIOServerCnxnFactory@197<ht...@197>]
>> -
>> Accepted socket connection from /10.0.0.66:49984
>> 2013-02-20 14:25:59,554 [myid:] - WARN
>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**793<ht...@793>]
>> -
>> Connection request from old client /10.0.0.66:49984; will be dropped if
>> server is in r-o mode
>> 2013-02-20 14:25:59,554 [myid:] - INFO
>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**839<ht...@839>]
>> - Client
>> attempting to establish new session at /10.0.0.66:49984
>> 2013-02-20 14:25:59,556 [myid:] - INFO
>> [SyncThread:0:ZooKeeperServer@**595] - Established session
>> 0x13cc3f3bcdc0095 with negotiated timeout 40000 for client /
>> 10.0.0.66:49984
>> 2013-02-20 14:25:59,598 [myid:] - INFO [ProcessThread(sid:0
>> cport:-1)::**PrepRequestProcessor@476] - Processed session termination
>> for
>> sessionid: 0x13cc3f3bcdc0095
>> 2013-02-20 14:25:59,599 [myid:] - INFO
>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:NIOServerCnxn@**1001<ht...@1001>]
>> - Closed
>> socket connection for client /10.0.0.66:49984 which had sessionid
>> 0x13cc3f3bcdc0095
>> 2013-02-20 14:25:59,600 [myid:] - INFO [ProcessThread(sid:0
>> cport:-1)::**PrepRequestProcessor@476] - Processed session termination
>> for
>> sessionid: 0x13cc3f3bcdc0094
>> 2013-02-20 14:25:59,601 [myid:] - INFO
>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:NIOServerCnxn@**1001<ht...@1001>]
>> - Closed
>> socket connection for client /10.0.0.66:49971 which had sessionid
>> 0x13cc3f3bcdc0094
>> 2013-02-20 14:26:00,218 [myid:] - INFO
>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:**NIOServerCnxnFactory@197<ht...@197>]
>> -
>> Accepted socket connection from /10.0.0.13:48286
>> 2013-02-20 14:26:00,220 [myid:] - WARN
>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**793<ht...@793>]
>> -
>> Connection request from old client /10.0.0.13:48286; will be dropped if
>> server is in r-o mode
>> 2013-02-20 14:26:00,221 [myid:] - INFO
>> [NIOServerCxn.Factory:0.0.0.0/**0.0.0.0:2181:ZooKeeperServer@**839<ht...@839>]
>> - Client
>> attempting to establish new session at /10.0.0.13:48286
>> 2013-02-20 14:26:00,222 [myid:] - INFO
>> [SyncThread:0:ZooKeeperServer@**595] - Established session
>> 0x13cc3f3bcdc0096 with negotiated timeout 40000 for client /
>> 10.0.0.13:48286
>>
>>
>> 10.0.0.13 is one of the regionservers.
>>
>> thanks,
>>
>>
>>
>> On 02/20/2013 02:48 PM, Ted Yu wrote:
>>
>>> The error seemed to be related to zookeeper.
>>>
>>> Can you post log from zk1 around 14:25:59 ?
>>>
>>> BTW stable release is 0.94.5, can you upgrade ?
>>>
>>> Cheers
>>>
>>> On Wed, Feb 20, 2013 at 2:46 PM, kaveh minooie <ka...@plutoz.com> wrote:
>>>
>>> Hello everyone
>>>> I was wondering if any one has tried to run hbase 0.90.x on a hadoop
>>>> 1.1.1 cluster? I have replaced a few of the jar files with the ones that
>>>> came with the packages that I am using in my cluster:
>>>>
>>>> commons-configuration-1.6.jar
>>>> hadoop-core-1.1.1.jar
>>>> slf4j-api-1.6.1.jar
>>>> slf4j-log4j12-1.6.1.jar
>>>> zookeeper-3.4.5.jar (tried with both 3.3.2 and 3.4.5. same result)
>>>>
>>>> after these changes I am able to run the region servers but the master
>>>> node still crashes when I try to start it:
>>>>
>>>> 2013-02-20 14:25:59,554 INFO org.apache.zookeeper.****ClientCnxn:
>>>> Socket
>>>>
>>>>> connection established to zk1/10.0.0.66:2181, initiating session
>>>>> 2013-02-20 14:25:59,556 INFO org.apache.zookeeper.****ClientCnxn:
>>>>> Session
>>>>> establishment complete on server zk1/10.0.0.66:2181, sessionid =
>>>>> 0x13cc3f3bcdc0095, negotiated timeout = 40000
>>>>> 2013-02-20 14:25:59,569 DEBUG
>>>>> org.apache.hadoop.hbase.****catalog.CatalogTracker:
>>>>> Starting catalog tracker org.apache.hadoop.hbase.**
>>>>> catalog.CatalogTracker@****151205ab
>>>>> 2013-02-20 14:25:59,587 FATAL org.apache.hadoop.hbase.****
>>>>> master.HMaster:
>>>>> Unhandled exception. Starting shutdown.
>>>>> java.lang.****IllegalArgumentException: �^@^@^@^\31321@d1r2n2.prod.**
>>>>> plutoz.comENABLED
>>>>> at org.apache.hadoop.hbase.****zookeeper.ZKTable.**
>>>>> getTableState(ZKTable.java:****119)
>>>>> at org.apache.hadoop.hbase.****zookeeper.ZKTable.**
>>>>> populateTableStates(ZKTable.****java:94)
>>>>> at org.apache.hadoop.hbase.****zookeeper.ZKTable.<init>(**
>>>>> ZKTable.java:79)
>>>>> at org.apache.hadoop.hbase.****master.AssignmentManager.<**
>>>>> init>(AssignmentManager.java:****171)
>>>>> at org.apache.hadoop.hbase.****master.HMaster.**
>>>>> finishInitialization(HMaster.****java:353)
>>>>> at org.apache.hadoop.hbase.****master.HMaster.run(HMaster.**
>>>>> java:281)
>>>>> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.hbase.****
>>>>> master.HMaster:
>>>>> Aborting
>>>>> 2013-02-20 14:25:59,590 DEBUG org.apache.hadoop.hbase.****
>>>>> master.HMaster:
>>>>> Stopping service threads
>>>>> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.ipc.****HBaseServer:
>>>>> Stopping server on 60000
>>>>> 2013-02-20 14:25:59,591 INFO org.apache.hadoop.ipc.****HBaseServer:
>>>>> IPC
>>>>> Server handler 0 on 60000: exiting
>>>>>
>>>>>
>>>>
>>>> does anyone have any idea or suggestion about what I can do to be
>>>> able to
>>>> run a hbase 0.90.x on hadoop 1.1.1?
>>>>
>>>> thanks,
>>>>
>>>> Kaveh Minooie
>>>>
>>>> www.plutoz.com
>>>>
>>>>
>>>
>>
> --
> Kaveh Minooie
>
> www.plutoz.com
>
Re: HBase newbie
Posted by kaveh minooie <ka...@plutoz.com>.
ok so I used to run hbase 0.94.4 before i realized that i had to
downgrade. my zookeeper hbase directory had the files from that version
before. once I cleaned the zookeeper directory using zkCli, master node
came up. and after that i checked and all the data directories on the
cluster are empty (there was no data in hbase). but now when i
start-hbase the master node log file just keeps saying this :
2013-02-20 15:45:39,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing
plan for region -ROOT-,,0.70236052; plan=hri=-ROOT-,,0.70236052, src=,
dest=d1r1n19.prod.plutoz.com,60020,1361403838824
2013-02-20 15:45:39,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
-ROOT-,,0.70236052 to d1r1n19.prod.plutoz.com,60020,1361403838824
2013-02-20 15:45:39,524 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=d1r1n19.prod.plutoz.com,60020,1361403838824, region=70236052/-ROOT-
2013-02-20 15:45:39,532 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_FAILED_OPEN,
server=d1r1n19.prod.plutoz.com,60020,1361403838824, region=70236052/-ROOT-
2013-02-20 15:45:39,532 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling
CLOSED event for 70236052
2013-02-20 15:45:39,532 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
was=-ROOT-,,0.70236052 state=CLOSED, ts=1361403939527
2013-02-20 15:45:39,532 DEBUG
org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:60000-0x13cc3f3bcdc00b9 Creating (or updating) unassigned node
for 70236052 with OFFLINE state
again and again until i kill the java process.
any idea what this means ?
On 02/20/2013 03:16 PM, kaveh minooie wrote:
> I can't upgrade. I need hbase for nutch and it currently only supports
> 0.90.x ( i am using 0.90.6)
>
> the zookeeper log is:
> 2013-02-20 14:25:59,035 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] -
> Accepted socket connection from /10.0.0.66:49971
> 2013-02-20 14:25:59,040 [myid:] - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@793] -
> Connection request from old client /10.0.0.66:49971; will be dropped if
> server
> is in r-o mode
> 2013-02-20 14:25:59,040 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client
> attempting to establish new session at /10.0.0.66:49971
> 2013-02-20 14:25:59,054 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@595] - Established session
> 0x13cc3f3bcdc0094 with negotiated timeout 40000 for client /10.0.0.66:49971
> 2013-02-20 14:25:59,554 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] -
> Accepted socket connection from /10.0.0.66:49984
> 2013-02-20 14:25:59,554 [myid:] - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@793] -
> Connection request from old client /10.0.0.66:49984; will be dropped if
> server is in r-o mode
> 2013-02-20 14:25:59,554 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client
> attempting to establish new session at /10.0.0.66:49984
> 2013-02-20 14:25:59,556 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@595] - Established session
> 0x13cc3f3bcdc0095 with negotiated timeout 40000 for client /10.0.0.66:49984
> 2013-02-20 14:25:59,598 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@476] - Processed session termination for
> sessionid: 0x13cc3f3bcdc0095
> 2013-02-20 14:25:59,599 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed
> socket connection for client /10.0.0.66:49984 which had sessionid
> 0x13cc3f3bcdc0095
> 2013-02-20 14:25:59,600 [myid:] - INFO [ProcessThread(sid:0
> cport:-1)::PrepRequestProcessor@476] - Processed session termination for
> sessionid: 0x13cc3f3bcdc0094
> 2013-02-20 14:25:59,601 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed
> socket connection for client /10.0.0.66:49971 which had sessionid
> 0x13cc3f3bcdc0094
> 2013-02-20 14:26:00,218 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] -
> Accepted socket connection from /10.0.0.13:48286
> 2013-02-20 14:26:00,220 [myid:] - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@793] -
> Connection request from old client /10.0.0.13:48286; will be dropped if
> server is in r-o mode
> 2013-02-20 14:26:00,221 [myid:] - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client
> attempting to establish new session at /10.0.0.13:48286
> 2013-02-20 14:26:00,222 [myid:] - INFO
> [SyncThread:0:ZooKeeperServer@595] - Established session
> 0x13cc3f3bcdc0096 with negotiated timeout 40000 for client /10.0.0.13:48286
>
>
> 10.0.0.13 is one of the regionservers.
>
> thanks,
>
>
>
> On 02/20/2013 02:48 PM, Ted Yu wrote:
>> The error seemed to be related to zookeeper.
>>
>> Can you post log from zk1 around 14:25:59 ?
>>
>> BTW stable release is 0.94.5, can you upgrade ?
>>
>> Cheers
>>
>> On Wed, Feb 20, 2013 at 2:46 PM, kaveh minooie <ka...@plutoz.com> wrote:
>>
>>> Hello everyone
>>> I was wondering if any one has tried to run hbase 0.90.x on a hadoop
>>> 1.1.1 cluster? I have replaced a few of the jar files with the ones that
>>> came with the packages that I am using in my cluster:
>>>
>>> commons-configuration-1.6.jar
>>> hadoop-core-1.1.1.jar
>>> slf4j-api-1.6.1.jar
>>> slf4j-log4j12-1.6.1.jar
>>> zookeeper-3.4.5.jar (tried with both 3.3.2 and 3.4.5. same result)
>>>
>>> after these changes I am able to run the region servers but the master
>>> node still crashes when I try to start it:
>>>
>>> 2013-02-20 14:25:59,554 INFO org.apache.zookeeper.**ClientCnxn: Socket
>>>> connection established to zk1/10.0.0.66:2181, initiating session
>>>> 2013-02-20 14:25:59,556 INFO org.apache.zookeeper.**ClientCnxn: Session
>>>> establishment complete on server zk1/10.0.0.66:2181, sessionid =
>>>> 0x13cc3f3bcdc0095, negotiated timeout = 40000
>>>> 2013-02-20 14:25:59,569 DEBUG
>>>> org.apache.hadoop.hbase.**catalog.CatalogTracker:
>>>> Starting catalog tracker org.apache.hadoop.hbase.**
>>>> catalog.CatalogTracker@**151205ab
>>>> 2013-02-20 14:25:59,587 FATAL org.apache.hadoop.hbase.**master.HMaster:
>>>> Unhandled exception. Starting shutdown.
>>>> java.lang.**IllegalArgumentException: �^@^@^@^\31321@d1r2n2.prod.**
>>>> plutoz.comENABLED
>>>> at org.apache.hadoop.hbase.**zookeeper.ZKTable.**
>>>> getTableState(ZKTable.java:**119)
>>>> at org.apache.hadoop.hbase.**zookeeper.ZKTable.**
>>>> populateTableStates(ZKTable.**java:94)
>>>> at org.apache.hadoop.hbase.**zookeeper.ZKTable.<init>(**
>>>> ZKTable.java:79)
>>>> at org.apache.hadoop.hbase.**master.AssignmentManager.<**
>>>> init>(AssignmentManager.java:**171)
>>>> at org.apache.hadoop.hbase.**master.HMaster.**
>>>> finishInitialization(HMaster.**java:353)
>>>> at org.apache.hadoop.hbase.**master.HMaster.run(HMaster.**
>>>> java:281)
>>>> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.hbase.**master.HMaster:
>>>> Aborting
>>>> 2013-02-20 14:25:59,590 DEBUG org.apache.hadoop.hbase.**master.HMaster:
>>>> Stopping service threads
>>>> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.ipc.**HBaseServer:
>>>> Stopping server on 60000
>>>> 2013-02-20 14:25:59,591 INFO org.apache.hadoop.ipc.**HBaseServer: IPC
>>>> Server handler 0 on 60000: exiting
>>>>
>>>
>>>
>>> does anyone have any idea or suggestion about what I can do to be
>>> able to
>>> run a hbase 0.90.x on hadoop 1.1.1?
>>>
>>> thanks,
>>>
>>> Kaveh Minooie
>>>
>>> www.plutoz.com
>>>
>>
>
--
Kaveh Minooie
www.plutoz.com
Re: HBase newbie
Posted by Ted Yu <yu...@gmail.com>.
What's IP address for d1r2n2.prod.**plutoz.com ?
Cheers
On Wed, Feb 20, 2013 at 3:16 PM, kaveh minooie <ka...@plutoz.com> wrote:
> I can't upgrade. I need hbase for nutch and it currently only supports
> 0.90.x ( i am using 0.90.6)
>
> the zookeeper log is:
> 2013-02-20 14:25:59,035 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/**
> 0.0.0.0:2181:**NIOServerCnxnFactory@197<ht...@197>]
> - Accepted socket connection from /10.0.0.66:49971
> 2013-02-20 14:25:59,040 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/**
> 0.0.0.0:2181:ZooKeeperServer@**793<ht...@793>]
> - Connection request from old client /10.0.0.66:49971; will be dropped if
> server
> is in r-o mode
> 2013-02-20 14:25:59,040 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/**
> 0.0.0.0:2181:ZooKeeperServer@**839<ht...@839>]
> - Client attempting to establish new session at /10.0.0.66:49971
> 2013-02-20 14:25:59,054 [myid:] - INFO [SyncThread:0:ZooKeeperServer@**595]
> - Established session 0x13cc3f3bcdc0094 with negotiated timeout 40000 for
> client /10.0.0.66:49971
> 2013-02-20 14:25:59,554 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/**
> 0.0.0.0:2181:**NIOServerCnxnFactory@197<ht...@197>]
> - Accepted socket connection from /10.0.0.66:49984
> 2013-02-20 14:25:59,554 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/**
> 0.0.0.0:2181:ZooKeeperServer@**793<ht...@793>]
> - Connection request from old client /10.0.0.66:49984; will be dropped if
> server is in r-o mode
> 2013-02-20 14:25:59,554 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/**
> 0.0.0.0:2181:ZooKeeperServer@**839<ht...@839>]
> - Client attempting to establish new session at /10.0.0.66:49984
> 2013-02-20 14:25:59,556 [myid:] - INFO [SyncThread:0:ZooKeeperServer@**595]
> - Established session 0x13cc3f3bcdc0095 with negotiated timeout 40000 for
> client /10.0.0.66:49984
> 2013-02-20 14:25:59,598 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::**
> PrepRequestProcessor@476] - Processed session termination for sessionid:
> 0x13cc3f3bcdc0095
> 2013-02-20 14:25:59,599 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/**
> 0.0.0.0:2181:NIOServerCnxn@**1001<ht...@1001>]
> - Closed socket connection for client /10.0.0.66:49984 which had
> sessionid 0x13cc3f3bcdc0095
> 2013-02-20 14:25:59,600 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::**
> PrepRequestProcessor@476] - Processed session termination for sessionid:
> 0x13cc3f3bcdc0094
> 2013-02-20 14:25:59,601 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/**
> 0.0.0.0:2181:NIOServerCnxn@**1001<ht...@1001>]
> - Closed socket connection for client /10.0.0.66:49971 which had
> sessionid 0x13cc3f3bcdc0094
> 2013-02-20 14:26:00,218 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/**
> 0.0.0.0:2181:**NIOServerCnxnFactory@197<ht...@197>]
> - Accepted socket connection from /10.0.0.13:48286
> 2013-02-20 14:26:00,220 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/**
> 0.0.0.0:2181:ZooKeeperServer@**793<ht...@793>]
> - Connection request from old client /10.0.0.13:48286; will be dropped if
> server is in r-o mode
> 2013-02-20 14:26:00,221 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/**
> 0.0.0.0:2181:ZooKeeperServer@**839<ht...@839>]
> - Client attempting to establish new session at /10.0.0.13:48286
> 2013-02-20 14:26:00,222 [myid:] - INFO [SyncThread:0:ZooKeeperServer@**595]
> - Established session 0x13cc3f3bcdc0096 with negotiated timeout 40000 for
> client /10.0.0.13:48286
>
>
> 10.0.0.13 is one of the regionservers.
>
> thanks,
>
>
>
>
> On 02/20/2013 02:48 PM, Ted Yu wrote:
>
>> The error seemed to be related to zookeeper.
>>
>> Can you post log from zk1 around 14:25:59 ?
>>
>> BTW stable release is 0.94.5, can you upgrade ?
>>
>> Cheers
>>
>> On Wed, Feb 20, 2013 at 2:46 PM, kaveh minooie <ka...@plutoz.com> wrote:
>>
>> Hello everyone
>>> I was wondering if any one has tried to run hbase 0.90.x on a hadoop
>>> 1.1.1 cluster? I have replaced a few of the jar files with the ones that
>>> came with the packages that I am using in my cluster:
>>>
>>> commons-configuration-1.6.jar
>>> hadoop-core-1.1.1.jar
>>> slf4j-api-1.6.1.jar
>>> slf4j-log4j12-1.6.1.jar
>>> zookeeper-3.4.5.jar (tried with both 3.3.2 and 3.4.5. same result)
>>>
>>> after these changes I am able to run the region servers but the master
>>> node still crashes when I try to start it:
>>>
>>> 2013-02-20 14:25:59,554 INFO org.apache.zookeeper.****ClientCnxn:
>>> Socket
>>>
>>>> connection established to zk1/10.0.0.66:2181, initiating session
>>>> 2013-02-20 14:25:59,556 INFO org.apache.zookeeper.****ClientCnxn:
>>>> Session
>>>>
>>>> establishment complete on server zk1/10.0.0.66:2181, sessionid =
>>>> 0x13cc3f3bcdc0095, negotiated timeout = 40000
>>>> 2013-02-20 14:25:59,569 DEBUG org.apache.hadoop.hbase.****
>>>> catalog.CatalogTracker:
>>>> Starting catalog tracker org.apache.hadoop.hbase.**
>>>> catalog.CatalogTracker@****151205ab
>>>> 2013-02-20 14:25:59,587 FATAL org.apache.hadoop.hbase.****
>>>> master.HMaster:
>>>> Unhandled exception. Starting shutdown.
>>>> java.lang.****IllegalArgumentException: �^@^@^@^\31321@d1r2n2.prod.**
>>>> plutoz.comENABLED
>>>> at org.apache.hadoop.hbase.****zookeeper.ZKTable.**
>>>> getTableState(ZKTable.java:****119)
>>>> at org.apache.hadoop.hbase.****zookeeper.ZKTable.**
>>>> populateTableStates(ZKTable.****java:94)
>>>> at org.apache.hadoop.hbase.****zookeeper.ZKTable.<init>(**
>>>> ZKTable.java:79)
>>>> at org.apache.hadoop.hbase.****master.AssignmentManager.<**
>>>> init>(AssignmentManager.java:****171)
>>>> at org.apache.hadoop.hbase.****master.HMaster.**
>>>> finishInitialization(HMaster.****java:353)
>>>> at org.apache.hadoop.hbase.****master.HMaster.run(HMaster.**
>>>> java:281)
>>>> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.hbase.****
>>>> master.HMaster:
>>>> Aborting
>>>> 2013-02-20 14:25:59,590 DEBUG org.apache.hadoop.hbase.****
>>>> master.HMaster:
>>>> Stopping service threads
>>>> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.ipc.****HBaseServer:
>>>> Stopping server on 60000
>>>> 2013-02-20 14:25:59,591 INFO org.apache.hadoop.ipc.****HBaseServer: IPC
>>>>
>>>> Server handler 0 on 60000: exiting
>>>>
>>>>
>>>
>>> does anyone have any idea or suggestion about what I can do to be able to
>>> run a hbase 0.90.x on hadoop 1.1.1?
>>>
>>> thanks,
>>>
>>> Kaveh Minooie
>>>
>>> www.plutoz.com
>>>
>>>
>>
> --
> Kaveh Minooie
>
> www.plutoz.com
>
Re: HBase newbie
Posted by kaveh minooie <ka...@plutoz.com>.
I can't upgrade. I need hbase for nutch and it currently only supports
0.90.x ( i am using 0.90.6)
the zookeeper log is:
2013-02-20 14:25:59,035 [myid:] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] -
Accepted socket connection from /10.0.0.66:49971
2013-02-20 14:25:59,040 [myid:] - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@793] -
Connection request from old client /10.0.0.66:49971; will be dropped if
server
is in r-o mode
2013-02-20 14:25:59,040 [myid:] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client
attempting to establish new session at /10.0.0.66:49971
2013-02-20 14:25:59,054 [myid:] - INFO
[SyncThread:0:ZooKeeperServer@595] - Established session
0x13cc3f3bcdc0094 with negotiated timeout 40000 for client /10.0.0.66:49971
2013-02-20 14:25:59,554 [myid:] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] -
Accepted socket connection from /10.0.0.66:49984
2013-02-20 14:25:59,554 [myid:] - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@793] -
Connection request from old client /10.0.0.66:49984; will be dropped if
server is in r-o mode
2013-02-20 14:25:59,554 [myid:] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client
attempting to establish new session at /10.0.0.66:49984
2013-02-20 14:25:59,556 [myid:] - INFO
[SyncThread:0:ZooKeeperServer@595] - Established session
0x13cc3f3bcdc0095 with negotiated timeout 40000 for client /10.0.0.66:49984
2013-02-20 14:25:59,598 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequestProcessor@476] - Processed session termination for
sessionid: 0x13cc3f3bcdc0095
2013-02-20 14:25:59,599 [myid:] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.0.66:49984 which had sessionid
0x13cc3f3bcdc0095
2013-02-20 14:25:59,600 [myid:] - INFO [ProcessThread(sid:0
cport:-1)::PrepRequestProcessor@476] - Processed session termination for
sessionid: 0x13cc3f3bcdc0094
2013-02-20 14:25:59,601 [myid:] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.0.66:49971 which had sessionid
0x13cc3f3bcdc0094
2013-02-20 14:26:00,218 [myid:] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] -
Accepted socket connection from /10.0.0.13:48286
2013-02-20 14:26:00,220 [myid:] - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@793] -
Connection request from old client /10.0.0.13:48286; will be dropped if
server is in r-o mode
2013-02-20 14:26:00,221 [myid:] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client
attempting to establish new session at /10.0.0.13:48286
2013-02-20 14:26:00,222 [myid:] - INFO
[SyncThread:0:ZooKeeperServer@595] - Established session
0x13cc3f3bcdc0096 with negotiated timeout 40000 for client /10.0.0.13:48286
10.0.0.13 is one of the regionservers.
thanks,
On 02/20/2013 02:48 PM, Ted Yu wrote:
> The error seemed to be related to zookeeper.
>
> Can you post log from zk1 around 14:25:59 ?
>
> BTW stable release is 0.94.5, can you upgrade ?
>
> Cheers
>
> On Wed, Feb 20, 2013 at 2:46 PM, kaveh minooie <ka...@plutoz.com> wrote:
>
>> Hello everyone
>> I was wondering if any one has tried to run hbase 0.90.x on a hadoop
>> 1.1.1 cluster? I have replaced a few of the jar files with the ones that
>> came with the packages that I am using in my cluster:
>>
>> commons-configuration-1.6.jar
>> hadoop-core-1.1.1.jar
>> slf4j-api-1.6.1.jar
>> slf4j-log4j12-1.6.1.jar
>> zookeeper-3.4.5.jar (tried with both 3.3.2 and 3.4.5. same result)
>>
>> after these changes I am able to run the region servers but the master
>> node still crashes when I try to start it:
>>
>> 2013-02-20 14:25:59,554 INFO org.apache.zookeeper.**ClientCnxn: Socket
>>> connection established to zk1/10.0.0.66:2181, initiating session
>>> 2013-02-20 14:25:59,556 INFO org.apache.zookeeper.**ClientCnxn: Session
>>> establishment complete on server zk1/10.0.0.66:2181, sessionid =
>>> 0x13cc3f3bcdc0095, negotiated timeout = 40000
>>> 2013-02-20 14:25:59,569 DEBUG org.apache.hadoop.hbase.**catalog.CatalogTracker:
>>> Starting catalog tracker org.apache.hadoop.hbase.**
>>> catalog.CatalogTracker@**151205ab
>>> 2013-02-20 14:25:59,587 FATAL org.apache.hadoop.hbase.**master.HMaster:
>>> Unhandled exception. Starting shutdown.
>>> java.lang.**IllegalArgumentException: �^@^@^@^\31321@d1r2n2.prod.**
>>> plutoz.comENABLED
>>> at org.apache.hadoop.hbase.**zookeeper.ZKTable.**
>>> getTableState(ZKTable.java:**119)
>>> at org.apache.hadoop.hbase.**zookeeper.ZKTable.**
>>> populateTableStates(ZKTable.**java:94)
>>> at org.apache.hadoop.hbase.**zookeeper.ZKTable.<init>(**
>>> ZKTable.java:79)
>>> at org.apache.hadoop.hbase.**master.AssignmentManager.<**
>>> init>(AssignmentManager.java:**171)
>>> at org.apache.hadoop.hbase.**master.HMaster.**
>>> finishInitialization(HMaster.**java:353)
>>> at org.apache.hadoop.hbase.**master.HMaster.run(HMaster.**
>>> java:281)
>>> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.hbase.**master.HMaster:
>>> Aborting
>>> 2013-02-20 14:25:59,590 DEBUG org.apache.hadoop.hbase.**master.HMaster:
>>> Stopping service threads
>>> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.ipc.**HBaseServer:
>>> Stopping server on 60000
>>> 2013-02-20 14:25:59,591 INFO org.apache.hadoop.ipc.**HBaseServer: IPC
>>> Server handler 0 on 60000: exiting
>>>
>>
>>
>> does anyone have any idea or suggestion about what I can do to be able to
>> run a hbase 0.90.x on hadoop 1.1.1?
>>
>> thanks,
>>
>> Kaveh Minooie
>>
>> www.plutoz.com
>>
>
--
Kaveh Minooie
www.plutoz.com
Re: HBase newbie
Posted by Ted Yu <yu...@gmail.com>.
The error seemed to be related to zookeeper.
Can you post log from zk1 around 14:25:59 ?
BTW stable release is 0.94.5, can you upgrade ?
Cheers
On Wed, Feb 20, 2013 at 2:46 PM, kaveh minooie <ka...@plutoz.com> wrote:
> Hello everyone
> I was wondering if any one has tried to run hbase 0.90.x on a hadoop
> 1.1.1 cluster? I have replaced a few of the jar files with the ones that
> came with the packages that I am using in my cluster:
>
> commons-configuration-1.6.jar
> hadoop-core-1.1.1.jar
> slf4j-api-1.6.1.jar
> slf4j-log4j12-1.6.1.jar
> zookeeper-3.4.5.jar (tried with both 3.3.2 and 3.4.5. same result)
>
> after these changes I am able to run the region servers but the master
> node still crashes when I try to start it:
>
> 2013-02-20 14:25:59,554 INFO org.apache.zookeeper.**ClientCnxn: Socket
>> connection established to zk1/10.0.0.66:2181, initiating session
>> 2013-02-20 14:25:59,556 INFO org.apache.zookeeper.**ClientCnxn: Session
>> establishment complete on server zk1/10.0.0.66:2181, sessionid =
>> 0x13cc3f3bcdc0095, negotiated timeout = 40000
>> 2013-02-20 14:25:59,569 DEBUG org.apache.hadoop.hbase.**catalog.CatalogTracker:
>> Starting catalog tracker org.apache.hadoop.hbase.**
>> catalog.CatalogTracker@**151205ab
>> 2013-02-20 14:25:59,587 FATAL org.apache.hadoop.hbase.**master.HMaster:
>> Unhandled exception. Starting shutdown.
>> java.lang.**IllegalArgumentException: �^@^@^@^\31321@d1r2n2.prod.**
>> plutoz.comENABLED
>> at org.apache.hadoop.hbase.**zookeeper.ZKTable.**
>> getTableState(ZKTable.java:**119)
>> at org.apache.hadoop.hbase.**zookeeper.ZKTable.**
>> populateTableStates(ZKTable.**java:94)
>> at org.apache.hadoop.hbase.**zookeeper.ZKTable.<init>(**
>> ZKTable.java:79)
>> at org.apache.hadoop.hbase.**master.AssignmentManager.<**
>> init>(AssignmentManager.java:**171)
>> at org.apache.hadoop.hbase.**master.HMaster.**
>> finishInitialization(HMaster.**java:353)
>> at org.apache.hadoop.hbase.**master.HMaster.run(HMaster.**
>> java:281)
>> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.hbase.**master.HMaster:
>> Aborting
>> 2013-02-20 14:25:59,590 DEBUG org.apache.hadoop.hbase.**master.HMaster:
>> Stopping service threads
>> 2013-02-20 14:25:59,590 INFO org.apache.hadoop.ipc.**HBaseServer:
>> Stopping server on 60000
>> 2013-02-20 14:25:59,591 INFO org.apache.hadoop.ipc.**HBaseServer: IPC
>> Server handler 0 on 60000: exiting
>>
>
>
> does anyone have any idea or suggestion about what I can do to be able to
> run a hbase 0.90.x on hadoop 1.1.1?
>
> thanks,
>
> Kaveh Minooie
>
> www.plutoz.com
>