You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by praveenesh kumar <pr...@gmail.com> on 2011/07/05 10:45:02 UTC
Zookeeper exception while trying to access HBase table using pig.
Hi,
I have a latest pig build from trunk. I have configured it to run on 12-node
hadoop cluster.
I am trying to access a hbase table, the map job is running fine for
sometime...but after some time.. I am getting zookeeper exception
hadoop@ub13:/usr/local/hadoop/hbase/conf$ pig
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.conf.Configuration).
log4j:WARN Please initialize the log4j system properly.
2011-07-05 12:40:23,737 [main] INFO org.apache.pig.Main - Logging error
messages to: /usr/local/hadoop/hbase/conf/pig_1309849823721.log
2011-07-05 12:40:24,047 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: hdfs://ub13:54310
2011-07-05 12:40:24,241 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to map-reduce job tracker at: ub13:54311
grunt> ht = load 'hbase://tweetinfo' using
org.apache.pig.backend.hadoop.hbase.HBaseStorage('prim_tweetinfo:name',
'-loadKey') as (id,name);
grunt> dump ht | HEAD 100;
2011-07-05 12:40:38,234 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script:
UNKNOWN
2011-07-05 12:40:38,481 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2011-07-05 12:40:38,513 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2011-07-05 12:40:38,514 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2011-07-05 12:40:38,559 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to
the job
2011-07-05 12:40:38,572 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:host.name=ub13
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.version=1.6.0_20
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.vendor=Sun Microsystems Inc.
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.class.path=/usr/local/hadoop/pig-0.10/bin/../conf:/usr/lib/jvm/java-6-openjdk/lib/tools.jar:/usr/local/hadoop/pig-0.10/bin/../build/classes:/usr/local/hadoop/pig-0.10/bin/../build/test/classes:/usr/local/hadoop/pig-0.10/bin/../pig-*-core.jar:/usr/local/hadoop/pig-0.10/bin/../build/pig-0.10.0-SNAPSHOT.jar:/usr/local/hadoop/pig-0.10/bin/../lib/automaton.jar:/usr/local/hadoop/hadoop/hadoop-0.20-append-for-hbase-core.jar:/usr/local/hadoop/hbase/hbase-0.90.3.jar:/usr/local/hadoop/hbase/hbase-0.90.3-tests.jar:/usr/local/hadoop/hbase/lib/zookeeper-3.3.2.jar:/usr/local/hadoop/hadoop:/usr/local/hadoop/hadoop/conf
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/server:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.io.tmpdir=/tmp
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.compiler=<NA>
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.name=Linux
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.arch=i386
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.version=2.6.35-22-generic-pae
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.name=hadoop
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.home=/home/hadoop
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.dir=/usr/local/hadoop/hbase/conf
2011-07-05 12:40:38,739 [main] INFO org.apache.zookeeper.ZooKeeper -
Initiating client connection, connectString=localhost:2181
sessionTimeout=180000 watcher=hconnection
2011-07-05 12:40:38,791 [main-SendThread()] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
localhost/127.0.0.1:2181
2011-07-05 12:40:38,814 [main-SendThread(localhost:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
localhost/127.0.0.1:2181, initiating session
2011-07-05 12:40:38,833 [main-SendThread(localhost:2181)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c0009, negotiated timeout =
180000
2011-07-05 12:40:43,484 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2011-07-05 12:40:43,529 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2011-07-05 12:40:44,030 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2011-07-05 12:40:44,499 [Thread-13] INFO org.apache.zookeeper.ZooKeeper -
Initiating client connection, connectString=localhost:2181
sessionTimeout=180000 watcher=hconnection
2011-07-05 12:40:44,500 [Thread-13-SendThread()] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
localhost/127.0.0.1:2181
2011-07-05 12:40:44,501 [Thread-13-SendThread(localhost:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
localhost/127.0.0.1:2181, initiating session
2011-07-05 12:40:44,512 [Thread-13-SendThread(localhost:2181)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c000a, negotiated timeout =
180000
2011-07-05 12:40:44,554 [Thread-13] INFO org.apache.zookeeper.ZooKeeper -
Initiating client connection, connectString=localhost:2181
sessionTimeout=180000 watcher=hconnection
2011-07-05 12:40:44,556 [Thread-13-SendThread()] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
localhost/127.0.0.1:2181
2011-07-05 12:40:44,558 [Thread-13-SendThread(localhost:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
localhost/127.0.0.1:2181, initiating session
2011-07-05 12:40:44,569 [Thread-13-SendThread(localhost:2181)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c000b, negotiated timeout =
180000
2011-07-05 12:40:45,964 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201107051228_0001
2011-07-05 12:40:45,964 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://ub13:50030/jobdetails.jsp?jobid=job_201107051228_0001
2011-07-05 12:41:04,624 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1% complete
2011-07-05 12:41:07,638 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 3% complete
2011-07-05 12:41:08,141 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 4% complete
2011-07-05 12:41:10,652 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 7% complete
2011-07-05 12:41:11,155 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 9% complete
2011-07-05 12:41:16,681 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 12% complete
2011-07-05 12:41:17,183 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 13% complete
2011-07-05 12:41:19,933 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 15% complete
2011-07-05 12:41:22,444 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 18% complete
2011-07-05 12:41:22,947 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 19% complete
2011-07-05 12:41:26,463 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 21% complete
2011-07-05 12:41:28,473 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 24% complete
2011-07-05 12:41:29,478 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 25% complete
2011-07-05 12:41:31,488 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 27% complete
2011-07-05 12:41:37,516 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 28% complete
2011-07-05 12:41:43,544 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 30% complete
2011-07-05 12:41:56,124 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 31% complete
2011-07-05 12:41:58,635 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-07-05 12:41:59,140 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 34% complete
2011-07-05 12:42:02,155 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 36% complete
2011-07-05 12:42:10,700 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job job_201107051228_0001 has failed! Stop running all dependent jobs
2011-07-05 12:42:10,701 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2011-07-05 12:42:10,748 [main] ERROR
org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
recreate exception from backed error:
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could be a
sign that the server has too many connections (30 is the default). Consider
inspecting your ZK server logs for that error and then make sure you are
reusing HBaseConfiguration as often as you can. See HTable's javadoc for
more information.
2011-07-05 12:42:10,748 [main] ERROR
org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2011-07-05 12:42:10,750 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt
Features
0.20-append-for-hbase 0.10.0-SNAPSHOT hadoop 2011-07-05 12:40:38
2011-07-05 12:42:10 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
job_201107051228_0001 ht MAP_ONLY Message: Job failed!
hdfs://ub13:54310/tmp/temp-1630311766/tmp888118030,
Input(s):
Failed to read data from "hbase://tweetinfo"
Output(s):
Failed to produce result in
"hdfs://ub13:54310/tmp/temp-1630311766/tmp888118030"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_201107051228_0001
2011-07-05 12:42:10,750 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
2011-07-05 12:42:10,783 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 2997: Unable to recreate exception from backed error:
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could be a
sign that the server has too many connections (30 is the default). Consider
inspecting your ZK server logs for that error and then make sure you are
reusing HBaseConfiguration as often as you can. See HTable's javadoc for
more information.
Details at logfile: /usr/local/hadoop/hbase/conf/pig_1309849823721.log
LOG file details.....
Backend error message
---------------------
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could be a
sign that the server has too many connections (30 is the default). Consider
inspecting your ZK server logs for that error and then make sure you are
reusing HBaseConfiguration as often as you can. See HTable's javadoc for
more information.
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:990)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:303)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:294)
at
org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:168)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:146)
at
org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:457)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:588)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
... 12 more
Pig Stack Trace
---------------
ERROR 2997: Unable to recreate exception from backed error:
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could be a
sign that the server has too many connections (30 is the default). Consider
inspecting your ZK server logs for that error and then make sure you are
reusing HBaseConfiguration as often as you can. See HTable's javadoc for
more information.
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
open iterator for alias ht. Backend error : Unable to recreate exception
from backed error: org.apache.hadoop.hbase.ZooKeeperConnectionException:
HBase is able to connect to ZooKeeper but the connection closes immediately.
This could be a sign that the server has too many connections (30 is the
default). Consider inspecting your ZK server logs for that error and then
make sure you are reusing HBaseConfiguration as often as you can. See
HTable's javadoc for more information.
at org.apache.pig.PigServer.openIterator(PigServer.java:890)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:67)
at org.apache.pig.Main.run(Main.java:487)
at org.apache.pig.Main.main(Main.java:108)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2997:
Unable to recreate exception from backed error:
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could be a
sign that the server has too many connections (30 is the default). Consider
inspecting your ZK server logs for that error and then make sure you are
reusing HBaseConfiguration as often as you can. See HTable's javadoc for
more information.
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:341)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1313)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1298)
at org.apache.pig.PigServer.storeEx(PigServer.java:995)
at org.apache.pig.PigServer.store(PigServer.java:962)
at org.apache.pig.PigServer.openIterator(PigServer.java:875)
... 7 more
================================================================================
My hbase-site.xml looks like this
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://ub13:54310/hbase</value>
<final>true</final>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>ub13,ub12,ub11</value>
<final>true</final>
</property>
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>200</value>
<final>true</final>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum
</description>
</property>
Zookeeper log -- http://pastebin.com/RRdzZYui
I have increased the maximum number of zookeeper connections.. but still I
am getting the same error.. I am not able to get enough out of zookeeper
logs also.. its just saying.. end of stream exception..socket ended..
something like that..
The table I am trying to access consists of around 21 lacs record..
If I try to access small tables.. I am able to access it..
What can be the issue ?
Thanks,
Praveenesh
Re: Zookeeper exception while trying to access HBase table using pig.
Posted by praveenesh kumar <pr...@gmail.com>.
I am using HBase 0.90.3
On Wed, Jul 6, 2011 at 2:45 AM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> What version of HBase are you running?
>
>
> On Tue, Jul 5, 2011 at 1:45 AM, praveenesh kumar <pr...@gmail.com>
> wrote:
> > Hi,
> > I have a latest pig build from trunk. I have configured it to run on
> 12-node
> > hadoop cluster.
> > I am trying to access a hbase table, the map job is running fine for
> > sometime...but after some time.. I am getting zookeeper exception
> >
> > hadoop@ub13:/usr/local/hadoop/hbase/conf$ pig
> > log4j:WARN No appenders could be found for logger
> > (org.apache.hadoop.conf.Configuration).
> > log4j:WARN Please initialize the log4j system properly.
> > 2011-07-05 12:40:23,737 [main] INFO org.apache.pig.Main - Logging error
> > messages to: /usr/local/hadoop/hbase/conf/pig_1309849823721.log
> > 2011-07-05 12:40:24,047 [main] INFO
> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> Connecting
> > to hadoop file system at: hdfs://ub13:54310
> > 2011-07-05 12:40:24,241 [main] INFO
> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> Connecting
> > to map-reduce job tracker at: ub13:54311
> >
> > grunt> ht = load 'hbase://tweetinfo' using
> > org.apache.pig.backend.hadoop.hbase.HBaseStorage('prim_tweetinfo:name',
> > '-loadKey') as (id,name);
> > grunt> dump ht | HEAD 100;
> >
> > 2011-07-05 12:40:38,234 [main] INFO
> > org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script:
> > UNKNOWN
> > 2011-07-05 12:40:38,481 [main] INFO
> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> > File concatenation threshold: 100 optimistic? false
> > 2011-07-05 12:40:38,513 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> > - MR plan size before optimization: 1
> > 2011-07-05 12:40:38,514 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> > - MR plan size after optimization: 1
> > 2011-07-05 12:40:38,559 [main] INFO
> > org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
> to
> > the job
> > 2011-07-05 12:40:38,572 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> > - mapred.job.reduce.markreset.buffer.percent is not set, set to default
> 0.3
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> > environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32
> GMT
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> > environment:host.name=ub13
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> > environment:java.version=1.6.0_20
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> > environment:java.vendor=Sun Microsystems Inc.
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> > environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> >
> environment:java.class.path=/usr/local/hadoop/pig-0.10/bin/../conf:/usr/lib/jvm/java-6-openjdk/lib/tools.jar:/usr/local/hadoop/pig-0.10/bin/../build/classes:/usr/local/hadoop/pig-0.10/bin/../build/test/classes:/usr/local/hadoop/pig-0.10/bin/../pig-*-core.jar:/usr/local/hadoop/pig-0.10/bin/../build/pig-0.10.0-SNAPSHOT.jar:/usr/local/hadoop/pig-0.10/bin/../lib/automaton.jar:/usr/local/hadoop/hadoop/hadoop-0.20-append-for-hbase-core.jar:/usr/local/hadoop/hbase/hbase-0.90.3.jar:/usr/local/hadoop/hbase/hbase-0.90.3-tests.jar:/usr/local/hadoop/hbase/lib/zookeeper-3.3.2.jar:/usr/local/hadoop/hadoop:/usr/local/hadoop/hadoop/conf
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> >
> environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/server:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> > environment:java.io.tmpdir=/tmp
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> > environment:java.compiler=<NA>
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> > environment:os.name=Linux
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> > environment:os.arch=i386
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> > environment:os.version=2.6.35-22-generic-pae
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> > environment:user.name=hadoop
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> > environment:user.home=/home/hadoop
> > 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper -
> Client
> > environment:user.dir=/usr/local/hadoop/hbase/conf
> > 2011-07-05 12:40:38,739 [main] INFO org.apache.zookeeper.ZooKeeper -
> > Initiating client connection, connectString=localhost:2181
> > sessionTimeout=180000 watcher=hconnection
> > 2011-07-05 12:40:38,791 [main-SendThread()] INFO
> > org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> > localhost/127.0.0.1:2181
> > 2011-07-05 12:40:38,814 [main-SendThread(localhost:2181)] INFO
> > org.apache.zookeeper.ClientCnxn - Socket connection established to
> > localhost/127.0.0.1:2181, initiating session
> > 2011-07-05 12:40:38,833 [main-SendThread(localhost:2181)] INFO
> > org.apache.zookeeper.ClientCnxn - Session establishment complete on
> server
> > localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c0009, negotiated
> timeout =
> > 180000
> > 2011-07-05 12:40:43,484 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> > - Setting up single store job
> > 2011-07-05 12:40:43,529 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 1 map-reduce job(s) waiting for submission.
> > 2011-07-05 12:40:44,030 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 0% complete
> > 2011-07-05 12:40:44,499 [Thread-13] INFO org.apache.zookeeper.ZooKeeper
> -
> > Initiating client connection, connectString=localhost:2181
> > sessionTimeout=180000 watcher=hconnection
> > 2011-07-05 12:40:44,500 [Thread-13-SendThread()] INFO
> > org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> > localhost/127.0.0.1:2181
> > 2011-07-05 12:40:44,501 [Thread-13-SendThread(localhost:2181)] INFO
> > org.apache.zookeeper.ClientCnxn - Socket connection established to
> > localhost/127.0.0.1:2181, initiating session
> > 2011-07-05 12:40:44,512 [Thread-13-SendThread(localhost:2181)] INFO
> > org.apache.zookeeper.ClientCnxn - Session establishment complete on
> server
> > localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c000a, negotiated
> timeout =
> > 180000
> > 2011-07-05 12:40:44,554 [Thread-13] INFO org.apache.zookeeper.ZooKeeper
> -
> > Initiating client connection, connectString=localhost:2181
> > sessionTimeout=180000 watcher=hconnection
> > 2011-07-05 12:40:44,556 [Thread-13-SendThread()] INFO
> > org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> > localhost/127.0.0.1:2181
> > 2011-07-05 12:40:44,558 [Thread-13-SendThread(localhost:2181)] INFO
> > org.apache.zookeeper.ClientCnxn - Socket connection established to
> > localhost/127.0.0.1:2181, initiating session
> > 2011-07-05 12:40:44,569 [Thread-13-SendThread(localhost:2181)] INFO
> > org.apache.zookeeper.ClientCnxn - Session establishment complete on
> server
> > localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c000b, negotiated
> timeout =
> > 180000
> > 2011-07-05 12:40:45,964 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - HadoopJobId: job_201107051228_0001
> > 2011-07-05 12:40:45,964 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - More information at:
> > http://ub13:50030/jobdetails.jsp?jobid=job_201107051228_0001
> > 2011-07-05 12:41:04,624 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 1% complete
> > 2011-07-05 12:41:07,638 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 3% complete
> > 2011-07-05 12:41:08,141 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 4% complete
> > 2011-07-05 12:41:10,652 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 7% complete
> > 2011-07-05 12:41:11,155 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 9% complete
> > 2011-07-05 12:41:16,681 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 12% complete
> > 2011-07-05 12:41:17,183 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 13% complete
> > 2011-07-05 12:41:19,933 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 15% complete
> > 2011-07-05 12:41:22,444 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 18% complete
> > 2011-07-05 12:41:22,947 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 19% complete
> > 2011-07-05 12:41:26,463 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 21% complete
> > 2011-07-05 12:41:28,473 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 24% complete
> > 2011-07-05 12:41:29,478 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 25% complete
> > 2011-07-05 12:41:31,488 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 27% complete
> > 2011-07-05 12:41:37,516 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 28% complete
> > 2011-07-05 12:41:43,544 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 30% complete
> > 2011-07-05 12:41:56,124 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 31% complete
> > 2011-07-05 12:41:58,635 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 33% complete
> > 2011-07-05 12:41:59,140 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 34% complete
> > 2011-07-05 12:42:02,155 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 36% complete
> > 2011-07-05 12:42:10,700 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - job job_201107051228_0001 has failed! Stop running all dependent jobs
> > 2011-07-05 12:42:10,701 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 100% complete
> > 2011-07-05 12:42:10,748 [main] ERROR
> > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
> > recreate exception from backed error:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> > connect to ZooKeeper but the connection closes immediately. This could be
> a
> > sign that the server has too many connections (30 is the default).
> Consider
> > inspecting your ZK server logs for that error and then make sure you are
> > reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> > more information.
> > 2011-07-05 12:42:10,748 [main] ERROR
> > org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> > 2011-07-05 12:42:10,750 [main] INFO
> > org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
> >
> > HadoopVersion PigVersion UserId StartedAt FinishedAt
> > Features
> > 0.20-append-for-hbase 0.10.0-SNAPSHOT hadoop 2011-07-05 12:40:38
> > 2011-07-05 12:42:10 UNKNOWN
> >
> > Failed!
> >
> > Failed Jobs:
> > JobId Alias Feature Message Outputs
> > job_201107051228_0001 ht MAP_ONLY Message: Job failed!
> > hdfs://ub13:54310/tmp/temp-1630311766/tmp888118030,
> >
> > Input(s):
> > Failed to read data from "hbase://tweetinfo"
> >
> > Output(s):
> > Failed to produce result in
> > "hdfs://ub13:54310/tmp/temp-1630311766/tmp888118030"
> >
> > Counters:
> > Total records written : 0
> > Total bytes written : 0
> > Spillable Memory Manager spill count : 0
> > Total bags proactively spilled: 0
> > Total records proactively spilled: 0
> >
> > Job DAG:
> > job_201107051228_0001
> >
> >
> > 2011-07-05 12:42:10,750 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - Failed!
> > 2011-07-05 12:42:10,783 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> > ERROR 2997: Unable to recreate exception from backed error:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> > connect to ZooKeeper but the connection closes immediately. This could be
> a
> > sign that the server has too many connections (30 is the default).
> Consider
> > inspecting your ZK server logs for that error and then make sure you are
> > reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> > more information.
> > Details at logfile: /usr/local/hadoop/hbase/conf/pig_1309849823721.log
> >
> > LOG file details.....
> >
> >
> > Backend error message
> > ---------------------
> > org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> > connect to ZooKeeper but the connection closes immediately. This could be
> a
> > sign that the server has too many connections (30 is the default).
> Consider
> > inspecting your ZK server logs for that error and then make sure you are
> > reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> > more information.
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:990)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:303)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:294)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:168)
> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:146)
> > at
> >
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:457)
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:588)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase
> > at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> > at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
> > ... 12 more
> >
> > Pig Stack Trace
> > ---------------
> > ERROR 2997: Unable to recreate exception from backed error:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> > connect to ZooKeeper but the connection closes immediately. This could be
> a
> > sign that the server has too many connections (30 is the default).
> Consider
> > inspecting your ZK server logs for that error and then make sure you are
> > reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> > more information.
> >
> > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> > open iterator for alias ht. Backend error : Unable to recreate exception
> > from backed error: org.apache.hadoop.hbase.ZooKeeperConnectionException:
> > HBase is able to connect to ZooKeeper but the connection closes
> immediately.
> > This could be a sign that the server has too many connections (30 is the
> > default). Consider inspecting your ZK server logs for that error and then
> > make sure you are reusing HBaseConfiguration as often as you can. See
> > HTable's javadoc for more information.
> > at org.apache.pig.PigServer.openIterator(PigServer.java:890)
> > at
> > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
> > at
> >
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> > at
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> > at
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:67)
> > at org.apache.pig.Main.run(Main.java:487)
> > at org.apache.pig.Main.main(Main.java:108)
> > Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
> 2997:
> > Unable to recreate exception from backed error:
> > org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> > connect to ZooKeeper but the connection closes immediately. This could be
> a
> > sign that the server has too many connections (30 is the default).
> Consider
> > inspecting your ZK server logs for that error and then make sure you are
> > reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> > more information.
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:341)
> > at org.apache.pig.PigServer.launchPlan(PigServer.java:1313)
> > at
> > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1298)
> > at org.apache.pig.PigServer.storeEx(PigServer.java:995)
> > at org.apache.pig.PigServer.store(PigServer.java:962)
> > at org.apache.pig.PigServer.openIterator(PigServer.java:875)
> > ... 7 more
> >
> ================================================================================
> >
> >
> > My hbase-site.xml looks like this
> >
> > <configuration>
> > <property>
> > <name>hbase.rootdir</name>
> > <value>hdfs://ub13:54310/hbase</value>
> > <final>true</final>
> > </property>
> >
> >
> > <property>
> > <name>hbase.zookeeper.quorum</name>
> > <value>ub13,ub12,ub11</value>
> > <final>true</final>
> > </property>
> >
> > <property>
> > <name>hbase.zookeeper.property.maxClientCnxns</name>
> > <value>200</value>
> > <final>true</final>
> > </property>
> >
> > <property>
> > <name>hbase.cluster.distributed</name>
> > <value>true</value>
> > <description>The mode the cluster will be in. Possible values are
> > false: standalone and pseudo-distributed setups with managed Zookeeper
> > true: fully-distributed with unmanaged Zookeeper Quorum
> > </description>
> > </property>
> >
> >
> > Zookeeper log -- http://pastebin.com/RRdzZYui
> >
> > I have increased the maximum number of zookeeper connections.. but still
> I
> > am getting the same error.. I am not able to get enough out of zookeeper
> > logs also.. its just saying.. end of stream exception..socket ended..
> > something like that..
> > The table I am trying to access consists of around 21 lacs record..
> > If I try to access small tables.. I am able to access it..
> >
> > What can be the issue ?
> >
> > Thanks,
> > Praveenesh
> >
>
Re: Zookeeper exception while trying to access HBase table using pig.
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
What version of HBase are you running?
On Tue, Jul 5, 2011 at 1:45 AM, praveenesh kumar <pr...@gmail.com> wrote:
> Hi,
> I have a latest pig build from trunk. I have configured it to run on 12-node
> hadoop cluster.
> I am trying to access a hbase table, the map job is running fine for
> sometime...but after some time.. I am getting zookeeper exception
>
> hadoop@ub13:/usr/local/hadoop/hbase/conf$ pig
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.conf.Configuration).
> log4j:WARN Please initialize the log4j system properly.
> 2011-07-05 12:40:23,737 [main] INFO org.apache.pig.Main - Logging error
> messages to: /usr/local/hadoop/hbase/conf/pig_1309849823721.log
> 2011-07-05 12:40:24,047 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to hadoop file system at: hdfs://ub13:54310
> 2011-07-05 12:40:24,241 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to map-reduce job tracker at: ub13:54311
>
> grunt> ht = load 'hbase://tweetinfo' using
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('prim_tweetinfo:name',
> '-loadKey') as (id,name);
> grunt> dump ht | HEAD 100;
>
> 2011-07-05 12:40:38,234 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script:
> UNKNOWN
> 2011-07-05 12:40:38,481 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
> 2011-07-05 12:40:38,513 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2011-07-05 12:40:38,514 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2011-07-05 12:40:38,559 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to
> the job
> 2011-07-05 12:40:38,572 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:host.name=ub13
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.version=1.6.0_20
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.vendor=Sun Microsystems Inc.
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.class.path=/usr/local/hadoop/pig-0.10/bin/../conf:/usr/lib/jvm/java-6-openjdk/lib/tools.jar:/usr/local/hadoop/pig-0.10/bin/../build/classes:/usr/local/hadoop/pig-0.10/bin/../build/test/classes:/usr/local/hadoop/pig-0.10/bin/../pig-*-core.jar:/usr/local/hadoop/pig-0.10/bin/../build/pig-0.10.0-SNAPSHOT.jar:/usr/local/hadoop/pig-0.10/bin/../lib/automaton.jar:/usr/local/hadoop/hadoop/hadoop-0.20-append-for-hbase-core.jar:/usr/local/hadoop/hbase/hbase-0.90.3.jar:/usr/local/hadoop/hbase/hbase-0.90.3-tests.jar:/usr/local/hadoop/hbase/lib/zookeeper-3.3.2.jar:/usr/local/hadoop/hadoop:/usr/local/hadoop/hadoop/conf
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/server:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.io.tmpdir=/tmp
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:java.compiler=<NA>
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.name=Linux
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.arch=i386
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:os.version=2.6.35-22-generic-pae
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:user.name=hadoop
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:user.home=/home/hadoop
> 2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
> environment:user.dir=/usr/local/hadoop/hbase/conf
> 2011-07-05 12:40:38,739 [main] INFO org.apache.zookeeper.ZooKeeper -
> Initiating client connection, connectString=localhost:2181
> sessionTimeout=180000 watcher=hconnection
> 2011-07-05 12:40:38,791 [main-SendThread()] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> localhost/127.0.0.1:2181
> 2011-07-05 12:40:38,814 [main-SendThread(localhost:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> localhost/127.0.0.1:2181, initiating session
> 2011-07-05 12:40:38,833 [main-SendThread(localhost:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Session establishment complete on server
> localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c0009, negotiated timeout =
> 180000
> 2011-07-05 12:40:43,484 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2011-07-05 12:40:43,529 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map-reduce job(s) waiting for submission.
> 2011-07-05 12:40:44,030 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
> 2011-07-05 12:40:44,499 [Thread-13] INFO org.apache.zookeeper.ZooKeeper -
> Initiating client connection, connectString=localhost:2181
> sessionTimeout=180000 watcher=hconnection
> 2011-07-05 12:40:44,500 [Thread-13-SendThread()] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> localhost/127.0.0.1:2181
> 2011-07-05 12:40:44,501 [Thread-13-SendThread(localhost:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> localhost/127.0.0.1:2181, initiating session
> 2011-07-05 12:40:44,512 [Thread-13-SendThread(localhost:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Session establishment complete on server
> localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c000a, negotiated timeout =
> 180000
> 2011-07-05 12:40:44,554 [Thread-13] INFO org.apache.zookeeper.ZooKeeper -
> Initiating client connection, connectString=localhost:2181
> sessionTimeout=180000 watcher=hconnection
> 2011-07-05 12:40:44,556 [Thread-13-SendThread()] INFO
> org.apache.zookeeper.ClientCnxn - Opening socket connection to server
> localhost/127.0.0.1:2181
> 2011-07-05 12:40:44,558 [Thread-13-SendThread(localhost:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Socket connection established to
> localhost/127.0.0.1:2181, initiating session
> 2011-07-05 12:40:44,569 [Thread-13-SendThread(localhost:2181)] INFO
> org.apache.zookeeper.ClientCnxn - Session establishment complete on server
> localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c000b, negotiated timeout =
> 180000
> 2011-07-05 12:40:45,964 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - HadoopJobId: job_201107051228_0001
> 2011-07-05 12:40:45,964 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - More information at:
> http://ub13:50030/jobdetails.jsp?jobid=job_201107051228_0001
> 2011-07-05 12:41:04,624 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1% complete
> 2011-07-05 12:41:07,638 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 3% complete
> 2011-07-05 12:41:08,141 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 4% complete
> 2011-07-05 12:41:10,652 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 7% complete
> 2011-07-05 12:41:11,155 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 9% complete
> 2011-07-05 12:41:16,681 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 12% complete
> 2011-07-05 12:41:17,183 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 13% complete
> 2011-07-05 12:41:19,933 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 15% complete
> 2011-07-05 12:41:22,444 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 18% complete
> 2011-07-05 12:41:22,947 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 19% complete
> 2011-07-05 12:41:26,463 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 21% complete
> 2011-07-05 12:41:28,473 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 24% complete
> 2011-07-05 12:41:29,478 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 25% complete
> 2011-07-05 12:41:31,488 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 27% complete
> 2011-07-05 12:41:37,516 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 28% complete
> 2011-07-05 12:41:43,544 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 30% complete
> 2011-07-05 12:41:56,124 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 31% complete
> 2011-07-05 12:41:58,635 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 33% complete
> 2011-07-05 12:41:59,140 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 34% complete
> 2011-07-05 12:42:02,155 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 36% complete
> 2011-07-05 12:42:10,700 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - job job_201107051228_0001 has failed! Stop running all dependent jobs
> 2011-07-05 12:42:10,701 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 100% complete
> 2011-07-05 12:42:10,748 [main] ERROR
> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
> recreate exception from backed error:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
> 2011-07-05 12:42:10,748 [main] ERROR
> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2011-07-05 12:42:10,750 [main] INFO
> org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt
> Features
> 0.20-append-for-hbase 0.10.0-SNAPSHOT hadoop 2011-07-05 12:40:38
> 2011-07-05 12:42:10 UNKNOWN
>
> Failed!
>
> Failed Jobs:
> JobId Alias Feature Message Outputs
> job_201107051228_0001 ht MAP_ONLY Message: Job failed!
> hdfs://ub13:54310/tmp/temp-1630311766/tmp888118030,
>
> Input(s):
> Failed to read data from "hbase://tweetinfo"
>
> Output(s):
> Failed to produce result in
> "hdfs://ub13:54310/tmp/temp-1630311766/tmp888118030"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> job_201107051228_0001
>
>
> 2011-07-05 12:42:10,750 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Failed!
> 2011-07-05 12:42:10,783 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 2997: Unable to recreate exception from backed error:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
> Details at logfile: /usr/local/hadoop/hbase/conf/pig_1309849823721.log
>
> LOG file details.....
>
>
> Backend error message
> ---------------------
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:990)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:303)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:294)
> at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:168)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:146)
> at
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:457)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:588)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
> ... 12 more
>
> Pig Stack Trace
> ---------------
> ERROR 2997: Unable to recreate exception from backed error:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> open iterator for alias ht. Backend error : Unable to recreate exception
> from backed error: org.apache.hadoop.hbase.ZooKeeperConnectionException:
> HBase is able to connect to ZooKeeper but the connection closes immediately.
> This could be a sign that the server has too many connections (30 is the
> default). Consider inspecting your ZK server logs for that error and then
> make sure you are reusing HBaseConfiguration as often as you can. See
> HTable's javadoc for more information.
> at org.apache.pig.PigServer.openIterator(PigServer.java:890)
> at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:67)
> at org.apache.pig.Main.run(Main.java:487)
> at org.apache.pig.Main.main(Main.java:108)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2997:
> Unable to recreate exception from backed error:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:341)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1313)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1298)
> at org.apache.pig.PigServer.storeEx(PigServer.java:995)
> at org.apache.pig.PigServer.store(PigServer.java:962)
> at org.apache.pig.PigServer.openIterator(PigServer.java:875)
> ... 7 more
> ================================================================================
>
>
> My hbase-site.xml looks like this
>
> <configuration>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://ub13:54310/hbase</value>
> <final>true</final>
> </property>
>
>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>ub13,ub12,ub11</value>
> <final>true</final>
> </property>
>
> <property>
> <name>hbase.zookeeper.property.maxClientCnxns</name>
> <value>200</value>
> <final>true</final>
> </property>
>
> <property>
> <name>hbase.cluster.distributed</name>
> <value>true</value>
> <description>The mode the cluster will be in. Possible values are
> false: standalone and pseudo-distributed setups with managed Zookeeper
> true: fully-distributed with unmanaged Zookeeper Quorum
> </description>
> </property>
>
>
> Zookeeper log -- http://pastebin.com/RRdzZYui
>
> I have increased the maximum number of zookeeper connections.. but still I
> am getting the same error.. I am not able to get enough out of zookeeper
> logs also.. its just saying.. end of stream exception..socket ended..
> something like that..
> The table I am trying to access consists of around 21 lacs record..
> If I try to access small tables.. I am able to access it..
>
> What can be the issue ?
>
> Thanks,
> Praveenesh
>