You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Juan Martin Pampliega <jp...@gmail.com> on 2011/06/03 22:18:27 UTC
ZooKeeper ClassNotFound when loading data from HBase 0.9.3 with Pig 0.8.1
I'm trying to load data from HBase and process it with Pig. I'm running Pig
and HBase en local mode and they both run fine on their own.
I'm using the following script:
REGISTER ./zookeeper-3.3.3.jar;
my_data = LOAD 'hbase://table' using
org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:1', '-loadKey') ;
dump my_data;
When I ran it I get the following error:
ERROR 2998: Unhandled internal error. org/apache/zookeeper/ZooKeeper
java.lang.NoClassDefFoundError: org/apache/zookeeper/ZooKeeper
I have defined PIG_CLASSPATH pointing to HBase conf dir and the lib dir
where the zookeeper jar is.
Also I have the zookeeper jar in the Pig dir and where I'm running the pig
script.
I've also tried running both Pig and HBase in pseudo-distributed mode with
Hadoop but the same error pops out.
Also, I checked the HBaseStorage code and found the error happens in the
line where it registers the guava, hbase and zookeeper jar but
the loader finds the other two classes correctly.
The same thing happens using HBase 0.9.2 and Pig 0.8.0.
Anybody knows how to make this work? Any pointers would be greatly
appreciated.
Thanks!
Re: ZooKeeper ClassNotFound when loading data from HBase 0.9.3 with
Pig 0.8.1
Posted by Juan Martin Pampliega <jp...@gmail.com>.
I had finally got it working by compiling with "ant jar-withouthadoop" and
then adding the zookeeper jar to the classpath. As you said I had only
included the directory where it was and not the actual jar.
Thanks for the help!
On Wed, Jun 8, 2011 at 7:00 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> Looking at the values in classpath, the zookeeper jar is not listed.
> It should be bundled into
> pig-0.8.1-SNAPSHOT.jar (you can check by running `jar tf
> pig-0.8.1-SNAPSHOT.jar | grep Zoo`), but if not, you should be able to
> get this to work by ensuring that the zookeeper jar (not just the
> directory that contains it!) is on PIG_CLASSPATH
>
> D
>
> On Sun, Jun 5, 2011 at 6:47 PM, Juan Martin Pampliega
> <jp...@gmail.com> wrote:
> > Thanks for your help Dmitriy.
> >
> > When I run with the -secretDebugCmd I get the following output:
> > /usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
> > -Dpig.log.dir=/home/user1/pig-0.8.1/bin/../logs -Dpig.log.file=pig.log
> > -Dpig.home.dir=/home/user1/pig-0.8.1/bin/..
> > -Dpig.root.logger=INFO,console,DRFA -classpath
> >
> /home/user1/pig-0.8.1/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/user1/pig-0.8.1/bin/../build/classes:/home/user1/pig-0.8.1/bin/../build/test/classes:/home/user1/pig-0.8.1/bin/../pig-0.8.1-core.jar:/home/user1/pig-0.8.1/bin/../build/pig-0.8.1-SNAPSHOT.jar:/home/user1/pig-0.8.1/bin/../lib/automaton.jar:/home/user1/hadoop-0.20.203.0/conf:/home/user1/pig-0.8.1:/home/user1/pig-0.8.1/build/ivy/lib:/home/user1/hbase-0.90.3/lib:/home/user1/hbase-0.90.3/conf:
> > org.apache.pig.Main -x local hbaseTest.pig
> >
> > I'm not sure what you mean about fat jar or no-hadoop jar. I'm using the
> > zookeeper-3.3.3.jar that is in the
> > directory /home/user1/pig-0.8.1/build/ivy/lib/Pig when building pig with
> > ant. I moved that jar to the $PIG_HOME directory.
> >
> > In the different directories in the classpath there are many
> zookeeper-*.jar
> > I don't know if that can cause any problem...
> >
> > On Fri, Jun 3, 2011 at 7:35 PM, Dmitriy Ryaboy <dv...@gmail.com>
> wrote:
> >
> >> It should work if the ZK jar is in your classpath. Try "pig
> >> -secretDebugCmd" to check how the classpath is set.
> >> Are you using the fat jar or the no-hadoop jar?
> >>
> >> D
> >>
> >> On Fri, Jun 3, 2011 at 1:18 PM, Juan Martin Pampliega
> >> <jp...@gmail.com> wrote:
> >> > I'm trying to load data from HBase and process it with Pig. I'm
> running
> >> Pig
> >> > and HBase en local mode and they both run fine on their own.
> >> > I'm using the following script:
> >> > REGISTER ./zookeeper-3.3.3.jar;
> >> > my_data = LOAD 'hbase://table' using
> >> > org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:1', '-loadKey') ;
> >> > dump my_data;
> >> >
> >> > When I ran it I get the following error:
> >> > ERROR 2998: Unhandled internal error. org/apache/zookeeper/ZooKeeper
> >> > java.lang.NoClassDefFoundError: org/apache/zookeeper/ZooKeeper
> >> >
> >> > I have defined PIG_CLASSPATH pointing to HBase conf dir and the lib
> dir
> >> > where the zookeeper jar is.
> >> > Also I have the zookeeper jar in the Pig dir and where I'm running the
> >> pig
> >> > script.
> >> > I've also tried running both Pig and HBase in pseudo-distributed mode
> >> with
> >> > Hadoop but the same error pops out.
> >> > Also, I checked the HBaseStorage code and found the error happens in
> the
> >> > line where it registers the guava, hbase and zookeeper jar but
> >> > the loader finds the other two classes correctly.
> >> >
> >> > The same thing happens using HBase 0.9.2 and Pig 0.8.0.
> >> >
> >> > Anybody knows how to make this work? Any pointers would be greatly
> >> > appreciated.
> >> >
> >> > Thanks!
> >> >
> >>
> >
>
Re: ZooKeeper ClassNotFound when loading data from HBase 0.9.3 with
Pig 0.8.1
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Looking at the values in classpath, the zookeeper jar is not listed.
It should be bundled into
pig-0.8.1-SNAPSHOT.jar (you can check by running `jar tf
pig-0.8.1-SNAPSHOT.jar | grep Zoo`), but if not, you should be able to
get this to work by ensuring that the zookeeper jar (not just the
directory that contains it!) is on PIG_CLASSPATH
D
On Sun, Jun 5, 2011 at 6:47 PM, Juan Martin Pampliega
<jp...@gmail.com> wrote:
> Thanks for your help Dmitriy.
>
> When I run with the -secretDebugCmd I get the following output:
> /usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
> -Dpig.log.dir=/home/user1/pig-0.8.1/bin/../logs -Dpig.log.file=pig.log
> -Dpig.home.dir=/home/user1/pig-0.8.1/bin/..
> -Dpig.root.logger=INFO,console,DRFA -classpath
> /home/user1/pig-0.8.1/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/user1/pig-0.8.1/bin/../build/classes:/home/user1/pig-0.8.1/bin/../build/test/classes:/home/user1/pig-0.8.1/bin/../pig-0.8.1-core.jar:/home/user1/pig-0.8.1/bin/../build/pig-0.8.1-SNAPSHOT.jar:/home/user1/pig-0.8.1/bin/../lib/automaton.jar:/home/user1/hadoop-0.20.203.0/conf:/home/user1/pig-0.8.1:/home/user1/pig-0.8.1/build/ivy/lib:/home/user1/hbase-0.90.3/lib:/home/user1/hbase-0.90.3/conf:
> org.apache.pig.Main -x local hbaseTest.pig
>
> I'm not sure what you mean about fat jar or no-hadoop jar. I'm using the
> zookeeper-3.3.3.jar that is in the
> directory /home/user1/pig-0.8.1/build/ivy/lib/Pig when building pig with
> ant. I moved that jar to the $PIG_HOME directory.
>
> In the different directories in the classpath there are many zookeeper-*.jar
> I don't know if that can cause any problem...
>
> On Fri, Jun 3, 2011 at 7:35 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>
>> It should work if the ZK jar is in your classpath. Try "pig
>> -secretDebugCmd" to check how the classpath is set.
>> Are you using the fat jar or the no-hadoop jar?
>>
>> D
>>
>> On Fri, Jun 3, 2011 at 1:18 PM, Juan Martin Pampliega
>> <jp...@gmail.com> wrote:
>> > I'm trying to load data from HBase and process it with Pig. I'm running
>> Pig
>> > and HBase en local mode and they both run fine on their own.
>> > I'm using the following script:
>> > REGISTER ./zookeeper-3.3.3.jar;
>> > my_data = LOAD 'hbase://table' using
>> > org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:1', '-loadKey') ;
>> > dump my_data;
>> >
>> > When I ran it I get the following error:
>> > ERROR 2998: Unhandled internal error. org/apache/zookeeper/ZooKeeper
>> > java.lang.NoClassDefFoundError: org/apache/zookeeper/ZooKeeper
>> >
>> > I have defined PIG_CLASSPATH pointing to HBase conf dir and the lib dir
>> > where the zookeeper jar is.
>> > Also I have the zookeeper jar in the Pig dir and where I'm running the
>> pig
>> > script.
>> > I've also tried running both Pig and HBase in pseudo-distributed mode
>> with
>> > Hadoop but the same error pops out.
>> > Also, I checked the HBaseStorage code and found the error happens in the
>> > line where it registers the guava, hbase and zookeeper jar but
>> > the loader finds the other two classes correctly.
>> >
>> > The same thing happens using HBase 0.9.2 and Pig 0.8.0.
>> >
>> > Anybody knows how to make this work? Any pointers would be greatly
>> > appreciated.
>> >
>> > Thanks!
>> >
>>
>
Re: ZooKeeper ClassNotFound when loading data from HBase 0.9.3 with
Pig 0.8.1
Posted by Juan Martin Pampliega <jp...@gmail.com>.
Thanks for your help Dmitriy.
When I run with the -secretDebugCmd I get the following output:
/usr/lib/jvm/java-6-sun/bin/java -Xmx1000m
-Dpig.log.dir=/home/user1/pig-0.8.1/bin/../logs -Dpig.log.file=pig.log
-Dpig.home.dir=/home/user1/pig-0.8.1/bin/..
-Dpig.root.logger=INFO,console,DRFA -classpath
/home/user1/pig-0.8.1/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/user1/pig-0.8.1/bin/../build/classes:/home/user1/pig-0.8.1/bin/../build/test/classes:/home/user1/pig-0.8.1/bin/../pig-0.8.1-core.jar:/home/user1/pig-0.8.1/bin/../build/pig-0.8.1-SNAPSHOT.jar:/home/user1/pig-0.8.1/bin/../lib/automaton.jar:/home/user1/hadoop-0.20.203.0/conf:/home/user1/pig-0.8.1:/home/user1/pig-0.8.1/build/ivy/lib:/home/user1/hbase-0.90.3/lib:/home/user1/hbase-0.90.3/conf:
org.apache.pig.Main -x local hbaseTest.pig
I'm not sure what you mean about fat jar or no-hadoop jar. I'm using the
zookeeper-3.3.3.jar that is in the
directory /home/user1/pig-0.8.1/build/ivy/lib/Pig when building pig with
ant. I moved that jar to the $PIG_HOME directory.
In the different directories in the classpath there are many zookeeper-*.jar
I don't know if that can cause any problem...
On Fri, Jun 3, 2011 at 7:35 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> It should work if the ZK jar is in your classpath. Try "pig
> -secretDebugCmd" to check how the classpath is set.
> Are you using the fat jar or the no-hadoop jar?
>
> D
>
> On Fri, Jun 3, 2011 at 1:18 PM, Juan Martin Pampliega
> <jp...@gmail.com> wrote:
> > I'm trying to load data from HBase and process it with Pig. I'm running
> Pig
> > and HBase en local mode and they both run fine on their own.
> > I'm using the following script:
> > REGISTER ./zookeeper-3.3.3.jar;
> > my_data = LOAD 'hbase://table' using
> > org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:1', '-loadKey') ;
> > dump my_data;
> >
> > When I ran it I get the following error:
> > ERROR 2998: Unhandled internal error. org/apache/zookeeper/ZooKeeper
> > java.lang.NoClassDefFoundError: org/apache/zookeeper/ZooKeeper
> >
> > I have defined PIG_CLASSPATH pointing to HBase conf dir and the lib dir
> > where the zookeeper jar is.
> > Also I have the zookeeper jar in the Pig dir and where I'm running the
> pig
> > script.
> > I've also tried running both Pig and HBase in pseudo-distributed mode
> with
> > Hadoop but the same error pops out.
> > Also, I checked the HBaseStorage code and found the error happens in the
> > line where it registers the guava, hbase and zookeeper jar but
> > the loader finds the other two classes correctly.
> >
> > The same thing happens using HBase 0.9.2 and Pig 0.8.0.
> >
> > Anybody knows how to make this work? Any pointers would be greatly
> > appreciated.
> >
> > Thanks!
> >
>
Re: ZooKeeper ClassNotFound when loading data from HBase 0.9.3 with
Pig 0.8.1
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
It should work if the ZK jar is in your classpath. Try "pig
-secretDebugCmd" to check how the classpath is set.
Are you using the fat jar or the no-hadoop jar?
D
On Fri, Jun 3, 2011 at 1:18 PM, Juan Martin Pampliega
<jp...@gmail.com> wrote:
> I'm trying to load data from HBase and process it with Pig. I'm running Pig
> and HBase en local mode and they both run fine on their own.
> I'm using the following script:
> REGISTER ./zookeeper-3.3.3.jar;
> my_data = LOAD 'hbase://table' using
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:1', '-loadKey') ;
> dump my_data;
>
> When I ran it I get the following error:
> ERROR 2998: Unhandled internal error. org/apache/zookeeper/ZooKeeper
> java.lang.NoClassDefFoundError: org/apache/zookeeper/ZooKeeper
>
> I have defined PIG_CLASSPATH pointing to HBase conf dir and the lib dir
> where the zookeeper jar is.
> Also I have the zookeeper jar in the Pig dir and where I'm running the pig
> script.
> I've also tried running both Pig and HBase in pseudo-distributed mode with
> Hadoop but the same error pops out.
> Also, I checked the HBaseStorage code and found the error happens in the
> line where it registers the guava, hbase and zookeeper jar but
> the loader finds the other two classes correctly.
>
> The same thing happens using HBase 0.9.2 and Pig 0.8.0.
>
> Anybody knows how to make this work? Any pointers would be greatly
> appreciated.
>
> Thanks!
>