You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Royston Sellman <ro...@googlemail.com> on 2012/02/02 21:43:39 UTC
Cannot STORE from Pig to HBase 0.9x
Hi,
I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from HBase
trunk) and following the tutorial.
This line loads the sample file from HDFS successfully:
raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS (user, time,
query);
This line seems to work:
T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time), query;
Because when I do:
DUMP T;
I get the response I expected.
But if I then do:
STORE T INTO 'excite' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query');
I get (in the log):
Pig Stack Trace
---------------
ERROR 2998: Unhandled internal error.
org/apache/hadoop/hbase/filter/WritableByteArrayComparable
java.lang.NoClassDefFoundError:
org/apache/hadoop/hbase/filter/WritableByteArrayComparable
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
at
org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder
.java:723)
at
org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.ja
va:712)
at
org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
java:4340)
at
org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator
.java:5956)
at
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.ja
va:1122)
at
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGene
rator.java:683)
at
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.ja
va:483)
at
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:3
69)
at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:171)
at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
at
org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.
java:386)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188
)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164
)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:495)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.filter.WritableByteArrayComparable
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 28 more
Grateful for any help with this.
Thanks,
Royston
RE: Cannot STORE from Pig to HBase 0.9x
Posted by Royston Sellman <ro...@googlemail.com>.
Hi Dmitriy,
Well thanks, that fixed it.
A few days ago I WAS registering the HBase jar and getting the same error.
It turns out that you also need the jar in the PIG_CLASSPATH and I didn't.
So I was never at a point where both things were simultaneously true. Many
thanks for the reminder.
I think it would be helpful if the sample Pig code in the javadoc for
HBaseStorage included something like "register etc/hbase/hbase-0.92.jar"
before the LOAD and STORE examples as this is one of the first places one
naturally looks for help on HBaseStorage.
Many of the other examples that turn up in a Google search are also missing
the "register" statement. So is the HBase book (not your problem). I only
stumbled across it by luck.
Of course, none of the above means I'm not an idiot.
Regards,
Royston
-----Original Message-----
From: Dmitriy Ryaboy [mailto:dvryaboy@gmail.com]
Sent: 06 February 2012 21:22
To: user@pig.apache.org
Subject: Re: Cannot STORE from Pig to HBase 0.9x
Did you explicitly "register" the hbase jar?
D
On Mon, Feb 6, 2012 at 3:37 AM, Royston Sellman <
royston.sellman@googlemail.com> wrote:
> Hi Praveenesh,
>
> Thanks for your suggestion but I checked and the HBase/ZK jars and
> dirs are correct in our CLASSPATH and in PIG_CLASSPATH.
>
> A few more details might help someone to point out where we are going
> wrong:
>
> We have the following script called 'hbase_sample.pig'...
>
> raw_data = LOAD 'sample_data.csv' USING PigStorage( ',' ) AS (
> listing_id: chararray,
> fname: chararray,
> lname: chararray );
>
> STORE raw_data INTO 'hbase://hello_world' USING
> org.apache.pig.backend.hadoop.hbase.HBaseStorage (
> 'info:fname info:lname');
>
> We have created the 'hello_world' table in HBase with the following
> command:
>
> create 'hello_world','info'
>
> There is an input file called 'sample_data.csv' containing the following:
>
> 1, John, Smith
> 2, Jane, Doe
> 3, George, Washington
> 4, Ben, Franklin
>
> We have stored 'sample_data.csv' in the same directory as the
> 'hbase_sample.pig' script and we have also stored a copy of the CSV
> file on HDFS under the home directory for the current user
> "hdfs://user/hadoop1/sample_data.csv".
>
> We ran the script in local mode SUCCESSFULLY with the following command:
>
> pig -x local -f hbase_sample.pig
>
> We then ran the script using the mapreduce mode (default for pig):
>
> pig -x mapreduce -f hbase_sample.pig
>
> The jobtracker shows the following error(repeatedly) in the log output
> for both map and reduce tasks:
>
> Error: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.filter.WritableByteArrayComparable
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
> at
>
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
> at
>
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperat
ors.POStore.getStoreFunc(POStore.java:232)
> at
>
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommit
ter.getCommitters(PigOutputCommitter.java:87)
> at
>
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommit
ter.<init>(PigOutputCommitter.java:69)
> at
>
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat
.getOutputCommitter(PigOutputFormat.java:279)
> at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
>
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1083)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>
> In addition, we see the following errors from pig (on the stdout):
>
> Input(s):
> Failed to read data from "hdfs://
> 10.0.0.235:8020/user/hadoop1/sample_data.csv"
>
> Output(s):
> Failed to produce result in "hbase://hello_world"
>
>
> So we have established that we can run a pig script successfully in
> local mode to store data in HBase (and we have verified that this has
> actually worked), however we cannot seem to get the same script to
> work with HBase in MR mode.
>
> Here is the complete output from pig:
>
> http://pastebin.com/vVvsCHXv
>
> Thanks,
> Royston
>
> On 3 Feb 2012, at 14:38, praveenesh kumar wrote:
>
> > Try adding Hbase-core.jar,Zookeeper.jar in Hadoop-classpath Also you
> > can try editing PIG_CLASSPATH in $PIG_HOME/bin/pig script. Just Add
> > your Hbase-core.jar and HBase_Conf dir in PiG_CLASSPATH inside pig
> > script.
> > Don't know whether these methods are the best method, but they work
> > for
> me
> > :-)
> >
> > Thanks,
> > Praveenesh
> >
> > On Fri, Feb 3, 2012 at 5:25 AM, Royston Sellman <
> > royston.sellman@googlemail.com> wrote:
> >
> >> Thanks for your reply. WritableByteArrayComparable is in the same
> >> place
> in
> >> HBase 0.93. I also registered my HBase jar from within Pig i.e
> >> register /opt/hbase/hbase-0.93.jar and that command returned
> >> without error. I
> also
> >> edited the line in the pig startup script that specifies the path
> >> to
> HBase.
> >> Is there some other config file that controls how Pig finds HBase jar?
> Is
> >> there a default location Pig searches for jars that I could copy my
> >> jar into?
> >>
> >> Cheers,
> >> Royston
> >>
> >>
> >>
> >> On 2 Feb 2012, at 23:38, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> >>
> >>> "Caused by: java.lang.ClassNotFoundException:
> >>> org.apache.hadoop.hbase.filter.WritableByteArrayComparable"
> >>> indicates
> >> that
> >>> you don't have HBase on your classpath, or that the version of
> >>> HBase
> you
> >>> are testing against moved this class someplace else. We've tested
> against
> >>> the 0.90 series, but not 0.92+... did they refactor something?
> >>>
> >>> D
> >>>
> >>> On Thu, Feb 2, 2012 at 12:43 PM, Royston Sellman <
> >>> royston.sellman@googlemail.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>>
> >>>>
> >>>> I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from
> HBase
> >>>> trunk) and following the tutorial.
> >>>>
> >>>>
> >>>>
> >>>> This line loads the sample file from HDFS successfully:
> >>>>
> >>>> raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS
> >>>> (user,
> >> time,
> >>>> query);
> >>>>
> >>>>
> >>>>
> >>>> This line seems to work:
> >>>>
> >>>> T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time),
> >>>> query;
> >>>>
> >>>>
> >>>>
> >>>> Because when I do:
> >>>>
> >>>> DUMP T;
> >>>>
> >>>> I get the response I expected.
> >>>>
> >>>>
> >>>>
> >>>> But if I then do:
> >>>>
> >>>> STORE T INTO 'excite' USING
> >>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query')
> >>>> ;
> >>>>
> >>>> I get (in the log):
> >>>>
> >>>> Pig Stack Trace
> >>>>
> >>>> ---------------
> >>>>
> >>>> ERROR 2998: Unhandled internal error.
> >>>> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
> >>>>
> >>>>
> >>>>
> >>>> java.lang.NoClassDefFoundError:
> >>>> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
> >>>>
> >>>> at java.lang.Class.forName0(Native Method)
> >>>>
> >>>> at java.lang.Class.forName(Class.java:247)
> >>>>
> >>>> at
> >>>> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:4
> >>>> 28)
> >>>>
> >>>> at
> >>>>
> >>
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java
> :458)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanB
> uilder
> >>>> .java:723)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuil
> der.ja
> >>>> va:712)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
>
org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
> >>>> java:4340)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGen
> erator
> >>>> .java:5956)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenera
> tor.ja
> >>>> va:1122)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPl
> anGene
> >>>> rator.java:683)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenera
> tor.ja
> >>>> va:483)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.
> java:3
> >>>> 69)
> >>>>
> >>>> at
> >>>>
> >>
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:1
> 71)
> >>>>
> >>>> at
> >> org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
> >>>>
> >>>> at
> >> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
> >>>>
> >>>> at
> >>>> org.apache.pig.PigServer.registerQuery(PigServer.java:584)
> >>>>
> >>>> at
> >>>>
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942
> )
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
>
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.
> >>>> java:386)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.ja
> va:188
> >>>> )
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.ja
> va:164
> >>>> )
> >>>>
> >>>> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> >>>>
> >>>> at org.apache.pig.Main.run(Main.java:495)
> >>>>
> >>>> at org.apache.pig.Main.main(Main.java:111)
> >>>>
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:39
> >>>> )
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl
> >>>> .java:25)
> >>>>
> >>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>
> >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >>>>
> >>>> Caused by: java.lang.ClassNotFoundException:
> >>>> org.apache.hadoop.hbase.filter.WritableByteArrayComparable
> >>>>
> >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>>>
> >>>> at java.security.AccessController.doPrivileged(Native
> >>>> Method)
> >>>>
> >>>> at
> >>>> java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>>>
> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>>>
> >>>> at
> >>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>>>
> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>>>
> >>>> ... 28 more
> >>>>
> >>>>
> >>>>
> >>>> Grateful for any help with this.
> >>>>
> >>>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Royston
> >>>>
> >>>>
> >>
>
>
Re: Cannot STORE from Pig to HBase 0.9x
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Did you explicitly "register" the hbase jar?
D
On Mon, Feb 6, 2012 at 3:37 AM, Royston Sellman <
royston.sellman@googlemail.com> wrote:
> Hi Praveenesh,
>
> Thanks for your suggestion but I checked and the HBase/ZK jars and dirs
> are correct in our CLASSPATH and in PIG_CLASSPATH.
>
> A few more details might help someone to point out where we are going
> wrong:
>
> We have the following script called 'hbase_sample.pig'...
>
> raw_data = LOAD 'sample_data.csv' USING PigStorage( ',' ) AS (
> listing_id: chararray,
> fname: chararray,
> lname: chararray );
>
> STORE raw_data INTO 'hbase://hello_world' USING
> org.apache.pig.backend.hadoop.hbase.HBaseStorage (
> 'info:fname info:lname');
>
> We have created the 'hello_world' table in HBase with the following
> command:
>
> create 'hello_world','info'
>
> There is an input file called 'sample_data.csv' containing the following:
>
> 1, John, Smith
> 2, Jane, Doe
> 3, George, Washington
> 4, Ben, Franklin
>
> We have stored 'sample_data.csv' in the same directory as the
> 'hbase_sample.pig' script and we have also stored a copy of the CSV file on
> HDFS under the home directory for the current user
> "hdfs://user/hadoop1/sample_data.csv".
>
> We ran the script in local mode SUCCESSFULLY with the following command:
>
> pig -x local -f hbase_sample.pig
>
> We then ran the script using the mapreduce mode (default for pig):
>
> pig -x mapreduce -f hbase_sample.pig
>
> The jobtracker shows the following error(repeatedly) in the log output for
> both map and reduce tasks:
>
> Error: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.filter.WritableByteArrayComparable
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
> at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:87)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:69)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
> at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
>
> In addition, we see the following errors from pig (on the stdout):
>
> Input(s):
> Failed to read data from "hdfs://
> 10.0.0.235:8020/user/hadoop1/sample_data.csv"
>
> Output(s):
> Failed to produce result in "hbase://hello_world"
>
>
> So we have established that we can run a pig script successfully in local
> mode to store data in HBase (and we have verified that this has actually
> worked), however we cannot seem to get the same script to work with HBase
> in MR mode.
>
> Here is the complete output from pig:
>
> http://pastebin.com/vVvsCHXv
>
> Thanks,
> Royston
>
> On 3 Feb 2012, at 14:38, praveenesh kumar wrote:
>
> > Try adding Hbase-core.jar,Zookeeper.jar in Hadoop-classpath
> > Also you can try editing PIG_CLASSPATH in $PIG_HOME/bin/pig script. Just
> > Add your Hbase-core.jar and HBase_Conf dir in PiG_CLASSPATH inside pig
> > script.
> > Don't know whether these methods are the best method, but they work for
> me
> > :-)
> >
> > Thanks,
> > Praveenesh
> >
> > On Fri, Feb 3, 2012 at 5:25 AM, Royston Sellman <
> > royston.sellman@googlemail.com> wrote:
> >
> >> Thanks for your reply. WritableByteArrayComparable is in the same place
> in
> >> HBase 0.93. I also registered my HBase jar from within Pig i.e register
> >> /opt/hbase/hbase-0.93.jar and that command returned without error. I
> also
> >> edited the line in the pig startup script that specifies the path to
> HBase.
> >> Is there some other config file that controls how Pig finds HBase jar?
> Is
> >> there a default location Pig searches for jars that I could copy my jar
> >> into?
> >>
> >> Cheers,
> >> Royston
> >>
> >>
> >>
> >> On 2 Feb 2012, at 23:38, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> >>
> >>> "Caused by: java.lang.ClassNotFoundException:
> >>> org.apache.hadoop.hbase.filter.WritableByteArrayComparable" indicates
> >> that
> >>> you don't have HBase on your classpath, or that the version of HBase
> you
> >>> are testing against moved this class someplace else. We've tested
> against
> >>> the 0.90 series, but not 0.92+... did they refactor something?
> >>>
> >>> D
> >>>
> >>> On Thu, Feb 2, 2012 at 12:43 PM, Royston Sellman <
> >>> royston.sellman@googlemail.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>>
> >>>>
> >>>> I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from
> HBase
> >>>> trunk) and following the tutorial.
> >>>>
> >>>>
> >>>>
> >>>> This line loads the sample file from HDFS successfully:
> >>>>
> >>>> raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS (user,
> >> time,
> >>>> query);
> >>>>
> >>>>
> >>>>
> >>>> This line seems to work:
> >>>>
> >>>> T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time), query;
> >>>>
> >>>>
> >>>>
> >>>> Because when I do:
> >>>>
> >>>> DUMP T;
> >>>>
> >>>> I get the response I expected.
> >>>>
> >>>>
> >>>>
> >>>> But if I then do:
> >>>>
> >>>> STORE T INTO 'excite' USING
> >>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query');
> >>>>
> >>>> I get (in the log):
> >>>>
> >>>> Pig Stack Trace
> >>>>
> >>>> ---------------
> >>>>
> >>>> ERROR 2998: Unhandled internal error.
> >>>> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
> >>>>
> >>>>
> >>>>
> >>>> java.lang.NoClassDefFoundError:
> >>>> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
> >>>>
> >>>> at java.lang.Class.forName0(Native Method)
> >>>>
> >>>> at java.lang.Class.forName(Class.java:247)
> >>>>
> >>>> at
> >>>> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
> >>>>
> >>>> at
> >>>>
> >>
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder
> >>>> .java:723)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.ja
> >>>> va:712)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
> >>>> java:4340)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator
> >>>> .java:5956)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.ja
> >>>> va:1122)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGene
> >>>> rator.java:683)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.ja
> >>>> va:483)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:3
> >>>> 69)
> >>>>
> >>>> at
> >>>>
> >>
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:171)
> >>>>
> >>>> at
> >> org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
> >>>>
> >>>> at
> >> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
> >>>>
> >>>> at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
> >>>>
> >>>> at
> >>>>
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.
> >>>> java:386)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188
> >>>> )
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164
> >>>> )
> >>>>
> >>>> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> >>>>
> >>>> at org.apache.pig.Main.run(Main.java:495)
> >>>>
> >>>> at org.apache.pig.Main.main(Main.java:111)
> >>>>
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> >>>> )
> >>>>
> >>>> at
> >>>>
> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> >>>> .java:25)
> >>>>
> >>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>
> >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >>>>
> >>>> Caused by: java.lang.ClassNotFoundException:
> >>>> org.apache.hadoop.hbase.filter.WritableByteArrayComparable
> >>>>
> >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>>>
> >>>> at java.security.AccessController.doPrivileged(Native Method)
> >>>>
> >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>>>
> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>>>
> >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>>>
> >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>>>
> >>>> ... 28 more
> >>>>
> >>>>
> >>>>
> >>>> Grateful for any help with this.
> >>>>
> >>>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Royston
> >>>>
> >>>>
> >>
>
>
Re: Cannot STORE from Pig to HBase 0.9x
Posted by Royston Sellman <ro...@googlemail.com>.
Hi Praveenesh,
Thanks for your suggestion but I checked and the HBase/ZK jars and dirs are correct in our CLASSPATH and in PIG_CLASSPATH.
A few more details might help someone to point out where we are going wrong:
We have the following script called 'hbase_sample.pig'...
raw_data = LOAD 'sample_data.csv' USING PigStorage( ',' ) AS (
listing_id: chararray,
fname: chararray,
lname: chararray );
STORE raw_data INTO 'hbase://hello_world' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage (
'info:fname info:lname');
We have created the 'hello_world' table in HBase with the following command:
create 'hello_world','info'
There is an input file called 'sample_data.csv' containing the following:
1, John, Smith
2, Jane, Doe
3, George, Washington
4, Ben, Franklin
We have stored 'sample_data.csv' in the same directory as the 'hbase_sample.pig' script and we have also stored a copy of the CSV file on HDFS under the home directory for the current user "hdfs://user/hadoop1/sample_data.csv".
We ran the script in local mode SUCCESSFULLY with the following command:
pig -x local -f hbase_sample.pig
We then ran the script using the mapreduce mode (default for pig):
pig -x mapreduce -f hbase_sample.pig
The jobtracker shows the following error(repeatedly) in the log output for both map and reduce tasks:
Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.filter.WritableByteArrayComparable
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStore.getStoreFunc(POStore.java:232)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.getCommitters(PigOutputCommitter.java:87)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.<init>(PigOutputCommitter.java:69)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:279)
at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
In addition, we see the following errors from pig (on the stdout):
Input(s):
Failed to read data from "hdfs://10.0.0.235:8020/user/hadoop1/sample_data.csv"
Output(s):
Failed to produce result in "hbase://hello_world"
So we have established that we can run a pig script successfully in local mode to store data in HBase (and we have verified that this has actually worked), however we cannot seem to get the same script to work with HBase in MR mode.
Here is the complete output from pig:
http://pastebin.com/vVvsCHXv
Thanks,
Royston
On 3 Feb 2012, at 14:38, praveenesh kumar wrote:
> Try adding Hbase-core.jar,Zookeeper.jar in Hadoop-classpath
> Also you can try editing PIG_CLASSPATH in $PIG_HOME/bin/pig script. Just
> Add your Hbase-core.jar and HBase_Conf dir in PiG_CLASSPATH inside pig
> script.
> Don't know whether these methods are the best method, but they work for me
> :-)
>
> Thanks,
> Praveenesh
>
> On Fri, Feb 3, 2012 at 5:25 AM, Royston Sellman <
> royston.sellman@googlemail.com> wrote:
>
>> Thanks for your reply. WritableByteArrayComparable is in the same place in
>> HBase 0.93. I also registered my HBase jar from within Pig i.e register
>> /opt/hbase/hbase-0.93.jar and that command returned without error. I also
>> edited the line in the pig startup script that specifies the path to HBase.
>> Is there some other config file that controls how Pig finds HBase jar? Is
>> there a default location Pig searches for jars that I could copy my jar
>> into?
>>
>> Cheers,
>> Royston
>>
>>
>>
>> On 2 Feb 2012, at 23:38, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>>
>>> "Caused by: java.lang.ClassNotFoundException:
>>> org.apache.hadoop.hbase.filter.WritableByteArrayComparable" indicates
>> that
>>> you don't have HBase on your classpath, or that the version of HBase you
>>> are testing against moved this class someplace else. We've tested against
>>> the 0.90 series, but not 0.92+... did they refactor something?
>>>
>>> D
>>>
>>> On Thu, Feb 2, 2012 at 12:43 PM, Royston Sellman <
>>> royston.sellman@googlemail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from HBase
>>>> trunk) and following the tutorial.
>>>>
>>>>
>>>>
>>>> This line loads the sample file from HDFS successfully:
>>>>
>>>> raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS (user,
>> time,
>>>> query);
>>>>
>>>>
>>>>
>>>> This line seems to work:
>>>>
>>>> T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time), query;
>>>>
>>>>
>>>>
>>>> Because when I do:
>>>>
>>>> DUMP T;
>>>>
>>>> I get the response I expected.
>>>>
>>>>
>>>>
>>>> But if I then do:
>>>>
>>>> STORE T INTO 'excite' USING
>>>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query');
>>>>
>>>> I get (in the log):
>>>>
>>>> Pig Stack Trace
>>>>
>>>> ---------------
>>>>
>>>> ERROR 2998: Unhandled internal error.
>>>> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
>>>>
>>>>
>>>>
>>>> java.lang.NoClassDefFoundError:
>>>> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
>>>>
>>>> at java.lang.Class.forName0(Native Method)
>>>>
>>>> at java.lang.Class.forName(Class.java:247)
>>>>
>>>> at
>>>> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
>>>>
>>>> at
>>>>
>> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder
>>>> .java:723)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.ja
>>>> va:712)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
>>>> java:4340)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator
>>>> .java:5956)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.ja
>>>> va:1122)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGene
>>>> rator.java:683)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.ja
>>>> va:483)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:3
>>>> 69)
>>>>
>>>> at
>>>>
>> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:171)
>>>>
>>>> at
>> org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
>>>>
>>>> at
>> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
>>>>
>>>> at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
>>>>
>>>> at
>>>> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.
>>>> java:386)
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188
>>>> )
>>>>
>>>> at
>>>>
>>>>
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164
>>>> )
>>>>
>>>> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>>>>
>>>> at org.apache.pig.Main.run(Main.java:495)
>>>>
>>>> at org.apache.pig.Main.main(Main.java:111)
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>
>>>> at
>>>>
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
>>>> )
>>>>
>>>> at
>>>>
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
>>>> .java:25)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>
>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> org.apache.hadoop.hbase.filter.WritableByteArrayComparable
>>>>
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>
>>>> ... 28 more
>>>>
>>>>
>>>>
>>>> Grateful for any help with this.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Royston
>>>>
>>>>
>>
RE: Setting HDFS configuration in PigServer
Posted by rakesh sharma <ra...@hotmail.com>.
Hi Prashant,
I needed a way to control the hadoop configuration based on the deployed environment. I found a work-around by putting empty core-site.xml in the classpath and providing properties in the constructor.
Thanks for your help!
-Rakesh
> Date: Mon, 6 Feb 2012 16:27:03 -0800
> Subject: Re: Setting HDFS configuration in PigServer
> From: prash1784@gmail.com
> To: user@pig.apache.org
>
> I think I jumped the gun here, I was under the impression you want to
> override certain properties.
>
> In MapReduce mode, Pig always looks for "hadoop-site.xml" or
> "core-site.xml". I am not aware of anyway to bypass this.
>
> Curious, why do you want to not provide the config files?
>
> Thanks,
> Prashant
>
> On Mon, Feb 6, 2012 at 4:17 PM, rakesh sharma
> <ra...@hotmail.com>wrote:
>
> >
> > Hi Prashant,
> > I tried it before posting the question. Here is the exception that I get
> > when "new PigServer(ExecType.MAPREDUCE, properties)" gets executed. Note
> > that properties has the value of property "fs.default.name":
> > org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Cannot
> > find hadoop configurations in classpath (neither hadoop-site.xml nor
> > core-site.xml was found in the classpath).If you plan to use local mode,
> > please put -x local option in command line at
> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:169)
> > at
> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
> > at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
> > at org.apache.pig.PigServer.<init>(PigServer.java:226) at
> > org.apache.pig.PigServer.<init>(PigServer.java:215) at
> > org.apache.pig.PigServer.<init>(PigServer.java:211)
> > Any suggestions?
> > Thanks,Rakesh
> >
> > > Date: Mon, 6 Feb 2012 15:00:24 -0800
> > > Subject: Re: Setting HDFS configuration in PigServer
> > > From: prash1784@gmail.com
> > > To: user@pig.apache.org
> > >
> > > Rakesh,
> > >
> > > You could use
> > >
> > http://pig.apache.org/docs/r0.9.1/api/org/apache/pig/PigServer.html#PigServer%28org.apache.pig.ExecType,%20java.util.Properties%29
> > >
> > > You can specify your own properties via this constructor.
> > >
> > > Something like,
> > >
> > > Properties properties = PropertiesUtil.loadDefaultProperties(); //If
> > > you want to load defaults
> > > properties.set("foo","bar");
> > > PigServer pigServer = new PigServer(ExecType.MAPREDUCE, properties);
> > >
> > >
> > > Thanks,
> > > Prashant
> > >
> > > On Mon, Feb 6, 2012 at 2:56 PM, rakesh sharma
> > > <ra...@hotmail.com>wrote:
> > >
> > > >
> > > > Hi All,
> > > > I am using PigServer to execute pig scripts. It picks up the HDFS
> > > > configuration from hadoop configuration files residing in the
> > classpath. I
> > > > am wondering if there is a way to pass these values to PigServer and
> > > > eliminate the need of having HDFS configuration files.
> > > > Thanks,Rakesh
> >
> >
Re: Setting HDFS configuration in PigServer
Posted by Prashant Kommireddi <pr...@gmail.com>.
I think I jumped the gun here, I was under the impression you want to
override certain properties.
In MapReduce mode, Pig always looks for "hadoop-site.xml" or
"core-site.xml". I am not aware of anyway to bypass this.
Curious, why do you want to not provide the config files?
Thanks,
Prashant
On Mon, Feb 6, 2012 at 4:17 PM, rakesh sharma
<ra...@hotmail.com>wrote:
>
> Hi Prashant,
> I tried it before posting the question. Here is the exception that I get
> when "new PigServer(ExecType.MAPREDUCE, properties)" gets executed. Note
> that properties has the value of property "fs.default.name":
> org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Cannot
> find hadoop configurations in classpath (neither hadoop-site.xml nor
> core-site.xml was found in the classpath).If you plan to use local mode,
> please put -x local option in command line at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:169)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
> at org.apache.pig.PigServer.<init>(PigServer.java:226) at
> org.apache.pig.PigServer.<init>(PigServer.java:215) at
> org.apache.pig.PigServer.<init>(PigServer.java:211)
> Any suggestions?
> Thanks,Rakesh
>
> > Date: Mon, 6 Feb 2012 15:00:24 -0800
> > Subject: Re: Setting HDFS configuration in PigServer
> > From: prash1784@gmail.com
> > To: user@pig.apache.org
> >
> > Rakesh,
> >
> > You could use
> >
> http://pig.apache.org/docs/r0.9.1/api/org/apache/pig/PigServer.html#PigServer%28org.apache.pig.ExecType,%20java.util.Properties%29
> >
> > You can specify your own properties via this constructor.
> >
> > Something like,
> >
> > Properties properties = PropertiesUtil.loadDefaultProperties(); //If
> > you want to load defaults
> > properties.set("foo","bar");
> > PigServer pigServer = new PigServer(ExecType.MAPREDUCE, properties);
> >
> >
> > Thanks,
> > Prashant
> >
> > On Mon, Feb 6, 2012 at 2:56 PM, rakesh sharma
> > <ra...@hotmail.com>wrote:
> >
> > >
> > > Hi All,
> > > I am using PigServer to execute pig scripts. It picks up the HDFS
> > > configuration from hadoop configuration files residing in the
> classpath. I
> > > am wondering if there is a way to pass these values to PigServer and
> > > eliminate the need of having HDFS configuration files.
> > > Thanks,Rakesh
>
>
RE: Setting HDFS configuration in PigServer
Posted by rakesh sharma <ra...@hotmail.com>.
Hi Prashant,
I tried it before posting the question. Here is the exception that I get when "new PigServer(ExecType.MAPREDUCE, properties)" gets executed. Note that properties has the value of property "fs.default.name":
org.apache.pig.backend.executionengine.ExecException: ERROR 4010: Cannot find hadoop configurations in classpath (neither hadoop-site.xml nor core-site.xml was found in the classpath).If you plan to use local mode, please put -x local option in command line at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:169) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134) at org.apache.pig.impl.PigContext.connect(PigContext.java:183) at org.apache.pig.PigServer.<init>(PigServer.java:226) at org.apache.pig.PigServer.<init>(PigServer.java:215) at org.apache.pig.PigServer.<init>(PigServer.java:211)
Any suggestions?
Thanks,Rakesh
> Date: Mon, 6 Feb 2012 15:00:24 -0800
> Subject: Re: Setting HDFS configuration in PigServer
> From: prash1784@gmail.com
> To: user@pig.apache.org
>
> Rakesh,
>
> You could use
> http://pig.apache.org/docs/r0.9.1/api/org/apache/pig/PigServer.html#PigServer%28org.apache.pig.ExecType,%20java.util.Properties%29
>
> You can specify your own properties via this constructor.
>
> Something like,
>
> Properties properties = PropertiesUtil.loadDefaultProperties(); //If
> you want to load defaults
> properties.set("foo","bar");
> PigServer pigServer = new PigServer(ExecType.MAPREDUCE, properties);
>
>
> Thanks,
> Prashant
>
> On Mon, Feb 6, 2012 at 2:56 PM, rakesh sharma
> <ra...@hotmail.com>wrote:
>
> >
> > Hi All,
> > I am using PigServer to execute pig scripts. It picks up the HDFS
> > configuration from hadoop configuration files residing in the classpath. I
> > am wondering if there is a way to pass these values to PigServer and
> > eliminate the need of having HDFS configuration files.
> > Thanks,Rakesh
Re: Setting HDFS configuration in PigServer
Posted by Prashant Kommireddi <pr...@gmail.com>.
Rakesh,
You could use
http://pig.apache.org/docs/r0.9.1/api/org/apache/pig/PigServer.html#PigServer%28org.apache.pig.ExecType,%20java.util.Properties%29
You can specify your own properties via this constructor.
Something like,
Properties properties = PropertiesUtil.loadDefaultProperties(); //If
you want to load defaults
properties.set("foo","bar");
PigServer pigServer = new PigServer(ExecType.MAPREDUCE, properties);
Thanks,
Prashant
On Mon, Feb 6, 2012 at 2:56 PM, rakesh sharma
<ra...@hotmail.com>wrote:
>
> Hi All,
> I am using PigServer to execute pig scripts. It picks up the HDFS
> configuration from hadoop configuration files residing in the classpath. I
> am wondering if there is a way to pass these values to PigServer and
> eliminate the need of having HDFS configuration files.
> Thanks,Rakesh
Setting HDFS configuration in PigServer
Posted by rakesh sharma <ra...@hotmail.com>.
Hi All,
I am using PigServer to execute pig scripts. It picks up the HDFS configuration from hadoop configuration files residing in the classpath. I am wondering if there is a way to pass these values to PigServer and eliminate the need of having HDFS configuration files.
Thanks,Rakesh
RE: Cannot STORE from Pig to HBase 0.9x
Posted by Royston Sellman <ro...@googlemail.com>.
[resending as previous attempt did not make it through]
Hi Praveenesh,
Thanks for your suggestion but I checked and the HBase/ZK jars and dirs are
correct in our CLASSPATH and in PIG_CLASSPATH.
A few more details might help someone to point out where we are going wrong:
We have the following script called 'hbase_sample.pig'...
raw_data = LOAD 'sample_data.csv' USING PigStorage( ',' ) AS (
listing_id: chararray,
fname: chararray,
lname: chararray );
STORE raw_data INTO 'hbase://hello_world' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage (
'info:fname info:lname');
We have created the 'hello_world' table in HBase with the following command:
create 'hello_world','info'
There is an input file called 'sample_data.csv' containing the following:
1, John, Smith
2, Jane, Doe
3, George, Washington
4, Ben, Franklin
We have stored 'sample_data.csv' in the same directory as the
'hbase_sample.pig' script and we have also stored a copy of the CSV file on
HDFS under the home directory for the current user
"hdfs://user/hadoop1/sample_data.csv".
We ran the script in local mode SUCCESSFULLY with the following command:
pig -x local -f hbase_sample.pig
We then ran the script using the mapreduce mode (default for pig):
pig -x mapreduce -f hbase_sample.pig
The jobtracker shows the following error(repeatedly) in the log output for
both map and reduce tasks:
Error: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.filter.WritableByteArrayComparable
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperat
ors.POStore.getStoreFunc(POStore.java:232)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommit
ter.getCommitters(PigOutputCommitter.java:87)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommit
ter.<init>(PigOutputCommitter.java:69)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat
.getOutputCommitter(PigOutputFormat.java:279)
at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
In addition, we see the following errors from pig (on the stdout):
Input(s):
Failed to read data from
"hdfs://10.0.0.235:8020/user/hadoop1/sample_data.csv"
Output(s):
Failed to produce result in "hbase://hello_world"
So we have established that we can run a pig script successfully in local
mode to store data in HBase (and we have verified that this has actually
worked), however we cannot seem to get the same script to work with HBase in
MR mode.
Here is the complete output from pig:
http://pastebin.com/vVvsCHXv
Thanks,
Royston
-----Original Message-----
From: praveenesh kumar [mailto:praveenesh@gmail.com]
Sent: 03 February 2012 14:38
To: user@pig.apache.org
Subject: Re: Cannot STORE from Pig to HBase 0.9x
Try adding Hbase-core.jar,Zookeeper.jar in Hadoop-classpath Also you can try
editing PIG_CLASSPATH in $PIG_HOME/bin/pig script. Just Add your
Hbase-core.jar and HBase_Conf dir in PiG_CLASSPATH inside pig script.
Don't know whether these methods are the best method, but they work for me
:-)
Thanks,
Praveenesh
On Fri, Feb 3, 2012 at 5:25 AM, Royston Sellman <
royston.sellman@googlemail.com> wrote:
> Thanks for your reply. WritableByteArrayComparable is in the same
> place in HBase 0.93. I also registered my HBase jar from within Pig
> i.e register /opt/hbase/hbase-0.93.jar and that command returned
> without error. I also edited the line in the pig startup script that
specifies the path to HBase.
> Is there some other config file that controls how Pig finds HBase jar?
> Is there a default location Pig searches for jars that I could copy my
> jar into?
>
> Cheers,
> Royston
>
>
>
> On 2 Feb 2012, at 23:38, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>
> > "Caused by: java.lang.ClassNotFoundException:
> > org.apache.hadoop.hbase.filter.WritableByteArrayComparable"
> > indicates
> that
> > you don't have HBase on your classpath, or that the version of HBase
> > you are testing against moved this class someplace else. We've
> > tested against the 0.90 series, but not 0.92+... did they refactor
something?
> >
> > D
> >
> > On Thu, Feb 2, 2012 at 12:43 PM, Royston Sellman <
> > royston.sellman@googlemail.com> wrote:
> >
> >> Hi,
> >>
> >>
> >>
> >> I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from
> >> HBase
> >> trunk) and following the tutorial.
> >>
> >>
> >>
> >> This line loads the sample file from HDFS successfully:
> >>
> >> raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS (user,
> time,
> >> query);
> >>
> >>
> >>
> >> This line seems to work:
> >>
> >> T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time),
> >> query;
> >>
> >>
> >>
> >> Because when I do:
> >>
> >> DUMP T;
> >>
> >> I get the response I expected.
> >>
> >>
> >>
> >> But if I then do:
> >>
> >> STORE T INTO 'excite' USING
> >> org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query');
> >>
> >> I get (in the log):
> >>
> >> Pig Stack Trace
> >>
> >> ---------------
> >>
> >> ERROR 2998: Unhandled internal error.
> >> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
> >>
> >>
> >>
> >> java.lang.NoClassDefFoundError:
> >> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
> >>
> >> at java.lang.Class.forName0(Native Method)
> >>
> >> at java.lang.Class.forName(Class.java:247)
> >>
> >> at
> >> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428
> >> )
> >>
> >> at
> >>
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java
> :458)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanB
> uilder
> >> .java:723)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuil
> der.ja
> >> va:712)
> >>
> >> at
> >>
> >>
>
org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
> >> java:4340)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGen
> erator
> >> .java:5956)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenera
> tor.ja
> >> va:1122)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPl
> anGene
> >> rator.java:683)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenera
> tor.ja
> >> va:483)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.
> java:3
> >> 69)
> >>
> >> at
> >>
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:1
> 71)
> >>
> >> at
> org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
> >>
> >> at
> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
> >>
> >> at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
> >>
> >> at
> >> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:
> >> 942)
> >>
> >> at
> >>
> >>
>
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.
> >> java:386)
> >>
> >> at
> >>
> >>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.ja
> va:188
> >> )
> >>
> >> at
> >>
> >>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.ja
> va:164
> >> )
> >>
> >> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> >>
> >> at org.apache.pig.Main.run(Main.java:495)
> >>
> >> at org.apache.pig.Main.main(Main.java:111)
> >>
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> Method)
> >>
> >> at
> >>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:39
> >> )
> >>
> >> at
> >>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl
> >> .java:25)
> >>
> >> at java.lang.reflect.Method.invoke(Method.java:597)
> >>
> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >>
> >> Caused by: java.lang.ClassNotFoundException:
> >> org.apache.hadoop.hbase.filter.WritableByteArrayComparable
> >>
> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>
> >> at java.security.AccessController.doPrivileged(Native Method)
> >>
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>
> >> at
> >> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>
> >> ... 28 more
> >>
> >>
> >>
> >> Grateful for any help with this.
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Royston
> >>
> >>
>
Re: Cannot STORE from Pig to HBase 0.9x
Posted by praveenesh kumar <pr...@gmail.com>.
Try adding Hbase-core.jar,Zookeeper.jar in Hadoop-classpath
Also you can try editing PIG_CLASSPATH in $PIG_HOME/bin/pig script. Just
Add your Hbase-core.jar and HBase_Conf dir in PiG_CLASSPATH inside pig
script.
Don't know whether these methods are the best method, but they work for me
:-)
Thanks,
Praveenesh
On Fri, Feb 3, 2012 at 5:25 AM, Royston Sellman <
royston.sellman@googlemail.com> wrote:
> Thanks for your reply. WritableByteArrayComparable is in the same place in
> HBase 0.93. I also registered my HBase jar from within Pig i.e register
> /opt/hbase/hbase-0.93.jar and that command returned without error. I also
> edited the line in the pig startup script that specifies the path to HBase.
> Is there some other config file that controls how Pig finds HBase jar? Is
> there a default location Pig searches for jars that I could copy my jar
> into?
>
> Cheers,
> Royston
>
>
>
> On 2 Feb 2012, at 23:38, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>
> > "Caused by: java.lang.ClassNotFoundException:
> > org.apache.hadoop.hbase.filter.WritableByteArrayComparable" indicates
> that
> > you don't have HBase on your classpath, or that the version of HBase you
> > are testing against moved this class someplace else. We've tested against
> > the 0.90 series, but not 0.92+... did they refactor something?
> >
> > D
> >
> > On Thu, Feb 2, 2012 at 12:43 PM, Royston Sellman <
> > royston.sellman@googlemail.com> wrote:
> >
> >> Hi,
> >>
> >>
> >>
> >> I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from HBase
> >> trunk) and following the tutorial.
> >>
> >>
> >>
> >> This line loads the sample file from HDFS successfully:
> >>
> >> raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS (user,
> time,
> >> query);
> >>
> >>
> >>
> >> This line seems to work:
> >>
> >> T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time), query;
> >>
> >>
> >>
> >> Because when I do:
> >>
> >> DUMP T;
> >>
> >> I get the response I expected.
> >>
> >>
> >>
> >> But if I then do:
> >>
> >> STORE T INTO 'excite' USING
> >> org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query');
> >>
> >> I get (in the log):
> >>
> >> Pig Stack Trace
> >>
> >> ---------------
> >>
> >> ERROR 2998: Unhandled internal error.
> >> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
> >>
> >>
> >>
> >> java.lang.NoClassDefFoundError:
> >> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
> >>
> >> at java.lang.Class.forName0(Native Method)
> >>
> >> at java.lang.Class.forName(Class.java:247)
> >>
> >> at
> >> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
> >>
> >> at
> >>
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder
> >> .java:723)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.ja
> >> va:712)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
> >> java:4340)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator
> >> .java:5956)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.ja
> >> va:1122)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGene
> >> rator.java:683)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.ja
> >> va:483)
> >>
> >> at
> >>
> >>
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:3
> >> 69)
> >>
> >> at
> >>
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:171)
> >>
> >> at
> org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
> >>
> >> at
> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
> >>
> >> at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
> >>
> >> at
> >> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
> >>
> >> at
> >>
> >>
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.
> >> java:386)
> >>
> >> at
> >>
> >>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188
> >> )
> >>
> >> at
> >>
> >>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164
> >> )
> >>
> >> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> >>
> >> at org.apache.pig.Main.run(Main.java:495)
> >>
> >> at org.apache.pig.Main.main(Main.java:111)
> >>
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>
> >> at
> >>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> >> )
> >>
> >> at
> >>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> >> .java:25)
> >>
> >> at java.lang.reflect.Method.invoke(Method.java:597)
> >>
> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >>
> >> Caused by: java.lang.ClassNotFoundException:
> >> org.apache.hadoop.hbase.filter.WritableByteArrayComparable
> >>
> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>
> >> at java.security.AccessController.doPrivileged(Native Method)
> >>
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >>
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>
> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >>
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>
> >> ... 28 more
> >>
> >>
> >>
> >> Grateful for any help with this.
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Royston
> >>
> >>
>
Re: Cannot STORE from Pig to HBase 0.9x
Posted by Royston Sellman <ro...@googlemail.com>.
Thanks for your reply. WritableByteArrayComparable is in the same place in HBase 0.93. I also registered my HBase jar from within Pig i.e register /opt/hbase/hbase-0.93.jar and that command returned without error. I also edited the line in the pig startup script that specifies the path to HBase.
Is there some other config file that controls how Pig finds HBase jar? Is there a default location Pig searches for jars that I could copy my jar into?
Cheers,
Royston
On 2 Feb 2012, at 23:38, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> "Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.filter.WritableByteArrayComparable" indicates that
> you don't have HBase on your classpath, or that the version of HBase you
> are testing against moved this class someplace else. We've tested against
> the 0.90 series, but not 0.92+... did they refactor something?
>
> D
>
> On Thu, Feb 2, 2012 at 12:43 PM, Royston Sellman <
> royston.sellman@googlemail.com> wrote:
>
>> Hi,
>>
>>
>>
>> I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from HBase
>> trunk) and following the tutorial.
>>
>>
>>
>> This line loads the sample file from HDFS successfully:
>>
>> raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS (user, time,
>> query);
>>
>>
>>
>> This line seems to work:
>>
>> T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time), query;
>>
>>
>>
>> Because when I do:
>>
>> DUMP T;
>>
>> I get the response I expected.
>>
>>
>>
>> But if I then do:
>>
>> STORE T INTO 'excite' USING
>> org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query');
>>
>> I get (in the log):
>>
>> Pig Stack Trace
>>
>> ---------------
>>
>> ERROR 2998: Unhandled internal error.
>> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
>>
>>
>>
>> java.lang.NoClassDefFoundError:
>> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
>>
>> at java.lang.Class.forName0(Native Method)
>>
>> at java.lang.Class.forName(Class.java:247)
>>
>> at
>> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
>>
>> at
>> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
>>
>> at
>>
>> org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder
>> .java:723)
>>
>> at
>>
>> org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.ja
>> va:712)
>>
>> at
>>
>> org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
>> java:4340)
>>
>> at
>>
>> org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator
>> .java:5956)
>>
>> at
>>
>> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.ja
>> va:1122)
>>
>> at
>>
>> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGene
>> rator.java:683)
>>
>> at
>>
>> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.ja
>> va:483)
>>
>> at
>>
>> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:3
>> 69)
>>
>> at
>> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:171)
>>
>> at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
>>
>> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
>>
>> at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
>>
>> at
>> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
>>
>> at
>>
>> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.
>> java:386)
>>
>> at
>>
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188
>> )
>>
>> at
>>
>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164
>> )
>>
>> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>>
>> at org.apache.pig.Main.run(Main.java:495)
>>
>> at org.apache.pig.Main.main(Main.java:111)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
>> )
>>
>> at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
>> .java:25)
>>
>> at java.lang.reflect.Method.invoke(Method.java:597)
>>
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.hbase.filter.WritableByteArrayComparable
>>
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>
>> ... 28 more
>>
>>
>>
>> Grateful for any help with this.
>>
>>
>>
>> Thanks,
>>
>> Royston
>>
>>
Re: Cannot STORE from Pig to HBase 0.9x
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
"Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.filter.WritableByteArrayComparable" indicates that
you don't have HBase on your classpath, or that the version of HBase you
are testing against moved this class someplace else. We've tested against
the 0.90 series, but not 0.92+... did they refactor something?
D
On Thu, Feb 2, 2012 at 12:43 PM, Royston Sellman <
royston.sellman@googlemail.com> wrote:
> Hi,
>
>
>
> I'm trying to use Pig 0.9.2 with HBase 0.93 (i.e. the latest from HBase
> trunk) and following the tutorial.
>
>
>
> This line loads the sample file from HDFS successfully:
>
> raw = LOAD 'test/excite-small.log' USING PigStorage('\t') AS (user, time,
> query);
>
>
>
> This line seems to work:
>
> T= FOREACH raw GENERATE CONCAT(CONCAT(user, '\u0000'), time), query;
>
>
>
> Because when I do:
>
> DUMP T;
>
> I get the response I expected.
>
>
>
> But if I then do:
>
> STORE T INTO 'excite' USING
> org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam1:query');
>
> I get (in the log):
>
> Pig Stack Trace
>
> ---------------
>
> ERROR 2998: Unhandled internal error.
> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
>
>
>
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/filter/WritableByteArrayComparable
>
> at java.lang.Class.forName0(Native Method)
>
> at java.lang.Class.forName(Class.java:247)
>
> at
> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:428)
>
> at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:458)
>
> at
>
> org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder
> .java:723)
>
> at
>
> org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.ja
> va:712)
>
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.
> java:4340)
>
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator
> .java:5956)
>
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.ja
> va:1122)
>
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGene
> rator.java:683)
>
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.ja
> va:483)
>
> at
>
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:3
> 69)
>
> at
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:171)
>
> at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
>
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
>
> at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
>
> at
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
>
> at
>
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.
> java:386)
>
> at
>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188
> )
>
> at
>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164
> )
>
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>
> at org.apache.pig.Main.run(Main.java:495)
>
> at org.apache.pig.Main.main(Main.java:111)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> )
>
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.filter.WritableByteArrayComparable
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>
> ... 28 more
>
>
>
> Grateful for any help with this.
>
>
>
> Thanks,
>
> Royston
>
>