You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Kris Coward <kr...@melon.org> on 2011/02/11 19:47:55 UTC
Problems with
com.twitter.elephantbird.pig.store.LzoTokenizedStorage
So in the interest of being a little less i/o bound, and saving a whole
mess of disk, I've started using
com.twitter.elephantbird.pig.store.LzoTokenizedStorage for storage... or
more accurately, will be using it as soon as I stop getting the
following error (with stack trace):
ERROR 2998: Unhandled internal error. Implementing class
java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:422)
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:452)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.NonEvalFuncSpec(QueryParser.java:5087)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.StoreClause(QueryParser.java:3568)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1369)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:911)
at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:724)
at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1336)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1286)
at org.apache.pig.PigServer.registerQuery(PigServer.java:460)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:738)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:324)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:163)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:139)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
at org.apache.pig.Main.main(Main.java:414)
Anyhow, the host machine is running CentOS, with the Cloudera
distribution of hadoop and pig (pig version: Apache Pig version 0.7.0+16),
and elephant-bird from the hadoop-gpl-packing RPM. The tweaks described
at http://code.google.com/p/hadoop-gpl-packing/#Using_in_Pig have been
applied, and the class still seems to be failing to load.
Anyone have any idea what the problem might be (or how to solve it)?
Thanks,
Kris
--
Kris Coward http://unripe.melon.org/
GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
RE: LzoTokenizedStorage working, but now I can't get the data back
out with LzoTokenizedLoader
Posted by Gerrit van Vuuren <gv...@specificmedia.com>.
Hi,
The hadoop-gpl-packaging jars have been tested using the standard apache pig 0.8.0 release, I've not tested this with the cloudera cdh pig.
Its important that you verify you've got lzo setup on all of the datanodes.
The configuration we use is:
For each data node:
Install lzo (yum install lzo.x86_64)
Configure:
$HADOOP_HOME/conf/mapred-site.xml (its important to set -Djava.library change the path to where you've got the native libs )
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx2048m -Xms1024m -Djava.library.path=/opt/hadoopgpl/native/Linux-amd64-64</value>
<final>true</final>
</property>
For intermediate compression setup do:
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
<final>true</final>
</property>
<property>
<name>mapred.output.compression.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
<property>
<name>mapred.map.output.compression.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
$HADOOP_HOME/conf/core-site.xml
<property>
<name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value>
<description>A list of the compression codec classes that can be used for compression/decompression.</description>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
You can also set the library property in $PIG_HOME/conf/pig.properties if the property mapred.child.java.opts it is not declared final on your cluster nodes.
e.g.
mapred.child.java.opts=-Xmx2048m -Xms1024m -Djava.library.path=/opt/hadoopgpl/native/Linux-amd64-64
(again change the path to where you've got the native libraries from the hadoop-gpl-packaging)
Cheers,
Gerrit
-----Original Message-----
From: Dmitriy Ryaboy [mailto:dvryaboy@gmail.com]
Sent: Wednesday, February 16, 2011 7:21 PM
To: user@pig.apache.org
Cc: Kris Coward
Subject: Re: LzoTokenizedStorage working, but now I can't get the data back out with LzoTokenizedLoader
"no codec" means you didn't get LZO set up right on the cluster.
There are instructions on the wiki of the googlecode project for hadoop lzo.
D
On Wed, Feb 16, 2011 at 11:05 AM, Kris Coward <kr...@melon.org> wrote:
>
> After a bunch of fiddling around (including some pretty heavy use of the
> secretDebugCmd--thanks), I finally got the LzoTokenizedStorage working,
> but now I'm having problems with the LzoTokenizedLoader.
>
> I'm still using pig 0.8.0-CDH3B4-SNAPSHOT, and for storage, have only
> seemed to have luck with the hirohanin jarfile in hadoop-gpl; registered
> all the other jarfiles from that package (i.e. google-collect-1.0.jar,
> hadoop-lzo-0.4.8.jar, protobuf-java-2.3.0.jar, slf4j-api-1.5.8.jar,
> slf4j-log4j12-1.5.10.jar, and yamlbeans-0.9.3.jar) and am now getting
> the error ans stack trace:
>
> ERROR 2997: Unable to recreate exception from backed error:
> java.io.IOException: No codec for file
>
> hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
> not found, cannot run
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
> to open iterator for alias test. Backend error : Unable to recreate
> exception from backed error: java.io.IOException: No codec for file
>
> hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
> not found, cannot run
> at org.apache.pig.PigServer.openIterator(PigServer.java:742)
> at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
> at org.apache.pig.Main.run(Main.java:465)
> at org.apache.pig.Main.main(Main.java:107)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
> 2997: Unable to recreate exception from backed error:
> java.io.IOException: No codec for file
>
> hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
> not found, cannot run
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:337)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
> at org.apache.pig.PigServer.storeEx(PigServer.java:874)
> at org.apache.pig.PigServer.store(PigServer.java:816)
> at org.apache.pig.PigServer.openIterator(PigServer.java:728)
> ... 7 more
>
> I'd think that with all the jarfiles I've loaded in (and the fact that
> LzoTokenizedStorage worked fine), that there should be a codec kicking
> around, so I'm kinda confused here..
>
> Any help would be appreciated.
>
> Thanks,
> Kris
>
> On Sun, Feb 13, 2011 at 03:00:13PM -0800, Dmitriy Ryaboy wrote:
> > You are still getting an IncompatibleClassChangeError?
> > That usually indicates there's something wrong with the classpath,
> > like perhaps your version of Pig is not the one your UDFs (in this
> > case, EB) was compiled against.
> >
> > Can you check your classpath? You can print out exactly what Pig is
> > running by passing the "-secretDebugCmd" flag to bin/pig.
> >
> > D
> >
> > On Sun, Feb 13, 2011 at 11:00 AM, Kris Coward <kr...@melon.org> wrote:
> > >
> > > Upgrading to pig 0.8.0-CDH3B4-SNAPSHOT, hasn't solved the problem, even
> > > trying with each of of the 0.8.0 jar files (the hadoopgpl package used
> > > provides 3 of them, one tagged as hirohanin's, another as gerritjvv's
> > > and another as yours). I'm going to keep banging away at this, but
> would
> > > be happy to hear any other ideas that might help.
> > >
> > > Thanks,
> > > Kris
> > >
> > > On Fri, Feb 11, 2011 at 11:15:56AM -0800, Dmitriy Ryaboy wrote:
> > >> The 0.7 branch of EB is not a real thing, it's hirohanin's first
> > >> attempt at porting that has a billion bugs.
> > >> Most things in it do not, in fact, work.
> > >> Use the 0.8 branch and Pig 0.8...
> > >>
> > >> D
> > >>
> > >> On Fri, Feb 11, 2011 at 10:47 AM, Kris Coward <kr...@melon.org> wrote:
> > >> >
> > >> > So in the interest of being a little less i/o bound, and saving a
> whole
> > >> > mess of disk, I've started using
> > >> > com.twitter.elephantbird.pig.store.LzoTokenizedStorage for
> storage... or
> > >> > more accurately, will be using it as soon as I stop getting the
> > >> > following error (with stack trace):
> > >> >
> > >> > ERROR 2998: Unhandled internal error. Implementing class
> > >> >
> > >> > java.lang.IncompatibleClassChangeError: Implementing class
> > >> > at java.lang.ClassLoader.defineClass1(Native Method)
> > >> > at
> java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> > >> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> > >> > at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> > >> > at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> > >> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> > >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> > >> > at java.security.AccessController.doPrivileged(Native Method)
> > >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > >> > at java.lang.ClassLoader.defineClass1(Native Method)
> > >> > at
> java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> > >> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> > >> > at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> > >> > at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> > >> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> > >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> > >> > at java.security.AccessController.doPrivileged(Native Method)
> > >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > >> > at java.lang.Class.forName0(Native Method)
> > >> > at java.lang.Class.forName(Class.java:247)
> > >> > at
> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:422)
> > >> > at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:452)
> > >> > at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.NonEvalFuncSpec(QueryParser.java:5087)
> > >> > at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.StoreClause(QueryParser.java:3568)
> > >> > at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1369)
> > >> > at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:911)
> > >> > at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:724)
> > >> > at
> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
> > >> > at
> org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1336)
> > >> > at
> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1286)
> > >> > at org.apache.pig.PigServer.registerQuery(PigServer.java:460)
> > >> > at
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:738)
> > >> > at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:324)
> > >> > at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:163)
> > >> > at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:139)
> > >> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
> > >> > at org.apache.pig.Main.main(Main.java:414)
> > >> >
> > >> > Anyhow, the host machine is running CentOS, with the Cloudera
> > >> > distribution of hadoop and pig (pig version: Apache Pig version
> 0.7.0+16),
> > >> > and elephant-bird from the hadoop-gpl-packing RPM. The tweaks
> described
> > >> > at http://code.google.com/p/hadoop-gpl-packing/#Using_in_Pig have
> been
> > >> > applied, and the class still seems to be failing to load.
> > >> >
> > >> > Anyone have any idea what the problem might be (or how to solve it)?
> > >> >
> > >> > Thanks,
> > >> > Kris
> > >> >
> > >> >
> > >> > --
> > >> > Kris Coward
> http://unripe.melon.org/
> > >> > GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
> > >> >
> > >
> > > --
> > > Kris Coward
> http://unripe.melon.org/
> > > GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
> > >
>
> --
> Kris Coward http://unripe.melon.org/
> GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
>
Re: LzoTokenizedStorage working, but now I can't get the data back
out with LzoTokenizedLoader
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
"no codec" means you didn't get LZO set up right on the cluster.
There are instructions on the wiki of the googlecode project for hadoop lzo.
D
On Wed, Feb 16, 2011 at 11:05 AM, Kris Coward <kr...@melon.org> wrote:
>
> After a bunch of fiddling around (including some pretty heavy use of the
> secretDebugCmd--thanks), I finally got the LzoTokenizedStorage working,
> but now I'm having problems with the LzoTokenizedLoader.
>
> I'm still using pig 0.8.0-CDH3B4-SNAPSHOT, and for storage, have only
> seemed to have luck with the hirohanin jarfile in hadoop-gpl; registered
> all the other jarfiles from that package (i.e. google-collect-1.0.jar,
> hadoop-lzo-0.4.8.jar, protobuf-java-2.3.0.jar, slf4j-api-1.5.8.jar,
> slf4j-log4j12-1.5.10.jar, and yamlbeans-0.9.3.jar) and am now getting
> the error ans stack trace:
>
> ERROR 2997: Unable to recreate exception from backed error:
> java.io.IOException: No codec for file
>
> hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
> not found, cannot run
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
> to open iterator for alias test. Backend error : Unable to recreate
> exception from backed error: java.io.IOException: No codec for file
>
> hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
> not found, cannot run
> at org.apache.pig.PigServer.openIterator(PigServer.java:742)
> at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
> at org.apache.pig.Main.run(Main.java:465)
> at org.apache.pig.Main.main(Main.java:107)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
> 2997: Unable to recreate exception from backed error:
> java.io.IOException: No codec for file
>
> hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
> not found, cannot run
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:337)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
> at org.apache.pig.PigServer.storeEx(PigServer.java:874)
> at org.apache.pig.PigServer.store(PigServer.java:816)
> at org.apache.pig.PigServer.openIterator(PigServer.java:728)
> ... 7 more
>
> I'd think that with all the jarfiles I've loaded in (and the fact that
> LzoTokenizedStorage worked fine), that there should be a codec kicking
> around, so I'm kinda confused here..
>
> Any help would be appreciated.
>
> Thanks,
> Kris
>
> On Sun, Feb 13, 2011 at 03:00:13PM -0800, Dmitriy Ryaboy wrote:
> > You are still getting an IncompatibleClassChangeError?
> > That usually indicates there's something wrong with the classpath,
> > like perhaps your version of Pig is not the one your UDFs (in this
> > case, EB) was compiled against.
> >
> > Can you check your classpath? You can print out exactly what Pig is
> > running by passing the "-secretDebugCmd" flag to bin/pig.
> >
> > D
> >
> > On Sun, Feb 13, 2011 at 11:00 AM, Kris Coward <kr...@melon.org> wrote:
> > >
> > > Upgrading to pig 0.8.0-CDH3B4-SNAPSHOT, hasn't solved the problem, even
> > > trying with each of of the 0.8.0 jar files (the hadoopgpl package used
> > > provides 3 of them, one tagged as hirohanin's, another as gerritjvv's
> > > and another as yours). I'm going to keep banging away at this, but
> would
> > > be happy to hear any other ideas that might help.
> > >
> > > Thanks,
> > > Kris
> > >
> > > On Fri, Feb 11, 2011 at 11:15:56AM -0800, Dmitriy Ryaboy wrote:
> > >> The 0.7 branch of EB is not a real thing, it's hirohanin's first
> > >> attempt at porting that has a billion bugs.
> > >> Most things in it do not, in fact, work.
> > >> Use the 0.8 branch and Pig 0.8...
> > >>
> > >> D
> > >>
> > >> On Fri, Feb 11, 2011 at 10:47 AM, Kris Coward <kr...@melon.org> wrote:
> > >> >
> > >> > So in the interest of being a little less i/o bound, and saving a
> whole
> > >> > mess of disk, I've started using
> > >> > com.twitter.elephantbird.pig.store.LzoTokenizedStorage for
> storage... or
> > >> > more accurately, will be using it as soon as I stop getting the
> > >> > following error (with stack trace):
> > >> >
> > >> > ERROR 2998: Unhandled internal error. Implementing class
> > >> >
> > >> > java.lang.IncompatibleClassChangeError: Implementing class
> > >> > at java.lang.ClassLoader.defineClass1(Native Method)
> > >> > at
> java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> > >> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> > >> > at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> > >> > at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> > >> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> > >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> > >> > at java.security.AccessController.doPrivileged(Native Method)
> > >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > >> > at java.lang.ClassLoader.defineClass1(Native Method)
> > >> > at
> java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> > >> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> > >> > at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> > >> > at
> java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> > >> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> > >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> > >> > at java.security.AccessController.doPrivileged(Native Method)
> > >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > >> > at java.lang.Class.forName0(Native Method)
> > >> > at java.lang.Class.forName(Class.java:247)
> > >> > at
> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:422)
> > >> > at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:452)
> > >> > at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.NonEvalFuncSpec(QueryParser.java:5087)
> > >> > at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.StoreClause(QueryParser.java:3568)
> > >> > at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1369)
> > >> > at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:911)
> > >> > at
> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:724)
> > >> > at
> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
> > >> > at
> org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1336)
> > >> > at
> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1286)
> > >> > at org.apache.pig.PigServer.registerQuery(PigServer.java:460)
> > >> > at
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:738)
> > >> > at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:324)
> > >> > at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:163)
> > >> > at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:139)
> > >> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
> > >> > at org.apache.pig.Main.main(Main.java:414)
> > >> >
> > >> > Anyhow, the host machine is running CentOS, with the Cloudera
> > >> > distribution of hadoop and pig (pig version: Apache Pig version
> 0.7.0+16),
> > >> > and elephant-bird from the hadoop-gpl-packing RPM. The tweaks
> described
> > >> > at http://code.google.com/p/hadoop-gpl-packing/#Using_in_Pig have
> been
> > >> > applied, and the class still seems to be failing to load.
> > >> >
> > >> > Anyone have any idea what the problem might be (or how to solve it)?
> > >> >
> > >> > Thanks,
> > >> > Kris
> > >> >
> > >> >
> > >> > --
> > >> > Kris Coward
> http://unripe.melon.org/
> > >> > GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
> > >> >
> > >
> > > --
> > > Kris Coward
> http://unripe.melon.org/
> > > GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
> > >
>
> --
> Kris Coward http://unripe.melon.org/
> GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
>
LzoTokenizedStorage working, but now I can't get the data back out
with LzoTokenizedLoader
Posted by Kris Coward <kr...@melon.org>.
After a bunch of fiddling around (including some pretty heavy use of the
secretDebugCmd--thanks), I finally got the LzoTokenizedStorage working,
but now I'm having problems with the LzoTokenizedLoader.
I'm still using pig 0.8.0-CDH3B4-SNAPSHOT, and for storage, have only
seemed to have luck with the hirohanin jarfile in hadoop-gpl; registered
all the other jarfiles from that package (i.e. google-collect-1.0.jar,
hadoop-lzo-0.4.8.jar, protobuf-java-2.3.0.jar, slf4j-api-1.5.8.jar,
slf4j-log4j12-1.5.10.jar, and yamlbeans-0.9.3.jar) and am now getting
the error ans stack trace:
ERROR 2997: Unable to recreate exception from backed error:
java.io.IOException: No codec for file
hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
not found, cannot run
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
to open iterator for alias test. Backend error : Unable to recreate
exception from backed error: java.io.IOException: No codec for file
hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
not found, cannot run
at org.apache.pig.PigServer.openIterator(PigServer.java:742)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
at org.apache.pig.Main.run(Main.java:465)
at org.apache.pig.Main.main(Main.java:107)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
2997: Unable to recreate exception from backed error:
java.io.IOException: No codec for file
hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
not found, cannot run
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:337)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
at org.apache.pig.PigServer.storeEx(PigServer.java:874)
at org.apache.pig.PigServer.store(PigServer.java:816)
at org.apache.pig.PigServer.openIterator(PigServer.java:728)
... 7 more
I'd think that with all the jarfiles I've loaded in (and the fact that
LzoTokenizedStorage worked fine), that there should be a codec kicking
around, so I'm kinda confused here..
Any help would be appreciated.
Thanks,
Kris
On Sun, Feb 13, 2011 at 03:00:13PM -0800, Dmitriy Ryaboy wrote:
> You are still getting an IncompatibleClassChangeError?
> That usually indicates there's something wrong with the classpath,
> like perhaps your version of Pig is not the one your UDFs (in this
> case, EB) was compiled against.
>
> Can you check your classpath? You can print out exactly what Pig is
> running by passing the "-secretDebugCmd" flag to bin/pig.
>
> D
>
> On Sun, Feb 13, 2011 at 11:00 AM, Kris Coward <kr...@melon.org> wrote:
> >
> > Upgrading to pig 0.8.0-CDH3B4-SNAPSHOT, hasn't solved the problem, even
> > trying with each of of the 0.8.0 jar files (the hadoopgpl package used
> > provides 3 of them, one tagged as hirohanin's, another as gerritjvv's
> > and another as yours). I'm going to keep banging away at this, but would
> > be happy to hear any other ideas that might help.
> >
> > Thanks,
> > Kris
> >
> > On Fri, Feb 11, 2011 at 11:15:56AM -0800, Dmitriy Ryaboy wrote:
> >> The 0.7 branch of EB is not a real thing, it's hirohanin's first
> >> attempt at porting that has a billion bugs.
> >> Most things in it do not, in fact, work.
> >> Use the 0.8 branch and Pig 0.8...
> >>
> >> D
> >>
> >> On Fri, Feb 11, 2011 at 10:47 AM, Kris Coward <kr...@melon.org> wrote:
> >> >
> >> > So in the interest of being a little less i/o bound, and saving a whole
> >> > mess of disk, I've started using
> >> > com.twitter.elephantbird.pig.store.LzoTokenizedStorage for storage... or
> >> > more accurately, will be using it as soon as I stop getting the
> >> > following error (with stack trace):
> >> >
> >> > ERROR 2998: Unhandled internal error. Implementing class
> >> >
> >> > java.lang.IncompatibleClassChangeError: Implementing class
> >> > at java.lang.ClassLoader.defineClass1(Native Method)
> >> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> >> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> >> > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> >> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> >> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> >> > at java.security.AccessController.doPrivileged(Native Method)
> >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >> > at java.lang.ClassLoader.defineClass1(Native Method)
> >> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> >> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> >> > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> >> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> >> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> >> > at java.security.AccessController.doPrivileged(Native Method)
> >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >> > at java.lang.Class.forName0(Native Method)
> >> > at java.lang.Class.forName(Class.java:247)
> >> > at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:422)
> >> > at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:452)
> >> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.NonEvalFuncSpec(QueryParser.java:5087)
> >> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.StoreClause(QueryParser.java:3568)
> >> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1369)
> >> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:911)
> >> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:724)
> >> > at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
> >> > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1336)
> >> > at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1286)
> >> > at org.apache.pig.PigServer.registerQuery(PigServer.java:460)
> >> > at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:738)
> >> > at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:324)
> >> > at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:163)
> >> > at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:139)
> >> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
> >> > at org.apache.pig.Main.main(Main.java:414)
> >> >
> >> > Anyhow, the host machine is running CentOS, with the Cloudera
> >> > distribution of hadoop and pig (pig version: Apache Pig version 0.7.0+16),
> >> > and elephant-bird from the hadoop-gpl-packing RPM. The tweaks described
> >> > at http://code.google.com/p/hadoop-gpl-packing/#Using_in_Pig have been
> >> > applied, and the class still seems to be failing to load.
> >> >
> >> > Anyone have any idea what the problem might be (or how to solve it)?
> >> >
> >> > Thanks,
> >> > Kris
> >> >
> >> >
> >> > --
> >> > Kris Coward http://unripe.melon.org/
> >> > GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
> >> >
> >
> > --
> > Kris Coward http://unripe.melon.org/
> > GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
> >
--
Kris Coward http://unripe.melon.org/
GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
Re: Problems with com.twitter.elephantbird.pig.store.LzoTokenizedStorage
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
You are still getting an IncompatibleClassChangeError?
That usually indicates there's something wrong with the classpath,
like perhaps your version of Pig is not the one your UDFs (in this
case, EB) was compiled against.
Can you check your classpath? You can print out exactly what Pig is
running by passing the "-secretDebugCmd" flag to bin/pig.
D
On Sun, Feb 13, 2011 at 11:00 AM, Kris Coward <kr...@melon.org> wrote:
>
> Upgrading to pig 0.8.0-CDH3B4-SNAPSHOT, hasn't solved the problem, even
> trying with each of of the 0.8.0 jar files (the hadoopgpl package used
> provides 3 of them, one tagged as hirohanin's, another as gerritjvv's
> and another as yours). I'm going to keep banging away at this, but would
> be happy to hear any other ideas that might help.
>
> Thanks,
> Kris
>
> On Fri, Feb 11, 2011 at 11:15:56AM -0800, Dmitriy Ryaboy wrote:
>> The 0.7 branch of EB is not a real thing, it's hirohanin's first
>> attempt at porting that has a billion bugs.
>> Most things in it do not, in fact, work.
>> Use the 0.8 branch and Pig 0.8...
>>
>> D
>>
>> On Fri, Feb 11, 2011 at 10:47 AM, Kris Coward <kr...@melon.org> wrote:
>> >
>> > So in the interest of being a little less i/o bound, and saving a whole
>> > mess of disk, I've started using
>> > com.twitter.elephantbird.pig.store.LzoTokenizedStorage for storage... or
>> > more accurately, will be using it as soon as I stop getting the
>> > following error (with stack trace):
>> >
>> > ERROR 2998: Unhandled internal error. Implementing class
>> >
>> > java.lang.IncompatibleClassChangeError: Implementing class
>> > at java.lang.ClassLoader.defineClass1(Native Method)
>> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
>> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
>> > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> > at java.lang.ClassLoader.defineClass1(Native Method)
>> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
>> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
>> > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> > at java.lang.Class.forName0(Native Method)
>> > at java.lang.Class.forName(Class.java:247)
>> > at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:422)
>> > at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:452)
>> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.NonEvalFuncSpec(QueryParser.java:5087)
>> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.StoreClause(QueryParser.java:3568)
>> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1369)
>> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:911)
>> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:724)
>> > at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
>> > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1336)
>> > at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1286)
>> > at org.apache.pig.PigServer.registerQuery(PigServer.java:460)
>> > at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:738)
>> > at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:324)
>> > at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:163)
>> > at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:139)
>> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
>> > at org.apache.pig.Main.main(Main.java:414)
>> >
>> > Anyhow, the host machine is running CentOS, with the Cloudera
>> > distribution of hadoop and pig (pig version: Apache Pig version 0.7.0+16),
>> > and elephant-bird from the hadoop-gpl-packing RPM. The tweaks described
>> > at http://code.google.com/p/hadoop-gpl-packing/#Using_in_Pig have been
>> > applied, and the class still seems to be failing to load.
>> >
>> > Anyone have any idea what the problem might be (or how to solve it)?
>> >
>> > Thanks,
>> > Kris
>> >
>> >
>> > --
>> > Kris Coward http://unripe.melon.org/
>> > GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
>> >
>
> --
> Kris Coward http://unripe.melon.org/
> GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
>
Re: Problems with
com.twitter.elephantbird.pig.store.LzoTokenizedStorage
Posted by Kris Coward <kr...@melon.org>.
Upgrading to pig 0.8.0-CDH3B4-SNAPSHOT, hasn't solved the problem, even
trying with each of of the 0.8.0 jar files (the hadoopgpl package used
provides 3 of them, one tagged as hirohanin's, another as gerritjvv's
and another as yours). I'm going to keep banging away at this, but would
be happy to hear any other ideas that might help.
Thanks,
Kris
On Fri, Feb 11, 2011 at 11:15:56AM -0800, Dmitriy Ryaboy wrote:
> The 0.7 branch of EB is not a real thing, it's hirohanin's first
> attempt at porting that has a billion bugs.
> Most things in it do not, in fact, work.
> Use the 0.8 branch and Pig 0.8...
>
> D
>
> On Fri, Feb 11, 2011 at 10:47 AM, Kris Coward <kr...@melon.org> wrote:
> >
> > So in the interest of being a little less i/o bound, and saving a whole
> > mess of disk, I've started using
> > com.twitter.elephantbird.pig.store.LzoTokenizedStorage for storage... or
> > more accurately, will be using it as soon as I stop getting the
> > following error (with stack trace):
> >
> > ERROR 2998: Unhandled internal error. Implementing class
> >
> > java.lang.IncompatibleClassChangeError: Implementing class
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:247)
> > at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:422)
> > at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:452)
> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.NonEvalFuncSpec(QueryParser.java:5087)
> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.StoreClause(QueryParser.java:3568)
> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1369)
> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:911)
> > at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:724)
> > at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
> > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1336)
> > at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1286)
> > at org.apache.pig.PigServer.registerQuery(PigServer.java:460)
> > at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:738)
> > at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:324)
> > at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:163)
> > at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:139)
> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
> > at org.apache.pig.Main.main(Main.java:414)
> >
> > Anyhow, the host machine is running CentOS, with the Cloudera
> > distribution of hadoop and pig (pig version: Apache Pig version 0.7.0+16),
> > and elephant-bird from the hadoop-gpl-packing RPM. The tweaks described
> > at http://code.google.com/p/hadoop-gpl-packing/#Using_in_Pig have been
> > applied, and the class still seems to be failing to load.
> >
> > Anyone have any idea what the problem might be (or how to solve it)?
> >
> > Thanks,
> > Kris
> >
> >
> > --
> > Kris Coward http://unripe.melon.org/
> > GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
> >
--
Kris Coward http://unripe.melon.org/
GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
Re: Problems with com.twitter.elephantbird.pig.store.LzoTokenizedStorage
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
The 0.7 branch of EB is not a real thing, it's hirohanin's first
attempt at porting that has a billion bugs.
Most things in it do not, in fact, work.
Use the 0.8 branch and Pig 0.8...
D
On Fri, Feb 11, 2011 at 10:47 AM, Kris Coward <kr...@melon.org> wrote:
>
> So in the interest of being a little less i/o bound, and saving a whole
> mess of disk, I've started using
> com.twitter.elephantbird.pig.store.LzoTokenizedStorage for storage... or
> more accurately, will be using it as soon as I stop getting the
> following error (with stack trace):
>
> ERROR 2998: Unhandled internal error. Implementing class
>
> java.lang.IncompatibleClassChangeError: Implementing class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
> at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:422)
> at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:452)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.NonEvalFuncSpec(QueryParser.java:5087)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.StoreClause(QueryParser.java:3568)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1369)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:911)
> at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:724)
> at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1336)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1286)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:460)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:738)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:324)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:163)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:139)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
> at org.apache.pig.Main.main(Main.java:414)
>
> Anyhow, the host machine is running CentOS, with the Cloudera
> distribution of hadoop and pig (pig version: Apache Pig version 0.7.0+16),
> and elephant-bird from the hadoop-gpl-packing RPM. The tweaks described
> at http://code.google.com/p/hadoop-gpl-packing/#Using_in_Pig have been
> applied, and the class still seems to be failing to load.
>
> Anyone have any idea what the problem might be (or how to solve it)?
>
> Thanks,
> Kris
>
>
> --
> Kris Coward http://unripe.melon.org/
> GPG Fingerprint: 2BF3 957D 310A FEEC 4733 830E 21A4 05C7 1FEB 12B3
>