Re: load from cassandra

[Kevin Burton <bu...@spinn3r.com>]

 java.lang.NoSuchMethodError:

… that's your problem. You have have a version conflict in a .jar
somewhere.

It's expecting an addJob method on
org.apache.hadoop.mapred.jobcontrol.JobControl

… which doesn't exist?

I had to recompile pig for YARN..  That might be your issue.


On Fri, Feb 7, 2014 at 2:17 PM, Irooniam <ir...@gmail.com> wrote:

> Hello, I'm having a hard time using Pig to extract data from Cassandra.
>
>
>  Cassandra: [cqlsh 4.1.0 | Cassandra 2.0.4 | CQL spec 3.1.1 | Thrift
> protocol 19.39.0]
>
> Hadoop (Cloudera): 2.0.0+1518
>
> Map Reduce: v2 (Yarn)
>
> Pig: Apache Pig version 0.11.0-cdh4.5.0
>
>
>  I can use pig fine to run mapreduce jobs.
>
>
>  The test schema is very simple:
>
> cqlsh:main> create table a (id int, name varchar, primary key (id));
>
> cqlsh:main> insert into a (id, name) values (1, 'blah');
>
> cqlsh:main> select * from a;
>
>  id | name
>
> ----+------
>
> 1 | blah
>
>  (1 rows)
>
>
>
>  The problem I run into is when I'm trying to extract data from Cassandra:
>
> bash-4.2$ ./apache-cassandra-2.0.4-src/examples/pig/bin/pig_cassandra -x
> local
>
> Using /home/hdfs/pig-0.12.0-src/pig-withouthadoop.jar.
>
> 2014-02-07 17:09:18,948 [main] INFO org.apache.pig.Main - Apache Pig
> version 0.10.0 (r1328203) compiled Apr 20 2012, 00:33:25
>
> 2014-02-07 17:09:18,949 [main] INFO org.apache.pig.Main - Logging error
> messages to: /home/hdfs/pig_1391810958945.log
>
> 2014-02-07 17:09:19,373 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to hadoop file system at: file:///
>
> 2014-02-07 17:09:19,377 [main] WARN org.apache.hadoop.conf.Configuration -
> mapred.used.genericoptionsparser is deprecated. Instead, use
> mapreduce.client.genericoptionsparser.used
>
> 2014-02-07 17:09:19,394 [main] WARN org.apache.hadoop.conf.Configuration -
> fs.default.name is deprecated. Instead, use fs.defaultFS
>
> 2014-02-07 17:09:19,395 [main] WARN org.apache.hadoop.conf.Configuration -
> mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in
>
> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
>
> SLF4J: Found binding in
>
> [jar:file:/home/hdfs/apache-cassandra-2.0.4-src/lib/slf4j-log4j12-1.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> 2014-02-07 17:09:20,026 [main] WARN org.apache.hadoop.conf.Configuration -
> io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
>
> 2014-02-07 17:09:20,030 [main] WARN org.apache.hadoop.conf.Configuration -
> fs.default.name is deprecated. Instead, use fs.defaultFS
>
> 2014-02-07 17:09:20,030 [main] WARN org.apache.hadoop.conf.Configuration -
> mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
>
> grunt> rows = LOAD 'cql://main/a' USING CqlStorage();
>
> grunt> describe rows;
>
> rows: {id: int,name: chararray}
>
>
>  Pig can get the schema out the table.
>
>
>  However trying to dump the data is when it all goes south:
>
> grunt> data = foreach rows generate $1;
>
> grunt> dump data;
>
> 2014-02-07 17:09:47,347 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
>
> 2014-02-07 17:09:47,416 [main] INFO
> org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned
> for rows: $0
>
> 2014-02-07 17:09:47,548 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
>
> 2014-02-07 17:09:47,589 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
>
> 2014-02-07 17:09:47,589 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
>
> 2014-02-07 17:09:47,960 [main] WARN org.apache.hadoop.conf.Configuration -
> session.id is deprecated. Instead, use dfs.metrics.session-id
>
> 2014-02-07 17:09:47,968 [main] INFO
> org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
> processName=JobTracker, sessionId=
>
> 2014-02-07 17:09:48,055 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
> to the job
>
> 2014-02-07 17:09:48,075 [main] WARN org.apache.hadoop.conf.Configuration -
> mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use
> mapreduce.reduce.markreset.buffer.percent
>
> 2014-02-07 17:09:48,075 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
>
> 2014-02-07 17:09:48,075 [main] WARN org.apache.hadoop.conf.Configuration -
> mapred.output.compress is deprecated. Instead, use
> mapreduce.output.fileoutputformat.compress
>
> 2014-02-07 17:09:48,206 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
>
> 2014-02-07 17:09:48,330 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 2998: Unhandled internal error.
>
> org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
>
>
> Details at logfile: /home/hdfs/pig_1391810958945.log
>
>
>  The log says:
>
> Pig Stack Trace
>
> ---------------
>
> ERROR 2998: Unhandled internal error.
>
> org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
>
>
>  java.lang.NoSuchMethodError:
>
> org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
>
>
> at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:261)
>
>
> at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:180)
>
>
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1270)
>
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1255)
>
> at org.apache.pig.PigServer.storeEx(PigServer.java:952)
>
> at org.apache.pig.PigServer.store(PigServer.java:919)
>
> at org.apache.pig.PigServer.openIterator(PigServer.java:832)
>
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:682)
>
> at
>
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
>
>
> at
>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
>
>
> at
>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
>
>
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>
> at org.apache.pig.Main.run(Main.java:490)
>
> at org.apache.pig.Main.main(Main.java:111)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
>
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>
>
>  I'd appreciate any help on this.
>



-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
Skype: *burtonator*
blog: http://burtonator.wordpress.com
… or check out my Google+
profile<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
War is peace. Freedom is slavery. Ignorance is strength. Corporations are
people.