PigServer and CDH4 (Hadoop 0.23 Yarn)?

["Tieman,Brian" <Br...@Cerner.com>]

Has anyone tried using PigServer with Hadoop 0.23 and Yarn-specifically the CDH4b1 release?

Running Pig scripts through Grunt works without hiccup but I have a utility program that runs commands through PigServer.  When running that way, I run into two problems.

The first, which I can hack around, is that I need to pass in to PigServer the hbase configuration.  When I do this with ConfigurationUtil.toProperties(hbaseConfig), the new properties get passed in-not the deprecated ones.  Later on down the road, Pig looks for fs.default.name which is the deprecated property for fs.defaultfs and throws [1]  The hack is to back map the fs.default.name property to the value of fs.defaultfs.

That brings me to the second problem which I haven't been able to hack around :(  When PigServer runs, it throws the exception[2].  Poking through the code, there is no longer an org.apache.hadoop.mapred.jobcontrol.JobControl.addJob() method distributed with hadoop 0.23-only an org.apache.hadoop.mapred.jobcontrol.JobControl.addJobs() method-which also appears to be deprecated.

Again, Grunt works fine-just PigServer shows these problems.  Has anyone resolved this?  Or should I log a bug?

Thanks!

[1]
java.lang.NullPointerException
        at java.net.URI$Parser.parse(URI.java:3003)
        at java.net.URI.<init>(URI.java:578)
        at org.apache.pig.impl.logicalLayer.parser.QueryParser.setHdfsServers(QueryParser.java:485)
        at org.apache.pig.impl.logicalLayer.parser.QueryParser.LoadClause(QueryParser.java:1618)
        at org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1366)
        at org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:1013)
        at org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:800)
        at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
        at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1612)
        at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1562)
        at org.apache.pig.PigServer.registerQuery(PigServer.java:534)
        at org.apache.pig.PigServer.registerQuery(PigServer.java:558)
        at com.cerner.kepler.pig.console.KeyDump.dumpKeys(KeyDump.java:227)


[2]
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Lja
va/lang/String;
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:258)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:147)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:382)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1209)
        at org.apache.pig.PigServer.storeEx(PigServer.java:885)
        at org.apache.pig.PigServer.store(PigServer.java:827)
        at com.cerner.kepler.pig.console.KeyDump.dumpKeys(KeyDump.java:231)

Brian Tieman
Software Architect
Brian.tieman@cerner.com<ma...@cerner.com>
www.cerner.com




----------------------------------------------------------------------
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.