You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by 李建伟 <le...@126.com> on 2013/04/03 03:04:56 UTC
viewfs on pig
Hi,
Do you know which version of Pig support Viewfs? I tried 0.11.0, got the following error:
grunt> A = load '/tmp/passwd' using PigStorage(':');
grunt> B = foreach A generate $0 as id;
grunt> dump B;
2013-04-03 01:04:20,489 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2013-04-03 01:04:20,608 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2013-04-03 01:04:20,645 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2013-04-03 01:04:20,646 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2013-04-03 01:04:20,776 [main] WARN org.apache.pig.backend.hadoop20.PigJobControl - falling back to default JobControl (not using hadoop 0.20 ?)
java.lang.NoSuchFieldException: runnerState
at java.lang.Class.getDeclaredField(Class.java:1882)
at org.apache.pig.backend.hadoop20.PigJobControl.<clinit>(PigJobControl.java:51)
at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.newJobControl(HadoopShims.java:97)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:285)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
at org.apache.pig.PigServer.storeEx(PigServer.java:931)
at org.apache.pig.PigServer.store(PigServer.java:898)
at org.apache.pig.PigServer.openIterator(PigServer.java:811)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:538)
at org.apache.pig.Main.main(Main.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
2013-04-03 01:04:20,778 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-04-03 01:04:20,792 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-04-03 01:04:20,798 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2013-04-03 01:04:20,800 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=1513
2013-04-03 01:04:20,800 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2013-04-03 01:04:20,801 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job1924576292248079915.jar
2013-04-03 01:04:23,873 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job1924576292248079915.jar created
2013-04-03 01:04:23,889 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2013-04-03 01:04:23,894 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2013-04-03 01:04:23,894 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2013-04-03 01:04:23,895 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
Re: viewfs on pig
Posted by Johnny Zhang <xi...@cloudera.com>.
Hi,
Pig 0.11 should start supporting viewfs
https://issues.apache.org/jira/browse/PIG-2791
Johnny
On Tue, Apr 2, 2013 at 6:04 PM, 李建伟 <le...@126.com> wrote:
> Hi,
>
>
> Do you know which version of Pig support Viewfs? I tried 0.11.0, got the
> following error:
> grunt> A = load '/tmp/passwd' using PigStorage(':');
> grunt> B = foreach A generate $0 as id;
> grunt> dump B;
> 2013-04-03 01:04:20,489 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2013-04-03 01:04:20,608 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
> 2013-04-03 01:04:20,645 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2013-04-03 01:04:20,646 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2013-04-03 01:04:20,776 [main] WARN
> org.apache.pig.backend.hadoop20.PigJobControl - falling back to default
> JobControl (not using hadoop 0.20 ?)
> java.lang.NoSuchFieldException: runnerState
> at java.lang.Class.getDeclaredField(Class.java:1882)
> at
> org.apache.pig.backend.hadoop20.PigJobControl.<clinit>(PigJobControl.java:51)
> at
> org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.newJobControl(HadoopShims.java:97)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:285)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
> at org.apache.pig.PigServer.storeEx(PigServer.java:931)
> at org.apache.pig.PigServer.store(PigServer.java:898)
> at org.apache.pig.PigServer.openIterator(PigServer.java:811)
> at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> at org.apache.pig.Main.run(Main.java:538)
> at org.apache.pig.Main.main(Main.java:157)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> 2013-04-03 01:04:20,778 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
> to the job
> 2013-04-03 01:04:20,792 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2013-04-03 01:04:20,798 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Using reducer estimator:
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
> 2013-04-03 01:04:20,800 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
> - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=1513
> 2013-04-03 01:04:20,800 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting Parallelism to 1
> 2013-04-03 01:04:20,801 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - creating jar file Job1924576292248079915.jar
> 2013-04-03 01:04:23,873 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - jar file Job1924576292248079915.jar created
> 2013-04-03 01:04:23,889 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2013-04-03 01:04:23,894 [main] INFO
> org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false,
> will not generate code.
> 2013-04-03 01:04:23,894 [main] INFO
> org.apache.pig.data.SchemaTupleFrontend - Starting process to move
> generated code to distributed cacche
> 2013-04-03 01:04:23,895 [main] INFO
> org.apache.pig.data.SchemaTupleFrontend - Setting key
> [pig.schematuple.classes] with classes to deserialize []
>
>