You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2016/06/02 20:48:59 UTC
[jira] [Commented] (PIG-4918) Pig on Tez cannot switch pig.temp.dir
to another fs
[ https://issues.apache.org/jira/browse/PIG-4918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313033#comment-15313033 ]
Rohini Palaniswamy commented on PIG-4918:
-----------------------------------------
Can we still initialize once? i.e this.remoteFs = resourcesDir.getFileSystem(conf); It is an unnecessary call to initialize everytime and if FileSystem cache is disabled for some reason can be quite expensive.
> Pig on Tez cannot switch pig.temp.dir to another fs
> ---------------------------------------------------
>
> Key: PIG-4918
> URL: https://issues.apache.org/jira/browse/PIG-4918
> Project: Pig
> Issue Type: Bug
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.17.0, 0.16.1
>
> Attachments: PIG-4918-1.patch
>
>
> If pig.temp.dir points to another fs, Pig fails. One such case is the defaultFS is set to s3, but use hdfs as temp dir. Error message:
> {code}
> org.apache.pig.backend.hadoop.executionengine.JobCreationException: ERROR 2017: Internal error creating job configuration.
> at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:141)
> at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.compile(TezJobCompiler.java:79)
> at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:194)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:304)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1431)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1416)
> at org.apache.pig.PigServer.execute(PigServer.java:1405)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:456)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:439)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
> at org.apache.pig.Main.run(Main.java:631)
> at org.apache.pig.Main.main(Main.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalArgumentException: Wrong FS: hdfs://pig-aws-devenv-5.openstacklocal:8020/tmp/daijy/temp-265134702/automaton-1.11-8.jar, expected: s3a://pig-aws-devenv
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:658)
> at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:478)
> at org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.addTezResource(TezResourceManager.java:82)
> at org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.addTezResources(TezResourceManager.java:106)
> at org.apache.pig.backend.hadoop.executionengine.tez.plan.TezPlanContainer.getLocalResources(TezPlanContainer.java:107)
> at org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler.getJob(TezJobCompiler.java:95)
> ... 20 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)