You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Yan Zhou (JIRA)" <ji...@apache.org> on 2010/11/20 02:51:14 UTC

[jira] Updated: (PIG-1739) Pig 0.8 zero return code when pig script fails; also error is dumped on screen instead of logfile

     [ https://issues.apache.org/jira/browse/PIG-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yan Zhou updated PIG-1739:
--------------------------

    Attachment: PIG-1739.patch

The problem is that when a job status is checked, its output status is checked. But the temporary file output status is never checked, which makes sense in case of successful jobs since nobody is interested in temporary file stats. But if a job producing temporary output fails, the output status need to be added otherwise the job status check won't find the failed job.

> Pig 0.8 zero return code when pig script fails; also error is dumped on screen instead of logfile
> -------------------------------------------------------------------------------------------------
>
>                 Key: PIG-1739
>                 URL: https://issues.apache.org/jira/browse/PIG-1739
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.8.0
>            Reporter: Viraj Bhat
>             Fix For: 0.8.0
>
>         Attachments: PIG-1739.patch
>
>
> I have pig script where one input directory cannot be accessed. The pig script obviously fails but the return code is zero.
> {code}
> A = LOAD '/user/viraj/testdata1' USING PigStorage(':') AS (ia, na);
> B = FOREACH A GENERATE $0 AS id;
> C = LOAD '/user/tstusr/test/' USING PigStorage(':') AS (ib, nb);
> D = FOREACH C GENERATE $0 AS id;
> --dump B;
> E = JOIN A by ia, C by ib USING 'replicated';
> store E into 'id.out';
> {code}
> Here is the console output:
> {quote}
> $ java -cp $PIG_HOME/pig.jar org.apache.pig.Main script.pig
> 2010-11-19 06:51:32,780 [main] INFO  org.apache.pig.Main - Logging error messages to: /home/viraj/pigscripts/pig_1290149492775.log
> ...
> 2010-11-19 06:51:39,136 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: REPLICATED_JOIN
> 2010-11-19 06:51:39,187 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: E: Store(hdfs://mynamenode/user/viraj/id.out:org.apache.pig.builtin.PigStorage) - 1-38 Operator Key: 1-38)
> 2010-11-19 06:51:39,198 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
> 2010-11-19 06:51:39,344 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - failed to get number of input files
> org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>         at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
>         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
>         at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:678)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:521)
>         at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:692)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.hasTooManyInputFiles(MRCompiler.java:1302)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:1210)
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:188)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:472)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:451)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:333)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:469)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:117)
>         at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
>         at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
>         at org.apache.pig.PigServer.execute(PigServer.java:1190)
>         at org.apache.pig.PigServer.access$100(PigServer.java:128)
>         at org.apache.pig.PigServer$Graph.execute(PigServer.java:1517)
>         at org.apache.pig.PigServer.executeBatchEx(PigServer.java:362)
>         at org.apache.pig.PigServer.executeBatch(PigServer.java:329)
>         at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:112)
>         at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
>         at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
>         at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
>         at org.apache.pig.Main.run(Main.java:498)
>         at org.apache.pig.Main.main(Main.java:107)
> 2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> 2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2010-11-19 06:51:56,714 [main] INFO  org.apache.pig.tools.pigstats.PigStats - Script Statistics:
> ...
> HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt      Features
> 0.20.1   0.8.0..1011012300       viraj   2010-11-19 06:51:41     2010-11-19 06:51:56     REPLICATED_JOIN
> Failed!
> Failed Jobs:
> JobId   Alias   Feature Message Outputs
> N/A     C       MAP_ONLY        Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> Input(s):
> Failed to read data from "/user/tstusr/test/"
> Output(s):
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
> Job DAG:
> null    ->      null,
> null
> 2010-11-19 06:51:56,714 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
> $echo $?
> 0
> {quote}
> Clearly users depending on this return code to run their workflows are affected.
> Viraj

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.