You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Viraj Bhat (JIRA)" <ji...@apache.org> on 2010/11/19 08:13:14 UTC
[jira] Created: (PIG-1739) Pig 0.8 zero return code when pig script
fails; also error is dumped on screen instead of logfile
Pig 0.8 zero return code when pig script fails; also error is dumped on screen instead of logfile
-------------------------------------------------------------------------------------------------
Key: PIG-1739
URL: https://issues.apache.org/jira/browse/PIG-1739
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.8.0
Reporter: Viraj Bhat
Fix For: 0.8.0
I have pig script where one input directory cannot be accessed. The pig script obviously fails but the return code is zero.
{code}
A = LOAD '/user/viraj/testdata1' USING PigStorage(':') AS (ia, na);
B = FOREACH A GENERATE $0 AS id;
C = LOAD '/user/tstusr/test/' USING PigStorage(':') AS (ib, nb);
D = FOREACH C GENERATE $0 AS id;
--dump B;
E = JOIN A by ia, C by ib USING 'replicated';
store E into 'id.out';
{code}
Here is the console output:
{quote}
$ java -cp $PIG_HOME/pig.jar org.apache.pig.Main script.pig
2010-11-19 06:51:32,780 [main] INFO org.apache.pig.Main - Logging error messages to: /home/viraj/pigscripts/pig_1290149492775.log
...
2010-11-19 06:51:39,136 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: REPLICATED_JOIN
2010-11-19 06:51:39,187 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: E: Store(hdfs://mynamenode/user/viraj/id.out:org.apache.pig.builtin.PigStorage) - 1-38 Operator Key: 1-38)
2010-11-19 06:51:39,198 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2010-11-19 06:51:39,344 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - failed to get number of input files
org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:678)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:521)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:692)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.hasTooManyInputFiles(MRCompiler.java:1302)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:1210)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:188)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:472)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:451)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:333)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:469)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:117)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
at org.apache.pig.PigServer.execute(PigServer.java:1190)
at org.apache.pig.PigServer.access$100(PigServer.java:128)
at org.apache.pig.PigServer$Graph.execute(PigServer.java:1517)
at org.apache.pig.PigServer.executeBatchEx(PigServer.java:362)
at org.apache.pig.PigServer.executeBatch(PigServer.java:329)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:112)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:498)
at org.apache.pig.Main.main(Main.java:107)
2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2010-11-19 06:51:56,714 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
...
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.1 0.8.0..1011012300 viraj 2010-11-19 06:51:41 2010-11-19 06:51:56 REPLICATED_JOIN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
N/A C MAP_ONLY Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
Input(s):
Failed to read data from "/user/tstusr/test/"
Output(s):
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
null -> null,
null
2010-11-19 06:51:56,714 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
$echo $?
0
{quote}
Clearly users depending on this return code to run their workflows are affected.
Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1739) Pig 0.8 zero return code when pig script
fails; also error is dumped on screen instead of logfile
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1739:
------------------------------
Assignee: Yan Zhou
> Pig 0.8 zero return code when pig script fails; also error is dumped on screen instead of logfile
> -------------------------------------------------------------------------------------------------
>
> Key: PIG-1739
> URL: https://issues.apache.org/jira/browse/PIG-1739
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.8.0
> Reporter: Viraj Bhat
> Assignee: Yan Zhou
> Fix For: 0.8.0
>
> Attachments: PIG-1739.patch
>
>
> I have pig script where one input directory cannot be accessed. The pig script obviously fails but the return code is zero.
> {code}
> A = LOAD '/user/viraj/testdata1' USING PigStorage(':') AS (ia, na);
> B = FOREACH A GENERATE $0 AS id;
> C = LOAD '/user/tstusr/test/' USING PigStorage(':') AS (ib, nb);
> D = FOREACH C GENERATE $0 AS id;
> --dump B;
> E = JOIN A by ia, C by ib USING 'replicated';
> store E into 'id.out';
> {code}
> Here is the console output:
> {quote}
> $ java -cp $PIG_HOME/pig.jar org.apache.pig.Main script.pig
> 2010-11-19 06:51:32,780 [main] INFO org.apache.pig.Main - Logging error messages to: /home/viraj/pigscripts/pig_1290149492775.log
> ...
> 2010-11-19 06:51:39,136 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: REPLICATED_JOIN
> 2010-11-19 06:51:39,187 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: E: Store(hdfs://mynamenode/user/viraj/id.out:org.apache.pig.builtin.PigStorage) - 1-38 Operator Key: 1-38)
> 2010-11-19 06:51:39,198 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
> 2010-11-19 06:51:39,344 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - failed to get number of input files
> org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:521)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:692)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.hasTooManyInputFiles(MRCompiler.java:1302)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:1210)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:188)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:472)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:451)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:333)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:469)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:117)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
> at org.apache.pig.PigServer.execute(PigServer.java:1190)
> at org.apache.pig.PigServer.access$100(PigServer.java:128)
> at org.apache.pig.PigServer$Graph.execute(PigServer.java:1517)
> at org.apache.pig.PigServer.executeBatchEx(PigServer.java:362)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:329)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:112)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> at org.apache.pig.Main.run(Main.java:498)
> at org.apache.pig.Main.main(Main.java:107)
> 2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> 2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2010-11-19 06:51:56,714 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
> ...
> HadoopVersion PigVersion UserId StartedAt FinishedAt Features
> 0.20.1 0.8.0..1011012300 viraj 2010-11-19 06:51:41 2010-11-19 06:51:56 REPLICATED_JOIN
> Failed!
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A C MAP_ONLY Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> Input(s):
> Failed to read data from "/user/tstusr/test/"
> Output(s):
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
> Job DAG:
> null -> null,
> null
> 2010-11-19 06:51:56,714 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
> $echo $?
> 0
> {quote}
> Clearly users depending on this return code to run their workflows are affected.
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1739) Pig 0.8 zero return code when pig
script fails; also error is dumped on screen instead of logfile
Posted by "Yan Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934220#action_12934220 ]
Yan Zhou commented on PIG-1739:
-------------------------------
test-core passes ok. test-patch has the following output:
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
The test case is hard to automate, I manually verified the fix.
> Pig 0.8 zero return code when pig script fails; also error is dumped on screen instead of logfile
> -------------------------------------------------------------------------------------------------
>
> Key: PIG-1739
> URL: https://issues.apache.org/jira/browse/PIG-1739
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.8.0
> Reporter: Viraj Bhat
> Fix For: 0.8.0
>
> Attachments: PIG-1739.patch
>
>
> I have pig script where one input directory cannot be accessed. The pig script obviously fails but the return code is zero.
> {code}
> A = LOAD '/user/viraj/testdata1' USING PigStorage(':') AS (ia, na);
> B = FOREACH A GENERATE $0 AS id;
> C = LOAD '/user/tstusr/test/' USING PigStorage(':') AS (ib, nb);
> D = FOREACH C GENERATE $0 AS id;
> --dump B;
> E = JOIN A by ia, C by ib USING 'replicated';
> store E into 'id.out';
> {code}
> Here is the console output:
> {quote}
> $ java -cp $PIG_HOME/pig.jar org.apache.pig.Main script.pig
> 2010-11-19 06:51:32,780 [main] INFO org.apache.pig.Main - Logging error messages to: /home/viraj/pigscripts/pig_1290149492775.log
> ...
> 2010-11-19 06:51:39,136 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: REPLICATED_JOIN
> 2010-11-19 06:51:39,187 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: E: Store(hdfs://mynamenode/user/viraj/id.out:org.apache.pig.builtin.PigStorage) - 1-38 Operator Key: 1-38)
> 2010-11-19 06:51:39,198 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
> 2010-11-19 06:51:39,344 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - failed to get number of input files
> org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:521)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:692)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.hasTooManyInputFiles(MRCompiler.java:1302)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:1210)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:188)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:472)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:451)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:333)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:469)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:117)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
> at org.apache.pig.PigServer.execute(PigServer.java:1190)
> at org.apache.pig.PigServer.access$100(PigServer.java:128)
> at org.apache.pig.PigServer$Graph.execute(PigServer.java:1517)
> at org.apache.pig.PigServer.executeBatchEx(PigServer.java:362)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:329)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:112)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> at org.apache.pig.Main.run(Main.java:498)
> at org.apache.pig.Main.main(Main.java:107)
> 2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> 2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2010-11-19 06:51:56,714 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
> ...
> HadoopVersion PigVersion UserId StartedAt FinishedAt Features
> 0.20.1 0.8.0..1011012300 viraj 2010-11-19 06:51:41 2010-11-19 06:51:56 REPLICATED_JOIN
> Failed!
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A C MAP_ONLY Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> Input(s):
> Failed to read data from "/user/tstusr/test/"
> Output(s):
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
> Job DAG:
> null -> null,
> null
> 2010-11-19 06:51:56,714 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
> $echo $?
> 0
> {quote}
> Clearly users depending on this return code to run their workflows are affected.
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1739) Pig 0.8 zero return code when pig script
fails; also error is dumped on screen instead of logfile
Posted by "Yan Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yan Zhou updated PIG-1739:
--------------------------
Attachment: PIG-1739.patch
The problem is that when a job status is checked, its output status is checked. But the temporary file output status is never checked, which makes sense in case of successful jobs since nobody is interested in temporary file stats. But if a job producing temporary output fails, the output status need to be added otherwise the job status check won't find the failed job.
> Pig 0.8 zero return code when pig script fails; also error is dumped on screen instead of logfile
> -------------------------------------------------------------------------------------------------
>
> Key: PIG-1739
> URL: https://issues.apache.org/jira/browse/PIG-1739
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.8.0
> Reporter: Viraj Bhat
> Fix For: 0.8.0
>
> Attachments: PIG-1739.patch
>
>
> I have pig script where one input directory cannot be accessed. The pig script obviously fails but the return code is zero.
> {code}
> A = LOAD '/user/viraj/testdata1' USING PigStorage(':') AS (ia, na);
> B = FOREACH A GENERATE $0 AS id;
> C = LOAD '/user/tstusr/test/' USING PigStorage(':') AS (ib, nb);
> D = FOREACH C GENERATE $0 AS id;
> --dump B;
> E = JOIN A by ia, C by ib USING 'replicated';
> store E into 'id.out';
> {code}
> Here is the console output:
> {quote}
> $ java -cp $PIG_HOME/pig.jar org.apache.pig.Main script.pig
> 2010-11-19 06:51:32,780 [main] INFO org.apache.pig.Main - Logging error messages to: /home/viraj/pigscripts/pig_1290149492775.log
> ...
> 2010-11-19 06:51:39,136 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: REPLICATED_JOIN
> 2010-11-19 06:51:39,187 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: E: Store(hdfs://mynamenode/user/viraj/id.out:org.apache.pig.builtin.PigStorage) - 1-38 Operator Key: 1-38)
> 2010-11-19 06:51:39,198 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
> 2010-11-19 06:51:39,344 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - failed to get number of input files
> org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:521)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:692)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.hasTooManyInputFiles(MRCompiler.java:1302)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:1210)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:188)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:472)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:451)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:333)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:469)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:117)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
> at org.apache.pig.PigServer.execute(PigServer.java:1190)
> at org.apache.pig.PigServer.access$100(PigServer.java:128)
> at org.apache.pig.PigServer$Graph.execute(PigServer.java:1517)
> at org.apache.pig.PigServer.executeBatchEx(PigServer.java:362)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:329)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:112)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> at org.apache.pig.Main.run(Main.java:498)
> at org.apache.pig.Main.main(Main.java:107)
> 2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> 2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2010-11-19 06:51:56,714 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
> ...
> HadoopVersion PigVersion UserId StartedAt FinishedAt Features
> 0.20.1 0.8.0..1011012300 viraj 2010-11-19 06:51:41 2010-11-19 06:51:56 REPLICATED_JOIN
> Failed!
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A C MAP_ONLY Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> Input(s):
> Failed to read data from "/user/tstusr/test/"
> Output(s):
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
> Job DAG:
> null -> null,
> null
> 2010-11-19 06:51:56,714 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
> $echo $?
> 0
> {quote}
> Clearly users depending on this return code to run their workflows are affected.
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-1739) Pig 0.8 zero return code when pig
script fails; also error is dumped on screen instead of logfile
Posted by "Yan Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yan Zhou resolved PIG-1739.
---------------------------
Resolution: Fixed
Committed to both the trunk and the 0.8 branch.
> Pig 0.8 zero return code when pig script fails; also error is dumped on screen instead of logfile
> -------------------------------------------------------------------------------------------------
>
> Key: PIG-1739
> URL: https://issues.apache.org/jira/browse/PIG-1739
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.8.0
> Reporter: Viraj Bhat
> Assignee: Yan Zhou
> Fix For: 0.8.0
>
> Attachments: PIG-1739.patch
>
>
> I have pig script where one input directory cannot be accessed. The pig script obviously fails but the return code is zero.
> {code}
> A = LOAD '/user/viraj/testdata1' USING PigStorage(':') AS (ia, na);
> B = FOREACH A GENERATE $0 AS id;
> C = LOAD '/user/tstusr/test/' USING PigStorage(':') AS (ib, nb);
> D = FOREACH C GENERATE $0 AS id;
> --dump B;
> E = JOIN A by ia, C by ib USING 'replicated';
> store E into 'id.out';
> {code}
> Here is the console output:
> {quote}
> $ java -cp $PIG_HOME/pig.jar org.apache.pig.Main script.pig
> 2010-11-19 06:51:32,780 [main] INFO org.apache.pig.Main - Logging error messages to: /home/viraj/pigscripts/pig_1290149492775.log
> ...
> 2010-11-19 06:51:39,136 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: REPLICATED_JOIN
> 2010-11-19 06:51:39,187 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: E: Store(hdfs://mynamenode/user/viraj/id.out:org.apache.pig.builtin.PigStorage) - 1-38 Operator Key: 1-38)
> 2010-11-19 06:51:39,198 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
> 2010-11-19 06:51:39,344 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - failed to get number of input files
> org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:521)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:692)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.hasTooManyInputFiles(MRCompiler.java:1302)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:1210)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:188)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:472)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:451)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:333)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:469)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:117)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
> at org.apache.pig.PigServer.execute(PigServer.java:1190)
> at org.apache.pig.PigServer.access$100(PigServer.java:128)
> at org.apache.pig.PigServer$Graph.execute(PigServer.java:1517)
> at org.apache.pig.PigServer.executeBatchEx(PigServer.java:362)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:329)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:112)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> at org.apache.pig.Main.run(Main.java:498)
> at org.apache.pig.Main.main(Main.java:107)
> 2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> 2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2010-11-19 06:51:56,714 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
> ...
> HadoopVersion PigVersion UserId StartedAt FinishedAt Features
> 0.20.1 0.8.0..1011012300 viraj 2010-11-19 06:51:41 2010-11-19 06:51:56 REPLICATED_JOIN
> Failed!
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A C MAP_ONLY Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> Input(s):
> Failed to read data from "/user/tstusr/test/"
> Output(s):
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
> Job DAG:
> null -> null,
> null
> 2010-11-19 06:51:56,714 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
> $echo $?
> 0
> {quote}
> Clearly users depending on this return code to run their workflows are affected.
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1739) Pig 0.8 zero return code when pig
script fails; also error is dumped on screen instead of logfile
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934515#action_12934515 ]
Richard Ding commented on PIG-1739:
-----------------------------------
+1
> Pig 0.8 zero return code when pig script fails; also error is dumped on screen instead of logfile
> -------------------------------------------------------------------------------------------------
>
> Key: PIG-1739
> URL: https://issues.apache.org/jira/browse/PIG-1739
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.8.0
> Reporter: Viraj Bhat
> Assignee: Yan Zhou
> Fix For: 0.8.0
>
> Attachments: PIG-1739.patch
>
>
> I have pig script where one input directory cannot be accessed. The pig script obviously fails but the return code is zero.
> {code}
> A = LOAD '/user/viraj/testdata1' USING PigStorage(':') AS (ia, na);
> B = FOREACH A GENERATE $0 AS id;
> C = LOAD '/user/tstusr/test/' USING PigStorage(':') AS (ib, nb);
> D = FOREACH C GENERATE $0 AS id;
> --dump B;
> E = JOIN A by ia, C by ib USING 'replicated';
> store E into 'id.out';
> {code}
> Here is the console output:
> {quote}
> $ java -cp $PIG_HOME/pig.jar org.apache.pig.Main script.pig
> 2010-11-19 06:51:32,780 [main] INFO org.apache.pig.Main - Logging error messages to: /home/viraj/pigscripts/pig_1290149492775.log
> ...
> 2010-11-19 06:51:39,136 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: REPLICATED_JOIN
> 2010-11-19 06:51:39,187 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: E: Store(hdfs://mynamenode/user/viraj/id.out:org.apache.pig.builtin.PigStorage) - 1-38 Operator Key: 1-38)
> 2010-11-19 06:51:39,198 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
> 2010-11-19 06:51:39,344 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - failed to get number of input files
> org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:521)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:692)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.hasTooManyInputFiles(MRCompiler.java:1302)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:1210)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:188)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:472)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:451)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:333)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:469)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:117)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
> at org.apache.pig.PigServer.execute(PigServer.java:1190)
> at org.apache.pig.PigServer.access$100(PigServer.java:128)
> at org.apache.pig.PigServer$Graph.execute(PigServer.java:1517)
> at org.apache.pig.PigServer.executeBatchEx(PigServer.java:362)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:329)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:112)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> at org.apache.pig.Main.run(Main.java:498)
> at org.apache.pig.Main.main(Main.java:107)
> 2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> 2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2010-11-19 06:51:56,714 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
> ...
> HadoopVersion PigVersion UserId StartedAt FinishedAt Features
> 0.20.1 0.8.0..1011012300 viraj 2010-11-19 06:51:41 2010-11-19 06:51:56 REPLICATED_JOIN
> Failed!
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A C MAP_ONLY Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> Input(s):
> Failed to read data from "/user/tstusr/test/"
> Output(s):
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
> Job DAG:
> null -> null,
> null
> 2010-11-19 06:51:56,714 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
> $echo $?
> 0
> {quote}
> Clearly users depending on this return code to run their workflows are affected.
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1739) Pig 0.8 zero return code when pig
script fails; also error is dumped on screen instead of logfile
Posted by "Yan Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934079#action_12934079 ]
Yan Zhou commented on PIG-1739:
-------------------------------
With the fix, the log files contains the following, which I believe is good enough for informational and debugging purposes as well. The "failed to get number of input files" message is just a warning, by itself does not cause the failure, and only sent to the console.
Backend error message during job submission
-------------------------------------------
org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=yanz, access=EXECUTE, inode="test1":yanz:hdfs:---------
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:907)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:801)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:752)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=yanz, access=EXECUTE, inode="test1":yanz:hdfs:---------
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:58)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:617)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:453)
at org.apache.hadoop.fs.FileSystem.getFileStatus(FileSystem.java:1330)
at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:924)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:866)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:209)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:246)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
... 7 more
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Permission denied: user=yanz, access=EXECUTE, inode="test1":yanz:hdfs:---------
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:159)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:115)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:85)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4575)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkTraverse(FSNamesystem.java:4554)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:1751)
at org.apache.hadoop.hdfs.server.namenode.NameNode.getFileInfo(NameNode.java:572)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:965)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:963)
at org.apache.hadoop.ipc.Client.call(Client.java:740)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.getFileInfo(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:615)
... 15 more
Pig Stack Trace
---------------
ERROR 2244: Job failed, hadoop does not return any error message
org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job failed, hadoop does not return any error message
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:117)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:509)
at org.apache.pig.Main.main(Main.java:107)
================================================================================
> Pig 0.8 zero return code when pig script fails; also error is dumped on screen instead of logfile
> -------------------------------------------------------------------------------------------------
>
> Key: PIG-1739
> URL: https://issues.apache.org/jira/browse/PIG-1739
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.8.0
> Reporter: Viraj Bhat
> Fix For: 0.8.0
>
> Attachments: PIG-1739.patch
>
>
> I have pig script where one input directory cannot be accessed. The pig script obviously fails but the return code is zero.
> {code}
> A = LOAD '/user/viraj/testdata1' USING PigStorage(':') AS (ia, na);
> B = FOREACH A GENERATE $0 AS id;
> C = LOAD '/user/tstusr/test/' USING PigStorage(':') AS (ib, nb);
> D = FOREACH C GENERATE $0 AS id;
> --dump B;
> E = JOIN A by ia, C by ib USING 'replicated';
> store E into 'id.out';
> {code}
> Here is the console output:
> {quote}
> $ java -cp $PIG_HOME/pig.jar org.apache.pig.Main script.pig
> 2010-11-19 06:51:32,780 [main] INFO org.apache.pig.Main - Logging error messages to: /home/viraj/pigscripts/pig_1290149492775.log
> ...
> 2010-11-19 06:51:39,136 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: REPLICATED_JOIN
> 2010-11-19 06:51:39,187 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: E: Store(hdfs://mynamenode/user/viraj/id.out:org.apache.pig.builtin.PigStorage) - 1-38 Operator Key: 1-38)
> 2010-11-19 06:51:39,198 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
> 2010-11-19 06:51:39,344 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - failed to get number of input files
> org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:678)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:521)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:692)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.hasTooManyInputFiles(MRCompiler.java:1302)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.visitFRJoin(MRCompiler.java:1210)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POFRJoin.visit(POFRJoin.java:188)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:472)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:451)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:333)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:469)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:117)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
> at org.apache.pig.PigServer.execute(PigServer.java:1190)
> at org.apache.pig.PigServer.access$100(PigServer.java:128)
> at org.apache.pig.PigServer$Graph.execute(PigServer.java:1517)
> at org.apache.pig.PigServer.executeBatchEx(PigServer.java:362)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:329)
> at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:112)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> at org.apache.pig.Main.run(Main.java:498)
> at org.apache.pig.Main.main(Main.java:107)
> 2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> 2010-11-19 06:51:56,712 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2010-11-19 06:51:56,714 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
> ...
> HadoopVersion PigVersion UserId StartedAt FinishedAt Features
> 0.20.1 0.8.0..1011012300 viraj 2010-11-19 06:51:41 2010-11-19 06:51:56 REPLICATED_JOIN
> Failed!
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A C MAP_ONLY Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: org.apache.hadoop.security.AccessControlException: Permission denied: user=viraj, access=EXECUTE, inode="tstusr":tstusr:users:rwx------
> Input(s):
> Failed to read data from "/user/tstusr/test/"
> Output(s):
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
> Job DAG:
> null -> null,
> null
> 2010-11-19 06:51:56,714 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
> $echo $?
> 0
> {quote}
> Clearly users depending on this return code to run their workflows are affected.
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.