You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Koji Noguchi (JIRA)" <ji...@apache.org> on 2015/07/08 22:22:04 UTC
[jira] [Updated] (PIG-4628) Pig 0.14 job with order by fails in
mapreduce mode with Oozie
[ https://issues.apache.org/jira/browse/PIG-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-4628:
------------------------------
Attachment: pig-4628-v01.patch
I guess the fix can be made at where it's _reading_ this invalid token path from pigcontext (WeightedRangePartitioner.java), or where it's _writing_ this invalid path to serialized pigcontext form inside jobconf.
This pig-4628-v01.patch does the latter. Running tests.
> Pig 0.14 job with order by fails in mapreduce mode with Oozie
> -------------------------------------------------------------
>
> Key: PIG-4628
> URL: https://issues.apache.org/jira/browse/PIG-4628
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.14.0, 0.15.0
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Attachments: pig-4628-v01.patch
>
>
> A simple pig script with order-by submitted through oozie and running with mapreduce-mode
> {code}
> A = LOAD '$input' AS (a1:CHARARRAY,a2:CHARARRAY, );
> A_sorted = ORDER A BY url DESC PARALLEL 2;
> STORE A_sorted INTO '$output';
> {code}
> failed on our hadoop cluster which had security turned on. Part of the stack trace had
> {noformat}
> 2015-06-08 22:24:39,246 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Exception reading file:/tmp/2/yarn-local/usercache/userA/appcache/application_1432697993142_199266/container_e06_1432697993142_199266_01_000003/container_tokens
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.init(WeightedRangePartitioner.java:155)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.getPartition(WeightedRangePartitioner.java:75)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.partitioners.WeightedRangePartitioner.getPartition(WeightedRangePartitioner.java:58)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)
> at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:135)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:281)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:274)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1694)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {noformat}
> This failing job was from application_1432697993142_199305 and the error path was from application_1432697993142_199266 which was a oozie pig-launcher job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)