You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2014/07/25 01:35:38 UTC
[jira] [Comment Edited] (PIG-4064) Fix tez auto parallelism test
failures
[ https://issues.apache.org/jira/browse/PIG-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14072628#comment-14072628 ]
Daniel Dai edited comment on PIG-4064 at 7/24/14 11:34 PM:
-----------------------------------------------------------
Seems some tests do not hang, I see exception instead.
The patch fixes:
TestLimitVariable.testLimitVariable2: DataMovementType of all edges of the sort vertex is set to custom and wait for VertexManager to set it. However, this should only goes to the edge between partition vertex and sort vertex. The scalar edge should not be involved
TestEvalPipeline2.testLocalRearrangeInReducer: SecondaryOptimizer does not handle two predecessors. When we process the first predecessor, we remove the foreach inner operators from the reduce side, and the second predecessor cannot see them
TestEvalPipeline2.testLimitAutoReducer: the parallelism of sort vertex is set too high in TEZ (7). Reduce it to 2 fix the issue
TestEvalPipeline2.testEmptySort: Fixed 0 quantile issue in FindQuantiles
TestEvalPipeline2.testForEachSameOriginColumn1 seems to pass for me
TestEvalPipeline2.testOrderByLimitJoin should be skipped for TEZ-1257
was (Author: daijy):
Seems some tests do not hang, I see exception instead.
The patch fixes:
TestLimitVariable.testLimitVariable2: DataMovementType of all edges of the sort vertex is set to custom and wait for VertexManager to set it. However, this should only goes to the edge between partition vertex and sort vertex. The scalar edge should not be involved
TestEvalPipeline2.testLocalRearrangeInReducer: SecondaryOptimizer does not handle two predecessors. When we process the first predecessor, we remove the foreach inner operators from the reduce side, and the second predecessor cannot see them
TestEvalPipeline2.testLimitAutoReducer: the parallelism of sort vertex is set too high in TEZ (40+). Reduce it to 2 fix the issue
TestEvalPipeline2.testEmptySort: Fixed 0 quantile issue in FindQuantiles
TestEvalPipeline2.testForEachSameOriginColumn1 seems to pass for me
TestEvalPipeline2.testOrderByLimitJoin should be skipped for TEZ-1257
> Fix tez auto parallelism test failures
> --------------------------------------
>
> Key: PIG-4064
> URL: https://issues.apache.org/jira/browse/PIG-4064
> Project: Pig
> Issue Type: Sub-task
> Components: tez
> Reporter: Rohini Palaniswamy
> Assignee: Daniel Dai
> Fix For: 0.14.0
>
> Attachments: PIG-4064-1.patch, PIG-4064-2.patch
>
>
> Tests that hang:
> - TestLimitVariable.testLimitVariable2
> - TestEvalPipeline2.testLocalRearrangeInReducer
> - TestEvalPipeline2.testLimitAutoReducer
> - TestEvalPipeline2.testForEachSameOriginColumn1
> Tests that fail:
> - TestEvalPipeline2.testEmptySort (Divide by zero error)
--
This message was sent by Atlassian JIRA
(v6.2#6252)