You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Cheolsoo Park (JIRA)" <ji...@apache.org> on 2014/07/28 22:37:40 UTC
[jira] [Updated] (PIG-4079) Parallel clause is not honored in local
mode
[ https://issues.apache.org/jira/browse/PIG-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-4079:
-------------------------------
Description:
I want to debug my custom partitioner in local mode. But it's not possible with Pig because:
# MR does not invoke partitioner if # of reducers is equal to 1.
https://issues.apache.org/jira/browse/MAPREDUCE-1287
# Pig always forces a single reducers in local mode.
{code:title=LogicalPlanBuilder.java}
void setParallel(LogicalRelationalOperator op, Integer parallel) {
if( parallel != null ) {
op.setRequestedParallelism( pigContext.getExecType() == ExecType.LOCAL ? 1 : parallel );
}
}
{code}
After I change Pig to honor the parallel clause, my custom partitioner gets invoked in local mode.
was:
I want to debug my custom partitioner in local mode. But it's not possible with Pig because:
# MR does not invoke partitioner if # of reducers is equal to 1.
# Pig always forces a single reducers in local mode.
{code:title=LogicalPlanBuilder.java}
void setParallel(LogicalRelationalOperator op, Integer parallel) {
if( parallel != null ) {
op.setRequestedParallelism( pigContext.getExecType() == ExecType.LOCAL ? 1 : parallel );
}
}
{code}
After I change Pig to honor the parallel clause, my custom partitioner gets invoked in local mode.
> Parallel clause is not honored in local mode
> --------------------------------------------
>
> Key: PIG-4079
> URL: https://issues.apache.org/jira/browse/PIG-4079
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: 0.14.0
>
>
> I want to debug my custom partitioner in local mode. But it's not possible with Pig because:
> # MR does not invoke partitioner if # of reducers is equal to 1.
> https://issues.apache.org/jira/browse/MAPREDUCE-1287
> # Pig always forces a single reducers in local mode.
> {code:title=LogicalPlanBuilder.java}
> void setParallel(LogicalRelationalOperator op, Integer parallel) {
> if( parallel != null ) {
> op.setRequestedParallelism( pigContext.getExecType() == ExecType.LOCAL ? 1 : parallel );
> }
> }
> {code}
> After I change Pig to honor the parallel clause, my custom partitioner gets invoked in local mode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)