You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Cheolsoo Park (JIRA)" <ji...@apache.org> on 2014/09/03 23:56:51 UTC
[jira] [Commented] (PIG-4152) Enabling
pig.exec.estimate.parallelism causes runtime exception in Tez
[ https://issues.apache.org/jira/browse/PIG-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120541#comment-14120541 ]
Cheolsoo Park commented on PIG-4152:
------------------------------------
Looking at VertexManagerEvent and PigProcessor, vertexName is null when PigProcessor calls VertexManagerEvent-
{code}
private VertexManagerEvent(String vertexName, ByteBuffer userPayload) {
Preconditions.checkArgument(vertexName != null); <--- HERE
Preconditions.checkArgument(userPayload != null);
this.targetVertexName = vertexName;
this.userPayload = userPayload;
}
{code}
{code}
int parallelism = 1;
if (sampleMap!=null && sampleMap.containsKey(ESTIMATED_NUM_PARALLELISM)) {
parallelism = (Integer)sampleMap.get(ESTIMATED_NUM_PARALLELISM);
}
String sortingVertex = conf.get(SORT_VERTEX); <--- HERE
// Should contain only 1 output for sampleAggregation job
LOG.info("Sending numParallelism " + parallelism + " to " + sortingVertex);
VertexManagerEvent vmEvent = VertexManagerEvent.create(
sortingVertex, ByteBuffer.wrap(Ints.toByteArray(parallelism)));
{code}
> Enabling pig.exec.estimate.parallelism causes runtime exception in Tez
> -----------------------------------------------------------------------
>
> Key: PIG-4152
> URL: https://issues.apache.org/jira/browse/PIG-4152
> Project: Pig
> Issue Type: Sub-task
> Components: tez
> Reporter: Cheolsoo Park
> Fix For: 0.14.0
>
>
> I have a job that runs fine. But if I enable auto parallelism for sample job, it fails with an error.
> {code}
> SET pig.exec.estimate.parallelism true;
> {code}
> Here is the stack trace-
> {code}
> 2014-09-03 19:17:18,060 INFO [TezChild] org.apache.tez.runtime.task.TezTaskRunner: Encounted an error while executing task: attempt_1409152864826_35896_1_00_000031_0
> java.lang.IllegalArgumentException
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
> at org.apache.tez.runtime.api.events.VertexManagerEvent.<init>(VertexManagerEvent.java:49)
> at org.apache.tez.runtime.api.events.VertexManagerEvent.create(VertexManagerEvent.java:61)
> at org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.run(PigProcessor.java:231)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
> at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:180)
> at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:172)
> at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:167)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)