[jira] [Commented] (PIG-5179) BigData_Streaming_1 is failing

["Nandor Kollar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15901301#comment-15901301 ] 

Nandor Kollar commented on PIG-5179:
------------------------------------

If we use ExecutableManager instead of HadoopExecutableManager in spark mode, the test passes, though we loose some of the functionality provided by HadoopExecutionEngine, like setting execution permission on the file if it is not yet set, storing logs on HDFS if required etc. Instead of using ExecutableManager, maybe we should create a SparkExecutableManager similar to HadoopExecutableManager. [~kellyzly] what do you think? Do you prefer using ExecutableManager, or we should implement our own SparkExecutableManager?

> BigData_Streaming_1 is failing
> ------------------------------
>
>                 Key: PIG-5179
>                 URL: https://issues.apache.org/jira/browse/PIG-5179
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: Nandor Kollar
>            Assignee: Nandor Kollar
>             Fix For: spark-branch
>
>         Attachments: PIG-5179_1.patch
>
>
> BigData_Streaming_1 is failing, because Spark uses HadoopExecutableManager, and looks like spark task id is not a valid task attempt id.
> {code}
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2083: Error while trying to get next result in POStream.
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStream.getNextHelper(POStream.java:309)
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStream.getNextTuple(POStream.java:171)
>         ... 19 more
> Caused by: java.lang.IllegalArgumentException: TaskAttemptId string : 1 is not properly formed
>         at org.apache.hadoop.mapreduce.TaskAttemptID.forName(TaskAttemptID.java:201)
>         at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.writeErrorToHDFS(HadoopExecutableManager.java:177)
>         at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.exec(HadoopExecutableManager.java:107)
>         at org.apache.pig.impl.streaming.ExecutableManager.run(ExecutableManager.java:259)
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POStream.getNextHelper(POStream.java:281)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)