You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Nandor Kollar (JIRA)" <ji...@apache.org> on 2017/03/29 22:42:41 UTC
[jira] [Comment Edited] (PIG-5163) MultiQuery_Streaming_1 is
failing with spark exec type
[ https://issues.apache.org/jira/browse/PIG-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15948015#comment-15948015 ]
Nandor Kollar edited comment on PIG-5163 at 3/29/17 10:42 PM:
--------------------------------------------------------------
[~kellyzly] thanks you for the review! MapReduceLauncher was modified, because I think instead of hardcoded strings it is better to use existing constants, but reverting that change is fine, it wasn't related to this fix.
I've one more question regarding multiquery optimization: do we need multiQueryOptimizeConnectionMap? The plan after the optimization looked strange, I don't understand why do we connect the two spark operators via a map instead of replacing scope-34 (loading of temp file) with scope-8 (POStream) in the optimized plan. My patch fixes the e2e test, and if it doesn't break any other test case (I'll execute the entire e2e test suite to make sure it doesn't) then it is fine, though we might have to reconsider the need of multiQueryOptimizeConnectionMap later.
was (Author: nkollar):
[~kellyzly] thank you for the review! MapReduceLauncher was modified, because I think instead of hardcoded strings it is better to use existing constants, but reverting that change is fine, it wasn't related to this fix.
I've one more question regarding multiquery optimization: do we need multiQueryOptimizeConnectionMap? The plan after the optimization looked strange, I don't understand why do we connect the two spark operators via a map instead of replacing scope-34 (loading of temp file) with scope-8 (POStream) in the optimized plan. My patch fixes the e2e test, and if it doesn't break any other test case (I'll execute the entire e2e test suite to make sure it doesn't) then it is fine, though we might have to reconsider the need of multiQueryOptimizeConnectionMap later.
> MultiQuery_Streaming_1 is failing with spark exec type
> ------------------------------------------------------
>
> Key: PIG-5163
> URL: https://issues.apache.org/jira/browse/PIG-5163
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Nandor Kollar
> Assignee: Nandor Kollar
> Fix For: spark-branch
>
> Attachments: PIG-5163_1.patch
>
>
> 2nd output was empty, looks like pig on spark didn't generate any data.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)