You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pig.apache.org by "Jakov Rabinovits (JIRA)" <ji...@apache.org> on 2017/08/25 10:06:00 UTC

[jira] [Comment Edited] (PIG-4621) Enable Illustrate in spark

    [ https://issues.apache.org/jira/browse/PIG-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16141447#comment-16141447 ] 

Jakov Rabinovits edited comment on PIG-4621 at 8/25/17 10:05 AM:
-----------------------------------------------------------------

Uploaded a patch.
Knows issues:
This patch is mostly done for executing commands in Grunt and calling illustrate on an alias. Some of the tests don't produce the desired output because if there are multiple store commands they may invoke the illustrate markup for the same operator multiple times, in turn that operator will have duplicated data in it's illustrator object.
Also, stores in illustrate script produce a "Output directory already exists" exception, because those stores run multiple times during the execution of illustrate command.
Additionally, Union operator is not supported, because it's UnionConverter doesn't have an apply function.


was (Author: yashock):
Knows issues:
This patch is mostly done for executing commands in Grunt and calling illustrate on an alias. Some of the tests don't produce the desired output because if there are multiple store commands they may invoke the illustrate markup for the same operator multiple times, in turn that operator will have duplicated data in it's illustrator object.
Also, stores in illustrate script produce a "Output directory already exists" exception, because those stores run multiple times during the execution of illustrate command.
Additionally, Union operator is not supported, because it's UnionConverter doesn't have an apply function.

> Enable Illustrate in spark
> --------------------------
>
>                 Key: PIG-4621
>                 URL: https://issues.apache.org/jira/browse/PIG-4621
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: Jakov Rabinovits
>              Labels: MissingFeature
>             Fix For: spark-branch
>
>         Attachments: PIG-4621.patch
>
>
> Current we don't support illustrate in spark mode.
> How illustrate works see:http://pig.apache.org/docs/r0.7.0/piglatin_ref2.html#ILLUSTRATE



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)