[jira] [Commented] (PIG-4621) Enable Illustrate in spark

["Adam Szita (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150432#comment-16150432 ] 

Adam Szita commented on PIG-4621:
---------------------------------

[~YaShock]: the tests returned a few errors:
# Spark mode:
#* All rank related tests: this is because a missing nullcheck in the RankConverter around {{illustrator.getData().clear();  illustrator.getEquivalenceClasses().get(0).clear();}}
#* TestPigContext#testImportList: the issue is due to POLoad getting passed in LoadConverter#ToTupleFunction#constructor. Looks like if a POLoad instance is passed, then for some reason the SparkEngineConf instance (which is also passed here) is not deserialized on the Spark end in time and we need the UDF import list before we would have an idea what classes we need. This results in Pig being unable to resolve TestUDF2 UDF. I propose we make an exception for POLoad and do the work of sparkIllustrate method inside the LoadConverter#ToTupleFunction. (We can pass the operatorkey of POLoad insteadof the op itself so that we can find the appropriate Illustrator instance)
# Tez mode: 
#* we're missing some assumeFalse's in TestGrunt#testIllustrateScript(1-7), these cases should be skipped in Tez mode: {{Assume.assumeFalse("Skip this test for TEZ. See PIG-3993", Util.isTezExecType(cluster.getExecType()));}}
# MR mode:
#* no failures

+ a few smaller comments:
* POSort#sparkIllustrate method is indented with tabs in some lines, please change to spaces
* POJoinGroupSpark#sparkIllustratorMarkup when catching ExecException let's log an error instead of printStackTrace
* TezExecutionEngine please remove unused import LocalSparkSimulator

Can you please upload a patch with these fixes?


> Enable Illustrate in spark
> --------------------------
>
>                 Key: PIG-4621
>                 URL: https://issues.apache.org/jira/browse/PIG-4621
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: Jakov Rabinovits
>              Labels: MissingFeature
>             Fix For: spark-branch
>
>         Attachments: PIG-4621.patch
>
>
> Current we don't support illustrate in spark mode.
> How illustrate works see:http://pig.apache.org/docs/r0.7.0/piglatin_ref2.html#ILLUSTRATE



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)