[jira] [Updated] (PIG-4374) Add SparkPlan in spark package

["liyunzhang_intel (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liyunzhang_intel updated PIG-4374:
----------------------------------
    Attachment: PIG-4374_2.patch.1

After running in my jekins, compared with https://builds.apache.org/job/Pig-spark/82/#showFailuresLink(129 failures). See attached pig_spark#27_Jenkins
15 added unit test failures:
org.apache.pig.test.TestSkewedJoin.testSkewedJoinWithGroup
org.apache.pig.test.TestEvalPipeline.testCogroupWithInputFromGroup
org.apache.pig.test.TestEvalPipeline2.testOrderByLimitJoin
org.apache.pig.test.TestEvalPipeline2.testDereferenceInnerPlan
org.apache.pig.test.TestEvalPipeline2.testUnionOnSchemaUidGeneration
org.apache.pig.test.TestEvalPipeline2.testForEachSameOriginColumn1
org.apache.pig.test.TestImplicitSplit.testImplicitSplitInCoGroup
org.apache.pig.test.TestJoinSmoke.testSkewedJoinWithGroup
org.apache.pig.test.TestKeyTypeDiscoveryVisitor.testNullJoin
org.apache.pig.test.TestMergeJoin.testExpressionFail
org.apache.pig.test.TestNewPlanImplicitSplit.testImplicitSplitInCoGroup
org.apache.pig.test.TestScriptLanguage.NegativeTest2
org.apache.pig.test.TestScriptLanguage.NegativeTest3
org.apache.pig.test.TestScriptLanguage.NegativeTest
org.apache.pig.test.TestStoreOld.testStoreWithMultipleMRJobs

4 deleted unit tests :
org.apache.pig.test.TestStoreInstances.testBackendStoreCommunication
org.apache.pig.test.TestToolsPigServer.testToolsPigServer
org.apache.pig.test.TestToolsPigServer.testToolsPigServerRegister
org.apache.pig.backend.hadoop.accumulo.TestAccumuloPigCluster.test


Currently 15 adding unit test failures are caused by the imcomplete code for join situation(SparkCompile#visitFRJoin SparkCompile#visitLimit SparkCompile#visitCollectedGroup is marked “TODO”).  Need more time to fix all.


> Add SparkPlan in spark package
> ------------------------------
>
>                 Key: PIG-4374
>                 URL: https://issues.apache.org/jira/browse/PIG-4374
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: liyunzhang_intel
>         Attachments: PIG-4374_1.patch, PIG-4374_2.patch.1, Pig-spark #76 [Jenkins .png, jenkins_PIG-4374_1_patch.png.png
>
>
> in current code, there are following class in mapreduce and tez package.
> mapreduce:
> MRCompiler
> MROperPlan  MROpPlanVisitor
> MapReduceOper
> tez:
> TezCompiler
> TezOperPlan  TezOpPlanVisitor
> TezOperator
> Following class needs to be added in spark package:
> SparkCompiler
> SparkOperPlan  SparkOpPlanVisitor
> SparkOperator
> Current code need to be refactored because of above adding classes. Some unit tests like TestStoreInstances fail because of it.
> following are the error of unit test TestStoreInstances:
> Error Message
> num jobs expected:<1> but was:<0>
> Stacktrace
> junit.framework.AssertionFailedError: num jobs expected:<1> but was:<0>
> 	at org.apache.pig.test.TestStoreInstances.testBackendStoreCommunication(TestStoreInstances.java:122)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)