You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "liyunzhang_intel (JIRA)" <ji...@apache.org> on 2015/06/09 04:49:00 UTC
[jira] [Updated] (PIG-4593) Enable "TestMultiQueryLocal" in spark
mode
[ https://issues.apache.org/jira/browse/PIG-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liyunzhang_intel updated PIG-4593:
----------------------------------
Attachment: PIG-4593.patch
[~mohitsabharwal],[~kexianda],[~praveenr019],[~xuefuz]:
In PIG-4593.patch, changes are:
1. fix org.apache.pig.test.TestMultiQueryLocal.testMultiQueryWithTwoStores,
org.apache.pig.test.TestMultiQueryLocal.testMultiQueryWithThreeStores,
org.apache.pig.test.TestMultiQueryLocal.testMultiQueryWithTwoLoads
The reason why these three unit test fail are same, let's explain why org.apache.pig.test.TestMultiQueryLocal.testMultiQueryWithTwoStores fails:
when "ant -Dtestcase=TestMultiQueryLocal -Dexectype=spark -Dhadoopversion=23 test" is executed,
it throws exception in [e.printStackTrace();|https://github.com/apache/pig/blob/spark/test/org/apache/pig/test/TestMultiQueryLocal.java#L110]. The exception is "java.lang.RuntimeException: Unknown test.exec.type: spark"
The exception is from https://github.com/apache/pig/blob/spark/test/org/apache/pig/test/MiniGenericCluster.java#L164 where we don't deal with spark case in MiniGenericCluster#getLauncher. In PIG-4593.patch, it adds the spark case in MiniGenericCluster#getLauncher.
In PIG-4593.patch, it adds SparkMiniCluster#getLauncher in pig/shims/test/hadoop23/org/apache/pig/test/SparkMiniCluster.java not pig/shims/test/hadoop20/org/apache/pig/test/SparkMiniCluster.java. The reason why i only add SparkMiniCluster#getLauncher in hadoop23 directory is it seems that we don't support hadoop-1 anymore and from current pig release version, we only support hadoop2.
> Enable "TestMultiQueryLocal" in spark mode
> ------------------------------------------
>
> Key: PIG-4593
> URL: https://issues.apache.org/jira/browse/PIG-4593
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4593.patch
>
>
> in https://builds.apache.org/job/Pig-spark/211/#showFailuresLink, it shows that following unit tests fail:
> org.apache.pig.test.TestMultiQueryLocal.testMultiQueryWithTwoStores
> org.apache.pig.test.TestMultiQueryLocal.testMultiQueryWithThreeStores
> org.apache.pig.test.TestMultiQueryLocal.testMultiQueryWithTwoLoads
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)