You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "liyunzhang_intel (JIRA)" <ji...@apache.org> on 2015/07/07 09:23:05 UTC
[jira] [Updated] (PIG-4293) Enable unit test "TestNativeMapReduce"
for spark
[ https://issues.apache.org/jira/browse/PIG-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liyunzhang_intel updated PIG-4293:
----------------------------------
Attachment: PIG-4293.patch
[~mohitsabharwal],[~kexianda],[~xuefuz],[~praveenr019]
PIG-4293.patch fixes following unit test failures:
rg.apache.pig.test.TestNativeMapReduce.testNativeMRJobTypeCastInserter
org.apache.pig.test.TestNativeMapReduce.testNativeMRJobSimple
org.apache.pig.test.TestNativeMapReduce.testNativeMRJobMultiStoreOnPred
org.apache.pig.test.TestNativeMapReduce.testNativeMRJobMultiQueryOpt
org.apache.pig.test.TestNativeMapReduce.testNativeMRJobSimpleFailure
Let's make an example to show how to use native map reduce in spark mode:
cat bin/native.pig
{code}
A = load './TestNMapReduceInputFile';
B = mapreduce '../test//org/apache/pig/test/data/TestWordCount.jar' Store A into 'table_testNativeMRJobSimple_input' Load 'table_testNativeMRJobSimple_output' `org.apache.pig.test.utils.WordCount -Dmapred.child.java.opts='-Xmx1536m -Xms128m' -files ./TestNMapReduceStopwFile table_testNativeMRJobSimple_input table_testNativeMRJobSimple_output TestNMapReduceStopwFile`;
Store B into './native.out'
{code}
cat bin/TestNMapReduceInputFile
{code}
one
two
three
three
two
three
{code}
cat bin/TestNMapReduceStopwFile
{code}
one
{code}
$PIG_HOME/bin/pig -x spark $PIG_HOME/bin/native.pig
the result:
cat native.out/part-r-00000
{code}
three 3
two 2
{code}
Changes in PIG-4293.patch:
1.add NativeSparkOperator#runJob
2.add SparkPigStatsSparkPigStats#addNativeJobStats
3.reformat SparkStatsUtil, before it uses 2 space indent.
> Enable unit test "TestNativeMapReduce" for spark
> ------------------------------------------------
>
> Key: PIG-4293
> URL: https://issues.apache.org/jira/browse/PIG-4293
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4293.patch, TEST-org.apache.pig.test.TestNativeMapReduce.txt
>
>
> error log is attached
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)