[jira] [Updated] (PIG-4281) Fix TestFinish for Spark engine

["liyunzhang_intel (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liyunzhang_intel updated PIG-4281:
----------------------------------
    Attachment: PIG-4281_3.patch

[~kexianda],[~mohitsabharwal],[~pallavi.rao]
PIG-4281_3.patch is to fix the comments from community about the merge work.

Changes:
1.           {code: title=org.apache.pig.test.TestFinish.MyEvalFunction#finish} 
	  @Override
        public void finish() {
            try {
                FileSystem fs = null;
                if (execType.equalsIgnoreCase("SPARK")) {
					//Kelly's comment: before we get the configuration by cluster#getConfiguration, but committee suggest that "The cluster object is not something a user UDF will have access to". In spark mode, PigMapReduce.sJobConfInternal.get() is null because we don't set the value for it.
					fs = FileSystem.get(cluster.getConfiguration());  
                } else {
                    fs = FileSystem.get(PigMapReduce.sJobConfInternal.get());
                }
                fs.create(new Path(expectedFileName));
            } catch (IOException e) {
                throw new RuntimeException("Unable to create file:" + expectedFileName);
            }
        }
		{code}
2. set value for PigMapReduce.sJobConfInternal in SparkLauncher#initialize

> Fix TestFinish for Spark engine
> -------------------------------
>
>                 Key: PIG-4281
>                 URL: https://issues.apache.org/jira/browse/PIG-4281
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: liyunzhang_intel
>             Fix For: spark-branch
>
>         Attachments: PIG-4281.patch, PIG-4281_2.patch, PIG-4281_3.patch, TEST-org.apache.pig.test.TestFinish.txt
>
>
> error log is attached



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)