[jira] [Commented] (PIG-4606) Enable "TestDefaultDateTimeZone" unit tests in spark mode

["Mohit Sabharwal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590386#comment-14590386 ] 

Mohit Sabharwal commented on PIG-4606:
--------------------------------------

Thanks, [~kellyzly], the fix LGTM.

While, we're here, it might be good to refactor some code here, because launchPig logic is getting bit crowded. 

For example, startSparkJob() name is confusing. The job actually gets started inside sparkPlanToRDD() method.

It might be cleaner to create a new initialize() method and put all the initialization steps inside that method:
 - saveUdfImporList
 - create and populate job conf
 - SchemaTupleBackend.initialize
- read time zone from conf and set it

And rename startSparkJob() to something like addFilesToSparkJob(SparkContext sc)

What do you think ?
 

> Enable "TestDefaultDateTimeZone" unit tests in spark mode
> ---------------------------------------------------------
>
>                 Key: PIG-4606
>                 URL: https://issues.apache.org/jira/browse/PIG-4606
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: liyunzhang_intel
>             Fix For: spark-branch
>
>         Attachments: PIG-4606.patch
>
>
> In https://builds.apache.org/job/Pig-spark/216/#showFailuresLink, unit tests about TestDefaultDateTimeZone fails:
> org.apache.pig.test.TestDefaultDateTimeZone.testDST
> org.apache.pig.test.TestDefaultDateTimeZone.testLocalExecution



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)