[jira] [Commented] (OOZIE-2554) Spark action tests on yarn-cluster, yarn-client mode

["Peter Cseh (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15456664#comment-15456664 ] 

Peter Cseh commented on OOZIE-2554:
-----------------------------------

I've looked into this today and got stuck. 
First, Spark wanted to have {{YARN_CONF_DIR}} env. varialbe to be set. As the MiniMRCluster runs in the same JVM as the test, this is not possible. I managed to work around this by setting the {{spark.testing}} property in TestSparkMain:
{code}        
System.setProperty("spark.testing",  "true");
{code}

But now, I get the error: 
{noformat}
Call From GezapetiMBP.local/192.168.1.4 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
{noformat}
This is because Spark still tries to get the Yarn configuration from that folder, I guess this is the fallback value.

I've checked the Spark tests and they are using [SparkLauncher|https://github.com/apache/spark/blob/branch-1.6/launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java], where it's possible to include environment variables for the Spark process it spawns. 
{{SparkMain}} might be changed to use {{SparkLauncher}}, but I rather replace the {{SparkSubmit.main}} call to a dynamic way to load and call Spark and we could replace it to have a thing to validate the parameters given to Spark in the tests. 
Any thoughts on this, [~satishsaley]?

> Spark action tests on yarn-cluster, yarn-client mode
> ----------------------------------------------------
>
>                 Key: OOZIE-2554
>                 URL: https://issues.apache.org/jira/browse/OOZIE-2554
>             Project: Oozie
>          Issue Type: New Feature
>            Reporter: Satish Subhashrao Saley
>            Assignee: Satish Subhashrao Saley
>
> Currently, we have unit test cases for Spark Action in local mode. To verify other functionality, we should have tests in yarn-cluster and yarn-client mode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)