[jira] [Commented] (TOREE-438) CLONE - How to support Spark on Yarn model?

["Ribamar Santarosa (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TOREE-438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162904#comment-16162904 ] 

Ribamar Santarosa commented on TOREE-438:
-----------------------------------------

https://github.com/apache/incubator-toree/pull/141 as proposed solution.

> CLONE - How to support Spark on Yarn model?
> -------------------------------------------
>
>                 Key: TOREE-438
>                 URL: https://issues.apache.org/jira/browse/TOREE-438
>             Project: TOREE
>          Issue Type: Bug
>            Reporter: Ribamar Santarosa
>
> It looks like the TOREE-97 issue -- support for Spark Yarn was closed without definitive solution (or something went wrong on the way). Toree does support it, but it won't work if a user don't add manually in their kernel.json definition, the env vars for `HADOOP_CONF_DIR`. Without that env var, Spark doesn't know what to do with the option `--master=yarn` (set in `__TOREE_SPARK_OPTS__`). It would be desirable to have it by default, and this patch provides this functionality. 
> Probably this is not the nicest way to solve the problem, because it  just hard codes more vars into the JSON file -- ideally it would be nice to have an interface to add or remove env vars from those files, however, `HADOOP_CONF_DIR`  and `SPARK_CONF_DIR`  look basic to be exported. Even for an Spark Standalone deployment, `HADOOP_CONF_DIR` won't hurt.  So, here it goes our 2 cents to improve a bit the situation.
> I cloned the TOREE-97 into TOREE-438 to sign this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)