You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@toree.apache.org by "Nitin Pasumarthy (JIRA)" <ji...@apache.org> on 2018/07/09 08:04:00 UTC
[jira] [Created] (TOREE-477) pip error
Nitin Pasumarthy created TOREE-477:
--------------------------------------
Summary: pip error
Key: TOREE-477
URL: https://issues.apache.org/jira/browse/TOREE-477
Project: TOREE
Issue Type: Bug
Reporter: Nitin Pasumarthy
It looks like the TOREE-97 issue -- support for Spark Yarn was closed without definitive solution (or something went wrong on the way). Toree does support it, but it won't work if a user doesn't add manually in their kernel.json definition, the env vars for {{HADOOP_CONF_DIR}}. Without that env var, Spark doesn't know what to do with the option {{--master=yarn}} (set in {{__TOREE_SPARK_OPTS__}}). It would be desirable to have it by default.
Probably this is not the nicest way to solve the problem, because it just hard codes more vars into the JSON file -- ideally it would be nice to have an interface to add or remove env vars from those files, however, {{HADOOP_CONF_DIR}} and {{SPARK_CONF_DIR}} look basic to be exported. Even for an Spark Standalone deployment, {{HADOOP_CONF_DIR}} won't hurt. So, here it goes our 2 cents to improve a bit the situation.
I cloned the TOREE-97 into TOREE-438 to sign this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)