You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@zeppelin.apache.org by "Semet (JIRA)" <ji...@apache.org> on 2016/09/08 09:41:20 UTC

[jira] [Created] (ZEPPELIN-1419) PySpark dependencies support

Semet created ZEPPELIN-1419:
-------------------------------

             Summary: PySpark dependencies support
                 Key: ZEPPELIN-1419
                 URL: https://issues.apache.org/jira/browse/ZEPPELIN-1419
             Project: Zeppelin
          Issue Type: Improvement
          Components: python-interpreter
            Reporter: Semet


Is it possible to add support for dependencies description on a notebook?

Ideally, one would describes its dependencies on top of the notebook, ie, when python developers write in requirements.txt).

PySpark would automatically handle the installation and deployment of it, inside a virtualenv or a conda.

This would allow PySpark jobs to be completely independent from each other. If one notebook needs a Python library that does not exist on the cluster the installation will be done automatically, and with all the transitive dependencies automatically downloaded as well from pypi.python.org.
Also, two different jobs might use the same library but in two different versions.

I am working on this support for PySpark, with the ticket SPARK-16367 and in Toree for Jupyter, with TOREE-337. Let me know what you think.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)