[jira] [Commented] (TOREE-337) %AddPythonDeps magic to install packages from Pypi

["Gino Bustelo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TOREE-337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15552894#comment-15552894 ] 

Gino Bustelo commented on TOREE-337:
------------------------------------

Makes total sense. PRs are welcomes.

> %AddPythonDeps magic to install packages from Pypi
> --------------------------------------------------
>
>                 Key: TOREE-337
>                 URL: https://issues.apache.org/jira/browse/TOREE-337
>             Project: TOREE
>          Issue Type: Improvement
>            Reporter: Semet
>              Labels: python,, python-wheel
>
> I would like to volunteer to work on add two "magic" to Toree, related to Python dependency managements:
> {code}
> %AddPythonDeps
> Usage: %AddPythonDeps pypi_package_name [version]
> {code}
> Download and install Python dependency from pypi.python.org and install it on the Spark cluster using pip. Transitive dependencies will be automatically retrieved.
> --- 
> {code}
> %AddPythonDist
> Usage: %AddPythonDist<url to dist>
> {code}
> Download the distribution package (source distribution package or binary distribution package or wheel) and install it on the master and workers using pip.
> ---
> Example:
> One would be able to specify something like
> {code}
> %AddPythonDeps numpy 1.1.1
> %AddPythonDeps requests 0.1.1
> {code}
> I am working to bring pip and virtualenv support to Spark since we need to work on clean virtualenv. Here is a [proposal on this subject|http://www.great-a-blog.co/wheel-deployment-for-pyspark/] and the [pull request|https://github.com/apache/spark/pull/14180].
> Let me know if it would make sense within Apache Toree to have such feature (I think it's cool to let each job describes its python dependencies and have Spark automatically handle them properly!)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)