users needs to install their own python and R libraries

[Manuel Sopena Ballesteros <ma...@garvan.org.au>]

Dear Zeppelin user community,

I am trying to setup python and R to submit jobs through Spark cluster. This is already done but now I need to enable the users to install their own libraries.

I was thinking to ask the users to setup conda in their home directory and modify the `zeppelin.pyspark.python` to full conda python path. Then user should be able to enable either python2 or 3 using `generic configuration interpreter`.

Is this the right way of doing what I am trying to do?

Thank you very much




NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

Re: users needs to install their own python and R libraries

[Jeff Zhang <zj...@gmail.com>]

If you only use these libraries in driver side, then this is the right
approach. If you use these libraries in spark UDF (which means they are
used in executors as well), then you need to install them in executor too.


Manuel Sopena Ballesteros <ma...@garvan.org.au> 于2019年8月14日周三 上午11:07写道：

> Dear Zeppelin user community,
>
>
>
> I am trying to setup python and R to submit jobs through Spark cluster.
> This is already done but now I need to enable the users to install their
> own libraries.
>
>
>
> I was thinking to ask the users to setup conda in their home directory and
> modify the `zeppelin.pyspark.python` to full conda python path. Then user
> should be able to enable either python2 or 3 using `generic configuration
> interpreter`.
>
>
>
> Is this the right way of doing what I am trying to do?
>
>
>
> Thank you very much
>
>
>
>
>
>
>
>
> NOTICE
> Please consider the environment before printing this email. This message
> and any attachments are intended for the addressee named and may contain
> legally privileged/confidential/copyright information. If you are not the
> intended recipient, you should not read, use, disclose, copy or distribute
> this communication. If you have received this message in error please
> notify us at once by return email and then delete both messages. We accept
> no liability for the distribution of viruses or similar in electronic
> communications. This notice should not be removed.
>


-- 
Best Regards

Jeff Zhang