python virtual environment on spark interpreter

[Manuel Sopena Ballesteros <ma...@garvan.org.au>]

Dear Zeppelin user community,

I have a zeppelin installation connected to a Spark cluster. I setup Zeppelin to submit jobs in yarn cluster mode and also impersonation is enabled. Now I would like to be able to use a python virtual environment instead of system one.
Is there a way I could specify the python parameter in the spark interpreter settings so is can point to specific folder use home folder (eg /home/{user_home}/python_virt_env/python) instead of a system one?

If not how should I achieve what I want?

Thank you

Manuel
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

Re: python virtual environment on spark interpreter

[Jeff Zhang <zj...@gmail.com>]

Use PYSPARK_PYTHON or spark.pyspark.python

Here's more details
https://www.zepl.com/viewer/notebooks/bm90ZTovL3pqZmZkdS9lNDYzY2NjMmRkODM0NTcwYjFiZTgwMzViMTBmNTUxZi9ub3RlLmpzb24


Manuel Sopena Ballesteros <ma...@garvan.org.au> 于2019年8月20日周二 上午7:57写道：

> Dear Zeppelin user community,
>
>
>
> I have a zeppelin installation connected to a Spark cluster. I setup
> Zeppelin to submit jobs in yarn cluster mode and also impersonation is
> enabled. Now I would like to be able to use a python virtual environment
> instead of system one.
>
> Is there a way I could specify the python parameter in the spark
> interpreter settings so is can point to specific folder use home folder (eg
> /home/{user_home}/python_virt_env/python) instead of a system one?
>
>
>
> If not how should I achieve what I want?
>
>
>
> Thank you
>
>
>
> Manuel
> NOTICE
> Please consider the environment before printing this email. This message
> and any attachments are intended for the addressee named and may contain
> legally privileged/confidential/copyright information. If you are not the
> intended recipient, you should not read, use, disclose, copy or distribute
> this communication. If you have received this message in error please
> notify us at once by return email and then delete both messages. We accept
> no liability for the distribution of viruses or similar in electronic
> communications. This notice should not be removed.
>


-- 
Best Regards

Jeff Zhang