You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Ia...@tdameritrade.com on 2016/11/17 22:22:33 UTC
Question about python scripts
Hi,
I’m currently running into an issue, where some of the Ambari related python scripts are being invoked from a python 3 environment:
/usr/bin/hdp-select
/etc/hadoop/conf/topology_script.py
The reason this happens is a bit complicated, I’m running jupyterhub, which requires a python 3 environment on my management node, where ambari is located. Ambari uses python 2 currently. For most kernels, it works fine, but Apache Toree, is for some reason calling those scripts from within the python 3 environment. Since the scripts use /usr/bin/env python, when they are in the conda environment for jupyterhub, they use python 3 and break. So, what I have done is update the scripts to be usable in both python 2 and 3. Unfortunately, it appears these scripts are recreated when the cluster goes down (or is built), so I’d have to swap the files out.
What I’m proposing is that my changes for the scripts (or someone else’s), get committed, so with future versions of Ambari, there are not issues of cross compatibility.
How does this sound?
Re: Question about python scripts
Posted by Ia...@tdameritrade.com.
I’m running Jupyterhub, which requires python 3. When I start up my spark
notebooks (which use yarn-client as the master),
those scripts are called and they use "/usr/bin/env python”. Since they’re
regenerated, I would like to find away to change
that (the first line in each file) to “/usr/bin/python”.
I would be happy to try to create a patch, but am unfamiliar with the code
base.
On 11/30/16, 2:21 PM, "Alejandro Fernandez" <af...@hortonworks.com>
wrote:
>Hi Ian,
>
>We would need more information about where that version of python is being
>invoked.
>You're welcome to submit a patch.
>
>Thanks,
>Alejandro
>
>
>On 11/29/16, 2:33 PM, "Ian.Maloney@tdameritrade.com"
><Ia...@tdameritrade.com> wrote:
>
>>Hi Alejandro,
>>
>>Sorry for the delay. I looked at that script and it didn’t seem like it
>>would allow for me to force the right python. What I need is something
>>like where the scripts get generated, I just need to change the top line
>>of the scripts below, but since they’re regenerated when services are
>>stopped, I need to do it programmatically. Maybe this could be made as a
>>property, which is optionally configurable?
>>
>>
>>/usr/bin/hdp-select
>>/etc/hadoop/conf/topology_script.py
>>
>>
>>
>>Thanks,
>>
>>Ian
>>
>>
>>
>>
>>
>>
>>
>>On 11/18/16, 6:44 PM, "Alejandro Fernandez" <af...@hortonworks.com>
>>wrote:
>>
>>>Hi Ian,
>>>
>>>Ambari currently supports Python 2.6 and 2.7.
>>>Take a look at
>>>https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_a
>>>m
>>>b
>>>ari_blob_trunk_ambari-2Dshell_ambari-2Dpython-2Dshel&d=DgIFAw&c=nulvIAQn
>>>C
>>>0
>>>yOOjC0e0NVa8TOcyq9jNhjZ156R-JJU10&r=CxpqDYMuQy-1uNI-UOyUbaX6BMPCZXH8d8ev
>>>u
>>>C
>>>oP_OA&m=9yDxGmbR83EbujEvcB36vHJPNAosEHXd763_ySz8hk0&s=PbZ99YTBAIx1e3-OmT
>>>8
>>>-
>>>6HZbd9isTV8yXA4SH0Tcxi4&e=
>>>l/conf/unix/ambari-shell to see if you can force it to always use Python
>>>2.#
>>>
>>>Thanks,
>>>Alejandro
>>>
>>>On 11/17/16, 2:22 PM, "Ian.Maloney@tdameritrade.com"
>>><Ia...@tdameritrade.com> wrote:
>>>
>>>>Hi,
>>>>
>>>>I¹m currently running into an issue, where some of the Ambari related
>>>>python scripts are being invoked from a python 3 environment:
>>>>
>>>>/usr/bin/hdp-select
>>>>/etc/hadoop/conf/topology_script.py
>>>>
>>>>The reason this happens is a bit complicated, I¹m running jupyterhub,
>>>>which requires a python 3 environment on my management node, where
>>>>ambari
>>>>is located. Ambari uses python 2 currently. For most kernels, it works
>>>>fine, but Apache Toree, is for some reason calling those scripts from
>>>>within the python 3 environment. Since the scripts use /usr/bin/env
>>>>python, when they are in the conda environment for jupyterhub, they use
>>>>python 3 and break. So, what I have done is update the scripts to be
>>>>usable in both python 2 and 3. Unfortunately, it appears these scripts
>>>>are recreated when the cluster goes down (or is built), so I¹d have to
>>>>swap the files out.
>>>>
>>>>What I¹m proposing is that my changes for the scripts (or someone
>>>>else¹s), get committed, so with future versions of Ambari, there are
>>>>not
>>>>issues of cross compatibility.
>>>>
>>>>How does this sound?
>>>>
>>>>
>>>>
>>>
>>
>
Re: Question about python scripts
Posted by Alejandro Fernandez <af...@hortonworks.com>.
Hi Ian,
We would need more information about where that version of python is being
invoked.
You're welcome to submit a patch.
Thanks,
Alejandro
On 11/29/16, 2:33 PM, "Ian.Maloney@tdameritrade.com"
<Ia...@tdameritrade.com> wrote:
>Hi Alejandro,
>
>Sorry for the delay. I looked at that script and it didn’t seem like it
>would allow for me to force the right python. What I need is something
>like where the scripts get generated, I just need to change the top line
>of the scripts below, but since they’re regenerated when services are
>stopped, I need to do it programmatically. Maybe this could be made as a
>property, which is optionally configurable?
>
>
>/usr/bin/hdp-select
>/etc/hadoop/conf/topology_script.py
>
>
>
>Thanks,
>
>Ian
>
>
>
>
>
>
>
>On 11/18/16, 6:44 PM, "Alejandro Fernandez" <af...@hortonworks.com>
>wrote:
>
>>Hi Ian,
>>
>>Ambari currently supports Python 2.6 and 2.7.
>>Take a look at
>>https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_am
>>b
>>ari_blob_trunk_ambari-2Dshell_ambari-2Dpython-2Dshel&d=DgIFAw&c=nulvIAQnC
>>0
>>yOOjC0e0NVa8TOcyq9jNhjZ156R-JJU10&r=CxpqDYMuQy-1uNI-UOyUbaX6BMPCZXH8d8evu
>>C
>>oP_OA&m=9yDxGmbR83EbujEvcB36vHJPNAosEHXd763_ySz8hk0&s=PbZ99YTBAIx1e3-OmT8
>>-
>>6HZbd9isTV8yXA4SH0Tcxi4&e=
>>l/conf/unix/ambari-shell to see if you can force it to always use Python
>>2.#
>>
>>Thanks,
>>Alejandro
>>
>>On 11/17/16, 2:22 PM, "Ian.Maloney@tdameritrade.com"
>><Ia...@tdameritrade.com> wrote:
>>
>>>Hi,
>>>
>>>I¹m currently running into an issue, where some of the Ambari related
>>>python scripts are being invoked from a python 3 environment:
>>>
>>>/usr/bin/hdp-select
>>>/etc/hadoop/conf/topology_script.py
>>>
>>>The reason this happens is a bit complicated, I¹m running jupyterhub,
>>>which requires a python 3 environment on my management node, where
>>>ambari
>>>is located. Ambari uses python 2 currently. For most kernels, it works
>>>fine, but Apache Toree, is for some reason calling those scripts from
>>>within the python 3 environment. Since the scripts use /usr/bin/env
>>>python, when they are in the conda environment for jupyterhub, they use
>>>python 3 and break. So, what I have done is update the scripts to be
>>>usable in both python 2 and 3. Unfortunately, it appears these scripts
>>>are recreated when the cluster goes down (or is built), so I¹d have to
>>>swap the files out.
>>>
>>>What I¹m proposing is that my changes for the scripts (or someone
>>>else¹s), get committed, so with future versions of Ambari, there are not
>>>issues of cross compatibility.
>>>
>>>How does this sound?
>>>
>>>
>>>
>>
>
Re: Question about python scripts
Posted by Ia...@tdameritrade.com.
Hi Alejandro,
Sorry for the delay. I looked at that script and it didn’t seem like it
would allow for me to force the right python. What I need is something
like where the scripts get generated, I just need to change the top line
of the scripts below, but since they’re regenerated when services are
stopped, I need to do it programmatically. Maybe this could be made as a
property, which is optionally configurable?
/usr/bin/hdp-select
/etc/hadoop/conf/topology_script.py
Thanks,
Ian
On 11/18/16, 6:44 PM, "Alejandro Fernandez" <af...@hortonworks.com>
wrote:
>Hi Ian,
>
>Ambari currently supports Python 2.6 and 2.7.
>Take a look at
>https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_amb
>ari_blob_trunk_ambari-2Dshell_ambari-2Dpython-2Dshel&d=DgIFAw&c=nulvIAQnC0
>yOOjC0e0NVa8TOcyq9jNhjZ156R-JJU10&r=CxpqDYMuQy-1uNI-UOyUbaX6BMPCZXH8d8evuC
>oP_OA&m=9yDxGmbR83EbujEvcB36vHJPNAosEHXd763_ySz8hk0&s=PbZ99YTBAIx1e3-OmT8-
>6HZbd9isTV8yXA4SH0Tcxi4&e=
>l/conf/unix/ambari-shell to see if you can force it to always use Python
>2.#
>
>Thanks,
>Alejandro
>
>On 11/17/16, 2:22 PM, "Ian.Maloney@tdameritrade.com"
><Ia...@tdameritrade.com> wrote:
>
>>Hi,
>>
>>I¹m currently running into an issue, where some of the Ambari related
>>python scripts are being invoked from a python 3 environment:
>>
>>/usr/bin/hdp-select
>>/etc/hadoop/conf/topology_script.py
>>
>>The reason this happens is a bit complicated, I¹m running jupyterhub,
>>which requires a python 3 environment on my management node, where ambari
>>is located. Ambari uses python 2 currently. For most kernels, it works
>>fine, but Apache Toree, is for some reason calling those scripts from
>>within the python 3 environment. Since the scripts use /usr/bin/env
>>python, when they are in the conda environment for jupyterhub, they use
>>python 3 and break. So, what I have done is update the scripts to be
>>usable in both python 2 and 3. Unfortunately, it appears these scripts
>>are recreated when the cluster goes down (or is built), so I¹d have to
>>swap the files out.
>>
>>What I¹m proposing is that my changes for the scripts (or someone
>>else¹s), get committed, so with future versions of Ambari, there are not
>>issues of cross compatibility.
>>
>>How does this sound?
>>
>>
>>
>
Re: Question about python scripts
Posted by Alejandro Fernandez <af...@hortonworks.com>.
Hi Ian,
Ambari currently supports Python 2.6 and 2.7.
Take a look at
https://github.com/apache/ambari/blob/trunk/ambari-shell/ambari-python-shel
l/conf/unix/ambari-shell to see if you can force it to always use Python
2.#
Thanks,
Alejandro
On 11/17/16, 2:22 PM, "Ian.Maloney@tdameritrade.com"
<Ia...@tdameritrade.com> wrote:
>Hi,
>
>I¹m currently running into an issue, where some of the Ambari related
>python scripts are being invoked from a python 3 environment:
>
>/usr/bin/hdp-select
>/etc/hadoop/conf/topology_script.py
>
>The reason this happens is a bit complicated, I¹m running jupyterhub,
>which requires a python 3 environment on my management node, where ambari
>is located. Ambari uses python 2 currently. For most kernels, it works
>fine, but Apache Toree, is for some reason calling those scripts from
>within the python 3 environment. Since the scripts use /usr/bin/env
>python, when they are in the conda environment for jupyterhub, they use
>python 3 and break. So, what I have done is update the scripts to be
>usable in both python 2 and 3. Unfortunately, it appears these scripts
>are recreated when the cluster goes down (or is built), so I¹d have to
>swap the files out.
>
>What I¹m proposing is that my changes for the scripts (or someone
>else¹s), get committed, so with future versions of Ambari, there are not
>issues of cross compatibility.
>
>How does this sound?
>
>
>