You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Msr Msr <ms...@gmail.com> on 2016/06/23 15:15:02 UTC
BashOperator strage behavior
Hi,
I installed airflow<python3> on my laptop and using BashOperator to login
to remote EC2 server and submit a python program. On my Ec2 machine I have
both Python2 <2.7.6> and Python3 <3.5.1>.
If I login manually to Ec2 instance and type just "python" it opens
Python3 shell.
But when I execute through BashOperator command using airflow it is going
to Python2.
Please see below command and output. The first command "python --version"
output is "{bash_operator.py:77} INFO - Python 2.7.6". for Python3
--version it gives "Python 3.4.3 " But I have 3.5.1
This is causing a bigger problem because I need to execute a Jupiter
notebook that has to process utf-8 chars. my notebook is in python3.
Python2 processing of same notebook gives utf-8 char related error and
fixing that is not possible from my side.
When I tried to execute the python program from BashOperator using
something like "python3 abc.py" it still uses python2 some where internally
so utf-8 char related error comes.
Please suggest me or point me out what I am missing. I have been trying
different options but didn't get any progress. Any help would be highly
appreciated.
Please see below log:
sparkcmd = """
ssh -i /users/msr/Downloads/EC2/abc.pem
ubuntu@ec2-xx-yyy-zz-aaa.us-west-1.compute.amazonaws.com 'python --version;
python3 --version;'
"""
t2 = BashOperator(
task_id='ClassifyNews',
bash_command=sparkcmd,
retries=3,
dag=dag)
[2016-06-23 07:50:25,522] {models.py:1219} INFO - Executing
<Task(BashOperator): ClassifyNews> on 2016-05-04 00:00:00
[2016-06-23 07:50:25,535] {bash_operator.py:55} INFO - tmp dir root
location:
/var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T
[2016-06-23 07:50:25,536] {bash_operator.py:64} INFO - Temporary script
location
:/var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T/airflowtmpa1cd9t9s//var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T/airflowtmpa1cd9t9s/ClassifyNewsgwx9mtr5
[2016-06-23 07:50:25,536] {bash_operator.py:65} INFO - Running command:
ssh -i /users/msr/Downloads/EC2/abc.pem
ubuntu@ec2-xx-yyy-zz-aaa.us-west-1.compute.amazonaws.com 'python --version;
python3 --version;'
[2016-06-23 07:50:25,539] {bash_operator.py:73} INFO - Output:
[2016-06-23 07:50:26,489] {bash_operator.py:77} INFO - Python 2.7.6
[2016-06-23 07:50:26,489] {bash_operator.py:77} INFO - Python 3.4.3
[2016-06-23 07:50:26,491] {bash_operator.py:80} INFO - Command exited with
return code 0
Re: BashOperator strage behavior
Posted by MSR M <ms...@gmail.com>.
Hi,
Please ignore this.
This is happening because remote EC2 has problem with some symbolic links
pointing python.
I have to give the full path of python in Bash command.
Thanks,
Madhu
On Thu, Jun 23, 2016 at 8:15 AM, Msr Msr <ms...@gmail.com> wrote:
> Hi,
>
> I installed airflow<python3> on my laptop and using BashOperator to login
> to remote EC2 server and submit a python program. On my Ec2 machine I have
> both Python2 <2.7.6> and Python3 <3.5.1>.
>
> If I login manually to Ec2 instance and type just "python" it opens
> Python3 shell.
>
> But when I execute through BashOperator command using airflow it is going
> to Python2.
>
> Please see below command and output. The first command "python --version"
> output is "{bash_operator.py:77} INFO - Python 2.7.6". for Python3
> --version it gives "Python 3.4.3 " But I have 3.5.1
>
> This is causing a bigger problem because I need to execute a Jupiter
> notebook that has to process utf-8 chars. my notebook is in python3.
> Python2 processing of same notebook gives utf-8 char related error and
> fixing that is not possible from my side.
>
> When I tried to execute the python program from BashOperator using
> something like "python3 abc.py" it still uses python2 some where internally
> so utf-8 char related error comes.
>
> Please suggest me or point me out what I am missing. I have been trying
> different options but didn't get any progress. Any help would be highly
> appreciated.
>
> Please see below log:
>
> sparkcmd = """
> ssh -i /users/msr/Downloads/EC2/abc.pem
> ubuntu@ec2-xx-yyy-zz-aaa.us-west-1.compute.amazonaws.com 'python
> --version; python3 --version;'
> """
>
> t2 = BashOperator(
> task_id='ClassifyNews',
> bash_command=sparkcmd,
> retries=3,
> dag=dag)
>
> [2016-06-23 07:50:25,522] {models.py:1219} INFO - Executing
> <Task(BashOperator): ClassifyNews> on 2016-05-04 00:00:00
> [2016-06-23 07:50:25,535] {bash_operator.py:55} INFO - tmp dir root
> location:
> /var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T
> [2016-06-23 07:50:25,536] {bash_operator.py:64} INFO - Temporary script
> location
> :/var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T/airflowtmpa1cd9t9s//var/folders/3x/2x0p55_n0x39fz28zq4drsnc0000gn/T/airflowtmpa1cd9t9s/ClassifyNewsgwx9mtr5
> [2016-06-23 07:50:25,536] {bash_operator.py:65} INFO - Running command:
> ssh -i /users/msr/Downloads/EC2/abc.pem
> ubuntu@ec2-xx-yyy-zz-aaa.us-west-1.compute.amazonaws.com 'python --version;
>
> python3 --version;'
>
> [2016-06-23 07:50:25,539] {bash_operator.py:73} INFO - Output:
> [2016-06-23 07:50:26,489] {bash_operator.py:77} INFO - Python 2.7.6
> [2016-06-23 07:50:26,489] {bash_operator.py:77} INFO - Python 3.4.3
> [2016-06-23 07:50:26,491] {bash_operator.py:80} INFO - Command exited with
> return code 0
>