[jira] [Resolved] (AIRFLOW-4367) When extending BaseSensorOperator getting exception in python 3

["Ash Berlin-Taylor (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ash Berlin-Taylor resolved AIRFLOW-4367.
----------------------------------------
    Resolution: Won't Fix

It is not recommended to import from airflow.sensors directly (and this style will go away on Airflow 2.0). Instead you should do something like:

{code:python}
from airflow.sensors.base_sensor_operator import BaseSensorOperator
{code}

That would not end up not trying to import all the sensors but just the ones that you specifically use.

(AIRFLOW-2697 will fix the specific error by replacing snakebite with a maintained Py3-compatible version)

> When extending BaseSensorOperator getting exception in python 3 
> ----------------------------------------------------------------
>
>                 Key: AIRFLOW-4367
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4367
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: plugins
>    Affects Versions: 1.10.3
>         Environment: python 3, airflow 1.10.3
>            Reporter: Nedko Hristov
>            Priority: Major
>
> When creating a custom sensor in 1.10.3 (EMRTaskSensor) I extend BaseSensorOperator (e.g. from airflow.operators.sensors import BaseSensorOperator) which in turn initializes all sensors including HdfsSensor which uses snakebite (not python 3 compatible) I get the following stacktrace due to the 1L implicits in Python 3. The same works as expected in 1.10.1 as I've tested it and have been running it for a while. Now, I know that snakebite is not python 3 compatible. I also know that I'm not using HDFS so not sure why the need to initialize it by default. A workaround will be to remove snakebite and use snakebite-py3 post install in my docker container but I'd prefer if there was a smarter way to not break my instance with features I don't use. Possible change the modules.py where things get loaded...
>  File "/git/airflow/dag/dp_etl_flow.py", line 8, in <module>
>  from plugins.sc_data_runner.sensors.emr_task_sensor import EMRTaskSensor
>  File "/git/airflow/plugins/sc_data_runner/sensors/emr_task_sensor.py", line 3, in <module>
>  from airflow.operators.sensors import BaseSensorOperator
>  File "/usr/local/airflow/.local/lib/python3.6/site-packages/airflow/operators/sensors.py", line 35, in <module>
>  from airflow.sensors.hdfs_sensor import HdfsSensor as HdfsSensorImp
>  File "/usr/local/airflow/.local/lib/python3.6/site-packages/airflow/sensors/hdfs_sensor.py", line 25, in <module>
>  from airflow.hooks.hdfs_hook import HDFSHook
>  File "/usr/local/airflow/.local/lib/python3.6/site-packages/airflow/hooks/hdfs_hook.py", line 24, in <module>
>  from snakebite.client import Client, HAClient, Namenode, AutoConfigClient
>  File "/usr/local/lib/python3.6/site-packages/snakebite/client.py", line 1473
>  baseTime = min(time * (1L << retries), cap);



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)