You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Cindy Rottinghuis <ci...@gmail.com> on 2018/05/16 03:36:57 UTC
Python3 and sensors module
Hi,
Are there any plans to update the HDFS_hook.py script to remove the reference to the snakebite python library? I’d like to run airflow on python3, and this is causing some issues. The hdfs_hook script is referenced in the sensors module.
Any suggestions?
Thanks,
Cindy
Re: Python3 and sensors module
Posted by Joy Gao <jo...@wepay.com>.
Not a full patch to replace snakebite with libhdfs, but should temporarily
unblock ci/cd: https://github.com/apache/incubator-airflow/pull/3365
On Wed, May 16, 2018 at 10:00 AM, Bolke de Bruin <bd...@gmail.com> wrote:
> Snakebite is not py3 compatible. We should move to libhdfs. Happy to take
> patches!
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> > Op 16 mei 2018 om 18:57 heeft Cindy Rottinghuis <
> cindyrottinghuis@gmail.com> het volgende geschreven:
> >
> > Here is a sample of what I am seeing.
> > I am running on ubuntu, python 3.5. This is a subset of the full
> message. Once it went into the snakebite package, checked and the library
> wasn’t supported in python3.
> > ….
> > File "/usr/local/airflow/dags/dim_date.py", line 5, in <module>
> > from airflow.operators.sensors import S3KeySensor
> > File "/usr/local/lib/python3.5/dist-packages/airflow/operators/sensors.py",
> line 34, in <module>
> > from airflow.hooks.hdfs_hook import HDFSHook
> > File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/hdfs_hook.py",
> line 20, in <module>
> > from snakebite.client import Client, HAClient, Namenode,
> AutoConfigClient
> > File "/usr/local/lib/python3.5/dist-packages/snakebite/client.py",
> line 1473
> > baseTime = min(time * (1L << retries), cap);
> > ^
> >
> >
> >> On May 16, 2018, at 9:24 AM, Cindy Rottinghuis <
> cindyrottinghuis@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> Yes, it looks like all of the other sensors will work under Python3,
> but not hdfs. I am planning to use the s3 sensor, which is wrapped up in
> the sensors.py module. My issue is that when I test my dag or airflow
> installation under python3, I get errors about the hdfs_hook, which I’m not
> using. Other then creating my own version of the sensors.py file and
> removing the hdfs related functions/libraries, is there any thing else I
> can do to work around this?
> >>
> >>
> >>
> >>> On May 16, 2018, at 1:33 AM, Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
> >>>
> >>> Hi Cindy,
> >>>
> >>> The other sensors should work under Python3. We try to support Python3
> as
> >>> much as possible, but sometimes libraries are used that are not
> compatible.
> >>> Could you describe what you are running into?
> >>>
> >>> Cheers, Fokko
> >>>
> >>> 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis <
> cindyrottinghuis@gmail.com>:
> >>>
> >>>> Hi,
> >>>>
> >>>> Are there any plans to update the HDFS_hook.py script to remove the
> >>>> reference to the snakebite python library? I’d like to run airflow on
> >>>> python3, and this is causing some issues. The hdfs_hook script is
> >>>> referenced in the sensors module.
> >>>>
> >>>> Any suggestions?
> >>>>
> >>>> Thanks,
> >>>> Cindy
> >>
> >
>
>
Re: Python3 and sensors module
Posted by Bolke de Bruin <bd...@gmail.com>.
Snakebite is not py3 compatible. We should move to libhdfs. Happy to take patches!
B.
Verstuurd vanaf mijn iPad
> Op 16 mei 2018 om 18:57 heeft Cindy Rottinghuis <ci...@gmail.com> het volgende geschreven:
>
> Here is a sample of what I am seeing.
> I am running on ubuntu, python 3.5. This is a subset of the full message. Once it went into the snakebite package, checked and the library wasn’t supported in python3.
> ….
> File "/usr/local/airflow/dags/dim_date.py", line 5, in <module>
> from airflow.operators.sensors import S3KeySensor
> File "/usr/local/lib/python3.5/dist-packages/airflow/operators/sensors.py", line 34, in <module>
> from airflow.hooks.hdfs_hook import HDFSHook
> File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/hdfs_hook.py", line 20, in <module>
> from snakebite.client import Client, HAClient, Namenode, AutoConfigClient
> File "/usr/local/lib/python3.5/dist-packages/snakebite/client.py", line 1473
> baseTime = min(time * (1L << retries), cap);
> ^
>
>
>> On May 16, 2018, at 9:24 AM, Cindy Rottinghuis <ci...@gmail.com> wrote:
>>
>> Hi,
>>
>> Yes, it looks like all of the other sensors will work under Python3, but not hdfs. I am planning to use the s3 sensor, which is wrapped up in the sensors.py module. My issue is that when I test my dag or airflow installation under python3, I get errors about the hdfs_hook, which I’m not using. Other then creating my own version of the sensors.py file and removing the hdfs related functions/libraries, is there any thing else I can do to work around this?
>>
>>
>>
>>> On May 16, 2018, at 1:33 AM, Driesprong, Fokko <fo...@driesprong.frl> wrote:
>>>
>>> Hi Cindy,
>>>
>>> The other sensors should work under Python3. We try to support Python3 as
>>> much as possible, but sometimes libraries are used that are not compatible.
>>> Could you describe what you are running into?
>>>
>>> Cheers, Fokko
>>>
>>> 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis <ci...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> Are there any plans to update the HDFS_hook.py script to remove the
>>>> reference to the snakebite python library? I’d like to run airflow on
>>>> python3, and this is causing some issues. The hdfs_hook script is
>>>> referenced in the sensors module.
>>>>
>>>> Any suggestions?
>>>>
>>>> Thanks,
>>>> Cindy
>>
>
Re: Python3 and sensors module
Posted by Cindy Rottinghuis <ci...@gmail.com>.
Here is a sample of what I am seeing.
I am running on ubuntu, python 3.5. This is a subset of the full message. Once it went into the snakebite package, checked and the library wasn’t supported in python3.
….
File "/usr/local/airflow/dags/dim_date.py", line 5, in <module>
from airflow.operators.sensors import S3KeySensor
File "/usr/local/lib/python3.5/dist-packages/airflow/operators/sensors.py", line 34, in <module>
from airflow.hooks.hdfs_hook import HDFSHook
File "/usr/local/lib/python3.5/dist-packages/airflow/hooks/hdfs_hook.py", line 20, in <module>
from snakebite.client import Client, HAClient, Namenode, AutoConfigClient
File "/usr/local/lib/python3.5/dist-packages/snakebite/client.py", line 1473
baseTime = min(time * (1L << retries), cap);
^
> On May 16, 2018, at 9:24 AM, Cindy Rottinghuis <ci...@gmail.com> wrote:
>
> Hi,
>
> Yes, it looks like all of the other sensors will work under Python3, but not hdfs. I am planning to use the s3 sensor, which is wrapped up in the sensors.py module. My issue is that when I test my dag or airflow installation under python3, I get errors about the hdfs_hook, which I’m not using. Other then creating my own version of the sensors.py file and removing the hdfs related functions/libraries, is there any thing else I can do to work around this?
>
>
>
>> On May 16, 2018, at 1:33 AM, Driesprong, Fokko <fo...@driesprong.frl> wrote:
>>
>> Hi Cindy,
>>
>> The other sensors should work under Python3. We try to support Python3 as
>> much as possible, but sometimes libraries are used that are not compatible.
>> Could you describe what you are running into?
>>
>> Cheers, Fokko
>>
>> 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis <ci...@gmail.com>:
>>
>>> Hi,
>>>
>>> Are there any plans to update the HDFS_hook.py script to remove the
>>> reference to the snakebite python library? I’d like to run airflow on
>>> python3, and this is causing some issues. The hdfs_hook script is
>>> referenced in the sensors module.
>>>
>>> Any suggestions?
>>>
>>> Thanks,
>>> Cindy
>
Re: Python3 and sensors module
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi Cindy,
The other sensors should work under Python3. We try to support Python3 as
much as possible, but sometimes libraries are used that are not compatible.
Could you describe what you are running into?
Cheers, Fokko
2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis <ci...@gmail.com>:
> Hi,
>
> Are there any plans to update the HDFS_hook.py script to remove the
> reference to the snakebite python library? I’d like to run airflow on
> python3, and this is causing some issues. The hdfs_hook script is
> referenced in the sensors module.
>
> Any suggestions?
>
> Thanks,
> Cindy