You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by zhong jiajie <zh...@hotmail.com> on 2019/02/17 07:42:43 UTC
答复: [DISCUSS] AIP-3: Drop Python2 support in Airflow 2.0
I overlook the PR https://github.com/apache/airflow/pull/3560 still use **hdfs3** rather than **PyArrow**. So which package we choice finilly?
[https://avatars3.githubusercontent.com/u/307739?s=400&v=4]<https://github.com/apache/airflow/pull/3560>
[AIRFLOW-2697] Drop snakebite in favour of hdfs3 by jrderuiter ・ Pull Request #3560 ・ apache/airflow ・ GitHub<https://github.com/apache/airflow/pull/3560>
Commit #52a2ff3 changes the HdfsSensor classes for the new HdfsHook. The biggest change is that it rewrites the HdfsSensor + HdfsSensorFolder (from contrib) sensors into a HdfsFileSensor and a HdfsFolderSensor class, which aim to retain the functionality of the previous classes.
github.com
________________________________
One blocker is HDFS support. Right now we rely on Snakebite which is
Python2 only, so we need to update this as well. There is an open ticket:
https://jira.apache.org/jira/browse/AIRFLOW-2697
Op vr 15 feb. 2019 om 07:55 schreef zhong jiajie <zhongjiajie955@hotmail.com
>:
> Agree with XD's second idea, we should warn Airflow users.
>
> ________________________________
>
> My two cents:
>
> 1. Can we discuss on the timeline for 2.0? Another big change is FAB-UI
> ONLY, which is already in master branch. Personally I think it’s already
> worth considering 2.0, unless there is other existing expectation for 2.0.
>
> 2. Before we completely drop Python2 support, we can start adding
> warning/reminder mentioning that it’s stronly recommended to use py3 & py2
> drooping can happen “at any time” for Airflow (we can add these warnings
> in 1.10.3?)
>
>
> XD
>
> On Thu, Feb 14, 2019 at 18:42 Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > We talked about this in the past <
> >
> http://mail-archives.apache.org/mod_mbox/airflow-dev/201808.mbox/%3CCAK%2Be0oujpY1A0mD0G775CmmRneVDjQMTi99aqK%2BA_C07ZW8ySA%40mail.gmail.com%3E
> >
> > but I'd like to bring it up again with a more concrete proposal.
> >
> > At the time Airflow 2.0 wasn't very concrete, but now with a lot of
> > otherwise breaking changes already on master how about we use that time
> to
> > also drop support for Py2?
> >
> > (Reminder: Python 2 is reaching End of Life in January 1, 2020 and will
> > receive zero updates, even security ones past this date. At least not by
> > the Python team. I imagine RedHat will still be running Python 2.6 in
> 2120
> > ;) )
> >
> > -ash
>
Re: [DISCUSS] AIP-3: Drop Python2 support in Airflow 2.0
Posted by airflowuser <ai...@protonmail.com.INVALID>.
This ticket also suggest function that doesn't support Python 3
https://issues.apache.org/jira/browse/AIRFLOW-2243
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Sunday, February 24, 2019 9:25 PM, Bas Harenslak <ba...@godatadriven.com> wrote:
> Following the comments to warn users about dropping Python 2, I went ahead and added the deprecation library to Airflow in this PR: https://github.com/apache/airflow/pull/4763, and added the first @deprecated annotation.
>
> I checked a few major packages and all use some self-made annotation for showing deprecation warnings. I didn’t want to re-invent the wheel and used a tiny package for it. More details are in the PR itself. If you know a better way to do this, let me know :-)
>
> Bas
>
> On 17 Feb 2019, at 09:00, Bas Harenslak <ba...@godatadriven.com> wrote:
>
> The PR comments say drop hdfs3 and use pyarrow, however this is not implemented yet.
>
> On 17 Feb 2019, at 08:42, zhong jiajie <zh...@hotmail.com> wrote:
>
> I overlook the PRhttps://github.com/apache/airflow/pull/3560 still use hdfs3 rather than PyArrow. So which package we choice finilly?
> [https://avatars3.githubusercontent.com/u/307739?s=400&v=4]https://github.com/apache/airflow/pull/3560
>
> [AIRFLOW-2697] Drop snakebite in favour of hdfs3 by jrderuiter ・ Pull Request #3560 ・ apache/airflow ・ GitHubhttps://github.com/apache/airflow/pull/3560
> Commit #52a2ff3 changes the HdfsSensor classes for the new HdfsHook. The biggest change is that it rewrites the HdfsSensor + HdfsSensorFolder (from contrib) sensors into a HdfsFileSensor and a HdfsFolderSensor class, which aim to retain the functionality of the previous classes.
> github.comhttp://github.com
>
> One blocker is HDFS support. Right now we rely on Snakebite which is
> Python2 only, so we need to update this as well. There is an open ticket:
> https://jira.apache.org/jira/browse/AIRFLOW-2697
>
> Op vr 15 feb. 2019 om 07:55 schreef zhong jiajie <zhongjiajie955@hotmail.com
> :
>
> Agree with XD's second idea, we should warn Airflow users.
>
> My two cents:
>
> 1. Can we discuss on the timeline for 2.0? Another big change is FAB-UI
> ONLY, which is already in master branch. Personally I think it’s already
> worth considering 2.0, unless there is other existing expectation for 2.0.
>
> 2. Before we completely drop Python2 support, we can start adding
> warning/reminder mentioning that it’s stronly recommended to use py3 & py2
> drooping can happen “at any time” for Airflow (we can add these warnings
> in 1.10.3?)
>
> XD
>
> On Thu, Feb 14, 2019 at 18:42 Ash Berlin-Taylor ash@apache.org wrote:
>
>
> We talked about this in the past <
>
> http://mail-archives.apache.org/mod_mbox/airflow-dev/201808.mbox/<CAK%2Be0oujpY1A0mD0G775CmmRneVDjQMTi99aqK%2BA_C07ZW8ySA%40mail.gmail.com>
>
> but I'd like to bring it up again with a more concrete proposal.
>
> At the time Airflow 2.0 wasn't very concrete, but now with a lot of
> otherwise breaking changes already on master how about we use that time
> to
> also drop support for Py2?
>
> (Reminder: Python 2 is reaching End of Life in January 1, 2020 and will
> receive zero updates, even security ones past this date. At least not by
> the Python team. I imagine RedHat will still be running Python 2.6 in
> 2120
> ;) )
>
> -ash
Re: [DISCUSS] AIP-3: Drop Python2 support in Airflow 2.0
Posted by Bas Harenslak <ba...@godatadriven.com>.
Following the comments to warn users about dropping Python 2, I went ahead and added the deprecation library to Airflow in this PR: https://github.com/apache/airflow/pull/4763, and added the first @deprecated annotation.
I checked a few major packages and all use some self-made annotation for showing deprecation warnings. I didn’t want to re-invent the wheel and used a tiny package for it. More details are in the PR itself. If you know a better way to do this, let me know :-)
Bas
On 17 Feb 2019, at 09:00, Bas Harenslak <ba...@godatadriven.com>> wrote:
The PR comments say drop hdfs3 and use pyarrow, however this is not implemented yet.
On 17 Feb 2019, at 08:42, zhong jiajie <zh...@hotmail.com>> wrote:
I overlook the PR https://github.com/apache/airflow/pull/3560 still use **hdfs3** rather than **PyArrow**. So which package we choice finilly?
[https://avatars3.githubusercontent.com/u/307739?s=400&v=4]<https://github.com/apache/airflow/pull/3560>
[AIRFLOW-2697] Drop snakebite in favour of hdfs3 by jrderuiter ・ Pull Request #3560 ・ apache/airflow ・ GitHub<https://github.com/apache/airflow/pull/3560>
Commit #52a2ff3 changes the HdfsSensor classes for the new HdfsHook. The biggest change is that it rewrites the HdfsSensor + HdfsSensorFolder (from contrib) sensors into a HdfsFileSensor and a HdfsFolderSensor class, which aim to retain the functionality of the previous classes.
github.com<http://github.com>
________________________________
One blocker is HDFS support. Right now we rely on Snakebite which is
Python2 only, so we need to update this as well. There is an open ticket:
https://jira.apache.org/jira/browse/AIRFLOW-2697
Op vr 15 feb. 2019 om 07:55 schreef zhong jiajie <zhongjiajie955@hotmail.com
:
Agree with XD's second idea, we should warn Airflow users.
________________________________
My two cents:
1. Can we discuss on the timeline for 2.0? Another big change is FAB-UI
ONLY, which is already in master branch. Personally I think it’s already
worth considering 2.0, unless there is other existing expectation for 2.0.
2. Before we completely drop Python2 support, we can start adding
warning/reminder mentioning that it’s stronly recommended to use py3 & py2
drooping can happen “at any time” for Airflow (we can add these warnings
in 1.10.3?)
XD
On Thu, Feb 14, 2019 at 18:42 Ash Berlin-Taylor <as...@apache.org> wrote:
We talked about this in the past <
http://mail-archives.apache.org/mod_mbox/airflow-dev/201808.mbox/%3CCAK%2Be0oujpY1A0mD0G775CmmRneVDjQMTi99aqK%2BA_C07ZW8ySA%40mail.gmail.com%3E
but I'd like to bring it up again with a more concrete proposal.
At the time Airflow 2.0 wasn't very concrete, but now with a lot of
otherwise breaking changes already on master how about we use that time
to
also drop support for Py2?
(Reminder: Python 2 is reaching End of Life in January 1, 2020 and will
receive zero updates, even security ones past this date. At least not by
the Python team. I imagine RedHat will still be running Python 2.6 in
2120
;) )
-ash
Re: [DISCUSS] AIP-3: Drop Python2 support in Airflow 2.0
Posted by Bas Harenslak <ba...@godatadriven.com>.
The PR comments say drop hdfs3 and use pyarrow, however this is not implemented yet.
> On 17 Feb 2019, at 08:42, zhong jiajie <zh...@hotmail.com> wrote:
>
> I overlook the PR https://github.com/apache/airflow/pull/3560 still use **hdfs3** rather than **PyArrow**. So which package we choice finilly?
> [https://avatars3.githubusercontent.com/u/307739?s=400&v=4]<https://github.com/apache/airflow/pull/3560>
>
> [AIRFLOW-2697] Drop snakebite in favour of hdfs3 by jrderuiter ・ Pull Request #3560 ・ apache/airflow ・ GitHub<https://github.com/apache/airflow/pull/3560>
> Commit #52a2ff3 changes the HdfsSensor classes for the new HdfsHook. The biggest change is that it rewrites the HdfsSensor + HdfsSensorFolder (from contrib) sensors into a HdfsFileSensor and a HdfsFolderSensor class, which aim to retain the functionality of the previous classes.
> github.com
>
>
>
> ________________________________
>
> One blocker is HDFS support. Right now we rely on Snakebite which is
> Python2 only, so we need to update this as well. There is an open ticket:
> https://jira.apache.org/jira/browse/AIRFLOW-2697
>
> Op vr 15 feb. 2019 om 07:55 schreef zhong jiajie <zhongjiajie955@hotmail.com
>> :
>
>> Agree with XD's second idea, we should warn Airflow users.
>>
>> ________________________________
>>
>> My two cents:
>>
>> 1. Can we discuss on the timeline for 2.0? Another big change is FAB-UI
>> ONLY, which is already in master branch. Personally I think it’s already
>> worth considering 2.0, unless there is other existing expectation for 2.0.
>>
>> 2. Before we completely drop Python2 support, we can start adding
>> warning/reminder mentioning that it’s stronly recommended to use py3 & py2
>> drooping can happen “at any time” for Airflow (we can add these warnings
>> in 1.10.3?)
>>
>>
>> XD
>>
>> On Thu, Feb 14, 2019 at 18:42 Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>>> We talked about this in the past <
>>>
>> http://mail-archives.apache.org/mod_mbox/airflow-dev/201808.mbox/%3CCAK%2Be0oujpY1A0mD0G775CmmRneVDjQMTi99aqK%2BA_C07ZW8ySA%40mail.gmail.com%3E
>>>
>>> but I'd like to bring it up again with a more concrete proposal.
>>>
>>> At the time Airflow 2.0 wasn't very concrete, but now with a lot of
>>> otherwise breaking changes already on master how about we use that time
>> to
>>> also drop support for Py2?
>>>
>>> (Reminder: Python 2 is reaching End of Life in January 1, 2020 and will
>>> receive zero updates, even security ones past this date. At least not by
>>> the Python team. I imagine RedHat will still be running Python 2.6 in
>> 2120
>>> ;) )
>>>
>>> -ash
>>