You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Chris Riccomini <cr...@apache.org> on 2018/03/26 18:52:46 UTC
Re: Airflow 1.10
Hey all,
Now that the RBAC UI has been merged in, I wanted to revive this thread
again. Timezone stuff has been in for a while. I'm not sure on the status
of the K8s stuff, but I do feel it's time to start getting the ball rolling
on a 1.10 release. What do others think? Any volunteers for a release
manager? :)
Cheers,
Chris
On Tue, Jan 16, 2018 at 12:42 PM, George Leslie-Waksman <
george@cloverhealth.com.invalid> wrote:
> +1
>
> On Tue, Jan 16, 2018 at 11:08 AM Joy Gao <jo...@wepay.com> wrote:
>
> > The new FAB UI does not modify the existing API (i.e. www2/api/ will be a
> > copy of www/api/), and the endpoints are registered as blueprints to the
> > flask app the same way as before, so it is fully backward compatible.
> >
> > Although FAB offers REST APIs on the models out-of-the-box, it currently
> is
> > still in BETA and does not support any HTTP authentication scheme,
> meaning
> > it would require a cookie to mimic a normal user session in order to be
> > used. In the long run (for Airflow 2.X+), I believe the best approach is
> to
> > patch FAB with auth backends, so we can apply @has_access annotation on
> all
> > rest endpoints to streamline authentication. For 1.10, the current plan
> is
> > to leave the API as is. I wouldn't recommend folks to use the FAB-based
> > REST APIs yet. I would also like to release the new UI as an alpha
> version,
> > and wait for 2.0 before promoting it to the default version. This will
> give
> > us some time to address any new UI bugs which I overlooked.
> >
> > +1 on polishing! (With the exception of "*Rest Api should standardise and
> > have proper swagger definitions" and any other bugs that require major
> > overhaul*, which I think can wait until 2.0)
> >
> >
> > On Sun, Jan 14, 2018 at 12:18 PM, Bolke de Bruin <bd...@gmail.com>
> > wrote:
> >
> > > Chris, Joy,
> > >
> > > Can you shed some light on the backward compatibility of the new UI,
> > > particularly with regards to the API? The API for example cannot use
> the
> > > login from FAB afaik.
> > >
> > > As much of the work is already in for 1.10 I think focus should be on
> > > polishing. There are some minor quirks and slightly annoying bugs.
> > >
> > > - It seems a dag with schedule “none” can still run when turned on from
> > > the UI (unconfirmed)
> > > - Exceptions are swallowed when importing a custom logging conf
> > > - UI only displays UTC
> > > - Logging ends up duplicated (fixed in master)
> > > - Tasks instantiated outside airflow do not set default time zone
> > (@msumit)
> > > - Log file retrieval feels archaic (non local?)
> > > - Rest Api should standardise and have proper swagger definitions
> > >
> > > And probably some others.
> > >
> > > Bolke
> > >
> > > > On 14 Jan 2018, at 15:41, Driesprong, Fokko <fo...@driesprong.frl>
> > > wrote:
> > > >
> > > > I think 1.10 is a good idea. I'm working on this refactoring of the
> > > sensor
> > > > structure: https://github.com/apache/incubator-airflow/pull/2875
> > > >
> > > > Would be awesome to get this in. At my current project we use sensors
> > in
> > > a
> > > > few places, but still there is some work to be done. For example,
> don't
> > > > allocate an executor slot to the sensors, but have a more
> sophisticated
> > > way
> > > > of poking.
> > > >
> > > > Cheers, Fokko
> > > >
> > > >
> > > >
> > > > 2018-01-12 21:19 GMT+01:00 Chris Riccomini <cr...@apache.org>:
> > > >
> > > >> Just the operator (AIRFLOW-1517)
> > > >>
> > > >> On Fri, Jan 12, 2018 at 11:21 AM, Anirudh Ramanathan <
> > > >> ramanathana@google.com.invalid> wrote:
> > > >>
> > > >>> Sounds awesome. Is k8s support here referring to both the executor
> > and
> > > >> the
> > > >>> operator?
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>>
> > > >>> On Jan 12, 2018 11:18 AM, "Sid Anand" <sa...@apache.org> wrote:
> > > >>>
> > > >>>> +1
> > > >>>>
> > > >>>>
> > > >>>> On Fri, Jan 12, 2018 at 10:56 AM, Chris Riccomini <
> > > >> criccomini@apache.org
> > > >>>>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hey all,
> > > >>>>>
> > > >>>>> After some past discussion on Airflow 1.10 vs 2.0, I think we've
> > > >>>> converged
> > > >>>>> on a 1.10 as the next step. 1.10 will include:
> > > >>>>>
> > > >>>>> * Timezone changes
> > > >>>>> * Kubernetes support
> > > >>>>> * New UI
> > > >>>>>
> > > >>>>> The first two have been merged in, as I saw Bolke just merged K8s
> > (I
> > > >>> saw
> > > >>>> a
> > > >>>>> few follow-on patches coming, though), and I think the new UI is
> > > >>>> probably a
> > > >>>>> couple of weeks out from a PR on master.
> > > >>>>>
> > > >>>>> What do people think of starting the release process on master in
> > > >> Feb?
> > > >>>>> Given that it took a month last time, I expect 1.10 to be
> released
> > in
> > > >>>>> March. Thoughts?
> > > >>>>>
> > > >>>>> Cheers,
> > > >>>>> Chris
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: Airflow 1.10
Posted by Naik Kaxil <k....@reply.com>.
+1 for 1.10
On 27/03/2018, 21:56, "Feng Lu" <fe...@google.com.INVALID> wrote:
+1 for 1.10, thank you Chris!
On Mon, Mar 26, 2018 at 12:44 PM Milan van der Meer <
milan.vandermeer@riaktr.com> wrote:
> The fix for the on_kill method for all the operators should really be
> included in the 1.10 release.
> https://github.com/apache/incubator-airflow/pull/2975
>
> On Mon, Mar 26, 2018 at 8:52 PM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > Hey all,
> >
> > Now that the RBAC UI has been merged in, I wanted to revive this thread
> > again. Timezone stuff has been in for a while. I'm not sure on the status
> > of the K8s stuff, but I do feel it's time to start getting the ball
> rolling
> > on a 1.10 release. What do others think? Any volunteers for a release
> > manager? :)
> >
> > Cheers,
> > Chris
> >
> > On Tue, Jan 16, 2018 at 12:42 PM, George Leslie-Waksman <
> > george@cloverhealth.com.invalid> wrote:
> >
> > > +1
> > >
> > > On Tue, Jan 16, 2018 at 11:08 AM Joy Gao <jo...@wepay.com> wrote:
> > >
> > > > The new FAB UI does not modify the existing API (i.e. www2/api/ will
> > be a
> > > > copy of www/api/), and the endpoints are registered as blueprints to
> > the
> > > > flask app the same way as before, so it is fully backward compatible.
> > > >
> > > > Although FAB offers REST APIs on the models out-of-the-box, it
> > currently
> > > is
> > > > still in BETA and does not support any HTTP authentication scheme,
> > > meaning
> > > > it would require a cookie to mimic a normal user session in order to
> be
> > > > used. In the long run (for Airflow 2.X+), I believe the best approach
> > is
> > > to
> > > > patch FAB with auth backends, so we can apply @has_access annotation
> on
> > > all
> > > > rest endpoints to streamline authentication. For 1.10, the current
> > plan
> > > is
> > > > to leave the API as is. I wouldn't recommend folks to use the
> FAB-based
> > > > REST APIs yet. I would also like to release the new UI as an alpha
> > > version,
> > > > and wait for 2.0 before promoting it to the default version. This
> will
> > > give
> > > > us some time to address any new UI bugs which I overlooked.
> > > >
> > > > +1 on polishing! (With the exception of "*Rest Api should standardise
> > and
> > > > have proper swagger definitions" and any other bugs that require
> major
> > > > overhaul*, which I think can wait until 2.0)
> > > >
> > > >
> > > > On Sun, Jan 14, 2018 at 12:18 PM, Bolke de Bruin <bd...@gmail.com>
> > > > wrote:
> > > >
> > > > > Chris, Joy,
> > > > >
> > > > > Can you shed some light on the backward compatibility of the new
> UI,
> > > > > particularly with regards to the API? The API for example cannot
> use
> > > the
> > > > > login from FAB afaik.
> > > > >
> > > > > As much of the work is already in for 1.10 I think focus should be
> on
> > > > > polishing. There are some minor quirks and slightly annoying bugs.
> > > > >
> > > > > - It seems a dag with schedule “none” can still run when turned on
> > from
> > > > > the UI (unconfirmed)
> > > > > - Exceptions are swallowed when importing a custom logging conf
> > > > > - UI only displays UTC
> > > > > - Logging ends up duplicated (fixed in master)
> > > > > - Tasks instantiated outside airflow do not set default time zone
> > > > (@msumit)
> > > > > - Log file retrieval feels archaic (non local?)
> > > > > - Rest Api should standardise and have proper swagger definitions
> > > > >
> > > > > And probably some others.
> > > > >
> > > > > Bolke
> > > > >
> > > > > > On 14 Jan 2018, at 15:41, Driesprong, Fokko <fokko@driesprong.frl
> >
> > > > > wrote:
> > > > > >
> > > > > > I think 1.10 is a good idea. I'm working on this refactoring of
> the
> > > > > sensor
> > > > > > structure: https://github.com/apache/incubator-airflow/pull/2875
> > > > > >
> > > > > > Would be awesome to get this in. At my current project we use
> > sensors
> > > > in
> > > > > a
> > > > > > few places, but still there is some work to be done. For example,
> > > don't
> > > > > > allocate an executor slot to the sensors, but have a more
> > > sophisticated
> > > > > way
> > > > > > of poking.
> > > > > >
> > > > > > Cheers, Fokko
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2018-01-12 21:19 GMT+01:00 Chris Riccomini <
> criccomini@apache.org
> > >:
> > > > > >
> > > > > >> Just the operator (AIRFLOW-1517)
> > > > > >>
> > > > > >> On Fri, Jan 12, 2018 at 11:21 AM, Anirudh Ramanathan <
> > > > > >> ramanathana@google.com.invalid> wrote:
> > > > > >>
> > > > > >>> Sounds awesome. Is k8s support here referring to both the
> > executor
> > > > and
> > > > > >> the
> > > > > >>> operator?
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>>
> > > > > >>>
> > > > > >>> On Jan 12, 2018 11:18 AM, "Sid Anand" <sa...@apache.org>
> wrote:
> > > > > >>>
> > > > > >>>> +1
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Fri, Jan 12, 2018 at 10:56 AM, Chris Riccomini <
> > > > > >> criccomini@apache.org
> > > > > >>>>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>>> Hey all,
> > > > > >>>>>
> > > > > >>>>> After some past discussion on Airflow 1.10 vs 2.0, I think
> > we've
> > > > > >>>> converged
> > > > > >>>>> on a 1.10 as the next step. 1.10 will include:
> > > > > >>>>>
> > > > > >>>>> * Timezone changes
> > > > > >>>>> * Kubernetes support
> > > > > >>>>> * New UI
> > > > > >>>>>
> > > > > >>>>> The first two have been merged in, as I saw Bolke just merged
> > K8s
> > > > (I
> > > > > >>> saw
> > > > > >>>> a
> > > > > >>>>> few follow-on patches coming, though), and I think the new UI
> > is
> > > > > >>>> probably a
> > > > > >>>>> couple of weeks out from a PR on master.
> > > > > >>>>>
> > > > > >>>>> What do people think of starting the release process on
> master
> > in
> > > > > >> Feb?
> > > > > >>>>> Given that it took a month last time, I expect 1.10 to be
> > > released
> > > > in
> > > > > >>>>> March. Thoughts?
> > > > > >>>>>
> > > > > >>>>> Cheers,
> > > > > >>>>> Chris
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
>
>
>
>
> *Milan van der Meer*
>
>
> *Software Engineer*www.riaktr.com
> *| Brussels Office**BE *+32 498 45 96 22 <0032498459622>* | Skype *
> *milan.vandermeer.ria*
>
Kaxil Naik
Data Reply
38 Grosvenor Gardens
London SW1W 0EB - UK
phone: +44 (0)20 7730 6000
k.naik@reply.com
www.reply.com
Re: Airflow 1.10
Posted by Feng Lu <fe...@google.com.INVALID>.
+1 for 1.10, thank you Chris!
On Mon, Mar 26, 2018 at 12:44 PM Milan van der Meer <
milan.vandermeer@riaktr.com> wrote:
> The fix for the on_kill method for all the operators should really be
> included in the 1.10 release.
> https://github.com/apache/incubator-airflow/pull/2975
>
> On Mon, Mar 26, 2018 at 8:52 PM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > Hey all,
> >
> > Now that the RBAC UI has been merged in, I wanted to revive this thread
> > again. Timezone stuff has been in for a while. I'm not sure on the status
> > of the K8s stuff, but I do feel it's time to start getting the ball
> rolling
> > on a 1.10 release. What do others think? Any volunteers for a release
> > manager? :)
> >
> > Cheers,
> > Chris
> >
> > On Tue, Jan 16, 2018 at 12:42 PM, George Leslie-Waksman <
> > george@cloverhealth.com.invalid> wrote:
> >
> > > +1
> > >
> > > On Tue, Jan 16, 2018 at 11:08 AM Joy Gao <jo...@wepay.com> wrote:
> > >
> > > > The new FAB UI does not modify the existing API (i.e. www2/api/ will
> > be a
> > > > copy of www/api/), and the endpoints are registered as blueprints to
> > the
> > > > flask app the same way as before, so it is fully backward compatible.
> > > >
> > > > Although FAB offers REST APIs on the models out-of-the-box, it
> > currently
> > > is
> > > > still in BETA and does not support any HTTP authentication scheme,
> > > meaning
> > > > it would require a cookie to mimic a normal user session in order to
> be
> > > > used. In the long run (for Airflow 2.X+), I believe the best approach
> > is
> > > to
> > > > patch FAB with auth backends, so we can apply @has_access annotation
> on
> > > all
> > > > rest endpoints to streamline authentication. For 1.10, the current
> > plan
> > > is
> > > > to leave the API as is. I wouldn't recommend folks to use the
> FAB-based
> > > > REST APIs yet. I would also like to release the new UI as an alpha
> > > version,
> > > > and wait for 2.0 before promoting it to the default version. This
> will
> > > give
> > > > us some time to address any new UI bugs which I overlooked.
> > > >
> > > > +1 on polishing! (With the exception of "*Rest Api should standardise
> > and
> > > > have proper swagger definitions" and any other bugs that require
> major
> > > > overhaul*, which I think can wait until 2.0)
> > > >
> > > >
> > > > On Sun, Jan 14, 2018 at 12:18 PM, Bolke de Bruin <bd...@gmail.com>
> > > > wrote:
> > > >
> > > > > Chris, Joy,
> > > > >
> > > > > Can you shed some light on the backward compatibility of the new
> UI,
> > > > > particularly with regards to the API? The API for example cannot
> use
> > > the
> > > > > login from FAB afaik.
> > > > >
> > > > > As much of the work is already in for 1.10 I think focus should be
> on
> > > > > polishing. There are some minor quirks and slightly annoying bugs.
> > > > >
> > > > > - It seems a dag with schedule “none” can still run when turned on
> > from
> > > > > the UI (unconfirmed)
> > > > > - Exceptions are swallowed when importing a custom logging conf
> > > > > - UI only displays UTC
> > > > > - Logging ends up duplicated (fixed in master)
> > > > > - Tasks instantiated outside airflow do not set default time zone
> > > > (@msumit)
> > > > > - Log file retrieval feels archaic (non local?)
> > > > > - Rest Api should standardise and have proper swagger definitions
> > > > >
> > > > > And probably some others.
> > > > >
> > > > > Bolke
> > > > >
> > > > > > On 14 Jan 2018, at 15:41, Driesprong, Fokko <fokko@driesprong.frl
> >
> > > > > wrote:
> > > > > >
> > > > > > I think 1.10 is a good idea. I'm working on this refactoring of
> the
> > > > > sensor
> > > > > > structure: https://github.com/apache/incubator-airflow/pull/2875
> > > > > >
> > > > > > Would be awesome to get this in. At my current project we use
> > sensors
> > > > in
> > > > > a
> > > > > > few places, but still there is some work to be done. For example,
> > > don't
> > > > > > allocate an executor slot to the sensors, but have a more
> > > sophisticated
> > > > > way
> > > > > > of poking.
> > > > > >
> > > > > > Cheers, Fokko
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2018-01-12 21:19 GMT+01:00 Chris Riccomini <
> criccomini@apache.org
> > >:
> > > > > >
> > > > > >> Just the operator (AIRFLOW-1517)
> > > > > >>
> > > > > >> On Fri, Jan 12, 2018 at 11:21 AM, Anirudh Ramanathan <
> > > > > >> ramanathana@google.com.invalid> wrote:
> > > > > >>
> > > > > >>> Sounds awesome. Is k8s support here referring to both the
> > executor
> > > > and
> > > > > >> the
> > > > > >>> operator?
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>>
> > > > > >>>
> > > > > >>> On Jan 12, 2018 11:18 AM, "Sid Anand" <sa...@apache.org>
> wrote:
> > > > > >>>
> > > > > >>>> +1
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Fri, Jan 12, 2018 at 10:56 AM, Chris Riccomini <
> > > > > >> criccomini@apache.org
> > > > > >>>>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>>> Hey all,
> > > > > >>>>>
> > > > > >>>>> After some past discussion on Airflow 1.10 vs 2.0, I think
> > we've
> > > > > >>>> converged
> > > > > >>>>> on a 1.10 as the next step. 1.10 will include:
> > > > > >>>>>
> > > > > >>>>> * Timezone changes
> > > > > >>>>> * Kubernetes support
> > > > > >>>>> * New UI
> > > > > >>>>>
> > > > > >>>>> The first two have been merged in, as I saw Bolke just merged
> > K8s
> > > > (I
> > > > > >>> saw
> > > > > >>>> a
> > > > > >>>>> few follow-on patches coming, though), and I think the new UI
> > is
> > > > > >>>> probably a
> > > > > >>>>> couple of weeks out from a PR on master.
> > > > > >>>>>
> > > > > >>>>> What do people think of starting the release process on
> master
> > in
> > > > > >> Feb?
> > > > > >>>>> Given that it took a month last time, I expect 1.10 to be
> > > released
> > > > in
> > > > > >>>>> March. Thoughts?
> > > > > >>>>>
> > > > > >>>>> Cheers,
> > > > > >>>>> Chris
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
>
>
>
>
> *Milan van der Meer*
>
>
> *Software Engineer*www.riaktr.com
> *| Brussels Office**BE *+32 498 45 96 22 <0032498459622>* | Skype *
> *milan.vandermeer.ria*
>
Re: Airflow 1.10
Posted by Milan van der Meer <mi...@riaktr.com>.
The fix for the on_kill method for all the operators should really be
included in the 1.10 release.
https://github.com/apache/incubator-airflow/pull/2975
On Mon, Mar 26, 2018 at 8:52 PM, Chris Riccomini <cr...@apache.org>
wrote:
> Hey all,
>
> Now that the RBAC UI has been merged in, I wanted to revive this thread
> again. Timezone stuff has been in for a while. I'm not sure on the status
> of the K8s stuff, but I do feel it's time to start getting the ball rolling
> on a 1.10 release. What do others think? Any volunteers for a release
> manager? :)
>
> Cheers,
> Chris
>
> On Tue, Jan 16, 2018 at 12:42 PM, George Leslie-Waksman <
> george@cloverhealth.com.invalid> wrote:
>
> > +1
> >
> > On Tue, Jan 16, 2018 at 11:08 AM Joy Gao <jo...@wepay.com> wrote:
> >
> > > The new FAB UI does not modify the existing API (i.e. www2/api/ will
> be a
> > > copy of www/api/), and the endpoints are registered as blueprints to
> the
> > > flask app the same way as before, so it is fully backward compatible.
> > >
> > > Although FAB offers REST APIs on the models out-of-the-box, it
> currently
> > is
> > > still in BETA and does not support any HTTP authentication scheme,
> > meaning
> > > it would require a cookie to mimic a normal user session in order to be
> > > used. In the long run (for Airflow 2.X+), I believe the best approach
> is
> > to
> > > patch FAB with auth backends, so we can apply @has_access annotation on
> > all
> > > rest endpoints to streamline authentication. For 1.10, the current
> plan
> > is
> > > to leave the API as is. I wouldn't recommend folks to use the FAB-based
> > > REST APIs yet. I would also like to release the new UI as an alpha
> > version,
> > > and wait for 2.0 before promoting it to the default version. This will
> > give
> > > us some time to address any new UI bugs which I overlooked.
> > >
> > > +1 on polishing! (With the exception of "*Rest Api should standardise
> and
> > > have proper swagger definitions" and any other bugs that require major
> > > overhaul*, which I think can wait until 2.0)
> > >
> > >
> > > On Sun, Jan 14, 2018 at 12:18 PM, Bolke de Bruin <bd...@gmail.com>
> > > wrote:
> > >
> > > > Chris, Joy,
> > > >
> > > > Can you shed some light on the backward compatibility of the new UI,
> > > > particularly with regards to the API? The API for example cannot use
> > the
> > > > login from FAB afaik.
> > > >
> > > > As much of the work is already in for 1.10 I think focus should be on
> > > > polishing. There are some minor quirks and slightly annoying bugs.
> > > >
> > > > - It seems a dag with schedule “none” can still run when turned on
> from
> > > > the UI (unconfirmed)
> > > > - Exceptions are swallowed when importing a custom logging conf
> > > > - UI only displays UTC
> > > > - Logging ends up duplicated (fixed in master)
> > > > - Tasks instantiated outside airflow do not set default time zone
> > > (@msumit)
> > > > - Log file retrieval feels archaic (non local?)
> > > > - Rest Api should standardise and have proper swagger definitions
> > > >
> > > > And probably some others.
> > > >
> > > > Bolke
> > > >
> > > > > On 14 Jan 2018, at 15:41, Driesprong, Fokko <fo...@driesprong.frl>
> > > > wrote:
> > > > >
> > > > > I think 1.10 is a good idea. I'm working on this refactoring of the
> > > > sensor
> > > > > structure: https://github.com/apache/incubator-airflow/pull/2875
> > > > >
> > > > > Would be awesome to get this in. At my current project we use
> sensors
> > > in
> > > > a
> > > > > few places, but still there is some work to be done. For example,
> > don't
> > > > > allocate an executor slot to the sensors, but have a more
> > sophisticated
> > > > way
> > > > > of poking.
> > > > >
> > > > > Cheers, Fokko
> > > > >
> > > > >
> > > > >
> > > > > 2018-01-12 21:19 GMT+01:00 Chris Riccomini <criccomini@apache.org
> >:
> > > > >
> > > > >> Just the operator (AIRFLOW-1517)
> > > > >>
> > > > >> On Fri, Jan 12, 2018 at 11:21 AM, Anirudh Ramanathan <
> > > > >> ramanathana@google.com.invalid> wrote:
> > > > >>
> > > > >>> Sounds awesome. Is k8s support here referring to both the
> executor
> > > and
> > > > >> the
> > > > >>> operator?
> > > > >>>
> > > > >>> Thanks,
> > > > >>>
> > > > >>>
> > > > >>> On Jan 12, 2018 11:18 AM, "Sid Anand" <sa...@apache.org> wrote:
> > > > >>>
> > > > >>>> +1
> > > > >>>>
> > > > >>>>
> > > > >>>> On Fri, Jan 12, 2018 at 10:56 AM, Chris Riccomini <
> > > > >> criccomini@apache.org
> > > > >>>>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Hey all,
> > > > >>>>>
> > > > >>>>> After some past discussion on Airflow 1.10 vs 2.0, I think
> we've
> > > > >>>> converged
> > > > >>>>> on a 1.10 as the next step. 1.10 will include:
> > > > >>>>>
> > > > >>>>> * Timezone changes
> > > > >>>>> * Kubernetes support
> > > > >>>>> * New UI
> > > > >>>>>
> > > > >>>>> The first two have been merged in, as I saw Bolke just merged
> K8s
> > > (I
> > > > >>> saw
> > > > >>>> a
> > > > >>>>> few follow-on patches coming, though), and I think the new UI
> is
> > > > >>>> probably a
> > > > >>>>> couple of weeks out from a PR on master.
> > > > >>>>>
> > > > >>>>> What do people think of starting the release process on master
> in
> > > > >> Feb?
> > > > >>>>> Given that it took a month last time, I expect 1.10 to be
> > released
> > > in
> > > > >>>>> March. Thoughts?
> > > > >>>>>
> > > > >>>>> Cheers,
> > > > >>>>> Chris
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>
--
*Milan van der Meer*
*Software Engineer*www.riaktr.com
*| Brussels Office**BE *+32 498 45 96 22 <0032498459622>* | Skype *
*milan.vandermeer.ria*
Re: Airflow 1.10
Posted by Bolke de Bruin <bd...@gmail.com>.
If we can make sure we take this as an opportunity to make this our graduation release that would be perfect. This would mean making sure no licensing issue anymore, security vulnerabilities properly documented and published then I am all for it!
Verstuurd vanaf mijn iPad
> Op 26 mrt. 2018 om 20:52 heeft Chris Riccomini <cr...@apache.org> het volgende geschreven:
>
> Hey all,
>
> Now that the RBAC UI has been merged in, I wanted to revive this thread
> again. Timezone stuff has been in for a while. I'm not sure on the status
> of the K8s stuff, but I do feel it's time to start getting the ball rolling
> on a 1.10 release. What do others think? Any volunteers for a release
> manager? :)
>
> Cheers,
> Chris
>
> On Tue, Jan 16, 2018 at 12:42 PM, George Leslie-Waksman <
> george@cloverhealth.com.invalid> wrote:
>
>> +1
>>
>>> On Tue, Jan 16, 2018 at 11:08 AM Joy Gao <jo...@wepay.com> wrote:
>>>
>>> The new FAB UI does not modify the existing API (i.e. www2/api/ will be a
>>> copy of www/api/), and the endpoints are registered as blueprints to the
>>> flask app the same way as before, so it is fully backward compatible.
>>>
>>> Although FAB offers REST APIs on the models out-of-the-box, it currently
>> is
>>> still in BETA and does not support any HTTP authentication scheme,
>> meaning
>>> it would require a cookie to mimic a normal user session in order to be
>>> used. In the long run (for Airflow 2.X+), I believe the best approach is
>> to
>>> patch FAB with auth backends, so we can apply @has_access annotation on
>> all
>>> rest endpoints to streamline authentication. For 1.10, the current plan
>> is
>>> to leave the API as is. I wouldn't recommend folks to use the FAB-based
>>> REST APIs yet. I would also like to release the new UI as an alpha
>> version,
>>> and wait for 2.0 before promoting it to the default version. This will
>> give
>>> us some time to address any new UI bugs which I overlooked.
>>>
>>> +1 on polishing! (With the exception of "*Rest Api should standardise and
>>> have proper swagger definitions" and any other bugs that require major
>>> overhaul*, which I think can wait until 2.0)
>>>
>>>
>>> On Sun, Jan 14, 2018 at 12:18 PM, Bolke de Bruin <bd...@gmail.com>
>>> wrote:
>>>
>>>> Chris, Joy,
>>>>
>>>> Can you shed some light on the backward compatibility of the new UI,
>>>> particularly with regards to the API? The API for example cannot use
>> the
>>>> login from FAB afaik.
>>>>
>>>> As much of the work is already in for 1.10 I think focus should be on
>>>> polishing. There are some minor quirks and slightly annoying bugs.
>>>>
>>>> - It seems a dag with schedule “none” can still run when turned on from
>>>> the UI (unconfirmed)
>>>> - Exceptions are swallowed when importing a custom logging conf
>>>> - UI only displays UTC
>>>> - Logging ends up duplicated (fixed in master)
>>>> - Tasks instantiated outside airflow do not set default time zone
>>> (@msumit)
>>>> - Log file retrieval feels archaic (non local?)
>>>> - Rest Api should standardise and have proper swagger definitions
>>>>
>>>> And probably some others.
>>>>
>>>> Bolke
>>>>
>>>>> On 14 Jan 2018, at 15:41, Driesprong, Fokko <fo...@driesprong.frl>
>>>> wrote:
>>>>>
>>>>> I think 1.10 is a good idea. I'm working on this refactoring of the
>>>> sensor
>>>>> structure: https://github.com/apache/incubator-airflow/pull/2875
>>>>>
>>>>> Would be awesome to get this in. At my current project we use sensors
>>> in
>>>> a
>>>>> few places, but still there is some work to be done. For example,
>> don't
>>>>> allocate an executor slot to the sensors, but have a more
>> sophisticated
>>>> way
>>>>> of poking.
>>>>>
>>>>> Cheers, Fokko
>>>>>
>>>>>
>>>>>
>>>>> 2018-01-12 21:19 GMT+01:00 Chris Riccomini <cr...@apache.org>:
>>>>>
>>>>>> Just the operator (AIRFLOW-1517)
>>>>>>
>>>>>> On Fri, Jan 12, 2018 at 11:21 AM, Anirudh Ramanathan <
>>>>>> ramanathana@google.com.invalid> wrote:
>>>>>>
>>>>>>> Sounds awesome. Is k8s support here referring to both the executor
>>> and
>>>>>> the
>>>>>>> operator?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>>
>>>>>>>> On Jan 12, 2018 11:18 AM, "Sid Anand" <sa...@apache.org> wrote:
>>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 12, 2018 at 10:56 AM, Chris Riccomini <
>>>>>> criccomini@apache.org
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hey all,
>>>>>>>>>
>>>>>>>>> After some past discussion on Airflow 1.10 vs 2.0, I think we've
>>>>>>>> converged
>>>>>>>>> on a 1.10 as the next step. 1.10 will include:
>>>>>>>>>
>>>>>>>>> * Timezone changes
>>>>>>>>> * Kubernetes support
>>>>>>>>> * New UI
>>>>>>>>>
>>>>>>>>> The first two have been merged in, as I saw Bolke just merged K8s
>>> (I
>>>>>>> saw
>>>>>>>> a
>>>>>>>>> few follow-on patches coming, though), and I think the new UI is
>>>>>>>> probably a
>>>>>>>>> couple of weeks out from a PR on master.
>>>>>>>>>
>>>>>>>>> What do people think of starting the release process on master in
>>>>>> Feb?
>>>>>>>>> Given that it took a month last time, I expect 1.10 to be
>> released
>>> in
>>>>>>>>> March. Thoughts?
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Chris
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>