[2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[Kamil Breguła <ka...@polidea.com>]

Hello,

I think we should remove remote mode in CLI and in-core API Client
(airflow.api.client package).
Here is docs about remote mode:
https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance

Since these features were introduced, it has never been actively developed
and I don't think it's widely used. At the same time, Apache Airflow is
evolving, and this code stands out more and more from the rest.

My main reservations about these features:
- Remote mode/in-core API Client is rarely used. I asked a few people and
none of them used it in production. Does anyone use it?
- A very small number of commands are available (7 pools command and 2 dags
command only)
- Remote mode/API Client depends on experimental REST API.
- Remote mode/API Client is a handwritten code that is difficult to
maintain.
- No documentation for API client
- Remote mode/API Client has low test coverage.
- Remote mode does not provide a good level of security, because it depends
on experimental API. There is the only authentication, but the
authenticated user can perform any operation.
- Requires full Airflow to be installed along with a large number of
unnecessary dependencies. Some of them are difficult to install in some
environments, e.g. setproctitle on Windows
- Using this client API changes the logger configuration because it
requires importing the airflow package.

I think this remote mode in CLI is something valuable, but I think we can
do it in a different way in the future, e.g. generate a CLI/API Client
based on the OpenAPI specification.

Generated API clients can be installed independently of airflow and will be
easier to maintain. We already have one API client for golang implemented
in this way, so new languages will only be developing this idea.
- https://github.com/apache/airflow-client-go

I will be happy to discuss the vision of the development of these two
things. How do we want to develop these two things?

Best regards,
Kamil Bregula

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[Jarek Potiuk <Ja...@polidea.com>]

Fine for me. Deprecating might be deferred, I have no problem with that :).

On Thu, Aug 13, 2020 at 11:08 PM Kaxil Naik <ka...@gmail.com> wrote:

> I feel deprecating without having a replacement will not aid anything.
>
> "deprecations" are marked just to give users a warning and I feel we have
> enough warnings for the user in the API endpoint itself where we say it is
> "experimental".
>
> https://airflow.readthedocs.io/en/stable/rest-api-ref.html says "The API
> structure is not stable. We expect the endpoint definitions to change.
>
> I feel we should keep the endpoints until 2.1.0 until users migrate to new
> endpoints or just remove them in 2.0, we already have a table that map old
> endpoints to new endpoints:
> https://github.com/apache/airflow/blob/master/UPDATING.md#base-endpoint
>
> Regards,
> Kaxil
>
> "
>
> On Thu, Aug 13, 2020 at 8:55 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > Yep I agree removal (especially now in 1.10) is not a good idea.That's
> fine
> > not to remove it in 2.0 but disable by default.
> >
> > But I still think we should mark the whole API deprecated NOW (in
> 1.10.12).
> >
> > It would be great to have at least one release in 1.10. where we
> explicitly
> > mark it as deprecated. We already know that we are going to use the new
> API
> > in 2.0 and it's pretty much "ready" to be released - so I'd say - let's
> > mark it as deprecated even if we do not provide a direct replacement
> > immediately, just to let know people to whe will try to use it now, that
> > they should think twice before doing it.  I don't think "@deprecated"
> means
> > "we provide a direct replacement already". It's more " do not use it
> > because it will disappear". Whether it is replaced by something else or
> not
> > is an entirely different story.
> >
> > J.
> >
> >
> > On Thu, Aug 13, 2020 at 9:18 PM Ry Walker <ry...@rywalker.com> wrote:
> >
> > > Yes I agree the new API would be made default, and experimental would
> be
> > > disabled by default.
> > >
> > > Before the new API goes live, we should allow minor improvements to the
> > > Experimental API (for example
> > https://github.com/apache/airflow/pull/10308
> > > ,
> > > which addresses to some degree the security hole in the experimental
> API
> > > for users of Airflow RBAC) which could change your last sentence a
> > little,
> > > Kamil.
> > >
> > >
> > > On Thu, Aug 13, 2020 at 1:54 PM Kamil Breguła <
> kamil.bregula@polidea.com
> > >
> > > wrote:
> > >
> > > > I agree. We do not have to completely delete the experimental API in
> > > > Airflow 2.0, but I think it is worth turning off by default so that
> the
> > > > user has to make a conscious decision that they want to use the API,
> > > which
> > > > provides a limited level of security-  no permission control, all
> > > > authorized users have full power.
> > > >
> > > > On Thu, Aug 13, 2020 at 7:42 PM QP Hou <qp...@scribd.com> wrote:
> > > >
> > > > > Yeah, i am also curious to know more about the reason why we want
> to
> > > nuke
> > > > > the experimental api code soon instead of just marking it as
> > > deprecated.
> > > > >
> > > > > As for getting more insights into remote mode cli usage, would it
> > make
> > > > > sense to make it part of the airflow user survey?
> > > > >
> > > > > Thanks,
> > > > > QP Hou
> > > > >
> > > > >
> > > > > On Thu, Aug 13, 2020 at 7:18 AM Ry Walker <ry...@rywalker.com> wrote:
> > > > >
> > > > > > I would think we would deprecate the old API once we say the new
> > API
> > > is
> > > > > > “ready to go” - and leave it in place a while as users transition
> > to
> > > > new
> > > > > > API. Why is there an urgency to remove it from codebase?
> > > > > >
> > > > > > On Thu, Aug 13, 2020 at 5:46 AM Ash Berlin-Taylor <
> ash@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Removing the experimental is a fundamental breaking change to
> > > users'
> > > > > > > workflows, and so we should remove it before 2.0.
> > > > > > >
> > > > > > > -ash
> > > > > > >
> > > > > > > On 13 August 2020 10:14:02 BST, Jarek Potiuk <
> > > > Jarek.Potiuk@polidea.com
> > > > > >
> > > > > > > wrote:
> > > > > > > >And I think we should make the whole experimenta API
> deprecated
> > in
> > > > > > > >1.10.12
> > > > > > > >possibly ?
> > > > > > > >
> > > > > > > >On Thu, Aug 13, 2020 at 11:12 AM Jarek Potiuk
> > > > > > > ><Ja...@polidea.com>
> > > > > > > >wrote:
> > > > > > > >
> > > > > > > >> I think no matter what, Maybe we should simply make it
> > > deprecated
> > > > in
> > > > > > > >the
> > > > > > > >> upcoming (today?) release of 1.10.12 ? Then we can decide if
> > in
> > > -
> > > > > > > >> potential  - 1.10.13 we remove it or leave it as it is.
> > > > > > > >>
> > > > > > > >> On Wed, Aug 12, 2020 at 6:37 PM Kamil Breguła
> > > > > > > ><ka...@polidea.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> I started this thread mainly to discuss what we want to do
> > with
> > > > > this
> > > > > > > >>> remote
> > > > > > > >>> mode prior to the Airflow 2.0 release. This is mainly due
> to
> > > the
> > > > > > > >fact that
> > > > > > > >>> he is using an experimental API which will be deprecated.
> > > > > > > >>>
> > > > > > > >>> In my opinion, we have several solutions.
> > > > > > > >>> a) Delete this mode as unused and not supported.
> > > > > > > >>> b) Rewrite in-core API client to support stable API
> > > > > > > >>> c) Prepare OpenAPI based client and rewrite CLI to use it
> > > > > > > >>> d) Leave as-is
> > > > > > > >>>
> > > > > > > >>> There were various expectations on the mailing list about
> > this
> > > > > mode,
> > > > > > > >but I
> > > > > > > >>> haven't seen anyone actively contribute to it. This brings
> me
> > > > > > > >further
> > > > > > > >>> questions. Does anyone use this mode in its current form?
> If
> > no
> > > > one
> > > > > > > >is
> > > > > > > >>> using it, I think we can take more radical steps to start
> > with
> > > a
> > > > > > > >blank
> > > > > > > >>> page. This will make it easier to start work and be able to
> > > > iterate
> > > > > > > >over
> > > > > > > >>> it
> > > > > > > >>> faster in the future. This looks like a simple task, but if
> > we
> > > > want
> > > > > > > >to be
> > > > > > > >>> sure that no breaking-changes are made, we should pay off
> > some
> > > > > > > >technical
> > > > > > > >>> debt and increase testing coverage before we can think
> about
> > > > making
> > > > > > > >more
> > > > > > > >>> changes. It may not be necessary if we choose a different
> > path
> > > of
> > > > > > > >>> development.
> > > > > > > >>>
> > > > > > > >>> I think now is a good time to try to make some decisions if
> > > > someone
> > > > > > > >is
> > > > > > > >>> actually interested in developing these features. I do not
> > > think
> > > > we
> > > > > > > >need a
> > > > > > > >>> precise vision of the development, if currently this
> feature
> > is
> > > > not
> > > > > > > >used
> > > > > > > >>> by
> > > > > > > >>> anyone and no one is really interested in its development.
> > > > > > > >>>
> > > > > > > >>> On Wed, Aug 12, 2020 at 7:16 AM QP Hou <qp...@scribd.com>
> > wrote:
> > > > > > > >>>
> > > > > > > >>> > I think it's best to divide the discussion into two
> > separate
> > > > > > > >topics.
> > > > > > > >>> >
> > > > > > > >>> > First one is to replace the existing json_client with the
> > new
> > > > to
> > > > > > > >be
> > > > > > > >>> created
> > > > > > > >>> > official Airflow Python Client backed by the new RESTful
> > API.
> > > > > This
> > > > > > > >IMHO
> > > > > > > >>> is
> > > > > > > >>> > a must have considering we are to deprecate experimental
> > API
> > > > > going
> > > > > > > >>> forward.
> > > > > > > >>> >
> > > > > > > >>> > The second topic is to create a better CLI experience
> > > > leveraging
> > > > > > > >the new
> > > > > > > >>> > APIs. This is much more controversial. I remember us
> > having a
> > > > > > > >>> > similar discussion in the dev list a year ago, which
> didn't
> > > get
> > > > > > > >much
> > > > > > > >>> > traction. It's not possible to fit all existing CLI
> > > > > > > >functionalities into
> > > > > > > >>> > REST APIs. DB utils use-case that Ash mentioned is a
> great
> > > > > > > >example. So I
> > > > > > > >>> > think one potential solution is to split the CLI into
> two.
> > > Keep
> > > > > > > >the
> > > > > > > >>> > existing CLI as the admin/management CLI can communicate
> > > > directly
> > > > > > > >with
> > > > > > > >>> the
> > > > > > > >>> > DB and taps into airflow core code base. On the other
> hand,
> > > we
> > > > > can
> > > > > > > >>> create a
> > > > > > > >>> > separate user facing CLI that's light weight, fast and
> > remote
> > > > > > > >only. It
> > > > > > > >>> > doesn't even need to be written in python to make it
> easier
> > > to
> > > > > > > >>> distribute
> > > > > > > >>> > as a single binary.
> > > > > > > >>> >
> > > > > > > >>> > Thanks,
> > > > > > > >>> > QP Hou
> > > > > > > >>> >
> > > > > > > >>> >
> > > > > > > >>> > On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor
> > > > > > > ><as...@apache.org>
> > > > > > > >>> wrote:
> > > > > > > >>> >
> > > > > > > >>> > > -1 from me without a firm plan how we will replace it.
> > > > > > > >>> > >
> > > > > > > >>> > > I see keeping it and extending to use the new API would
> > > > ensure
> > > > > > > >that
> > > > > > > >>> > > everything the CLI can do locally (i.e. when airflow
> > > > webserver
> > > > > > > >isn't
> > > > > > > >>> up
> > > > > > > >>> > > yet, with the ) also works over the API with the
> > exception
> > > of
> > > > > db
> > > > > > > >>> > utilities.
> > > > > > > >>> > >
> > > > > > > >>> > > -ash
> > > > > > > >>> > >
> > > > > > > >>> > > On 11 August 2020 20:05:56 BST, QP Hou <qph@scribd.com
> >
> > > > wrote:
> > > > > > > >>> > > >+1 for replacing the existing remote mode client with
> > the
> > > > open
> > > > > > > >api
> > > > > > > >>> > > >based
> > > > > > > >>> > > >client. IMO, we don't really have other options here
> > > because
> > > > > > > >the
> > > > > > > >>> > > >experimental API will be deprecated in the future.
> > > > > > > >>> > > >
> > > > > > > >>> > > >For OpenAPI based Airflow REST clients, the current
> plan
> > > is
> > > > to
> > > > > > > >>> maintain
> > > > > > > >>> > > >all
> > > > > > > >>> > > >the code gen automation within the main source tree
> [1],
> > > > then
> > > > > > > >use it
> > > > > > > >>> to
> > > > > > > >>> > > >populate each individual language specific client repo
> > > like
> > > > > the
> > > > > > > >go
> > > > > > > >>> > > >client
> > > > > > > >>> > > >mentioned earlier. So far, we have the go client
> > completed
> > > > and
> > > > > > > >>> > > >validated to
> > > > > > > >>> > > >make sure this development flow will meet our needs.
> The
> > > > next
> > > > > > > >step I
> > > > > > > >>> > > >think
> > > > > > > >>> > > >the community should focus on is getting API auth
> > > > implemented
> > > > > > > >[2]
> > > > > > > >>> > > >before we
> > > > > > > >>> > > >move on to generate the python client. How we do API
> > auth
> > > > > could
> > > > > > > >have
> > > > > > > >>> a
> > > > > > > >>> > > >big
> > > > > > > >>> > > >impact on client code gen automation, so it is worth
> > > waiting
> > > > > > > >for.
> > > > > > > >>> > > >
> > > > > > > >>> > > >Once we have authentication implemented in both
> Airflow
> > > core
> > > > > > > >and
> > > > > > > >>> > > >clients,
> > > > > > > >>> > > >we should be all good to start doing version releases
> > for
> > > > our
> > > > > > > >API
> > > > > > > >>> > > >clients.
> > > > > > > >>> > > >
> > > > > > > >>> > > >That said, adopting open api based clients in the CLI
> > > alone
> > > > > > > >won't
> > > > > > > >>> > > >address
> > > > > > > >>> > > >the issue of CLI depending on full airflow
> installation.
> > > > Some
> > > > > > > >of the
> > > > > > > >>> > > >cli
> > > > > > > >>> > > >commands like `dags test` depend on a full airflow
> > > > > installation
> > > > > > > >by
> > > > > > > >>> > > >design.
> > > > > > > >>> > > >We will have to either develop a separate CLI intended
> > for
> > > > > > > >remote
> > > > > > > >>> only
> > > > > > > >>> > > >use
> > > > > > > >>> > > >or add a flag in the existing cli so it can run in a
> > pure
> > > > > > > >remote mode
> > > > > > > >>> > > >where
> > > > > > > >>> > > >it would disable loading of code that requires airflow
> > > > > > > >installation
> > > > > > > >>> > > >entirely.
> > > > > > > >>> > > >
> > > > > > > >>> > > >[1]:
> > > https://github.com/apache/airflow/tree/master/clients,
> > > > > > > >>> > > >[2]: https://github.com/apache/airflow/issues/8112
> > > > > > > >>> > > >
> > > > > > > >>> > > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła
> > > > > > > >>> > > ><ka...@polidea.com>
> > > > > > > >>> > > >wrote:
> > > > > > > >>> > > >
> > > > > > > >>> > > >> Hello,
> > > > > > > >>> > > >>
> > > > > > > >>> > > >> I think we should remove remote mode in CLI and
> > in-core
> > > > API
> > > > > > > >Client
> > > > > > > >>> > > >> (airflow.api.client package).
> > > > > > > >>> > > >> Here is docs about remote mode:
> > > > > > > >>> > > >>
> > > > > > > >>> > > >>
> > > > > > > >>> > > >
> > > > > > > >>> > >
> > > > > > > >>> >
> > > > > > > >>>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
> > > > > > > >>> > > >>
> > > > > > > >>> > > >> Since these features were introduced, it has never
> > been
> > > > > > > >actively
> > > > > > > >>> > > >developed
> > > > > > > >>> > > >> and I don't think it's widely used. At the same
> time,
> > > > Apache
> > > > > > > >>> Airflow
> > > > > > > >>> > > >is
> > > > > > > >>> > > >> evolving, and this code stands out more and more
> from
> > > the
> > > > > > > >rest.
> > > > > > > >>> > > >>
> > > > > > > >>> > > >> My main reservations about these features:
> > > > > > > >>> > > >> - Remote mode/in-core API Client is rarely used. I
> > > asked a
> > > > > > > >few
> > > > > > > >>> people
> > > > > > > >>> > > >and
> > > > > > > >>> > > >> none of them used it in production. Does anyone use
> > it?
> > > > > > > >>> > > >> - A very small number of commands are available (7
> > pools
> > > > > > > >command
> > > > > > > >>> and
> > > > > > > >>> > > >2 dags
> > > > > > > >>> > > >> command only)
> > > > > > > >>> > > >> - Remote mode/API Client depends on experimental
> REST
> > > API.
> > > > > > > >>> > > >> - Remote mode/API Client is a handwritten code that
> is
> > > > > > > >difficult to
> > > > > > > >>> > > >> maintain.
> > > > > > > >>> > > >> - No documentation for API client
> > > > > > > >>> > > >> - Remote mode/API Client has low test coverage.
> > > > > > > >>> > > >> - Remote mode does not provide a good level of
> > security,
> > > > > > > >because it
> > > > > > > >>> > > >depends
> > > > > > > >>> > > >> on experimental API. There is the only
> authentication,
> > > but
> > > > > > > >the
> > > > > > > >>> > > >> authenticated user can perform any operation.
> > > > > > > >>> > > >> - Requires full Airflow to be installed along with a
> > > large
> > > > > > > >number
> > > > > > > >>> of
> > > > > > > >>> > > >> unnecessary dependencies. Some of them are difficult
> > to
> > > > > > > >install in
> > > > > > > >>> > > >some
> > > > > > > >>> > > >> environments, e.g. setproctitle on Windows
> > > > > > > >>> > > >> - Using this client API changes the logger
> > configuration
> > > > > > > >because it
> > > > > > > >>> > > >> requires importing the airflow package.
> > > > > > > >>> > > >>
> > > > > > > >>> > > >> I think this remote mode in CLI is something
> valuable,
> > > > but I
> > > > > > > >think
> > > > > > > >>> we
> > > > > > > >>> > > >can
> > > > > > > >>> > > >> do it in a different way in the future, e.g.
> generate
> > a
> > > > > > > >CLI/API
> > > > > > > >>> > > >Client
> > > > > > > >>> > > >> based on the OpenAPI specification.
> > > > > > > >>> > > >>
> > > > > > > >>> > > >> Generated API clients can be installed independently
> > of
> > > > > > > >airflow and
> > > > > > > >>> > > >will be
> > > > > > > >>> > > >> easier to maintain. We already have one API client
> for
> > > > > golang
> > > > > > > >>> > > >implemented
> > > > > > > >>> > > >> in this way, so new languages will only be
> developing
> > > this
> > > > > > > >idea.
> > > > > > > >>> > > >> - https://github.com/apache/airflow-client-go
> > > > > > > >>> > > >>
> > > > > > > >>> > > >> I will be happy to discuss the vision of the
> > development
> > > > of
> > > > > > > >these
> > > > > > > >>> two
> > > > > > > >>> > > >> things. How do we want to develop these two things?
> > > > > > > >>> > > >>
> > > > > > > >>> > > >> Best regards,
> > > > > > > >>> > > >> Kamil Bregula
> > > > > > > >>> > > >>
> > > > > > > >>> > >
> > > > > > > >>> >
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >>
> > > > > > > >> Jarek Potiuk
> > > > > > > >> Polidea <https://www.polidea.com/> | Principal Software
> > > Engineer
> > > > > > > >>
> > > > > > > >> M: +48 660 796 129 <+48660796129>
> > > > > > > >> [image: Polidea] <https://www.polidea.com/>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >
> > > > > > > >--
> > > > > > > >
> > > > > > > >Jarek Potiuk
> > > > > > > >Polidea <https://www.polidea.com/> | Principal Software
> > Engineer
> > > > > > > >
> > > > > > > >M: +48 660 796 129 <+48660796129>
> > > > > > > >[image: Polidea] <https://www.polidea.com/>
> > > > > > >
> > > > > > --
> > > > > > Sent from Gmail Mobile
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>


-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[Kaxil Naik <ka...@gmail.com>]

I feel deprecating without having a replacement will not aid anything.

"deprecations" are marked just to give users a warning and I feel we have
enough warnings for the user in the API endpoint itself where we say it is
"experimental".

https://airflow.readthedocs.io/en/stable/rest-api-ref.html says "The API
structure is not stable. We expect the endpoint definitions to change.

I feel we should keep the endpoints until 2.1.0 until users migrate to new
endpoints or just remove them in 2.0, we already have a table that map old
endpoints to new endpoints:
https://github.com/apache/airflow/blob/master/UPDATING.md#base-endpoint

Regards,
Kaxil

"

On Thu, Aug 13, 2020 at 8:55 PM Jarek Potiuk <Ja...@polidea.com>
wrote:

> Yep I agree removal (especially now in 1.10) is not a good idea.That's fine
> not to remove it in 2.0 but disable by default.
>
> But I still think we should mark the whole API deprecated NOW (in 1.10.12).
>
> It would be great to have at least one release in 1.10. where we explicitly
> mark it as deprecated. We already know that we are going to use the new API
> in 2.0 and it's pretty much "ready" to be released - so I'd say - let's
> mark it as deprecated even if we do not provide a direct replacement
> immediately, just to let know people to whe will try to use it now, that
> they should think twice before doing it.  I don't think "@deprecated" means
> "we provide a direct replacement already". It's more " do not use it
> because it will disappear". Whether it is replaced by something else or not
> is an entirely different story.
>
> J.
>
>
> On Thu, Aug 13, 2020 at 9:18 PM Ry Walker <ry...@rywalker.com> wrote:
>
> > Yes I agree the new API would be made default, and experimental would be
> > disabled by default.
> >
> > Before the new API goes live, we should allow minor improvements to the
> > Experimental API (for example
> https://github.com/apache/airflow/pull/10308
> > ,
> > which addresses to some degree the security hole in the experimental API
> > for users of Airflow RBAC) which could change your last sentence a
> little,
> > Kamil.
> >
> >
> > On Thu, Aug 13, 2020 at 1:54 PM Kamil Breguła <kamil.bregula@polidea.com
> >
> > wrote:
> >
> > > I agree. We do not have to completely delete the experimental API in
> > > Airflow 2.0, but I think it is worth turning off by default so that the
> > > user has to make a conscious decision that they want to use the API,
> > which
> > > provides a limited level of security-  no permission control, all
> > > authorized users have full power.
> > >
> > > On Thu, Aug 13, 2020 at 7:42 PM QP Hou <qp...@scribd.com> wrote:
> > >
> > > > Yeah, i am also curious to know more about the reason why we want to
> > nuke
> > > > the experimental api code soon instead of just marking it as
> > deprecated.
> > > >
> > > > As for getting more insights into remote mode cli usage, would it
> make
> > > > sense to make it part of the airflow user survey?
> > > >
> > > > Thanks,
> > > > QP Hou
> > > >
> > > >
> > > > On Thu, Aug 13, 2020 at 7:18 AM Ry Walker <ry...@rywalker.com> wrote:
> > > >
> > > > > I would think we would deprecate the old API once we say the new
> API
> > is
> > > > > “ready to go” - and leave it in place a while as users transition
> to
> > > new
> > > > > API. Why is there an urgency to remove it from codebase?
> > > > >
> > > > > On Thu, Aug 13, 2020 at 5:46 AM Ash Berlin-Taylor <as...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Removing the experimental is a fundamental breaking change to
> > users'
> > > > > > workflows, and so we should remove it before 2.0.
> > > > > >
> > > > > > -ash
> > > > > >
> > > > > > On 13 August 2020 10:14:02 BST, Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com
> > > > >
> > > > > > wrote:
> > > > > > >And I think we should make the whole experimenta API deprecated
> in
> > > > > > >1.10.12
> > > > > > >possibly ?
> > > > > > >
> > > > > > >On Thu, Aug 13, 2020 at 11:12 AM Jarek Potiuk
> > > > > > ><Ja...@polidea.com>
> > > > > > >wrote:
> > > > > > >
> > > > > > >> I think no matter what, Maybe we should simply make it
> > deprecated
> > > in
> > > > > > >the
> > > > > > >> upcoming (today?) release of 1.10.12 ? Then we can decide if
> in
> > -
> > > > > > >> potential  - 1.10.13 we remove it or leave it as it is.
> > > > > > >>
> > > > > > >> On Wed, Aug 12, 2020 at 6:37 PM Kamil Breguła
> > > > > > ><ka...@polidea.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> I started this thread mainly to discuss what we want to do
> with
> > > > this
> > > > > > >>> remote
> > > > > > >>> mode prior to the Airflow 2.0 release. This is mainly due to
> > the
> > > > > > >fact that
> > > > > > >>> he is using an experimental API which will be deprecated.
> > > > > > >>>
> > > > > > >>> In my opinion, we have several solutions.
> > > > > > >>> a) Delete this mode as unused and not supported.
> > > > > > >>> b) Rewrite in-core API client to support stable API
> > > > > > >>> c) Prepare OpenAPI based client and rewrite CLI to use it
> > > > > > >>> d) Leave as-is
> > > > > > >>>
> > > > > > >>> There were various expectations on the mailing list about
> this
> > > > mode,
> > > > > > >but I
> > > > > > >>> haven't seen anyone actively contribute to it. This brings me
> > > > > > >further
> > > > > > >>> questions. Does anyone use this mode in its current form? If
> no
> > > one
> > > > > > >is
> > > > > > >>> using it, I think we can take more radical steps to start
> with
> > a
> > > > > > >blank
> > > > > > >>> page. This will make it easier to start work and be able to
> > > iterate
> > > > > > >over
> > > > > > >>> it
> > > > > > >>> faster in the future. This looks like a simple task, but if
> we
> > > want
> > > > > > >to be
> > > > > > >>> sure that no breaking-changes are made, we should pay off
> some
> > > > > > >technical
> > > > > > >>> debt and increase testing coverage before we can think about
> > > making
> > > > > > >more
> > > > > > >>> changes. It may not be necessary if we choose a different
> path
> > of
> > > > > > >>> development.
> > > > > > >>>
> > > > > > >>> I think now is a good time to try to make some decisions if
> > > someone
> > > > > > >is
> > > > > > >>> actually interested in developing these features. I do not
> > think
> > > we
> > > > > > >need a
> > > > > > >>> precise vision of the development, if currently this feature
> is
> > > not
> > > > > > >used
> > > > > > >>> by
> > > > > > >>> anyone and no one is really interested in its development.
> > > > > > >>>
> > > > > > >>> On Wed, Aug 12, 2020 at 7:16 AM QP Hou <qp...@scribd.com>
> wrote:
> > > > > > >>>
> > > > > > >>> > I think it's best to divide the discussion into two
> separate
> > > > > > >topics.
> > > > > > >>> >
> > > > > > >>> > First one is to replace the existing json_client with the
> new
> > > to
> > > > > > >be
> > > > > > >>> created
> > > > > > >>> > official Airflow Python Client backed by the new RESTful
> API.
> > > > This
> > > > > > >IMHO
> > > > > > >>> is
> > > > > > >>> > a must have considering we are to deprecate experimental
> API
> > > > going
> > > > > > >>> forward.
> > > > > > >>> >
> > > > > > >>> > The second topic is to create a better CLI experience
> > > leveraging
> > > > > > >the new
> > > > > > >>> > APIs. This is much more controversial. I remember us
> having a
> > > > > > >>> > similar discussion in the dev list a year ago, which didn't
> > get
> > > > > > >much
> > > > > > >>> > traction. It's not possible to fit all existing CLI
> > > > > > >functionalities into
> > > > > > >>> > REST APIs. DB utils use-case that Ash mentioned is a great
> > > > > > >example. So I
> > > > > > >>> > think one potential solution is to split the CLI into two.
> > Keep
> > > > > > >the
> > > > > > >>> > existing CLI as the admin/management CLI can communicate
> > > directly
> > > > > > >with
> > > > > > >>> the
> > > > > > >>> > DB and taps into airflow core code base. On the other hand,
> > we
> > > > can
> > > > > > >>> create a
> > > > > > >>> > separate user facing CLI that's light weight, fast and
> remote
> > > > > > >only. It
> > > > > > >>> > doesn't even need to be written in python to make it easier
> > to
> > > > > > >>> distribute
> > > > > > >>> > as a single binary.
> > > > > > >>> >
> > > > > > >>> > Thanks,
> > > > > > >>> > QP Hou
> > > > > > >>> >
> > > > > > >>> >
> > > > > > >>> > On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor
> > > > > > ><as...@apache.org>
> > > > > > >>> wrote:
> > > > > > >>> >
> > > > > > >>> > > -1 from me without a firm plan how we will replace it.
> > > > > > >>> > >
> > > > > > >>> > > I see keeping it and extending to use the new API would
> > > ensure
> > > > > > >that
> > > > > > >>> > > everything the CLI can do locally (i.e. when airflow
> > > webserver
> > > > > > >isn't
> > > > > > >>> up
> > > > > > >>> > > yet, with the ) also works over the API with the
> exception
> > of
> > > > db
> > > > > > >>> > utilities.
> > > > > > >>> > >
> > > > > > >>> > > -ash
> > > > > > >>> > >
> > > > > > >>> > > On 11 August 2020 20:05:56 BST, QP Hou <qp...@scribd.com>
> > > wrote:
> > > > > > >>> > > >+1 for replacing the existing remote mode client with
> the
> > > open
> > > > > > >api
> > > > > > >>> > > >based
> > > > > > >>> > > >client. IMO, we don't really have other options here
> > because
> > > > > > >the
> > > > > > >>> > > >experimental API will be deprecated in the future.
> > > > > > >>> > > >
> > > > > > >>> > > >For OpenAPI based Airflow REST clients, the current plan
> > is
> > > to
> > > > > > >>> maintain
> > > > > > >>> > > >all
> > > > > > >>> > > >the code gen automation within the main source tree [1],
> > > then
> > > > > > >use it
> > > > > > >>> to
> > > > > > >>> > > >populate each individual language specific client repo
> > like
> > > > the
> > > > > > >go
> > > > > > >>> > > >client
> > > > > > >>> > > >mentioned earlier. So far, we have the go client
> completed
> > > and
> > > > > > >>> > > >validated to
> > > > > > >>> > > >make sure this development flow will meet our needs. The
> > > next
> > > > > > >step I
> > > > > > >>> > > >think
> > > > > > >>> > > >the community should focus on is getting API auth
> > > implemented
> > > > > > >[2]
> > > > > > >>> > > >before we
> > > > > > >>> > > >move on to generate the python client. How we do API
> auth
> > > > could
> > > > > > >have
> > > > > > >>> a
> > > > > > >>> > > >big
> > > > > > >>> > > >impact on client code gen automation, so it is worth
> > waiting
> > > > > > >for.
> > > > > > >>> > > >
> > > > > > >>> > > >Once we have authentication implemented in both Airflow
> > core
> > > > > > >and
> > > > > > >>> > > >clients,
> > > > > > >>> > > >we should be all good to start doing version releases
> for
> > > our
> > > > > > >API
> > > > > > >>> > > >clients.
> > > > > > >>> > > >
> > > > > > >>> > > >That said, adopting open api based clients in the CLI
> > alone
> > > > > > >won't
> > > > > > >>> > > >address
> > > > > > >>> > > >the issue of CLI depending on full airflow installation.
> > > Some
> > > > > > >of the
> > > > > > >>> > > >cli
> > > > > > >>> > > >commands like `dags test` depend on a full airflow
> > > > installation
> > > > > > >by
> > > > > > >>> > > >design.
> > > > > > >>> > > >We will have to either develop a separate CLI intended
> for
> > > > > > >remote
> > > > > > >>> only
> > > > > > >>> > > >use
> > > > > > >>> > > >or add a flag in the existing cli so it can run in a
> pure
> > > > > > >remote mode
> > > > > > >>> > > >where
> > > > > > >>> > > >it would disable loading of code that requires airflow
> > > > > > >installation
> > > > > > >>> > > >entirely.
> > > > > > >>> > > >
> > > > > > >>> > > >[1]:
> > https://github.com/apache/airflow/tree/master/clients,
> > > > > > >>> > > >[2]: https://github.com/apache/airflow/issues/8112
> > > > > > >>> > > >
> > > > > > >>> > > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła
> > > > > > >>> > > ><ka...@polidea.com>
> > > > > > >>> > > >wrote:
> > > > > > >>> > > >
> > > > > > >>> > > >> Hello,
> > > > > > >>> > > >>
> > > > > > >>> > > >> I think we should remove remote mode in CLI and
> in-core
> > > API
> > > > > > >Client
> > > > > > >>> > > >> (airflow.api.client package).
> > > > > > >>> > > >> Here is docs about remote mode:
> > > > > > >>> > > >>
> > > > > > >>> > > >>
> > > > > > >>> > > >
> > > > > > >>> > >
> > > > > > >>> >
> > > > > > >>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
> > > > > > >>> > > >>
> > > > > > >>> > > >> Since these features were introduced, it has never
> been
> > > > > > >actively
> > > > > > >>> > > >developed
> > > > > > >>> > > >> and I don't think it's widely used. At the same time,
> > > Apache
> > > > > > >>> Airflow
> > > > > > >>> > > >is
> > > > > > >>> > > >> evolving, and this code stands out more and more from
> > the
> > > > > > >rest.
> > > > > > >>> > > >>
> > > > > > >>> > > >> My main reservations about these features:
> > > > > > >>> > > >> - Remote mode/in-core API Client is rarely used. I
> > asked a
> > > > > > >few
> > > > > > >>> people
> > > > > > >>> > > >and
> > > > > > >>> > > >> none of them used it in production. Does anyone use
> it?
> > > > > > >>> > > >> - A very small number of commands are available (7
> pools
> > > > > > >command
> > > > > > >>> and
> > > > > > >>> > > >2 dags
> > > > > > >>> > > >> command only)
> > > > > > >>> > > >> - Remote mode/API Client depends on experimental REST
> > API.
> > > > > > >>> > > >> - Remote mode/API Client is a handwritten code that is
> > > > > > >difficult to
> > > > > > >>> > > >> maintain.
> > > > > > >>> > > >> - No documentation for API client
> > > > > > >>> > > >> - Remote mode/API Client has low test coverage.
> > > > > > >>> > > >> - Remote mode does not provide a good level of
> security,
> > > > > > >because it
> > > > > > >>> > > >depends
> > > > > > >>> > > >> on experimental API. There is the only authentication,
> > but
> > > > > > >the
> > > > > > >>> > > >> authenticated user can perform any operation.
> > > > > > >>> > > >> - Requires full Airflow to be installed along with a
> > large
> > > > > > >number
> > > > > > >>> of
> > > > > > >>> > > >> unnecessary dependencies. Some of them are difficult
> to
> > > > > > >install in
> > > > > > >>> > > >some
> > > > > > >>> > > >> environments, e.g. setproctitle on Windows
> > > > > > >>> > > >> - Using this client API changes the logger
> configuration
> > > > > > >because it
> > > > > > >>> > > >> requires importing the airflow package.
> > > > > > >>> > > >>
> > > > > > >>> > > >> I think this remote mode in CLI is something valuable,
> > > but I
> > > > > > >think
> > > > > > >>> we
> > > > > > >>> > > >can
> > > > > > >>> > > >> do it in a different way in the future, e.g. generate
> a
> > > > > > >CLI/API
> > > > > > >>> > > >Client
> > > > > > >>> > > >> based on the OpenAPI specification.
> > > > > > >>> > > >>
> > > > > > >>> > > >> Generated API clients can be installed independently
> of
> > > > > > >airflow and
> > > > > > >>> > > >will be
> > > > > > >>> > > >> easier to maintain. We already have one API client for
> > > > golang
> > > > > > >>> > > >implemented
> > > > > > >>> > > >> in this way, so new languages will only be developing
> > this
> > > > > > >idea.
> > > > > > >>> > > >> - https://github.com/apache/airflow-client-go
> > > > > > >>> > > >>
> > > > > > >>> > > >> I will be happy to discuss the vision of the
> development
> > > of
> > > > > > >these
> > > > > > >>> two
> > > > > > >>> > > >> things. How do we want to develop these two things?
> > > > > > >>> > > >>
> > > > > > >>> > > >> Best regards,
> > > > > > >>> > > >> Kamil Bregula
> > > > > > >>> > > >>
> > > > > > >>> > >
> > > > > > >>> >
> > > > > > >>>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >>
> > > > > > >> Jarek Potiuk
> > > > > > >> Polidea <https://www.polidea.com/> | Principal Software
> > Engineer
> > > > > > >>
> > > > > > >> M: +48 660 796 129 <+48660796129>
> > > > > > >> [image: Polidea] <https://www.polidea.com/>
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > > >--
> > > > > > >
> > > > > > >Jarek Potiuk
> > > > > > >Polidea <https://www.polidea.com/> | Principal Software
> Engineer
> > > > > > >
> > > > > > >M: +48 660 796 129 <+48660796129>
> > > > > > >[image: Polidea] <https://www.polidea.com/>
> > > > > >
> > > > > --
> > > > > Sent from Gmail Mobile
> > > > >
> > > >
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[Jarek Potiuk <Ja...@polidea.com>]

Yep I agree removal (especially now in 1.10) is not a good idea.That's fine
not to remove it in 2.0 but disable by default.

But I still think we should mark the whole API deprecated NOW (in 1.10.12).

It would be great to have at least one release in 1.10. where we explicitly
mark it as deprecated. We already know that we are going to use the new API
in 2.0 and it's pretty much "ready" to be released - so I'd say - let's
mark it as deprecated even if we do not provide a direct replacement
immediately, just to let know people to whe will try to use it now, that
they should think twice before doing it.  I don't think "@deprecated" means
"we provide a direct replacement already". It's more " do not use it
because it will disappear". Whether it is replaced by something else or not
is an entirely different story.

J.


On Thu, Aug 13, 2020 at 9:18 PM Ry Walker <ry...@rywalker.com> wrote:

> Yes I agree the new API would be made default, and experimental would be
> disabled by default.
>
> Before the new API goes live, we should allow minor improvements to the
> Experimental API (for example https://github.com/apache/airflow/pull/10308
> ,
> which addresses to some degree the security hole in the experimental API
> for users of Airflow RBAC) which could change your last sentence a little,
> Kamil.
>
>
> On Thu, Aug 13, 2020 at 1:54 PM Kamil Breguła <ka...@polidea.com>
> wrote:
>
> > I agree. We do not have to completely delete the experimental API in
> > Airflow 2.0, but I think it is worth turning off by default so that the
> > user has to make a conscious decision that they want to use the API,
> which
> > provides a limited level of security-  no permission control, all
> > authorized users have full power.
> >
> > On Thu, Aug 13, 2020 at 7:42 PM QP Hou <qp...@scribd.com> wrote:
> >
> > > Yeah, i am also curious to know more about the reason why we want to
> nuke
> > > the experimental api code soon instead of just marking it as
> deprecated.
> > >
> > > As for getting more insights into remote mode cli usage, would it make
> > > sense to make it part of the airflow user survey?
> > >
> > > Thanks,
> > > QP Hou
> > >
> > >
> > > On Thu, Aug 13, 2020 at 7:18 AM Ry Walker <ry...@rywalker.com> wrote:
> > >
> > > > I would think we would deprecate the old API once we say the new API
> is
> > > > “ready to go” - and leave it in place a while as users transition to
> > new
> > > > API. Why is there an urgency to remove it from codebase?
> > > >
> > > > On Thu, Aug 13, 2020 at 5:46 AM Ash Berlin-Taylor <as...@apache.org>
> > > wrote:
> > > >
> > > > > Removing the experimental is a fundamental breaking change to
> users'
> > > > > workflows, and so we should remove it before 2.0.
> > > > >
> > > > > -ash
> > > > >
> > > > > On 13 August 2020 10:14:02 BST, Jarek Potiuk <
> > Jarek.Potiuk@polidea.com
> > > >
> > > > > wrote:
> > > > > >And I think we should make the whole experimenta API deprecated in
> > > > > >1.10.12
> > > > > >possibly ?
> > > > > >
> > > > > >On Thu, Aug 13, 2020 at 11:12 AM Jarek Potiuk
> > > > > ><Ja...@polidea.com>
> > > > > >wrote:
> > > > > >
> > > > > >> I think no matter what, Maybe we should simply make it
> deprecated
> > in
> > > > > >the
> > > > > >> upcoming (today?) release of 1.10.12 ? Then we can decide if in
> -
> > > > > >> potential  - 1.10.13 we remove it or leave it as it is.
> > > > > >>
> > > > > >> On Wed, Aug 12, 2020 at 6:37 PM Kamil Breguła
> > > > > ><ka...@polidea.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> I started this thread mainly to discuss what we want to do with
> > > this
> > > > > >>> remote
> > > > > >>> mode prior to the Airflow 2.0 release. This is mainly due to
> the
> > > > > >fact that
> > > > > >>> he is using an experimental API which will be deprecated.
> > > > > >>>
> > > > > >>> In my opinion, we have several solutions.
> > > > > >>> a) Delete this mode as unused and not supported.
> > > > > >>> b) Rewrite in-core API client to support stable API
> > > > > >>> c) Prepare OpenAPI based client and rewrite CLI to use it
> > > > > >>> d) Leave as-is
> > > > > >>>
> > > > > >>> There were various expectations on the mailing list about this
> > > mode,
> > > > > >but I
> > > > > >>> haven't seen anyone actively contribute to it. This brings me
> > > > > >further
> > > > > >>> questions. Does anyone use this mode in its current form? If no
> > one
> > > > > >is
> > > > > >>> using it, I think we can take more radical steps to start with
> a
> > > > > >blank
> > > > > >>> page. This will make it easier to start work and be able to
> > iterate
> > > > > >over
> > > > > >>> it
> > > > > >>> faster in the future. This looks like a simple task, but if we
> > want
> > > > > >to be
> > > > > >>> sure that no breaking-changes are made, we should pay off some
> > > > > >technical
> > > > > >>> debt and increase testing coverage before we can think about
> > making
> > > > > >more
> > > > > >>> changes. It may not be necessary if we choose a different path
> of
> > > > > >>> development.
> > > > > >>>
> > > > > >>> I think now is a good time to try to make some decisions if
> > someone
> > > > > >is
> > > > > >>> actually interested in developing these features. I do not
> think
> > we
> > > > > >need a
> > > > > >>> precise vision of the development, if currently this feature is
> > not
> > > > > >used
> > > > > >>> by
> > > > > >>> anyone and no one is really interested in its development.
> > > > > >>>
> > > > > >>> On Wed, Aug 12, 2020 at 7:16 AM QP Hou <qp...@scribd.com> wrote:
> > > > > >>>
> > > > > >>> > I think it's best to divide the discussion into two separate
> > > > > >topics.
> > > > > >>> >
> > > > > >>> > First one is to replace the existing json_client with the new
> > to
> > > > > >be
> > > > > >>> created
> > > > > >>> > official Airflow Python Client backed by the new RESTful API.
> > > This
> > > > > >IMHO
> > > > > >>> is
> > > > > >>> > a must have considering we are to deprecate experimental API
> > > going
> > > > > >>> forward.
> > > > > >>> >
> > > > > >>> > The second topic is to create a better CLI experience
> > leveraging
> > > > > >the new
> > > > > >>> > APIs. This is much more controversial. I remember us having a
> > > > > >>> > similar discussion in the dev list a year ago, which didn't
> get
> > > > > >much
> > > > > >>> > traction. It's not possible to fit all existing CLI
> > > > > >functionalities into
> > > > > >>> > REST APIs. DB utils use-case that Ash mentioned is a great
> > > > > >example. So I
> > > > > >>> > think one potential solution is to split the CLI into two.
> Keep
> > > > > >the
> > > > > >>> > existing CLI as the admin/management CLI can communicate
> > directly
> > > > > >with
> > > > > >>> the
> > > > > >>> > DB and taps into airflow core code base. On the other hand,
> we
> > > can
> > > > > >>> create a
> > > > > >>> > separate user facing CLI that's light weight, fast and remote
> > > > > >only. It
> > > > > >>> > doesn't even need to be written in python to make it easier
> to
> > > > > >>> distribute
> > > > > >>> > as a single binary.
> > > > > >>> >
> > > > > >>> > Thanks,
> > > > > >>> > QP Hou
> > > > > >>> >
> > > > > >>> >
> > > > > >>> > On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor
> > > > > ><as...@apache.org>
> > > > > >>> wrote:
> > > > > >>> >
> > > > > >>> > > -1 from me without a firm plan how we will replace it.
> > > > > >>> > >
> > > > > >>> > > I see keeping it and extending to use the new API would
> > ensure
> > > > > >that
> > > > > >>> > > everything the CLI can do locally (i.e. when airflow
> > webserver
> > > > > >isn't
> > > > > >>> up
> > > > > >>> > > yet, with the ) also works over the API with the exception
> of
> > > db
> > > > > >>> > utilities.
> > > > > >>> > >
> > > > > >>> > > -ash
> > > > > >>> > >
> > > > > >>> > > On 11 August 2020 20:05:56 BST, QP Hou <qp...@scribd.com>
> > wrote:
> > > > > >>> > > >+1 for replacing the existing remote mode client with the
> > open
> > > > > >api
> > > > > >>> > > >based
> > > > > >>> > > >client. IMO, we don't really have other options here
> because
> > > > > >the
> > > > > >>> > > >experimental API will be deprecated in the future.
> > > > > >>> > > >
> > > > > >>> > > >For OpenAPI based Airflow REST clients, the current plan
> is
> > to
> > > > > >>> maintain
> > > > > >>> > > >all
> > > > > >>> > > >the code gen automation within the main source tree [1],
> > then
> > > > > >use it
> > > > > >>> to
> > > > > >>> > > >populate each individual language specific client repo
> like
> > > the
> > > > > >go
> > > > > >>> > > >client
> > > > > >>> > > >mentioned earlier. So far, we have the go client completed
> > and
> > > > > >>> > > >validated to
> > > > > >>> > > >make sure this development flow will meet our needs. The
> > next
> > > > > >step I
> > > > > >>> > > >think
> > > > > >>> > > >the community should focus on is getting API auth
> > implemented
> > > > > >[2]
> > > > > >>> > > >before we
> > > > > >>> > > >move on to generate the python client. How we do API auth
> > > could
> > > > > >have
> > > > > >>> a
> > > > > >>> > > >big
> > > > > >>> > > >impact on client code gen automation, so it is worth
> waiting
> > > > > >for.
> > > > > >>> > > >
> > > > > >>> > > >Once we have authentication implemented in both Airflow
> core
> > > > > >and
> > > > > >>> > > >clients,
> > > > > >>> > > >we should be all good to start doing version releases for
> > our
> > > > > >API
> > > > > >>> > > >clients.
> > > > > >>> > > >
> > > > > >>> > > >That said, adopting open api based clients in the CLI
> alone
> > > > > >won't
> > > > > >>> > > >address
> > > > > >>> > > >the issue of CLI depending on full airflow installation.
> > Some
> > > > > >of the
> > > > > >>> > > >cli
> > > > > >>> > > >commands like `dags test` depend on a full airflow
> > > installation
> > > > > >by
> > > > > >>> > > >design.
> > > > > >>> > > >We will have to either develop a separate CLI intended for
> > > > > >remote
> > > > > >>> only
> > > > > >>> > > >use
> > > > > >>> > > >or add a flag in the existing cli so it can run in a pure
> > > > > >remote mode
> > > > > >>> > > >where
> > > > > >>> > > >it would disable loading of code that requires airflow
> > > > > >installation
> > > > > >>> > > >entirely.
> > > > > >>> > > >
> > > > > >>> > > >[1]:
> https://github.com/apache/airflow/tree/master/clients,
> > > > > >>> > > >[2]: https://github.com/apache/airflow/issues/8112
> > > > > >>> > > >
> > > > > >>> > > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła
> > > > > >>> > > ><ka...@polidea.com>
> > > > > >>> > > >wrote:
> > > > > >>> > > >
> > > > > >>> > > >> Hello,
> > > > > >>> > > >>
> > > > > >>> > > >> I think we should remove remote mode in CLI and in-core
> > API
> > > > > >Client
> > > > > >>> > > >> (airflow.api.client package).
> > > > > >>> > > >> Here is docs about remote mode:
> > > > > >>> > > >>
> > > > > >>> > > >>
> > > > > >>> > > >
> > > > > >>> > >
> > > > > >>> >
> > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
> > > > > >>> > > >>
> > > > > >>> > > >> Since these features were introduced, it has never been
> > > > > >actively
> > > > > >>> > > >developed
> > > > > >>> > > >> and I don't think it's widely used. At the same time,
> > Apache
> > > > > >>> Airflow
> > > > > >>> > > >is
> > > > > >>> > > >> evolving, and this code stands out more and more from
> the
> > > > > >rest.
> > > > > >>> > > >>
> > > > > >>> > > >> My main reservations about these features:
> > > > > >>> > > >> - Remote mode/in-core API Client is rarely used. I
> asked a
> > > > > >few
> > > > > >>> people
> > > > > >>> > > >and
> > > > > >>> > > >> none of them used it in production. Does anyone use it?
> > > > > >>> > > >> - A very small number of commands are available (7 pools
> > > > > >command
> > > > > >>> and
> > > > > >>> > > >2 dags
> > > > > >>> > > >> command only)
> > > > > >>> > > >> - Remote mode/API Client depends on experimental REST
> API.
> > > > > >>> > > >> - Remote mode/API Client is a handwritten code that is
> > > > > >difficult to
> > > > > >>> > > >> maintain.
> > > > > >>> > > >> - No documentation for API client
> > > > > >>> > > >> - Remote mode/API Client has low test coverage.
> > > > > >>> > > >> - Remote mode does not provide a good level of security,
> > > > > >because it
> > > > > >>> > > >depends
> > > > > >>> > > >> on experimental API. There is the only authentication,
> but
> > > > > >the
> > > > > >>> > > >> authenticated user can perform any operation.
> > > > > >>> > > >> - Requires full Airflow to be installed along with a
> large
> > > > > >number
> > > > > >>> of
> > > > > >>> > > >> unnecessary dependencies. Some of them are difficult to
> > > > > >install in
> > > > > >>> > > >some
> > > > > >>> > > >> environments, e.g. setproctitle on Windows
> > > > > >>> > > >> - Using this client API changes the logger configuration
> > > > > >because it
> > > > > >>> > > >> requires importing the airflow package.
> > > > > >>> > > >>
> > > > > >>> > > >> I think this remote mode in CLI is something valuable,
> > but I
> > > > > >think
> > > > > >>> we
> > > > > >>> > > >can
> > > > > >>> > > >> do it in a different way in the future, e.g. generate a
> > > > > >CLI/API
> > > > > >>> > > >Client
> > > > > >>> > > >> based on the OpenAPI specification.
> > > > > >>> > > >>
> > > > > >>> > > >> Generated API clients can be installed independently of
> > > > > >airflow and
> > > > > >>> > > >will be
> > > > > >>> > > >> easier to maintain. We already have one API client for
> > > golang
> > > > > >>> > > >implemented
> > > > > >>> > > >> in this way, so new languages will only be developing
> this
> > > > > >idea.
> > > > > >>> > > >> - https://github.com/apache/airflow-client-go
> > > > > >>> > > >>
> > > > > >>> > > >> I will be happy to discuss the vision of the development
> > of
> > > > > >these
> > > > > >>> two
> > > > > >>> > > >> things. How do we want to develop these two things?
> > > > > >>> > > >>
> > > > > >>> > > >> Best regards,
> > > > > >>> > > >> Kamil Bregula
> > > > > >>> > > >>
> > > > > >>> > >
> > > > > >>> >
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >>
> > > > > >> Jarek Potiuk
> > > > > >> Polidea <https://www.polidea.com/> | Principal Software
> Engineer
> > > > > >>
> > > > > >> M: +48 660 796 129 <+48660796129>
> > > > > >> [image: Polidea] <https://www.polidea.com/>
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >--
> > > > > >
> > > > > >Jarek Potiuk
> > > > > >Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > > >
> > > > > >M: +48 660 796 129 <+48660796129>
> > > > > >[image: Polidea] <https://www.polidea.com/>
> > > > >
> > > > --
> > > > Sent from Gmail Mobile
> > > >
> > >
> >
>


-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[Ry Walker <ry...@rywalker.com>]

Yes I agree the new API would be made default, and experimental would be
disabled by default.

Before the new API goes live, we should allow minor improvements to the
Experimental API (for example https://github.com/apache/airflow/pull/10308,
which addresses to some degree the security hole in the experimental API
for users of Airflow RBAC) which could change your last sentence a little,
Kamil.


On Thu, Aug 13, 2020 at 1:54 PM Kamil Breguła <ka...@polidea.com>
wrote:

> I agree. We do not have to completely delete the experimental API in
> Airflow 2.0, but I think it is worth turning off by default so that the
> user has to make a conscious decision that they want to use the API, which
> provides a limited level of security-  no permission control, all
> authorized users have full power.
>
> On Thu, Aug 13, 2020 at 7:42 PM QP Hou <qp...@scribd.com> wrote:
>
> > Yeah, i am also curious to know more about the reason why we want to nuke
> > the experimental api code soon instead of just marking it as deprecated.
> >
> > As for getting more insights into remote mode cli usage, would it make
> > sense to make it part of the airflow user survey?
> >
> > Thanks,
> > QP Hou
> >
> >
> > On Thu, Aug 13, 2020 at 7:18 AM Ry Walker <ry...@rywalker.com> wrote:
> >
> > > I would think we would deprecate the old API once we say the new API is
> > > “ready to go” - and leave it in place a while as users transition to
> new
> > > API. Why is there an urgency to remove it from codebase?
> > >
> > > On Thu, Aug 13, 2020 at 5:46 AM Ash Berlin-Taylor <as...@apache.org>
> > wrote:
> > >
> > > > Removing the experimental is a fundamental breaking change to users'
> > > > workflows, and so we should remove it before 2.0.
> > > >
> > > > -ash
> > > >
> > > > On 13 August 2020 10:14:02 BST, Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > >
> > > > wrote:
> > > > >And I think we should make the whole experimenta API deprecated in
> > > > >1.10.12
> > > > >possibly ?
> > > > >
> > > > >On Thu, Aug 13, 2020 at 11:12 AM Jarek Potiuk
> > > > ><Ja...@polidea.com>
> > > > >wrote:
> > > > >
> > > > >> I think no matter what, Maybe we should simply make it deprecated
> in
> > > > >the
> > > > >> upcoming (today?) release of 1.10.12 ? Then we can decide if in -
> > > > >> potential  - 1.10.13 we remove it or leave it as it is.
> > > > >>
> > > > >> On Wed, Aug 12, 2020 at 6:37 PM Kamil Breguła
> > > > ><ka...@polidea.com>
> > > > >> wrote:
> > > > >>
> > > > >>> I started this thread mainly to discuss what we want to do with
> > this
> > > > >>> remote
> > > > >>> mode prior to the Airflow 2.0 release. This is mainly due to the
> > > > >fact that
> > > > >>> he is using an experimental API which will be deprecated.
> > > > >>>
> > > > >>> In my opinion, we have several solutions.
> > > > >>> a) Delete this mode as unused and not supported.
> > > > >>> b) Rewrite in-core API client to support stable API
> > > > >>> c) Prepare OpenAPI based client and rewrite CLI to use it
> > > > >>> d) Leave as-is
> > > > >>>
> > > > >>> There were various expectations on the mailing list about this
> > mode,
> > > > >but I
> > > > >>> haven't seen anyone actively contribute to it. This brings me
> > > > >further
> > > > >>> questions. Does anyone use this mode in its current form? If no
> one
> > > > >is
> > > > >>> using it, I think we can take more radical steps to start with a
> > > > >blank
> > > > >>> page. This will make it easier to start work and be able to
> iterate
> > > > >over
> > > > >>> it
> > > > >>> faster in the future. This looks like a simple task, but if we
> want
> > > > >to be
> > > > >>> sure that no breaking-changes are made, we should pay off some
> > > > >technical
> > > > >>> debt and increase testing coverage before we can think about
> making
> > > > >more
> > > > >>> changes. It may not be necessary if we choose a different path of
> > > > >>> development.
> > > > >>>
> > > > >>> I think now is a good time to try to make some decisions if
> someone
> > > > >is
> > > > >>> actually interested in developing these features. I do not think
> we
> > > > >need a
> > > > >>> precise vision of the development, if currently this feature is
> not
> > > > >used
> > > > >>> by
> > > > >>> anyone and no one is really interested in its development.
> > > > >>>
> > > > >>> On Wed, Aug 12, 2020 at 7:16 AM QP Hou <qp...@scribd.com> wrote:
> > > > >>>
> > > > >>> > I think it's best to divide the discussion into two separate
> > > > >topics.
> > > > >>> >
> > > > >>> > First one is to replace the existing json_client with the new
> to
> > > > >be
> > > > >>> created
> > > > >>> > official Airflow Python Client backed by the new RESTful API.
> > This
> > > > >IMHO
> > > > >>> is
> > > > >>> > a must have considering we are to deprecate experimental API
> > going
> > > > >>> forward.
> > > > >>> >
> > > > >>> > The second topic is to create a better CLI experience
> leveraging
> > > > >the new
> > > > >>> > APIs. This is much more controversial. I remember us having a
> > > > >>> > similar discussion in the dev list a year ago, which didn't get
> > > > >much
> > > > >>> > traction. It's not possible to fit all existing CLI
> > > > >functionalities into
> > > > >>> > REST APIs. DB utils use-case that Ash mentioned is a great
> > > > >example. So I
> > > > >>> > think one potential solution is to split the CLI into two. Keep
> > > > >the
> > > > >>> > existing CLI as the admin/management CLI can communicate
> directly
> > > > >with
> > > > >>> the
> > > > >>> > DB and taps into airflow core code base. On the other hand, we
> > can
> > > > >>> create a
> > > > >>> > separate user facing CLI that's light weight, fast and remote
> > > > >only. It
> > > > >>> > doesn't even need to be written in python to make it easier to
> > > > >>> distribute
> > > > >>> > as a single binary.
> > > > >>> >
> > > > >>> > Thanks,
> > > > >>> > QP Hou
> > > > >>> >
> > > > >>> >
> > > > >>> > On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor
> > > > ><as...@apache.org>
> > > > >>> wrote:
> > > > >>> >
> > > > >>> > > -1 from me without a firm plan how we will replace it.
> > > > >>> > >
> > > > >>> > > I see keeping it and extending to use the new API would
> ensure
> > > > >that
> > > > >>> > > everything the CLI can do locally (i.e. when airflow
> webserver
> > > > >isn't
> > > > >>> up
> > > > >>> > > yet, with the ) also works over the API with the exception of
> > db
> > > > >>> > utilities.
> > > > >>> > >
> > > > >>> > > -ash
> > > > >>> > >
> > > > >>> > > On 11 August 2020 20:05:56 BST, QP Hou <qp...@scribd.com>
> wrote:
> > > > >>> > > >+1 for replacing the existing remote mode client with the
> open
> > > > >api
> > > > >>> > > >based
> > > > >>> > > >client. IMO, we don't really have other options here because
> > > > >the
> > > > >>> > > >experimental API will be deprecated in the future.
> > > > >>> > > >
> > > > >>> > > >For OpenAPI based Airflow REST clients, the current plan is
> to
> > > > >>> maintain
> > > > >>> > > >all
> > > > >>> > > >the code gen automation within the main source tree [1],
> then
> > > > >use it
> > > > >>> to
> > > > >>> > > >populate each individual language specific client repo like
> > the
> > > > >go
> > > > >>> > > >client
> > > > >>> > > >mentioned earlier. So far, we have the go client completed
> and
> > > > >>> > > >validated to
> > > > >>> > > >make sure this development flow will meet our needs. The
> next
> > > > >step I
> > > > >>> > > >think
> > > > >>> > > >the community should focus on is getting API auth
> implemented
> > > > >[2]
> > > > >>> > > >before we
> > > > >>> > > >move on to generate the python client. How we do API auth
> > could
> > > > >have
> > > > >>> a
> > > > >>> > > >big
> > > > >>> > > >impact on client code gen automation, so it is worth waiting
> > > > >for.
> > > > >>> > > >
> > > > >>> > > >Once we have authentication implemented in both Airflow core
> > > > >and
> > > > >>> > > >clients,
> > > > >>> > > >we should be all good to start doing version releases for
> our
> > > > >API
> > > > >>> > > >clients.
> > > > >>> > > >
> > > > >>> > > >That said, adopting open api based clients in the CLI alone
> > > > >won't
> > > > >>> > > >address
> > > > >>> > > >the issue of CLI depending on full airflow installation.
> Some
> > > > >of the
> > > > >>> > > >cli
> > > > >>> > > >commands like `dags test` depend on a full airflow
> > installation
> > > > >by
> > > > >>> > > >design.
> > > > >>> > > >We will have to either develop a separate CLI intended for
> > > > >remote
> > > > >>> only
> > > > >>> > > >use
> > > > >>> > > >or add a flag in the existing cli so it can run in a pure
> > > > >remote mode
> > > > >>> > > >where
> > > > >>> > > >it would disable loading of code that requires airflow
> > > > >installation
> > > > >>> > > >entirely.
> > > > >>> > > >
> > > > >>> > > >[1]: https://github.com/apache/airflow/tree/master/clients,
> > > > >>> > > >[2]: https://github.com/apache/airflow/issues/8112
> > > > >>> > > >
> > > > >>> > > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła
> > > > >>> > > ><ka...@polidea.com>
> > > > >>> > > >wrote:
> > > > >>> > > >
> > > > >>> > > >> Hello,
> > > > >>> > > >>
> > > > >>> > > >> I think we should remove remote mode in CLI and in-core
> API
> > > > >Client
> > > > >>> > > >> (airflow.api.client package).
> > > > >>> > > >> Here is docs about remote mode:
> > > > >>> > > >>
> > > > >>> > > >>
> > > > >>> > > >
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >
> > > >
> > >
> >
> https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
> > > > >>> > > >>
> > > > >>> > > >> Since these features were introduced, it has never been
> > > > >actively
> > > > >>> > > >developed
> > > > >>> > > >> and I don't think it's widely used. At the same time,
> Apache
> > > > >>> Airflow
> > > > >>> > > >is
> > > > >>> > > >> evolving, and this code stands out more and more from the
> > > > >rest.
> > > > >>> > > >>
> > > > >>> > > >> My main reservations about these features:
> > > > >>> > > >> - Remote mode/in-core API Client is rarely used. I asked a
> > > > >few
> > > > >>> people
> > > > >>> > > >and
> > > > >>> > > >> none of them used it in production. Does anyone use it?
> > > > >>> > > >> - A very small number of commands are available (7 pools
> > > > >command
> > > > >>> and
> > > > >>> > > >2 dags
> > > > >>> > > >> command only)
> > > > >>> > > >> - Remote mode/API Client depends on experimental REST API.
> > > > >>> > > >> - Remote mode/API Client is a handwritten code that is
> > > > >difficult to
> > > > >>> > > >> maintain.
> > > > >>> > > >> - No documentation for API client
> > > > >>> > > >> - Remote mode/API Client has low test coverage.
> > > > >>> > > >> - Remote mode does not provide a good level of security,
> > > > >because it
> > > > >>> > > >depends
> > > > >>> > > >> on experimental API. There is the only authentication, but
> > > > >the
> > > > >>> > > >> authenticated user can perform any operation.
> > > > >>> > > >> - Requires full Airflow to be installed along with a large
> > > > >number
> > > > >>> of
> > > > >>> > > >> unnecessary dependencies. Some of them are difficult to
> > > > >install in
> > > > >>> > > >some
> > > > >>> > > >> environments, e.g. setproctitle on Windows
> > > > >>> > > >> - Using this client API changes the logger configuration
> > > > >because it
> > > > >>> > > >> requires importing the airflow package.
> > > > >>> > > >>
> > > > >>> > > >> I think this remote mode in CLI is something valuable,
> but I
> > > > >think
> > > > >>> we
> > > > >>> > > >can
> > > > >>> > > >> do it in a different way in the future, e.g. generate a
> > > > >CLI/API
> > > > >>> > > >Client
> > > > >>> > > >> based on the OpenAPI specification.
> > > > >>> > > >>
> > > > >>> > > >> Generated API clients can be installed independently of
> > > > >airflow and
> > > > >>> > > >will be
> > > > >>> > > >> easier to maintain. We already have one API client for
> > golang
> > > > >>> > > >implemented
> > > > >>> > > >> in this way, so new languages will only be developing this
> > > > >idea.
> > > > >>> > > >> - https://github.com/apache/airflow-client-go
> > > > >>> > > >>
> > > > >>> > > >> I will be happy to discuss the vision of the development
> of
> > > > >these
> > > > >>> two
> > > > >>> > > >> things. How do we want to develop these two things?
> > > > >>> > > >>
> > > > >>> > > >> Best regards,
> > > > >>> > > >> Kamil Bregula
> > > > >>> > > >>
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >>
> > > > >> --
> > > > >>
> > > > >> Jarek Potiuk
> > > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > >>
> > > > >> M: +48 660 796 129 <+48660796129>
> > > > >> [image: Polidea] <https://www.polidea.com/>
> > > > >>
> > > > >>
> > > > >
> > > > >--
> > > > >
> > > > >Jarek Potiuk
> > > > >Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > >
> > > > >M: +48 660 796 129 <+48660796129>
> > > > >[image: Polidea] <https://www.polidea.com/>
> > > >
> > > --
> > > Sent from Gmail Mobile
> > >
> >
>

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[Kamil Breguła <ka...@polidea.com>]

I agree. We do not have to completely delete the experimental API in
Airflow 2.0, but I think it is worth turning off by default so that the
user has to make a conscious decision that they want to use the API, which
provides a limited level of security-  no permission control, all
authorized users have full power.

On Thu, Aug 13, 2020 at 7:42 PM QP Hou <qp...@scribd.com> wrote:

> Yeah, i am also curious to know more about the reason why we want to nuke
> the experimental api code soon instead of just marking it as deprecated.
>
> As for getting more insights into remote mode cli usage, would it make
> sense to make it part of the airflow user survey?
>
> Thanks,
> QP Hou
>
>
> On Thu, Aug 13, 2020 at 7:18 AM Ry Walker <ry...@rywalker.com> wrote:
>
> > I would think we would deprecate the old API once we say the new API is
> > “ready to go” - and leave it in place a while as users transition to new
> > API. Why is there an urgency to remove it from codebase?
> >
> > On Thu, Aug 13, 2020 at 5:46 AM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> > > Removing the experimental is a fundamental breaking change to users'
> > > workflows, and so we should remove it before 2.0.
> > >
> > > -ash
> > >
> > > On 13 August 2020 10:14:02 BST, Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > > wrote:
> > > >And I think we should make the whole experimenta API deprecated in
> > > >1.10.12
> > > >possibly ?
> > > >
> > > >On Thu, Aug 13, 2020 at 11:12 AM Jarek Potiuk
> > > ><Ja...@polidea.com>
> > > >wrote:
> > > >
> > > >> I think no matter what, Maybe we should simply make it deprecated in
> > > >the
> > > >> upcoming (today?) release of 1.10.12 ? Then we can decide if in -
> > > >> potential  - 1.10.13 we remove it or leave it as it is.
> > > >>
> > > >> On Wed, Aug 12, 2020 at 6:37 PM Kamil Breguła
> > > ><ka...@polidea.com>
> > > >> wrote:
> > > >>
> > > >>> I started this thread mainly to discuss what we want to do with
> this
> > > >>> remote
> > > >>> mode prior to the Airflow 2.0 release. This is mainly due to the
> > > >fact that
> > > >>> he is using an experimental API which will be deprecated.
> > > >>>
> > > >>> In my opinion, we have several solutions.
> > > >>> a) Delete this mode as unused and not supported.
> > > >>> b) Rewrite in-core API client to support stable API
> > > >>> c) Prepare OpenAPI based client and rewrite CLI to use it
> > > >>> d) Leave as-is
> > > >>>
> > > >>> There were various expectations on the mailing list about this
> mode,
> > > >but I
> > > >>> haven't seen anyone actively contribute to it. This brings me
> > > >further
> > > >>> questions. Does anyone use this mode in its current form? If no one
> > > >is
> > > >>> using it, I think we can take more radical steps to start with a
> > > >blank
> > > >>> page. This will make it easier to start work and be able to iterate
> > > >over
> > > >>> it
> > > >>> faster in the future. This looks like a simple task, but if we want
> > > >to be
> > > >>> sure that no breaking-changes are made, we should pay off some
> > > >technical
> > > >>> debt and increase testing coverage before we can think about making
> > > >more
> > > >>> changes. It may not be necessary if we choose a different path of
> > > >>> development.
> > > >>>
> > > >>> I think now is a good time to try to make some decisions if someone
> > > >is
> > > >>> actually interested in developing these features. I do not think we
> > > >need a
> > > >>> precise vision of the development, if currently this feature is not
> > > >used
> > > >>> by
> > > >>> anyone and no one is really interested in its development.
> > > >>>
> > > >>> On Wed, Aug 12, 2020 at 7:16 AM QP Hou <qp...@scribd.com> wrote:
> > > >>>
> > > >>> > I think it's best to divide the discussion into two separate
> > > >topics.
> > > >>> >
> > > >>> > First one is to replace the existing json_client with the new to
> > > >be
> > > >>> created
> > > >>> > official Airflow Python Client backed by the new RESTful API.
> This
> > > >IMHO
> > > >>> is
> > > >>> > a must have considering we are to deprecate experimental API
> going
> > > >>> forward.
> > > >>> >
> > > >>> > The second topic is to create a better CLI experience leveraging
> > > >the new
> > > >>> > APIs. This is much more controversial. I remember us having a
> > > >>> > similar discussion in the dev list a year ago, which didn't get
> > > >much
> > > >>> > traction. It's not possible to fit all existing CLI
> > > >functionalities into
> > > >>> > REST APIs. DB utils use-case that Ash mentioned is a great
> > > >example. So I
> > > >>> > think one potential solution is to split the CLI into two. Keep
> > > >the
> > > >>> > existing CLI as the admin/management CLI can communicate directly
> > > >with
> > > >>> the
> > > >>> > DB and taps into airflow core code base. On the other hand, we
> can
> > > >>> create a
> > > >>> > separate user facing CLI that's light weight, fast and remote
> > > >only. It
> > > >>> > doesn't even need to be written in python to make it easier to
> > > >>> distribute
> > > >>> > as a single binary.
> > > >>> >
> > > >>> > Thanks,
> > > >>> > QP Hou
> > > >>> >
> > > >>> >
> > > >>> > On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor
> > > ><as...@apache.org>
> > > >>> wrote:
> > > >>> >
> > > >>> > > -1 from me without a firm plan how we will replace it.
> > > >>> > >
> > > >>> > > I see keeping it and extending to use the new API would ensure
> > > >that
> > > >>> > > everything the CLI can do locally (i.e. when airflow webserver
> > > >isn't
> > > >>> up
> > > >>> > > yet, with the ) also works over the API with the exception of
> db
> > > >>> > utilities.
> > > >>> > >
> > > >>> > > -ash
> > > >>> > >
> > > >>> > > On 11 August 2020 20:05:56 BST, QP Hou <qp...@scribd.com> wrote:
> > > >>> > > >+1 for replacing the existing remote mode client with the open
> > > >api
> > > >>> > > >based
> > > >>> > > >client. IMO, we don't really have other options here because
> > > >the
> > > >>> > > >experimental API will be deprecated in the future.
> > > >>> > > >
> > > >>> > > >For OpenAPI based Airflow REST clients, the current plan is to
> > > >>> maintain
> > > >>> > > >all
> > > >>> > > >the code gen automation within the main source tree [1], then
> > > >use it
> > > >>> to
> > > >>> > > >populate each individual language specific client repo like
> the
> > > >go
> > > >>> > > >client
> > > >>> > > >mentioned earlier. So far, we have the go client completed and
> > > >>> > > >validated to
> > > >>> > > >make sure this development flow will meet our needs. The next
> > > >step I
> > > >>> > > >think
> > > >>> > > >the community should focus on is getting API auth implemented
> > > >[2]
> > > >>> > > >before we
> > > >>> > > >move on to generate the python client. How we do API auth
> could
> > > >have
> > > >>> a
> > > >>> > > >big
> > > >>> > > >impact on client code gen automation, so it is worth waiting
> > > >for.
> > > >>> > > >
> > > >>> > > >Once we have authentication implemented in both Airflow core
> > > >and
> > > >>> > > >clients,
> > > >>> > > >we should be all good to start doing version releases for our
> > > >API
> > > >>> > > >clients.
> > > >>> > > >
> > > >>> > > >That said, adopting open api based clients in the CLI alone
> > > >won't
> > > >>> > > >address
> > > >>> > > >the issue of CLI depending on full airflow installation. Some
> > > >of the
> > > >>> > > >cli
> > > >>> > > >commands like `dags test` depend on a full airflow
> installation
> > > >by
> > > >>> > > >design.
> > > >>> > > >We will have to either develop a separate CLI intended for
> > > >remote
> > > >>> only
> > > >>> > > >use
> > > >>> > > >or add a flag in the existing cli so it can run in a pure
> > > >remote mode
> > > >>> > > >where
> > > >>> > > >it would disable loading of code that requires airflow
> > > >installation
> > > >>> > > >entirely.
> > > >>> > > >
> > > >>> > > >[1]: https://github.com/apache/airflow/tree/master/clients,
> > > >>> > > >[2]: https://github.com/apache/airflow/issues/8112
> > > >>> > > >
> > > >>> > > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła
> > > >>> > > ><ka...@polidea.com>
> > > >>> > > >wrote:
> > > >>> > > >
> > > >>> > > >> Hello,
> > > >>> > > >>
> > > >>> > > >> I think we should remove remote mode in CLI and in-core API
> > > >Client
> > > >>> > > >> (airflow.api.client package).
> > > >>> > > >> Here is docs about remote mode:
> > > >>> > > >>
> > > >>> > > >>
> > > >>> > > >
> > > >>> > >
> > > >>> >
> > > >>>
> > > >
> > >
> >
> https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
> > > >>> > > >>
> > > >>> > > >> Since these features were introduced, it has never been
> > > >actively
> > > >>> > > >developed
> > > >>> > > >> and I don't think it's widely used. At the same time, Apache
> > > >>> Airflow
> > > >>> > > >is
> > > >>> > > >> evolving, and this code stands out more and more from the
> > > >rest.
> > > >>> > > >>
> > > >>> > > >> My main reservations about these features:
> > > >>> > > >> - Remote mode/in-core API Client is rarely used. I asked a
> > > >few
> > > >>> people
> > > >>> > > >and
> > > >>> > > >> none of them used it in production. Does anyone use it?
> > > >>> > > >> - A very small number of commands are available (7 pools
> > > >command
> > > >>> and
> > > >>> > > >2 dags
> > > >>> > > >> command only)
> > > >>> > > >> - Remote mode/API Client depends on experimental REST API.
> > > >>> > > >> - Remote mode/API Client is a handwritten code that is
> > > >difficult to
> > > >>> > > >> maintain.
> > > >>> > > >> - No documentation for API client
> > > >>> > > >> - Remote mode/API Client has low test coverage.
> > > >>> > > >> - Remote mode does not provide a good level of security,
> > > >because it
> > > >>> > > >depends
> > > >>> > > >> on experimental API. There is the only authentication, but
> > > >the
> > > >>> > > >> authenticated user can perform any operation.
> > > >>> > > >> - Requires full Airflow to be installed along with a large
> > > >number
> > > >>> of
> > > >>> > > >> unnecessary dependencies. Some of them are difficult to
> > > >install in
> > > >>> > > >some
> > > >>> > > >> environments, e.g. setproctitle on Windows
> > > >>> > > >> - Using this client API changes the logger configuration
> > > >because it
> > > >>> > > >> requires importing the airflow package.
> > > >>> > > >>
> > > >>> > > >> I think this remote mode in CLI is something valuable, but I
> > > >think
> > > >>> we
> > > >>> > > >can
> > > >>> > > >> do it in a different way in the future, e.g. generate a
> > > >CLI/API
> > > >>> > > >Client
> > > >>> > > >> based on the OpenAPI specification.
> > > >>> > > >>
> > > >>> > > >> Generated API clients can be installed independently of
> > > >airflow and
> > > >>> > > >will be
> > > >>> > > >> easier to maintain. We already have one API client for
> golang
> > > >>> > > >implemented
> > > >>> > > >> in this way, so new languages will only be developing this
> > > >idea.
> > > >>> > > >> - https://github.com/apache/airflow-client-go
> > > >>> > > >>
> > > >>> > > >> I will be happy to discuss the vision of the development of
> > > >these
> > > >>> two
> > > >>> > > >> things. How do we want to develop these two things?
> > > >>> > > >>
> > > >>> > > >> Best regards,
> > > >>> > > >> Kamil Bregula
> > > >>> > > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Jarek Potiuk
> > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >>
> > > >> M: +48 660 796 129 <+48660796129>
> > > >> [image: Polidea] <https://www.polidea.com/>
> > > >>
> > > >>
> > > >
> > > >--
> > > >
> > > >Jarek Potiuk
> > > >Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > >M: +48 660 796 129 <+48660796129>
> > > >[image: Polidea] <https://www.polidea.com/>
> > >
> > --
> > Sent from Gmail Mobile
> >
>

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[QP Hou <qp...@scribd.com>]

Yeah, i am also curious to know more about the reason why we want to nuke
the experimental api code soon instead of just marking it as deprecated.

As for getting more insights into remote mode cli usage, would it make
sense to make it part of the airflow user survey?

Thanks,
QP Hou


On Thu, Aug 13, 2020 at 7:18 AM Ry Walker <ry...@rywalker.com> wrote:

> I would think we would deprecate the old API once we say the new API is
> “ready to go” - and leave it in place a while as users transition to new
> API. Why is there an urgency to remove it from codebase?
>
> On Thu, Aug 13, 2020 at 5:46 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > Removing the experimental is a fundamental breaking change to users'
> > workflows, and so we should remove it before 2.0.
> >
> > -ash
> >
> > On 13 August 2020 10:14:02 BST, Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> > >And I think we should make the whole experimenta API deprecated in
> > >1.10.12
> > >possibly ?
> > >
> > >On Thu, Aug 13, 2020 at 11:12 AM Jarek Potiuk
> > ><Ja...@polidea.com>
> > >wrote:
> > >
> > >> I think no matter what, Maybe we should simply make it deprecated in
> > >the
> > >> upcoming (today?) release of 1.10.12 ? Then we can decide if in -
> > >> potential  - 1.10.13 we remove it or leave it as it is.
> > >>
> > >> On Wed, Aug 12, 2020 at 6:37 PM Kamil Breguła
> > ><ka...@polidea.com>
> > >> wrote:
> > >>
> > >>> I started this thread mainly to discuss what we want to do with this
> > >>> remote
> > >>> mode prior to the Airflow 2.0 release. This is mainly due to the
> > >fact that
> > >>> he is using an experimental API which will be deprecated.
> > >>>
> > >>> In my opinion, we have several solutions.
> > >>> a) Delete this mode as unused and not supported.
> > >>> b) Rewrite in-core API client to support stable API
> > >>> c) Prepare OpenAPI based client and rewrite CLI to use it
> > >>> d) Leave as-is
> > >>>
> > >>> There were various expectations on the mailing list about this mode,
> > >but I
> > >>> haven't seen anyone actively contribute to it. This brings me
> > >further
> > >>> questions. Does anyone use this mode in its current form? If no one
> > >is
> > >>> using it, I think we can take more radical steps to start with a
> > >blank
> > >>> page. This will make it easier to start work and be able to iterate
> > >over
> > >>> it
> > >>> faster in the future. This looks like a simple task, but if we want
> > >to be
> > >>> sure that no breaking-changes are made, we should pay off some
> > >technical
> > >>> debt and increase testing coverage before we can think about making
> > >more
> > >>> changes. It may not be necessary if we choose a different path of
> > >>> development.
> > >>>
> > >>> I think now is a good time to try to make some decisions if someone
> > >is
> > >>> actually interested in developing these features. I do not think we
> > >need a
> > >>> precise vision of the development, if currently this feature is not
> > >used
> > >>> by
> > >>> anyone and no one is really interested in its development.
> > >>>
> > >>> On Wed, Aug 12, 2020 at 7:16 AM QP Hou <qp...@scribd.com> wrote:
> > >>>
> > >>> > I think it's best to divide the discussion into two separate
> > >topics.
> > >>> >
> > >>> > First one is to replace the existing json_client with the new to
> > >be
> > >>> created
> > >>> > official Airflow Python Client backed by the new RESTful API. This
> > >IMHO
> > >>> is
> > >>> > a must have considering we are to deprecate experimental API going
> > >>> forward.
> > >>> >
> > >>> > The second topic is to create a better CLI experience leveraging
> > >the new
> > >>> > APIs. This is much more controversial. I remember us having a
> > >>> > similar discussion in the dev list a year ago, which didn't get
> > >much
> > >>> > traction. It's not possible to fit all existing CLI
> > >functionalities into
> > >>> > REST APIs. DB utils use-case that Ash mentioned is a great
> > >example. So I
> > >>> > think one potential solution is to split the CLI into two. Keep
> > >the
> > >>> > existing CLI as the admin/management CLI can communicate directly
> > >with
> > >>> the
> > >>> > DB and taps into airflow core code base. On the other hand, we can
> > >>> create a
> > >>> > separate user facing CLI that's light weight, fast and remote
> > >only. It
> > >>> > doesn't even need to be written in python to make it easier to
> > >>> distribute
> > >>> > as a single binary.
> > >>> >
> > >>> > Thanks,
> > >>> > QP Hou
> > >>> >
> > >>> >
> > >>> > On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor
> > ><as...@apache.org>
> > >>> wrote:
> > >>> >
> > >>> > > -1 from me without a firm plan how we will replace it.
> > >>> > >
> > >>> > > I see keeping it and extending to use the new API would ensure
> > >that
> > >>> > > everything the CLI can do locally (i.e. when airflow webserver
> > >isn't
> > >>> up
> > >>> > > yet, with the ) also works over the API with the exception of db
> > >>> > utilities.
> > >>> > >
> > >>> > > -ash
> > >>> > >
> > >>> > > On 11 August 2020 20:05:56 BST, QP Hou <qp...@scribd.com> wrote:
> > >>> > > >+1 for replacing the existing remote mode client with the open
> > >api
> > >>> > > >based
> > >>> > > >client. IMO, we don't really have other options here because
> > >the
> > >>> > > >experimental API will be deprecated in the future.
> > >>> > > >
> > >>> > > >For OpenAPI based Airflow REST clients, the current plan is to
> > >>> maintain
> > >>> > > >all
> > >>> > > >the code gen automation within the main source tree [1], then
> > >use it
> > >>> to
> > >>> > > >populate each individual language specific client repo like the
> > >go
> > >>> > > >client
> > >>> > > >mentioned earlier. So far, we have the go client completed and
> > >>> > > >validated to
> > >>> > > >make sure this development flow will meet our needs. The next
> > >step I
> > >>> > > >think
> > >>> > > >the community should focus on is getting API auth implemented
> > >[2]
> > >>> > > >before we
> > >>> > > >move on to generate the python client. How we do API auth could
> > >have
> > >>> a
> > >>> > > >big
> > >>> > > >impact on client code gen automation, so it is worth waiting
> > >for.
> > >>> > > >
> > >>> > > >Once we have authentication implemented in both Airflow core
> > >and
> > >>> > > >clients,
> > >>> > > >we should be all good to start doing version releases for our
> > >API
> > >>> > > >clients.
> > >>> > > >
> > >>> > > >That said, adopting open api based clients in the CLI alone
> > >won't
> > >>> > > >address
> > >>> > > >the issue of CLI depending on full airflow installation. Some
> > >of the
> > >>> > > >cli
> > >>> > > >commands like `dags test` depend on a full airflow installation
> > >by
> > >>> > > >design.
> > >>> > > >We will have to either develop a separate CLI intended for
> > >remote
> > >>> only
> > >>> > > >use
> > >>> > > >or add a flag in the existing cli so it can run in a pure
> > >remote mode
> > >>> > > >where
> > >>> > > >it would disable loading of code that requires airflow
> > >installation
> > >>> > > >entirely.
> > >>> > > >
> > >>> > > >[1]: https://github.com/apache/airflow/tree/master/clients,
> > >>> > > >[2]: https://github.com/apache/airflow/issues/8112
> > >>> > > >
> > >>> > > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła
> > >>> > > ><ka...@polidea.com>
> > >>> > > >wrote:
> > >>> > > >
> > >>> > > >> Hello,
> > >>> > > >>
> > >>> > > >> I think we should remove remote mode in CLI and in-core API
> > >Client
> > >>> > > >> (airflow.api.client package).
> > >>> > > >> Here is docs about remote mode:
> > >>> > > >>
> > >>> > > >>
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >
> >
> https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
> > >>> > > >>
> > >>> > > >> Since these features were introduced, it has never been
> > >actively
> > >>> > > >developed
> > >>> > > >> and I don't think it's widely used. At the same time, Apache
> > >>> Airflow
> > >>> > > >is
> > >>> > > >> evolving, and this code stands out more and more from the
> > >rest.
> > >>> > > >>
> > >>> > > >> My main reservations about these features:
> > >>> > > >> - Remote mode/in-core API Client is rarely used. I asked a
> > >few
> > >>> people
> > >>> > > >and
> > >>> > > >> none of them used it in production. Does anyone use it?
> > >>> > > >> - A very small number of commands are available (7 pools
> > >command
> > >>> and
> > >>> > > >2 dags
> > >>> > > >> command only)
> > >>> > > >> - Remote mode/API Client depends on experimental REST API.
> > >>> > > >> - Remote mode/API Client is a handwritten code that is
> > >difficult to
> > >>> > > >> maintain.
> > >>> > > >> - No documentation for API client
> > >>> > > >> - Remote mode/API Client has low test coverage.
> > >>> > > >> - Remote mode does not provide a good level of security,
> > >because it
> > >>> > > >depends
> > >>> > > >> on experimental API. There is the only authentication, but
> > >the
> > >>> > > >> authenticated user can perform any operation.
> > >>> > > >> - Requires full Airflow to be installed along with a large
> > >number
> > >>> of
> > >>> > > >> unnecessary dependencies. Some of them are difficult to
> > >install in
> > >>> > > >some
> > >>> > > >> environments, e.g. setproctitle on Windows
> > >>> > > >> - Using this client API changes the logger configuration
> > >because it
> > >>> > > >> requires importing the airflow package.
> > >>> > > >>
> > >>> > > >> I think this remote mode in CLI is something valuable, but I
> > >think
> > >>> we
> > >>> > > >can
> > >>> > > >> do it in a different way in the future, e.g. generate a
> > >CLI/API
> > >>> > > >Client
> > >>> > > >> based on the OpenAPI specification.
> > >>> > > >>
> > >>> > > >> Generated API clients can be installed independently of
> > >airflow and
> > >>> > > >will be
> > >>> > > >> easier to maintain. We already have one API client for golang
> > >>> > > >implemented
> > >>> > > >> in this way, so new languages will only be developing this
> > >idea.
> > >>> > > >> - https://github.com/apache/airflow-client-go
> > >>> > > >>
> > >>> > > >> I will be happy to discuss the vision of the development of
> > >these
> > >>> two
> > >>> > > >> things. How do we want to develop these two things?
> > >>> > > >>
> > >>> > > >> Best regards,
> > >>> > > >> Kamil Bregula
> > >>> > > >>
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >> --
> > >>
> > >> Jarek Potiuk
> > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>
> > >> M: +48 660 796 129 <+48660796129>
> > >> [image: Polidea] <https://www.polidea.com/>
> > >>
> > >>
> > >
> > >--
> > >
> > >Jarek Potiuk
> > >Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > >M: +48 660 796 129 <+48660796129>
> > >[image: Polidea] <https://www.polidea.com/>
> >
> --
> Sent from Gmail Mobile
>

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[Ry Walker <ry...@rywalker.com>]

I would think we would deprecate the old API once we say the new API is
“ready to go” - and leave it in place a while as users transition to new
API. Why is there an urgency to remove it from codebase?

On Thu, Aug 13, 2020 at 5:46 AM Ash Berlin-Taylor <as...@apache.org> wrote:

> Removing the experimental is a fundamental breaking change to users'
> workflows, and so we should remove it before 2.0.
>
> -ash
>
> On 13 August 2020 10:14:02 BST, Jarek Potiuk <Ja...@polidea.com>
> wrote:
> >And I think we should make the whole experimenta API deprecated in
> >1.10.12
> >possibly ?
> >
> >On Thu, Aug 13, 2020 at 11:12 AM Jarek Potiuk
> ><Ja...@polidea.com>
> >wrote:
> >
> >> I think no matter what, Maybe we should simply make it deprecated in
> >the
> >> upcoming (today?) release of 1.10.12 ? Then we can decide if in -
> >> potential  - 1.10.13 we remove it or leave it as it is.
> >>
> >> On Wed, Aug 12, 2020 at 6:37 PM Kamil Breguła
> ><ka...@polidea.com>
> >> wrote:
> >>
> >>> I started this thread mainly to discuss what we want to do with this
> >>> remote
> >>> mode prior to the Airflow 2.0 release. This is mainly due to the
> >fact that
> >>> he is using an experimental API which will be deprecated.
> >>>
> >>> In my opinion, we have several solutions.
> >>> a) Delete this mode as unused and not supported.
> >>> b) Rewrite in-core API client to support stable API
> >>> c) Prepare OpenAPI based client and rewrite CLI to use it
> >>> d) Leave as-is
> >>>
> >>> There were various expectations on the mailing list about this mode,
> >but I
> >>> haven't seen anyone actively contribute to it. This brings me
> >further
> >>> questions. Does anyone use this mode in its current form? If no one
> >is
> >>> using it, I think we can take more radical steps to start with a
> >blank
> >>> page. This will make it easier to start work and be able to iterate
> >over
> >>> it
> >>> faster in the future. This looks like a simple task, but if we want
> >to be
> >>> sure that no breaking-changes are made, we should pay off some
> >technical
> >>> debt and increase testing coverage before we can think about making
> >more
> >>> changes. It may not be necessary if we choose a different path of
> >>> development.
> >>>
> >>> I think now is a good time to try to make some decisions if someone
> >is
> >>> actually interested in developing these features. I do not think we
> >need a
> >>> precise vision of the development, if currently this feature is not
> >used
> >>> by
> >>> anyone and no one is really interested in its development.
> >>>
> >>> On Wed, Aug 12, 2020 at 7:16 AM QP Hou <qp...@scribd.com> wrote:
> >>>
> >>> > I think it's best to divide the discussion into two separate
> >topics.
> >>> >
> >>> > First one is to replace the existing json_client with the new to
> >be
> >>> created
> >>> > official Airflow Python Client backed by the new RESTful API. This
> >IMHO
> >>> is
> >>> > a must have considering we are to deprecate experimental API going
> >>> forward.
> >>> >
> >>> > The second topic is to create a better CLI experience leveraging
> >the new
> >>> > APIs. This is much more controversial. I remember us having a
> >>> > similar discussion in the dev list a year ago, which didn't get
> >much
> >>> > traction. It's not possible to fit all existing CLI
> >functionalities into
> >>> > REST APIs. DB utils use-case that Ash mentioned is a great
> >example. So I
> >>> > think one potential solution is to split the CLI into two. Keep
> >the
> >>> > existing CLI as the admin/management CLI can communicate directly
> >with
> >>> the
> >>> > DB and taps into airflow core code base. On the other hand, we can
> >>> create a
> >>> > separate user facing CLI that's light weight, fast and remote
> >only. It
> >>> > doesn't even need to be written in python to make it easier to
> >>> distribute
> >>> > as a single binary.
> >>> >
> >>> > Thanks,
> >>> > QP Hou
> >>> >
> >>> >
> >>> > On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor
> ><as...@apache.org>
> >>> wrote:
> >>> >
> >>> > > -1 from me without a firm plan how we will replace it.
> >>> > >
> >>> > > I see keeping it and extending to use the new API would ensure
> >that
> >>> > > everything the CLI can do locally (i.e. when airflow webserver
> >isn't
> >>> up
> >>> > > yet, with the ) also works over the API with the exception of db
> >>> > utilities.
> >>> > >
> >>> > > -ash
> >>> > >
> >>> > > On 11 August 2020 20:05:56 BST, QP Hou <qp...@scribd.com> wrote:
> >>> > > >+1 for replacing the existing remote mode client with the open
> >api
> >>> > > >based
> >>> > > >client. IMO, we don't really have other options here because
> >the
> >>> > > >experimental API will be deprecated in the future.
> >>> > > >
> >>> > > >For OpenAPI based Airflow REST clients, the current plan is to
> >>> maintain
> >>> > > >all
> >>> > > >the code gen automation within the main source tree [1], then
> >use it
> >>> to
> >>> > > >populate each individual language specific client repo like the
> >go
> >>> > > >client
> >>> > > >mentioned earlier. So far, we have the go client completed and
> >>> > > >validated to
> >>> > > >make sure this development flow will meet our needs. The next
> >step I
> >>> > > >think
> >>> > > >the community should focus on is getting API auth implemented
> >[2]
> >>> > > >before we
> >>> > > >move on to generate the python client. How we do API auth could
> >have
> >>> a
> >>> > > >big
> >>> > > >impact on client code gen automation, so it is worth waiting
> >for.
> >>> > > >
> >>> > > >Once we have authentication implemented in both Airflow core
> >and
> >>> > > >clients,
> >>> > > >we should be all good to start doing version releases for our
> >API
> >>> > > >clients.
> >>> > > >
> >>> > > >That said, adopting open api based clients in the CLI alone
> >won't
> >>> > > >address
> >>> > > >the issue of CLI depending on full airflow installation. Some
> >of the
> >>> > > >cli
> >>> > > >commands like `dags test` depend on a full airflow installation
> >by
> >>> > > >design.
> >>> > > >We will have to either develop a separate CLI intended for
> >remote
> >>> only
> >>> > > >use
> >>> > > >or add a flag in the existing cli so it can run in a pure
> >remote mode
> >>> > > >where
> >>> > > >it would disable loading of code that requires airflow
> >installation
> >>> > > >entirely.
> >>> > > >
> >>> > > >[1]: https://github.com/apache/airflow/tree/master/clients,
> >>> > > >[2]: https://github.com/apache/airflow/issues/8112
> >>> > > >
> >>> > > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła
> >>> > > ><ka...@polidea.com>
> >>> > > >wrote:
> >>> > > >
> >>> > > >> Hello,
> >>> > > >>
> >>> > > >> I think we should remove remote mode in CLI and in-core API
> >Client
> >>> > > >> (airflow.api.client package).
> >>> > > >> Here is docs about remote mode:
> >>> > > >>
> >>> > > >>
> >>> > > >
> >>> > >
> >>> >
> >>>
> >
> https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
> >>> > > >>
> >>> > > >> Since these features were introduced, it has never been
> >actively
> >>> > > >developed
> >>> > > >> and I don't think it's widely used. At the same time, Apache
> >>> Airflow
> >>> > > >is
> >>> > > >> evolving, and this code stands out more and more from the
> >rest.
> >>> > > >>
> >>> > > >> My main reservations about these features:
> >>> > > >> - Remote mode/in-core API Client is rarely used. I asked a
> >few
> >>> people
> >>> > > >and
> >>> > > >> none of them used it in production. Does anyone use it?
> >>> > > >> - A very small number of commands are available (7 pools
> >command
> >>> and
> >>> > > >2 dags
> >>> > > >> command only)
> >>> > > >> - Remote mode/API Client depends on experimental REST API.
> >>> > > >> - Remote mode/API Client is a handwritten code that is
> >difficult to
> >>> > > >> maintain.
> >>> > > >> - No documentation for API client
> >>> > > >> - Remote mode/API Client has low test coverage.
> >>> > > >> - Remote mode does not provide a good level of security,
> >because it
> >>> > > >depends
> >>> > > >> on experimental API. There is the only authentication, but
> >the
> >>> > > >> authenticated user can perform any operation.
> >>> > > >> - Requires full Airflow to be installed along with a large
> >number
> >>> of
> >>> > > >> unnecessary dependencies. Some of them are difficult to
> >install in
> >>> > > >some
> >>> > > >> environments, e.g. setproctitle on Windows
> >>> > > >> - Using this client API changes the logger configuration
> >because it
> >>> > > >> requires importing the airflow package.
> >>> > > >>
> >>> > > >> I think this remote mode in CLI is something valuable, but I
> >think
> >>> we
> >>> > > >can
> >>> > > >> do it in a different way in the future, e.g. generate a
> >CLI/API
> >>> > > >Client
> >>> > > >> based on the OpenAPI specification.
> >>> > > >>
> >>> > > >> Generated API clients can be installed independently of
> >airflow and
> >>> > > >will be
> >>> > > >> easier to maintain. We already have one API client for golang
> >>> > > >implemented
> >>> > > >> in this way, so new languages will only be developing this
> >idea.
> >>> > > >> - https://github.com/apache/airflow-client-go
> >>> > > >>
> >>> > > >> I will be happy to discuss the vision of the development of
> >these
> >>> two
> >>> > > >> things. How do we want to develop these two things?
> >>> > > >>
> >>> > > >> Best regards,
> >>> > > >> Kamil Bregula
> >>> > > >>
> >>> > >
> >>> >
> >>>
> >>
> >>
> >> --
> >>
> >> Jarek Potiuk
> >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>
> >> M: +48 660 796 129 <+48660796129>
> >> [image: Polidea] <https://www.polidea.com/>
> >>
> >>
> >
> >--
> >
> >Jarek Potiuk
> >Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> >M: +48 660 796 129 <+48660796129>
> >[image: Polidea] <https://www.polidea.com/>
>
-- 
Sent from Gmail Mobile

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[Ash Berlin-Taylor <as...@apache.org>]

Removing the experimental is a fundamental breaking change to users' workflows, and so we should remove it before 2.0.

-ash

On 13 August 2020 10:14:02 BST, Jarek Potiuk <Ja...@polidea.com> wrote:
>And I think we should make the whole experimenta API deprecated in
>1.10.12
>possibly ?
>
>On Thu, Aug 13, 2020 at 11:12 AM Jarek Potiuk
><Ja...@polidea.com>
>wrote:
>
>> I think no matter what, Maybe we should simply make it deprecated in
>the
>> upcoming (today?) release of 1.10.12 ? Then we can decide if in -
>> potential  - 1.10.13 we remove it or leave it as it is.
>>
>> On Wed, Aug 12, 2020 at 6:37 PM Kamil Breguła
><ka...@polidea.com>
>> wrote:
>>
>>> I started this thread mainly to discuss what we want to do with this
>>> remote
>>> mode prior to the Airflow 2.0 release. This is mainly due to the
>fact that
>>> he is using an experimental API which will be deprecated.
>>>
>>> In my opinion, we have several solutions.
>>> a) Delete this mode as unused and not supported.
>>> b) Rewrite in-core API client to support stable API
>>> c) Prepare OpenAPI based client and rewrite CLI to use it
>>> d) Leave as-is
>>>
>>> There were various expectations on the mailing list about this mode,
>but I
>>> haven't seen anyone actively contribute to it. This brings me
>further
>>> questions. Does anyone use this mode in its current form? If no one
>is
>>> using it, I think we can take more radical steps to start with a
>blank
>>> page. This will make it easier to start work and be able to iterate
>over
>>> it
>>> faster in the future. This looks like a simple task, but if we want
>to be
>>> sure that no breaking-changes are made, we should pay off some
>technical
>>> debt and increase testing coverage before we can think about making
>more
>>> changes. It may not be necessary if we choose a different path of
>>> development.
>>>
>>> I think now is a good time to try to make some decisions if someone
>is
>>> actually interested in developing these features. I do not think we
>need a
>>> precise vision of the development, if currently this feature is not
>used
>>> by
>>> anyone and no one is really interested in its development.
>>>
>>> On Wed, Aug 12, 2020 at 7:16 AM QP Hou <qp...@scribd.com> wrote:
>>>
>>> > I think it's best to divide the discussion into two separate
>topics.
>>> >
>>> > First one is to replace the existing json_client with the new to
>be
>>> created
>>> > official Airflow Python Client backed by the new RESTful API. This
>IMHO
>>> is
>>> > a must have considering we are to deprecate experimental API going
>>> forward.
>>> >
>>> > The second topic is to create a better CLI experience leveraging
>the new
>>> > APIs. This is much more controversial. I remember us having a
>>> > similar discussion in the dev list a year ago, which didn't get
>much
>>> > traction. It's not possible to fit all existing CLI
>functionalities into
>>> > REST APIs. DB utils use-case that Ash mentioned is a great
>example. So I
>>> > think one potential solution is to split the CLI into two. Keep
>the
>>> > existing CLI as the admin/management CLI can communicate directly
>with
>>> the
>>> > DB and taps into airflow core code base. On the other hand, we can
>>> create a
>>> > separate user facing CLI that's light weight, fast and remote
>only. It
>>> > doesn't even need to be written in python to make it easier to
>>> distribute
>>> > as a single binary.
>>> >
>>> > Thanks,
>>> > QP Hou
>>> >
>>> >
>>> > On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor
><as...@apache.org>
>>> wrote:
>>> >
>>> > > -1 from me without a firm plan how we will replace it.
>>> > >
>>> > > I see keeping it and extending to use the new API would ensure
>that
>>> > > everything the CLI can do locally (i.e. when airflow webserver
>isn't
>>> up
>>> > > yet, with the ) also works over the API with the exception of db
>>> > utilities.
>>> > >
>>> > > -ash
>>> > >
>>> > > On 11 August 2020 20:05:56 BST, QP Hou <qp...@scribd.com> wrote:
>>> > > >+1 for replacing the existing remote mode client with the open
>api
>>> > > >based
>>> > > >client. IMO, we don't really have other options here because
>the
>>> > > >experimental API will be deprecated in the future.
>>> > > >
>>> > > >For OpenAPI based Airflow REST clients, the current plan is to
>>> maintain
>>> > > >all
>>> > > >the code gen automation within the main source tree [1], then
>use it
>>> to
>>> > > >populate each individual language specific client repo like the
>go
>>> > > >client
>>> > > >mentioned earlier. So far, we have the go client completed and
>>> > > >validated to
>>> > > >make sure this development flow will meet our needs. The next
>step I
>>> > > >think
>>> > > >the community should focus on is getting API auth implemented
>[2]
>>> > > >before we
>>> > > >move on to generate the python client. How we do API auth could
>have
>>> a
>>> > > >big
>>> > > >impact on client code gen automation, so it is worth waiting
>for.
>>> > > >
>>> > > >Once we have authentication implemented in both Airflow core
>and
>>> > > >clients,
>>> > > >we should be all good to start doing version releases for our
>API
>>> > > >clients.
>>> > > >
>>> > > >That said, adopting open api based clients in the CLI alone
>won't
>>> > > >address
>>> > > >the issue of CLI depending on full airflow installation. Some
>of the
>>> > > >cli
>>> > > >commands like `dags test` depend on a full airflow installation
>by
>>> > > >design.
>>> > > >We will have to either develop a separate CLI intended for
>remote
>>> only
>>> > > >use
>>> > > >or add a flag in the existing cli so it can run in a pure
>remote mode
>>> > > >where
>>> > > >it would disable loading of code that requires airflow
>installation
>>> > > >entirely.
>>> > > >
>>> > > >[1]: https://github.com/apache/airflow/tree/master/clients,
>>> > > >[2]: https://github.com/apache/airflow/issues/8112
>>> > > >
>>> > > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła
>>> > > ><ka...@polidea.com>
>>> > > >wrote:
>>> > > >
>>> > > >> Hello,
>>> > > >>
>>> > > >> I think we should remove remote mode in CLI and in-core API
>Client
>>> > > >> (airflow.api.client package).
>>> > > >> Here is docs about remote mode:
>>> > > >>
>>> > > >>
>>> > > >
>>> > >
>>> >
>>>
>https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
>>> > > >>
>>> > > >> Since these features were introduced, it has never been
>actively
>>> > > >developed
>>> > > >> and I don't think it's widely used. At the same time, Apache
>>> Airflow
>>> > > >is
>>> > > >> evolving, and this code stands out more and more from the
>rest.
>>> > > >>
>>> > > >> My main reservations about these features:
>>> > > >> - Remote mode/in-core API Client is rarely used. I asked a
>few
>>> people
>>> > > >and
>>> > > >> none of them used it in production. Does anyone use it?
>>> > > >> - A very small number of commands are available (7 pools
>command
>>> and
>>> > > >2 dags
>>> > > >> command only)
>>> > > >> - Remote mode/API Client depends on experimental REST API.
>>> > > >> - Remote mode/API Client is a handwritten code that is
>difficult to
>>> > > >> maintain.
>>> > > >> - No documentation for API client
>>> > > >> - Remote mode/API Client has low test coverage.
>>> > > >> - Remote mode does not provide a good level of security,
>because it
>>> > > >depends
>>> > > >> on experimental API. There is the only authentication, but
>the
>>> > > >> authenticated user can perform any operation.
>>> > > >> - Requires full Airflow to be installed along with a large
>number
>>> of
>>> > > >> unnecessary dependencies. Some of them are difficult to
>install in
>>> > > >some
>>> > > >> environments, e.g. setproctitle on Windows
>>> > > >> - Using this client API changes the logger configuration
>because it
>>> > > >> requires importing the airflow package.
>>> > > >>
>>> > > >> I think this remote mode in CLI is something valuable, but I
>think
>>> we
>>> > > >can
>>> > > >> do it in a different way in the future, e.g. generate a
>CLI/API
>>> > > >Client
>>> > > >> based on the OpenAPI specification.
>>> > > >>
>>> > > >> Generated API clients can be installed independently of
>airflow and
>>> > > >will be
>>> > > >> easier to maintain. We already have one API client for golang
>>> > > >implemented
>>> > > >> in this way, so new languages will only be developing this
>idea.
>>> > > >> - https://github.com/apache/airflow-client-go
>>> > > >>
>>> > > >> I will be happy to discuss the vision of the development of
>these
>>> two
>>> > > >> things. How do we want to develop these two things?
>>> > > >>
>>> > > >> Best regards,
>>> > > >> Kamil Bregula
>>> > > >>
>>> > >
>>> >
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
>-- 
>
>Jarek Potiuk
>Polidea <https://www.polidea.com/> | Principal Software Engineer
>
>M: +48 660 796 129 <+48660796129>
>[image: Polidea] <https://www.polidea.com/>

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[Jarek Potiuk <Ja...@polidea.com>]

And I think we should make the whole experimenta API deprecated in 1.10.12
possibly ?

On Thu, Aug 13, 2020 at 11:12 AM Jarek Potiuk <Ja...@polidea.com>
wrote:

> I think no matter what, Maybe we should simply make it deprecated in the
> upcoming (today?) release of 1.10.12 ? Then we can decide if in -
> potential  - 1.10.13 we remove it or leave it as it is.
>
> On Wed, Aug 12, 2020 at 6:37 PM Kamil Breguła <ka...@polidea.com>
> wrote:
>
>> I started this thread mainly to discuss what we want to do with this
>> remote
>> mode prior to the Airflow 2.0 release. This is mainly due to the fact that
>> he is using an experimental API which will be deprecated.
>>
>> In my opinion, we have several solutions.
>> a) Delete this mode as unused and not supported.
>> b) Rewrite in-core API client to support stable API
>> c) Prepare OpenAPI based client and rewrite CLI to use it
>> d) Leave as-is
>>
>> There were various expectations on the mailing list about this mode, but I
>> haven't seen anyone actively contribute to it. This brings me further
>> questions. Does anyone use this mode in its current form? If no one is
>> using it, I think we can take more radical steps to start with a blank
>> page. This will make it easier to start work and be able to iterate over
>> it
>> faster in the future. This looks like a simple task, but if we want to be
>> sure that no breaking-changes are made, we should pay off some technical
>> debt and increase testing coverage before we can think about making more
>> changes. It may not be necessary if we choose a different path of
>> development.
>>
>> I think now is a good time to try to make some decisions if someone is
>> actually interested in developing these features. I do not think we need a
>> precise vision of the development, if currently this feature is not used
>> by
>> anyone and no one is really interested in its development.
>>
>> On Wed, Aug 12, 2020 at 7:16 AM QP Hou <qp...@scribd.com> wrote:
>>
>> > I think it's best to divide the discussion into two separate topics.
>> >
>> > First one is to replace the existing json_client with the new to be
>> created
>> > official Airflow Python Client backed by the new RESTful API. This IMHO
>> is
>> > a must have considering we are to deprecate experimental API going
>> forward.
>> >
>> > The second topic is to create a better CLI experience leveraging the new
>> > APIs. This is much more controversial. I remember us having a
>> > similar discussion in the dev list a year ago, which didn't get much
>> > traction. It's not possible to fit all existing CLI functionalities into
>> > REST APIs. DB utils use-case that Ash mentioned is a great example. So I
>> > think one potential solution is to split the CLI into two. Keep the
>> > existing CLI as the admin/management CLI can communicate directly with
>> the
>> > DB and taps into airflow core code base. On the other hand, we can
>> create a
>> > separate user facing CLI that's light weight, fast and remote only. It
>> > doesn't even need to be written in python to make it easier to
>> distribute
>> > as a single binary.
>> >
>> > Thanks,
>> > QP Hou
>> >
>> >
>> > On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor <as...@apache.org>
>> wrote:
>> >
>> > > -1 from me without a firm plan how we will replace it.
>> > >
>> > > I see keeping it and extending to use the new API would ensure that
>> > > everything the CLI can do locally (i.e. when airflow webserver isn't
>> up
>> > > yet, with the ) also works over the API with the exception of db
>> > utilities.
>> > >
>> > > -ash
>> > >
>> > > On 11 August 2020 20:05:56 BST, QP Hou <qp...@scribd.com> wrote:
>> > > >+1 for replacing the existing remote mode client with the open api
>> > > >based
>> > > >client. IMO, we don't really have other options here because the
>> > > >experimental API will be deprecated in the future.
>> > > >
>> > > >For OpenAPI based Airflow REST clients, the current plan is to
>> maintain
>> > > >all
>> > > >the code gen automation within the main source tree [1], then use it
>> to
>> > > >populate each individual language specific client repo like the go
>> > > >client
>> > > >mentioned earlier. So far, we have the go client completed and
>> > > >validated to
>> > > >make sure this development flow will meet our needs. The next step I
>> > > >think
>> > > >the community should focus on is getting API auth implemented [2]
>> > > >before we
>> > > >move on to generate the python client. How we do API auth could have
>> a
>> > > >big
>> > > >impact on client code gen automation, so it is worth waiting for.
>> > > >
>> > > >Once we have authentication implemented in both Airflow core and
>> > > >clients,
>> > > >we should be all good to start doing version releases for our API
>> > > >clients.
>> > > >
>> > > >That said, adopting open api based clients in the CLI alone won't
>> > > >address
>> > > >the issue of CLI depending on full airflow installation. Some of the
>> > > >cli
>> > > >commands like `dags test` depend on a full airflow installation by
>> > > >design.
>> > > >We will have to either develop a separate CLI intended for remote
>> only
>> > > >use
>> > > >or add a flag in the existing cli so it can run in a pure remote mode
>> > > >where
>> > > >it would disable loading of code that requires airflow installation
>> > > >entirely.
>> > > >
>> > > >[1]: https://github.com/apache/airflow/tree/master/clients,
>> > > >[2]: https://github.com/apache/airflow/issues/8112
>> > > >
>> > > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła
>> > > ><ka...@polidea.com>
>> > > >wrote:
>> > > >
>> > > >> Hello,
>> > > >>
>> > > >> I think we should remove remote mode in CLI and in-core API Client
>> > > >> (airflow.api.client package).
>> > > >> Here is docs about remote mode:
>> > > >>
>> > > >>
>> > > >
>> > >
>> >
>> https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
>> > > >>
>> > > >> Since these features were introduced, it has never been actively
>> > > >developed
>> > > >> and I don't think it's widely used. At the same time, Apache
>> Airflow
>> > > >is
>> > > >> evolving, and this code stands out more and more from the rest.
>> > > >>
>> > > >> My main reservations about these features:
>> > > >> - Remote mode/in-core API Client is rarely used. I asked a few
>> people
>> > > >and
>> > > >> none of them used it in production. Does anyone use it?
>> > > >> - A very small number of commands are available (7 pools command
>> and
>> > > >2 dags
>> > > >> command only)
>> > > >> - Remote mode/API Client depends on experimental REST API.
>> > > >> - Remote mode/API Client is a handwritten code that is difficult to
>> > > >> maintain.
>> > > >> - No documentation for API client
>> > > >> - Remote mode/API Client has low test coverage.
>> > > >> - Remote mode does not provide a good level of security, because it
>> > > >depends
>> > > >> on experimental API. There is the only authentication, but the
>> > > >> authenticated user can perform any operation.
>> > > >> - Requires full Airflow to be installed along with a large number
>> of
>> > > >> unnecessary dependencies. Some of them are difficult to install in
>> > > >some
>> > > >> environments, e.g. setproctitle on Windows
>> > > >> - Using this client API changes the logger configuration because it
>> > > >> requires importing the airflow package.
>> > > >>
>> > > >> I think this remote mode in CLI is something valuable, but I think
>> we
>> > > >can
>> > > >> do it in a different way in the future, e.g. generate a CLI/API
>> > > >Client
>> > > >> based on the OpenAPI specification.
>> > > >>
>> > > >> Generated API clients can be installed independently of airflow and
>> > > >will be
>> > > >> easier to maintain. We already have one API client for golang
>> > > >implemented
>> > > >> in this way, so new languages will only be developing this idea.
>> > > >> - https://github.com/apache/airflow-client-go
>> > > >>
>> > > >> I will be happy to discuss the vision of the development of these
>> two
>> > > >> things. How do we want to develop these two things?
>> > > >>
>> > > >> Best regards,
>> > > >> Kamil Bregula
>> > > >>
>> > >
>> >
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[Jarek Potiuk <Ja...@polidea.com>]

I think no matter what, Maybe we should simply make it deprecated in the
upcoming (today?) release of 1.10.12 ? Then we can decide if in -
potential  - 1.10.13 we remove it or leave it as it is.

On Wed, Aug 12, 2020 at 6:37 PM Kamil Breguła <ka...@polidea.com>
wrote:

> I started this thread mainly to discuss what we want to do with this remote
> mode prior to the Airflow 2.0 release. This is mainly due to the fact that
> he is using an experimental API which will be deprecated.
>
> In my opinion, we have several solutions.
> a) Delete this mode as unused and not supported.
> b) Rewrite in-core API client to support stable API
> c) Prepare OpenAPI based client and rewrite CLI to use it
> d) Leave as-is
>
> There were various expectations on the mailing list about this mode, but I
> haven't seen anyone actively contribute to it. This brings me further
> questions. Does anyone use this mode in its current form? If no one is
> using it, I think we can take more radical steps to start with a blank
> page. This will make it easier to start work and be able to iterate over it
> faster in the future. This looks like a simple task, but if we want to be
> sure that no breaking-changes are made, we should pay off some technical
> debt and increase testing coverage before we can think about making more
> changes. It may not be necessary if we choose a different path of
> development.
>
> I think now is a good time to try to make some decisions if someone is
> actually interested in developing these features. I do not think we need a
> precise vision of the development, if currently this feature is not used by
> anyone and no one is really interested in its development.
>
> On Wed, Aug 12, 2020 at 7:16 AM QP Hou <qp...@scribd.com> wrote:
>
> > I think it's best to divide the discussion into two separate topics.
> >
> > First one is to replace the existing json_client with the new to be
> created
> > official Airflow Python Client backed by the new RESTful API. This IMHO
> is
> > a must have considering we are to deprecate experimental API going
> forward.
> >
> > The second topic is to create a better CLI experience leveraging the new
> > APIs. This is much more controversial. I remember us having a
> > similar discussion in the dev list a year ago, which didn't get much
> > traction. It's not possible to fit all existing CLI functionalities into
> > REST APIs. DB utils use-case that Ash mentioned is a great example. So I
> > think one potential solution is to split the CLI into two. Keep the
> > existing CLI as the admin/management CLI can communicate directly with
> the
> > DB and taps into airflow core code base. On the other hand, we can
> create a
> > separate user facing CLI that's light weight, fast and remote only. It
> > doesn't even need to be written in python to make it easier to distribute
> > as a single binary.
> >
> > Thanks,
> > QP Hou
> >
> >
> > On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> > > -1 from me without a firm plan how we will replace it.
> > >
> > > I see keeping it and extending to use the new API would ensure that
> > > everything the CLI can do locally (i.e. when airflow webserver isn't up
> > > yet, with the ) also works over the API with the exception of db
> > utilities.
> > >
> > > -ash
> > >
> > > On 11 August 2020 20:05:56 BST, QP Hou <qp...@scribd.com> wrote:
> > > >+1 for replacing the existing remote mode client with the open api
> > > >based
> > > >client. IMO, we don't really have other options here because the
> > > >experimental API will be deprecated in the future.
> > > >
> > > >For OpenAPI based Airflow REST clients, the current plan is to
> maintain
> > > >all
> > > >the code gen automation within the main source tree [1], then use it
> to
> > > >populate each individual language specific client repo like the go
> > > >client
> > > >mentioned earlier. So far, we have the go client completed and
> > > >validated to
> > > >make sure this development flow will meet our needs. The next step I
> > > >think
> > > >the community should focus on is getting API auth implemented [2]
> > > >before we
> > > >move on to generate the python client. How we do API auth could have a
> > > >big
> > > >impact on client code gen automation, so it is worth waiting for.
> > > >
> > > >Once we have authentication implemented in both Airflow core and
> > > >clients,
> > > >we should be all good to start doing version releases for our API
> > > >clients.
> > > >
> > > >That said, adopting open api based clients in the CLI alone won't
> > > >address
> > > >the issue of CLI depending on full airflow installation. Some of the
> > > >cli
> > > >commands like `dags test` depend on a full airflow installation by
> > > >design.
> > > >We will have to either develop a separate CLI intended for remote only
> > > >use
> > > >or add a flag in the existing cli so it can run in a pure remote mode
> > > >where
> > > >it would disable loading of code that requires airflow installation
> > > >entirely.
> > > >
> > > >[1]: https://github.com/apache/airflow/tree/master/clients,
> > > >[2]: https://github.com/apache/airflow/issues/8112
> > > >
> > > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła
> > > ><ka...@polidea.com>
> > > >wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >> I think we should remove remote mode in CLI and in-core API Client
> > > >> (airflow.api.client package).
> > > >> Here is docs about remote mode:
> > > >>
> > > >>
> > > >
> > >
> >
> https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
> > > >>
> > > >> Since these features were introduced, it has never been actively
> > > >developed
> > > >> and I don't think it's widely used. At the same time, Apache Airflow
> > > >is
> > > >> evolving, and this code stands out more and more from the rest.
> > > >>
> > > >> My main reservations about these features:
> > > >> - Remote mode/in-core API Client is rarely used. I asked a few
> people
> > > >and
> > > >> none of them used it in production. Does anyone use it?
> > > >> - A very small number of commands are available (7 pools command and
> > > >2 dags
> > > >> command only)
> > > >> - Remote mode/API Client depends on experimental REST API.
> > > >> - Remote mode/API Client is a handwritten code that is difficult to
> > > >> maintain.
> > > >> - No documentation for API client
> > > >> - Remote mode/API Client has low test coverage.
> > > >> - Remote mode does not provide a good level of security, because it
> > > >depends
> > > >> on experimental API. There is the only authentication, but the
> > > >> authenticated user can perform any operation.
> > > >> - Requires full Airflow to be installed along with a large number of
> > > >> unnecessary dependencies. Some of them are difficult to install in
> > > >some
> > > >> environments, e.g. setproctitle on Windows
> > > >> - Using this client API changes the logger configuration because it
> > > >> requires importing the airflow package.
> > > >>
> > > >> I think this remote mode in CLI is something valuable, but I think
> we
> > > >can
> > > >> do it in a different way in the future, e.g. generate a CLI/API
> > > >Client
> > > >> based on the OpenAPI specification.
> > > >>
> > > >> Generated API clients can be installed independently of airflow and
> > > >will be
> > > >> easier to maintain. We already have one API client for golang
> > > >implemented
> > > >> in this way, so new languages will only be developing this idea.
> > > >> - https://github.com/apache/airflow-client-go
> > > >>
> > > >> I will be happy to discuss the vision of the development of these
> two
> > > >> things. How do we want to develop these two things?
> > > >>
> > > >> Best regards,
> > > >> Kamil Bregula
> > > >>
> > >
> >
>


-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[Kamil Breguła <ka...@polidea.com>]

I started this thread mainly to discuss what we want to do with this remote
mode prior to the Airflow 2.0 release. This is mainly due to the fact that
he is using an experimental API which will be deprecated.

In my opinion, we have several solutions.
a) Delete this mode as unused and not supported.
b) Rewrite in-core API client to support stable API
c) Prepare OpenAPI based client and rewrite CLI to use it
d) Leave as-is

There were various expectations on the mailing list about this mode, but I
haven't seen anyone actively contribute to it. This brings me further
questions. Does anyone use this mode in its current form? If no one is
using it, I think we can take more radical steps to start with a blank
page. This will make it easier to start work and be able to iterate over it
faster in the future. This looks like a simple task, but if we want to be
sure that no breaking-changes are made, we should pay off some technical
debt and increase testing coverage before we can think about making more
changes. It may not be necessary if we choose a different path of
development.

I think now is a good time to try to make some decisions if someone is
actually interested in developing these features. I do not think we need a
precise vision of the development, if currently this feature is not used by
anyone and no one is really interested in its development.

On Wed, Aug 12, 2020 at 7:16 AM QP Hou <qp...@scribd.com> wrote:

> I think it's best to divide the discussion into two separate topics.
>
> First one is to replace the existing json_client with the new to be created
> official Airflow Python Client backed by the new RESTful API. This IMHO is
> a must have considering we are to deprecate experimental API going forward.
>
> The second topic is to create a better CLI experience leveraging the new
> APIs. This is much more controversial. I remember us having a
> similar discussion in the dev list a year ago, which didn't get much
> traction. It's not possible to fit all existing CLI functionalities into
> REST APIs. DB utils use-case that Ash mentioned is a great example. So I
> think one potential solution is to split the CLI into two. Keep the
> existing CLI as the admin/management CLI can communicate directly with the
> DB and taps into airflow core code base. On the other hand, we can create a
> separate user facing CLI that's light weight, fast and remote only. It
> doesn't even need to be written in python to make it easier to distribute
> as a single binary.
>
> Thanks,
> QP Hou
>
>
> On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > -1 from me without a firm plan how we will replace it.
> >
> > I see keeping it and extending to use the new API would ensure that
> > everything the CLI can do locally (i.e. when airflow webserver isn't up
> > yet, with the ) also works over the API with the exception of db
> utilities.
> >
> > -ash
> >
> > On 11 August 2020 20:05:56 BST, QP Hou <qp...@scribd.com> wrote:
> > >+1 for replacing the existing remote mode client with the open api
> > >based
> > >client. IMO, we don't really have other options here because the
> > >experimental API will be deprecated in the future.
> > >
> > >For OpenAPI based Airflow REST clients, the current plan is to maintain
> > >all
> > >the code gen automation within the main source tree [1], then use it to
> > >populate each individual language specific client repo like the go
> > >client
> > >mentioned earlier. So far, we have the go client completed and
> > >validated to
> > >make sure this development flow will meet our needs. The next step I
> > >think
> > >the community should focus on is getting API auth implemented [2]
> > >before we
> > >move on to generate the python client. How we do API auth could have a
> > >big
> > >impact on client code gen automation, so it is worth waiting for.
> > >
> > >Once we have authentication implemented in both Airflow core and
> > >clients,
> > >we should be all good to start doing version releases for our API
> > >clients.
> > >
> > >That said, adopting open api based clients in the CLI alone won't
> > >address
> > >the issue of CLI depending on full airflow installation. Some of the
> > >cli
> > >commands like `dags test` depend on a full airflow installation by
> > >design.
> > >We will have to either develop a separate CLI intended for remote only
> > >use
> > >or add a flag in the existing cli so it can run in a pure remote mode
> > >where
> > >it would disable loading of code that requires airflow installation
> > >entirely.
> > >
> > >[1]: https://github.com/apache/airflow/tree/master/clients,
> > >[2]: https://github.com/apache/airflow/issues/8112
> > >
> > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła
> > ><ka...@polidea.com>
> > >wrote:
> > >
> > >> Hello,
> > >>
> > >> I think we should remove remote mode in CLI and in-core API Client
> > >> (airflow.api.client package).
> > >> Here is docs about remote mode:
> > >>
> > >>
> > >
> >
> https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
> > >>
> > >> Since these features were introduced, it has never been actively
> > >developed
> > >> and I don't think it's widely used. At the same time, Apache Airflow
> > >is
> > >> evolving, and this code stands out more and more from the rest.
> > >>
> > >> My main reservations about these features:
> > >> - Remote mode/in-core API Client is rarely used. I asked a few people
> > >and
> > >> none of them used it in production. Does anyone use it?
> > >> - A very small number of commands are available (7 pools command and
> > >2 dags
> > >> command only)
> > >> - Remote mode/API Client depends on experimental REST API.
> > >> - Remote mode/API Client is a handwritten code that is difficult to
> > >> maintain.
> > >> - No documentation for API client
> > >> - Remote mode/API Client has low test coverage.
> > >> - Remote mode does not provide a good level of security, because it
> > >depends
> > >> on experimental API. There is the only authentication, but the
> > >> authenticated user can perform any operation.
> > >> - Requires full Airflow to be installed along with a large number of
> > >> unnecessary dependencies. Some of them are difficult to install in
> > >some
> > >> environments, e.g. setproctitle on Windows
> > >> - Using this client API changes the logger configuration because it
> > >> requires importing the airflow package.
> > >>
> > >> I think this remote mode in CLI is something valuable, but I think we
> > >can
> > >> do it in a different way in the future, e.g. generate a CLI/API
> > >Client
> > >> based on the OpenAPI specification.
> > >>
> > >> Generated API clients can be installed independently of airflow and
> > >will be
> > >> easier to maintain. We already have one API client for golang
> > >implemented
> > >> in this way, so new languages will only be developing this idea.
> > >> - https://github.com/apache/airflow-client-go
> > >>
> > >> I will be happy to discuss the vision of the development of these two
> > >> things. How do we want to develop these two things?
> > >>
> > >> Best regards,
> > >> Kamil Bregula
> > >>
> >
>

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[QP Hou <qp...@scribd.com>]

I think it's best to divide the discussion into two separate topics.

First one is to replace the existing json_client with the new to be created
official Airflow Python Client backed by the new RESTful API. This IMHO is
a must have considering we are to deprecate experimental API going forward.

The second topic is to create a better CLI experience leveraging the new
APIs. This is much more controversial. I remember us having a
similar discussion in the dev list a year ago, which didn't get much
traction. It's not possible to fit all existing CLI functionalities into
REST APIs. DB utils use-case that Ash mentioned is a great example. So I
think one potential solution is to split the CLI into two. Keep the
existing CLI as the admin/management CLI can communicate directly with the
DB and taps into airflow core code base. On the other hand, we can create a
separate user facing CLI that's light weight, fast and remote only. It
doesn't even need to be written in python to make it easier to distribute
as a single binary.

Thanks,
QP Hou


On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor <as...@apache.org> wrote:

> -1 from me without a firm plan how we will replace it.
>
> I see keeping it and extending to use the new API would ensure that
> everything the CLI can do locally (i.e. when airflow webserver isn't up
> yet, with the ) also works over the API with the exception of db utilities.
>
> -ash
>
> On 11 August 2020 20:05:56 BST, QP Hou <qp...@scribd.com> wrote:
> >+1 for replacing the existing remote mode client with the open api
> >based
> >client. IMO, we don't really have other options here because the
> >experimental API will be deprecated in the future.
> >
> >For OpenAPI based Airflow REST clients, the current plan is to maintain
> >all
> >the code gen automation within the main source tree [1], then use it to
> >populate each individual language specific client repo like the go
> >client
> >mentioned earlier. So far, we have the go client completed and
> >validated to
> >make sure this development flow will meet our needs. The next step I
> >think
> >the community should focus on is getting API auth implemented [2]
> >before we
> >move on to generate the python client. How we do API auth could have a
> >big
> >impact on client code gen automation, so it is worth waiting for.
> >
> >Once we have authentication implemented in both Airflow core and
> >clients,
> >we should be all good to start doing version releases for our API
> >clients.
> >
> >That said, adopting open api based clients in the CLI alone won't
> >address
> >the issue of CLI depending on full airflow installation. Some of the
> >cli
> >commands like `dags test` depend on a full airflow installation by
> >design.
> >We will have to either develop a separate CLI intended for remote only
> >use
> >or add a flag in the existing cli so it can run in a pure remote mode
> >where
> >it would disable loading of code that requires airflow installation
> >entirely.
> >
> >[1]: https://github.com/apache/airflow/tree/master/clients,
> >[2]: https://github.com/apache/airflow/issues/8112
> >
> >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła
> ><ka...@polidea.com>
> >wrote:
> >
> >> Hello,
> >>
> >> I think we should remove remote mode in CLI and in-core API Client
> >> (airflow.api.client package).
> >> Here is docs about remote mode:
> >>
> >>
> >
> https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
> >>
> >> Since these features were introduced, it has never been actively
> >developed
> >> and I don't think it's widely used. At the same time, Apache Airflow
> >is
> >> evolving, and this code stands out more and more from the rest.
> >>
> >> My main reservations about these features:
> >> - Remote mode/in-core API Client is rarely used. I asked a few people
> >and
> >> none of them used it in production. Does anyone use it?
> >> - A very small number of commands are available (7 pools command and
> >2 dags
> >> command only)
> >> - Remote mode/API Client depends on experimental REST API.
> >> - Remote mode/API Client is a handwritten code that is difficult to
> >> maintain.
> >> - No documentation for API client
> >> - Remote mode/API Client has low test coverage.
> >> - Remote mode does not provide a good level of security, because it
> >depends
> >> on experimental API. There is the only authentication, but the
> >> authenticated user can perform any operation.
> >> - Requires full Airflow to be installed along with a large number of
> >> unnecessary dependencies. Some of them are difficult to install in
> >some
> >> environments, e.g. setproctitle on Windows
> >> - Using this client API changes the logger configuration because it
> >> requires importing the airflow package.
> >>
> >> I think this remote mode in CLI is something valuable, but I think we
> >can
> >> do it in a different way in the future, e.g. generate a CLI/API
> >Client
> >> based on the OpenAPI specification.
> >>
> >> Generated API clients can be installed independently of airflow and
> >will be
> >> easier to maintain. We already have one API client for golang
> >implemented
> >> in this way, so new languages will only be developing this idea.
> >> - https://github.com/apache/airflow-client-go
> >>
> >> I will be happy to discuss the vision of the development of these two
> >> things. How do we want to develop these two things?
> >>
> >> Best regards,
> >> Kamil Bregula
> >>
>

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[Ash Berlin-Taylor <as...@apache.org>]

-1 from me without a firm plan how we will replace it.

I see keeping it and extending to use the new API would ensure that everything the CLI can do locally (i.e. when airflow webserver isn't up yet, with the ) also works over the API with the exception of db utilities.

-ash

On 11 August 2020 20:05:56 BST, QP Hou <qp...@scribd.com> wrote:
>+1 for replacing the existing remote mode client with the open api
>based
>client. IMO, we don't really have other options here because the
>experimental API will be deprecated in the future.
>
>For OpenAPI based Airflow REST clients, the current plan is to maintain
>all
>the code gen automation within the main source tree [1], then use it to
>populate each individual language specific client repo like the go
>client
>mentioned earlier. So far, we have the go client completed and
>validated to
>make sure this development flow will meet our needs. The next step I
>think
>the community should focus on is getting API auth implemented [2]
>before we
>move on to generate the python client. How we do API auth could have a
>big
>impact on client code gen automation, so it is worth waiting for.
>
>Once we have authentication implemented in both Airflow core and
>clients,
>we should be all good to start doing version releases for our API
>clients.
>
>That said, adopting open api based clients in the CLI alone won't
>address
>the issue of CLI depending on full airflow installation. Some of the
>cli
>commands like `dags test` depend on a full airflow installation by
>design.
>We will have to either develop a separate CLI intended for remote only
>use
>or add a flag in the existing cli so it can run in a pure remote mode
>where
>it would disable loading of code that requires airflow installation
>entirely.
>
>[1]: https://github.com/apache/airflow/tree/master/clients,
>[2]: https://github.com/apache/airflow/issues/8112
>
>On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła
><ka...@polidea.com>
>wrote:
>
>> Hello,
>>
>> I think we should remove remote mode in CLI and in-core API Client
>> (airflow.api.client package).
>> Here is docs about remote mode:
>>
>>
>https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
>>
>> Since these features were introduced, it has never been actively
>developed
>> and I don't think it's widely used. At the same time, Apache Airflow
>is
>> evolving, and this code stands out more and more from the rest.
>>
>> My main reservations about these features:
>> - Remote mode/in-core API Client is rarely used. I asked a few people
>and
>> none of them used it in production. Does anyone use it?
>> - A very small number of commands are available (7 pools command and
>2 dags
>> command only)
>> - Remote mode/API Client depends on experimental REST API.
>> - Remote mode/API Client is a handwritten code that is difficult to
>> maintain.
>> - No documentation for API client
>> - Remote mode/API Client has low test coverage.
>> - Remote mode does not provide a good level of security, because it
>depends
>> on experimental API. There is the only authentication, but the
>> authenticated user can perform any operation.
>> - Requires full Airflow to be installed along with a large number of
>> unnecessary dependencies. Some of them are difficult to install in
>some
>> environments, e.g. setproctitle on Windows
>> - Using this client API changes the logger configuration because it
>> requires importing the airflow package.
>>
>> I think this remote mode in CLI is something valuable, but I think we
>can
>> do it in a different way in the future, e.g. generate a CLI/API
>Client
>> based on the OpenAPI specification.
>>
>> Generated API clients can be installed independently of airflow and
>will be
>> easier to maintain. We already have one API client for golang
>implemented
>> in this way, so new languages will only be developing this idea.
>> - https://github.com/apache/airflow-client-go
>>
>> I will be happy to discuss the vision of the development of these two
>> things. How do we want to develop these two things?
>>
>> Best regards,
>> Kamil Bregula
>>

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

[QP Hou <qp...@scribd.com>]

+1 for replacing the existing remote mode client with the open api based
client. IMO, we don't really have other options here because the
experimental API will be deprecated in the future.

For OpenAPI based Airflow REST clients, the current plan is to maintain all
the code gen automation within the main source tree [1], then use it to
populate each individual language specific client repo like the go client
mentioned earlier. So far, we have the go client completed and validated to
make sure this development flow will meet our needs. The next step I think
the community should focus on is getting API auth implemented [2] before we
move on to generate the python client. How we do API auth could have a big
impact on client code gen automation, so it is worth waiting for.

Once we have authentication implemented in both Airflow core and clients,
we should be all good to start doing version releases for our API clients.

That said, adopting open api based clients in the CLI alone won't address
the issue of CLI depending on full airflow installation. Some of the cli
commands like `dags test` depend on a full airflow installation by design.
We will have to either develop a separate CLI intended for remote only use
or add a flag in the existing cli so it can run in a pure remote mode where
it would disable loading of code that requires airflow installation
entirely.

[1]: https://github.com/apache/airflow/tree/master/clients,
[2]: https://github.com/apache/airflow/issues/8112

On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła <ka...@polidea.com>
wrote:

> Hello,
>
> I think we should remove remote mode in CLI and in-core API Client
> (airflow.api.client package).
> Here is docs about remote mode:
>
> https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance
>
> Since these features were introduced, it has never been actively developed
> and I don't think it's widely used. At the same time, Apache Airflow is
> evolving, and this code stands out more and more from the rest.
>
> My main reservations about these features:
> - Remote mode/in-core API Client is rarely used. I asked a few people and
> none of them used it in production. Does anyone use it?
> - A very small number of commands are available (7 pools command and 2 dags
> command only)
> - Remote mode/API Client depends on experimental REST API.
> - Remote mode/API Client is a handwritten code that is difficult to
> maintain.
> - No documentation for API client
> - Remote mode/API Client has low test coverage.
> - Remote mode does not provide a good level of security, because it depends
> on experimental API. There is the only authentication, but the
> authenticated user can perform any operation.
> - Requires full Airflow to be installed along with a large number of
> unnecessary dependencies. Some of them are difficult to install in some
> environments, e.g. setproctitle on Windows
> - Using this client API changes the logger configuration because it
> requires importing the airflow package.
>
> I think this remote mode in CLI is something valuable, but I think we can
> do it in a different way in the future, e.g. generate a CLI/API Client
> based on the OpenAPI specification.
>
> Generated API clients can be installed independently of airflow and will be
> easier to maintain. We already have one API client for golang implemented
> in this way, so new languages will only be developing this idea.
> - https://github.com/apache/airflow-client-go
>
> I will be happy to discuss the vision of the development of these two
> things. How do we want to develop these two things?
>
> Best regards,
> Kamil Bregula
>