You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/25 17:50:52 UTC

[GitHub] [airflow] mik-laj opened a new issue #10552: Airflow CLI client based on Stable API

mik-laj opened a new issue #10552:
URL: https://github.com/apache/airflow/issues/10552

**tl;dr;**: [Hyperone](https://www.hyperone.com/) and good citizen @ad-m will be happy to help build a new Airflliw CLI.
Here is docs for POC:
https://gist.github.com/ad-m/19cc06d91a7ee756f461c95f5d656eb6

If we have a [Stable API](https://airflow.readthedocs.io/en/latest/stable-rest-api-ref.html) ready, we can start working on tools that use it. A highly anticipated feature is the ability to manage Airflow from the CLI. Currently, there is [similar feature](https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance) in Airflow, but with many limitations:

- A very small number of commands are available (7 pool commands and 2 dag commands only).
- Use deprecated experimental REST API.
- Low test coverage.
- No access control. (Big security risks)
- Requires full Airflow to be installed along with a large number of unnecessary dependencies.
- [Doen't work on Windows](https://github.com/apache/airflow/issues/10388).
- Installation is via pip which is great. However, users expect such a tool to be available as a single package with all dependencies included.

I also suggested that this be deleted in Airflow 2.0 as it is rarely used.
https://lists.apache.org/thread.html/rfada8ac2fce87c0516d62923e35e3bebfc44ee5a379103b890f8c61c%40%3Cdev.airflow.apache.org%3E

Some users use the normal CLI, but this also has many of the disadvantages mentioned above, but for this it poses an even greater security risk as it requires a direct connection to the database. On the other hand, it has many more features, but not all that are available in Web UI, eg there is no access to remote logs but they are available in Web UI and stable API.

I spoke with good citizen @ad-m how [HyperOne](https://www.hyperone.com/) will develop CLI for its services. HyperOne is a cloud provider that develops services for polish good citizens in Poland. They use OpenAPI very intensively for build thier platform. They also strive for automation and self-maintenance. I think we can rely on their experience, knowledge and talent.

From a conversation with this expert, I learned about the existence of the [h1-cli](https://github.com/hyperonecom/h1-cli/) project. This is the CLI that makes it easy to manage HyperOne services. Have the key-features from our perspective:
- Use NodeJS (-/+)
- Uses OpenAPI (+)
- Available as a single binary(+). To achieve it, they used the [pkg](https://github.com/vercel/pkg) utility.
- Works on many multiple operating systems (+)

This is not the end of the story! They are working on [a new CLI v2](https://github.com/hyperonecom/h1-cli/tree/v2) that will use the API not only to verify the message format, but also **the new commands** will be built based on the OpenAPI specification. Yes! Adding a new service to the platform does not require any changes to the CLI.

The fantastic features don't stop there. CLI is written in NodeJS and thanks to the additional [cli-device-browser](https://github.com/hyperonecom/h1-cli/tree/v2/packages/cli-device-browser) module it is possible to use this CLI also **from the browser**. Yes! We can have an airflow console in the browser.

When a resource is created, a CLI command equivalent is also generated. The use of **automatic command generation** makes the maintenance of the CLI a minimum effort. The CLI developed in this way is able to automatically adapt to new API parameters, which reduces the effort required for its maintenance. The reference documentation of CLI is also updated.

CLI is integrated with the HyperOne Management Panel. When an operation is performed via the Panel, an **example of how to perform an analogous operation with CLI** is displayed. Besides the example developed in this way can be directly run in the CLI in the browser. An excellent way to educate users and popularize CLI. This mechanism is part of the CLI framework and we can integrate our web-UI similarly.

I spoke to @ad-m and HyperOne are eager to collaborate and share their experience and technology so we can similarly build our CLI. @ad-m even prepared a POC based on our specification. After writing ~300 lines of code, he had Airfllow CLI ready, which after some improvements, could be used more widely.

The CLI framework is under active development. New functionalities are planned that we will be able to use. I found out that autocomplete support is planned (including remote data). They also work as hard as possible to improve the documentation format, e.g. generate CLI context help based on examples from OpenAPI, provide documentation in new formats.

Here is the documentation that shows the idea of how it will be possible to use such a CLI.

https://gist.github.com/ad-m/19cc06d91a7ee756f461c95f5d656eb6

In a further development, we may write additional commands that address common use cases. The CLI framework assumes that basic operations are generated, and specific operations can additionally be added or even dynamically loaded (plugin-way). At this stage, these POC of CLIs are more advanced and more powerful than the remote mode in the current Airflow CLI.

As of now, I don't have the capacity to lead this idea, so I'm not starting a mailing list discussion. However, if I have positive opinions, I can try to book a few hours for its implementation.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org