You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by David Cavaletto <da...@gmail.com> on 2019/02/01 02:21:50 UTC
Proposal to remove json_client
Hello,
While trying to refactor the Connection API (both REST and CLI) I've
discovered the CLI has the ability to make REST API calls. The flow looks
like this:
[image: Apache Airflow API flow.jpg]
In a discussion with Ash in Slack, he said this architecture is so that a
user can use a local installation of Airflow (like on their laptop) to
issue CLI commands to a remote installation of Airflow. Examples would
include a cluster running Fargate, where docker exec and ssh are not
available, and so no CLI commands can be run on the installation.
I'd like to propose that we simplify this entire structure, removing the
boxes highlighted in red, like this:
[image: Apache Airflow API flow (1).jpg]
Users needing to issue REST commands to a cluster would use a REST client
(Postman, curl, etc) to issue the commands to the cluster, allowing us to
simplify this implementation of the APIs in Airflow.
Thoughts? Is there some other use case I've overlooked?
-Dave
Re: Proposal to remove json_client
Posted by Bolke de Bruin <bd...@gmail.com>.
The airlow cli can function as this rest client and does not need to be at the same server as where airflow is running.
Direct DB access is bad from a separation of concerns perspective as you can change task statusses, insert arbitrary things etc.
B.
Sent from my iPhone
> On 1 Feb 2019, at 13:44, David Cavaletto <da...@gmail.com> wrote:
>
> Actually the opposite. If we're going to have a REST API, users should
> interact with it over http(s), using a rest client. If a user has SSH
> access to the server running airflow I don't see the security concern of
> having the CLI access the DB in the same manner the REST API does.
>
> In my diagram all DB be access would be in the API/common/experimental/*.py
> files, with both REST and CLI APIs making calls to this internal API.
>
> Can you explain further the security concern with the CLI?
>
> -Dave
>
>
>> On Fri, Feb 1, 2019, 00:24 Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Hi David,
>>
>> I assume that you then want to ensure that all calls from the CLI are
>> being made through the Rest-API. In other words the json_client would be
>> the only client remaining. The local_client is a security problem as it
>> needs direct database access which makes it problematic to have normals
>> user use the cli.
>>
>> If this is the case then yes this would be great, but please note that
>> airflow backfills are more difficult to factor out then.
>>
>> Cheers
>> Bolke
>>
>> Verstuurd vanaf mijn iPad
>>
>>> Op 1 feb. 2019 om 03:26 heeft David Cavaletto <da...@gmail.com>
>> het volgende geschreven:
>>>
>>> In case my images didn't come through, here is a link that shows what
>> I'd like to remove (in red) and add (in green)
>>>
>>>
>> https://docs.google.com/drawings/d/1Ux8qGQUdRp2L6YWgqayliIzFiJv1nwev4JaG2rR7oiQ/edit?usp=sharing
>>>
>>>
>>>> On Thu, Jan 31, 2019 at 8:21 PM David Cavaletto <da...@gmail.com>
>> wrote:
>>>> Hello,
>>>>
>>>> While trying to refactor the Connection API (both REST and CLI) I've
>> discovered the CLI has the ability to make REST API calls. The flow looks
>> like this:
>>>>
>>>>
>>>> In a discussion with Ash in Slack, he said this architecture is so that
>> a user can use a local installation of Airflow (like on their laptop) to
>> issue CLI commands to a remote installation of Airflow. Examples would
>> include a cluster running Fargate, where docker exec and ssh are not
>> available, and so no CLI commands can be run on the installation.
>>>>
>>>> I'd like to propose that we simplify this entire structure, removing
>> the boxes highlighted in red, like this:
>>>>
>>>>
>>>> Users needing to issue REST commands to a cluster would use a REST
>> client (Postman, curl, etc) to issue the commands to the cluster, allowing
>> us to simplify this implementation of the APIs in Airflow.
>>>>
>>>> Thoughts? Is there some other use case I've overlooked?
>>>>
>>>> -Dave
>>
Re: Proposal to remove json_client
Posted by David Cavaletto <da...@gmail.com>.
Actually the opposite. If we're going to have a REST API, users should
interact with it over http(s), using a rest client. If a user has SSH
access to the server running airflow I don't see the security concern of
having the CLI access the DB in the same manner the REST API does.
In my diagram all DB be access would be in the API/common/experimental/*.py
files, with both REST and CLI APIs making calls to this internal API.
Can you explain further the security concern with the CLI?
-Dave
On Fri, Feb 1, 2019, 00:24 Bolke de Bruin <bd...@gmail.com> wrote:
> Hi David,
>
> I assume that you then want to ensure that all calls from the CLI are
> being made through the Rest-API. In other words the json_client would be
> the only client remaining. The local_client is a security problem as it
> needs direct database access which makes it problematic to have normals
> user use the cli.
>
> If this is the case then yes this would be great, but please note that
> airflow backfills are more difficult to factor out then.
>
> Cheers
> Bolke
>
> Verstuurd vanaf mijn iPad
>
> > Op 1 feb. 2019 om 03:26 heeft David Cavaletto <da...@gmail.com>
> het volgende geschreven:
> >
> > In case my images didn't come through, here is a link that shows what
> I'd like to remove (in red) and add (in green)
> >
> >
> https://docs.google.com/drawings/d/1Ux8qGQUdRp2L6YWgqayliIzFiJv1nwev4JaG2rR7oiQ/edit?usp=sharing
> >
> >
> >> On Thu, Jan 31, 2019 at 8:21 PM David Cavaletto <da...@gmail.com>
> wrote:
> >> Hello,
> >>
> >> While trying to refactor the Connection API (both REST and CLI) I've
> discovered the CLI has the ability to make REST API calls. The flow looks
> like this:
> >>
> >>
> >> In a discussion with Ash in Slack, he said this architecture is so that
> a user can use a local installation of Airflow (like on their laptop) to
> issue CLI commands to a remote installation of Airflow. Examples would
> include a cluster running Fargate, where docker exec and ssh are not
> available, and so no CLI commands can be run on the installation.
> >>
> >> I'd like to propose that we simplify this entire structure, removing
> the boxes highlighted in red, like this:
> >>
> >>
> >> Users needing to issue REST commands to a cluster would use a REST
> client (Postman, curl, etc) to issue the commands to the cluster, allowing
> us to simplify this implementation of the APIs in Airflow.
> >>
> >> Thoughts? Is there some other use case I've overlooked?
> >>
> >> -Dave
>
Re: Proposal to remove json_client
Posted by Bolke de Bruin <bd...@gmail.com>.
Hi David,
I assume that you then want to ensure that all calls from the CLI are being made through the Rest-API. In other words the json_client would be the only client remaining. The local_client is a security problem as it needs direct database access which makes it problematic to have normals user use the cli.
If this is the case then yes this would be great, but please note that airflow backfills are more difficult to factor out then.
Cheers
Bolke
Verstuurd vanaf mijn iPad
> Op 1 feb. 2019 om 03:26 heeft David Cavaletto <da...@gmail.com> het volgende geschreven:
>
> In case my images didn't come through, here is a link that shows what I'd like to remove (in red) and add (in green)
>
> https://docs.google.com/drawings/d/1Ux8qGQUdRp2L6YWgqayliIzFiJv1nwev4JaG2rR7oiQ/edit?usp=sharing
>
>
>> On Thu, Jan 31, 2019 at 8:21 PM David Cavaletto <da...@gmail.com> wrote:
>> Hello,
>>
>> While trying to refactor the Connection API (both REST and CLI) I've discovered the CLI has the ability to make REST API calls. The flow looks like this:
>>
>>
>> In a discussion with Ash in Slack, he said this architecture is so that a user can use a local installation of Airflow (like on their laptop) to issue CLI commands to a remote installation of Airflow. Examples would include a cluster running Fargate, where docker exec and ssh are not available, and so no CLI commands can be run on the installation.
>>
>> I'd like to propose that we simplify this entire structure, removing the boxes highlighted in red, like this:
>>
>>
>> Users needing to issue REST commands to a cluster would use a REST client (Postman, curl, etc) to issue the commands to the cluster, allowing us to simplify this implementation of the APIs in Airflow.
>>
>> Thoughts? Is there some other use case I've overlooked?
>>
>> -Dave
Re: Proposal to remove json_client
Posted by David Cavaletto <da...@gmail.com>.
In case my images didn't come through, here is a link that shows what I'd
like to remove (in red) and add (in green)
https://docs.google.com/drawings/d/1Ux8qGQUdRp2L6YWgqayliIzFiJv1nwev4JaG2rR7oiQ/edit?usp=sharing
On Thu, Jan 31, 2019 at 8:21 PM David Cavaletto <da...@gmail.com>
wrote:
> Hello,
>
> While trying to refactor the Connection API (both REST and CLI) I've
> discovered the CLI has the ability to make REST API calls. The flow looks
> like this:
> [image: Apache Airflow API flow.jpg]
>
> In a discussion with Ash in Slack, he said this architecture is so that a
> user can use a local installation of Airflow (like on their laptop) to
> issue CLI commands to a remote installation of Airflow. Examples would
> include a cluster running Fargate, where docker exec and ssh are not
> available, and so no CLI commands can be run on the installation.
>
> I'd like to propose that we simplify this entire structure, removing the
> boxes highlighted in red, like this:
> [image: Apache Airflow API flow (1).jpg]
>
> Users needing to issue REST commands to a cluster would use a REST client
> (Postman, curl, etc) to issue the commands to the cluster, allowing us to
> simplify this implementation of the APIs in Airflow.
>
> Thoughts? Is there some other use case I've overlooked?
>
> -Dave
>