You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Mateusz Henc <mh...@google.com.INVALID> on 2022/11/02 13:01:25 UTC
[AIP-44] PoC JSON RPC
Hi,
*JSON RPC API*
Finally I had some time to work on an Internal API based on JSON RPC.
I found that this solution can be really elegant and clean from a developer
perspective.
You can find the OpenAPi definition there
<https://github.com/apache/airflow/commit/e86c389a004b26f22953e804e365ffaee200107c#diff-e454d231c105c349be8619b3112c8bfda74f0d45b6f9503e206fc0c8d5f15351>
-
I expects there 3 parameters:
- jsonrpc - all examples had this, may be important for some versioning?
- method_name -(string) name of the method to call
- params - parameters for the method.
Storing this file next to the existing Airflow APi allows it to run as part
of Web Server. Of course it can still be run independently.
To switch a method to make calls to the Internal API I implemented
internal_api_call
<https://github.com/apache/airflow/compare/main...mhenc:airflow:poc_44#diff-0f4f6450d07fffead241765439dcae32988ba1aceac81ed54c884505b813e882>
decorator.
Then we just add it to the method like
@internal_api_call("sample_method")
def sample_method(dag_id: string...):
The decorator will either call the method body or send a request to the
Internal APi endpoint (depending on the configuration).
On the API-server side we have a rpc_api handler
<https://github.com/apache/airflow/commit/eaf3f8e418d8bf00641f09837c4cf9286b1877ab#diff-024141cc0f52a9f8e84b56c9de06b0487b1bc0cd0def4749ac6490816bab3890>
which
contains the methods mapping:
METHODS = {
'update_import_errors': DagFileProcessor.update_import_errors,
'process_file': processor.process_file
}
Note: `process_file`is just for testing purposes. All methods there should
be static.
Rpc endpoint handler then gets metod_name from the request and uses the
proper function from METHODS. We may add some presubmit to make sure this
array is properly populated.
*Args/Results serialization*
The main problem is parameters and results serialization and sending them
over-the-wire. As this is JSON RPC then serialization to JSON was my first
idea, however I found that it means we may need to define a custom
serialization for many methods (example
<https://github.com/apache/airflow/commit/e86c389a004b26f22953e804e365ffaee200107c#diff-024141cc0f52a9f8e84b56c9de06b0487b1bc0cd0def4749ac6490816bab3890R55>)
- and not single but 4 of them (params_to_json, params_from_json,
result_to_json, result_from_json).
On the other side we could just use Pickle - serialize the args dictionary,
convert to base64 (to send safely inside the JSON) then deserialize on the
other side.
This leads to the very elegant solution where all we need to do is change
the method to static and add a decorator (compare rpc_api handler: pickle
<https://github.com/apache/airflow/commit/eaf3f8e418d8bf00641f09837c4cf9286b1877ab#diff-024141cc0f52a9f8e84b56c9de06b0487b1bc0cd0def4749ac6490816bab3890>
vs json
<https://github.com/apache/airflow/commit/e86c389a004b26f22953e804e365ffaee200107c#diff-024141cc0f52a9f8e84b56c9de06b0487b1bc0cd0def4749ac6490816bab3890>
)
*Unless there are strong options not to use pickle, then I would
recommend this solution.*
There is also yet another idea: use Pydantic, however I found that it
requires all objects to inherit from its base class - not sure if we should
do that, but if you believe it would be useful, then I can check if it's
feasible. The only problem I can see is K8s v1Pod objects that are used in
multiple places (to be honest I didn't find a good solution to serialize
and deserialize them properly to/from JSON).
*Tests*
I ran some tests to compare performance - using Jarek's testing scenarios
<https://github.com/apache/airflow/pull/25094>:
Tests were executed on localhost with
time airflow internal-api test-client --num-repeats 20 --num-callbacks 20
--test file_processor
Results*:*
- No Internal API : 104 s
- JSON serialization: 119s
- Pickle: 115s
I exported my changes to PRs:
https://github.com/apache/airflow/pull/27468/files
https://github.com/apache/airflow/pull/27469/files
Any comments appreciated.
Best regards,
Mateusz Henc
Re: [AIP-44] PoC JSON RPC
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hey Giorgio:
We use JSONSchema for validation in our JSONSerializer when we need
it. We can do the same in this case. But we can also choose not to do
it - based on actual implementation and testing how much it costs.
This is typical practice when you have full control of both sides and
you can run a comprehensive test suite (which we will).
1) The method inventory is in the AIP docs -
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-44+Airflow+Internal+API.
It might be slightly outdated as Airflow is constantly being developed
and we deliberately have not put details in the docs - but you can
find all the methods in the code and check what parameters there are
to pass/return. Those are basically parameters of each method that we
are "remoting" and returning values.
2) Those are already provided. As you can see from the initial
communication Both POCs - mine
https://github.com/apache/airflow/pull/25094 and Mateusz''s (you can
find it at the beginning of the thread) contains the code that we used
for testing. If you want to experiment with those - feel free.
> P.S when we'll feel the need of speed, PyO3 + Rust is the way to go or also without going native, asyncio+uvloop.
Absolutely. If you need the best speed, those would be my favourites
too. PyO3 + Rust is precisely what Pydantic v2 uses (see this plan
that Samuel came up with
https://pydantic-docs.helpmanual.io/blog/pydantic-v2/). Unfortunately
Pydantic v2 is still months away (likely more than few). And maybe one
day we switch when we will not have to fight with its teething
problems.
We are in a little different situation than most "public" APIs out
there. All those methods of ours that we are going to remote will make
a (usually remote) Relational Database Query, converting Python
objects ORM to SQL - usually pretty heavy query at that. Executing
them in the DB and going back. Our tests confirm that and since then
optimising that part is non-goal for us (or rather has much lower
priority than familiarity with other parts of the codebase).
We do not want to micro-optimise the part of the process that can give
us low, single-percentage improvement. And we can always do it in the
future if we get to the point that this is our bottleneck - and it
will be easy to switch if we decide to.
J.
On Tue, Nov 8, 2022 at 6:13 PM Giorgio Zoppi <gi...@gmail.com> wrote:
>
> Makes sense.
> It's ok exchanging a json, but it's also important to provide a schema for input validation in those cases.
> Yes, you'll have to maintain the schema, but safer is better than sorry. Two questions:
> 1. which is the model that you want to serialize? I don't see a clear speration of concern between rpc rest call.
> 2. And also can you provide the tests for minimal experimentation?
> Best Regards,
> Giorgio
> P.S when we'll feel the need of speed, PyO3 + Rust is the way to go or also without going native, asyncio+uvloop.
> in my recent REST service tests, the latency became comparable to Go.
>
Re: [AIP-44] PoC JSON RPC
Posted by Giorgio Zoppi <gi...@gmail.com>.
Makes sense.
It's ok exchanging a json, but it's also important to provide a schema for
input validation in those cases.
Yes, you'll have to maintain the schema, but safer is better than sorry.
Two questions:
1. which is the model that you want to serialize? I don't see a clear
speration of concern between rpc rest call.
2. And also can you provide the tests for minimal experimentation?
Best Regards,
Giorgio
P.S when we'll feel the need of speed, PyO3 + Rust is the way to go or also
without going native, asyncio+uvloop.
in my recent REST service tests, the latency became comparable to Go.
Re: [AIP-44] PoC JSON RPC
Posted by Jarek Potiuk <ja...@potiuk.com>.
Also Giorgio - we are not ignoring Flatbuffers, but one of the
criteria we had when we discussed "GRPC vs. JSON" was that we did not
want to introduce new technologies that might require extra
learning/skills while debugging problems.
My thought process here:
We are using our own custom JSONSerialization in a few places, and it
can already handle some of "non-standard" structures for us (for
example K8S objects). If we go flatbuffers, this is basically
repeating the same story - any custom objects that are not directly
serializable need to be handled in a custom way.
Yes, our serialization not "standard" outside Airflow and likely any
"reusable" serialization might be better optimized. We looked at -
very popular in the Python API world Pydantic. The v2 has all the
internals rewritten in rust for example and it is sometimes 16 times
faster. but In a way that is a premature optimisation IMHO.
We are not aiming for cross-language (FlatBuffers, GRPC are). We are
not aiming for extremely complex and huge structures to send. We
already know that serialization will not be our bottleneck and the
serialisation speed has very limited impact. So getting a "faster" or
"more standard" solution is not as important as getting the "more
familiar" one.
I think if anything, that alone means our custom serialization wins.
And this is an easily replaceable choice. Shall we ever feel the need
to make it cross-language, it's just one component to replace :)
J.
On Tue, Nov 8, 2022 at 4:53 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Very good point. I have not thought about it but this is a very strong
> reason to use our JSONSerialization.
>
> J,
>
> On Tue, Nov 8, 2022 at 4:28 PM Mateusz Henc <mh...@google.com> wrote:
> >
> > I just learned from https://docs.python.org/3/library/pickle.html
> > Warning The pickle module is not secure. Only unpickle data you trust.
> > It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Never unpickle data that could have come from an untrusted source, or that could have been tampered with.
> >
> > So there we have a "trusted" component - Internal API, exposing an endpoint that can be called from Worker, so from any arbitrary code. Unless there are some ways to protect from this, it seems that JSON serialization is the only option there.
> >
> > Best regards,
> > Mateusz Henc
> >
> >
> > On Thu, Nov 3, 2022 at 9:45 AM Mateusz Henc <mh...@google.com> wrote:
> >>
> >> Thank you Giorgio.
> >>
> >> TBH I've never heard about FlatBuffer, but I will take a look.
> >> The big advantage I see for Pickle is seamless integration - no additional conversion code required for developers, especially that we get the arguments as a dictionary. Pickle handles it without any problem (at least in my tests). If FlatBuffer offers a similar experience then we definitely should consider it.
> >> The other question is if we should introduce yet another dependency to Airflow - the number of pypi packages is big anyway, which leads to many problems when users want to install their customer packages (dependency conflicts etc).
> >>
> >> Best regards,
> >> Mateusz Henc
> >>
> >>
> >> On Wed, Nov 2, 2022 at 6:59 PM Giorgio Zoppi <gi...@gmail.com> wrote:
> >>>
> >>> Hello,
> >>> this is something i'd like to work too in my spare time but some i'd rather use flatbuffers for the payload since its duality json/binary. Flatbuffers have the nice feature that they're able to parse JSON files that conform to a schema into FlatBuffer binary files, so you can have duality json -> binary_on_wire-> dataclasses at receiver side. I'd rather put pickle as last resort or avoid at all. @Mateus please feel free to sync with me privately on missing actions items to make this feature a success.
> >>> Just 1c,
> >>> Best Regads,
> >>> Giorgio
> >>>
Re: [AIP-44] PoC JSON RPC
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hey Ash,
JSON-RPC came from using best practices on how we can encode "actions"
in JSON. Mateusz posted tresult of his POC at the beginning of this
thread.
Unlike standard REST (which is not very useful in our case) it has a
way to POST JSON formatted requests (no parameters passed via URL) and
receive JSON Formatted responses both of which are not referring to
resources but actions to execute).
JSON RPC defines standard for how those should be encoded:
- single endpoint to handle it
- method name ("action") to execute
- parameters to pass
- unique id (useful when you want to use multiplexing to match request
with response over single stream)
- result - including error code handling standards
https://www.jsonrpc.org/specification
It has no other semantics included. It just handles the approach
where we are not operating on Resources but execute actions (which is
our case).
J.
On Tue, Nov 8, 2022 at 5:27 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> Where did JSON-RPC come from? I'd like to catch up to speed on the discussion/reason. It is a very... crufty protocol and not one I would ever personally choose for new work.
>
> (Sorry I've been out a bit and may have missed some context or discussion.)
>
> -ash
>
> On 8 November 2022 15:53:06 GMT, Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> Very good point. I have not thought about it but this is a very strong
>> reason to use our JSONSerialization.
>>
>> J,
>>
>> On Tue, Nov 8, 2022 at 4:28 PM Mateusz Henc <mh...@google.com> wrote:
>>>
>>>
>>> I just learned from https://docs.python.org/3/library/pickle.html
>>> Warning The pickle module is not secure. Only unpickle data you trust.
>>> It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Never unpickle data that could have come from an untrusted source, or that could have been tampered with.
>>>
>>> So there we have a "trusted" component - Internal API, exposing an endpoint that can be called from Worker, so from any arbitrary code. Unless there are some ways to protect from this, it seems that JSON serialization is the only option there.
>>>
>>> Best regards,
>>> Mateusz Henc
>>>
>>>
>>> On Thu, Nov 3, 2022 at 9:45 AM Mateusz Henc <mh...@google.com> wrote:
>>>>
>>>>
>>>> Thank you Giorgio.
>>>>
>>>> TBH I've never heard about FlatBuffer, but I will take a look.
>>>> The big advantage I see for Pickle is seamless integration - no additional conversion code required for developers, especially that we get the arguments as a dictionary. Pickle handles it without any problem (at least in my tests). If FlatBuffer offers a similar experience then we definitely should consider it.
>>>> The other question is if we should introduce yet another dependency to Airflow - the number of pypi packages is big anyway, which leads to many problems when users want to install their customer packages (dependency conflicts etc).
>>>>
>>>> Best regards,
>>>> Mateusz Henc
>>>>
>>>>
>>>> On Wed, Nov 2, 2022 at 6:59 PM Giorgio Zoppi <gi...@gmail.com> wrote:
>>>>>
>>>>>
>>>>> Hello,
>>>>> this is something i'd like to work too in my spare time but some i'd rather use flatbuffers for the payload since its duality json/binary. Flatbuffers have the nice feature that they're able to parse JSON files that conform to a schema into FlatBuffer binary files, so you can have duality json -> binary_on_wire-> dataclasses at receiver side. I'd rather put pickle as last resort or avoid at all. @Mateus please feel free to sync with me privately on missing actions items to make this feature a success.
>>>>> Just 1c,
>>>>> Best Regads,
>>>>> Giorgio
>>>>>
Re: [AIP-44] PoC JSON RPC
Posted by Ash Berlin-Taylor <as...@apache.org>.
Where did JSON-RPC come from? I'd like to catch up to speed on the discussion/reason. It is a very... crufty protocol and not one I would ever personally choose for new work.
(Sorry I've been out a bit and may have missed some context or discussion.)
-ash
On 8 November 2022 15:53:06 GMT, Jarek Potiuk <ja...@potiuk.com> wrote:
>Very good point. I have not thought about it but this is a very strong
>reason to use our JSONSerialization.
>
>J,
>
>On Tue, Nov 8, 2022 at 4:28 PM Mateusz Henc <mh...@google.com> wrote:
>>
>> I just learned from https://docs.python.org/3/library/pickle.html
>> Warning The pickle module is not secure. Only unpickle data you trust.
>> It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Never unpickle data that could have come from an untrusted source, or that could have been tampered with.
>>
>> So there we have a "trusted" component - Internal API, exposing an endpoint that can be called from Worker, so from any arbitrary code. Unless there are some ways to protect from this, it seems that JSON serialization is the only option there.
>>
>> Best regards,
>> Mateusz Henc
>>
>>
>> On Thu, Nov 3, 2022 at 9:45 AM Mateusz Henc <mh...@google.com> wrote:
>>>
>>> Thank you Giorgio.
>>>
>>> TBH I've never heard about FlatBuffer, but I will take a look.
>>> The big advantage I see for Pickle is seamless integration - no additional conversion code required for developers, especially that we get the arguments as a dictionary. Pickle handles it without any problem (at least in my tests). If FlatBuffer offers a similar experience then we definitely should consider it.
>>> The other question is if we should introduce yet another dependency to Airflow - the number of pypi packages is big anyway, which leads to many problems when users want to install their customer packages (dependency conflicts etc).
>>>
>>> Best regards,
>>> Mateusz Henc
>>>
>>>
>>> On Wed, Nov 2, 2022 at 6:59 PM Giorgio Zoppi <gi...@gmail.com> wrote:
>>>>
>>>> Hello,
>>>> this is something i'd like to work too in my spare time but some i'd rather use flatbuffers for the payload since its duality json/binary. Flatbuffers have the nice feature that they're able to parse JSON files that conform to a schema into FlatBuffer binary files, so you can have duality json -> binary_on_wire-> dataclasses at receiver side. I'd rather put pickle as last resort or avoid at all. @Mateus please feel free to sync with me privately on missing actions items to make this feature a success.
>>>> Just 1c,
>>>> Best Regads,
>>>> Giorgio
>>>>
Re: [AIP-44] PoC JSON RPC
Posted by Jarek Potiuk <ja...@potiuk.com>.
Very good point. I have not thought about it but this is a very strong
reason to use our JSONSerialization.
J,
On Tue, Nov 8, 2022 at 4:28 PM Mateusz Henc <mh...@google.com> wrote:
>
> I just learned from https://docs.python.org/3/library/pickle.html
> Warning The pickle module is not secure. Only unpickle data you trust.
> It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Never unpickle data that could have come from an untrusted source, or that could have been tampered with.
>
> So there we have a "trusted" component - Internal API, exposing an endpoint that can be called from Worker, so from any arbitrary code. Unless there are some ways to protect from this, it seems that JSON serialization is the only option there.
>
> Best regards,
> Mateusz Henc
>
>
> On Thu, Nov 3, 2022 at 9:45 AM Mateusz Henc <mh...@google.com> wrote:
>>
>> Thank you Giorgio.
>>
>> TBH I've never heard about FlatBuffer, but I will take a look.
>> The big advantage I see for Pickle is seamless integration - no additional conversion code required for developers, especially that we get the arguments as a dictionary. Pickle handles it without any problem (at least in my tests). If FlatBuffer offers a similar experience then we definitely should consider it.
>> The other question is if we should introduce yet another dependency to Airflow - the number of pypi packages is big anyway, which leads to many problems when users want to install their customer packages (dependency conflicts etc).
>>
>> Best regards,
>> Mateusz Henc
>>
>>
>> On Wed, Nov 2, 2022 at 6:59 PM Giorgio Zoppi <gi...@gmail.com> wrote:
>>>
>>> Hello,
>>> this is something i'd like to work too in my spare time but some i'd rather use flatbuffers for the payload since its duality json/binary. Flatbuffers have the nice feature that they're able to parse JSON files that conform to a schema into FlatBuffer binary files, so you can have duality json -> binary_on_wire-> dataclasses at receiver side. I'd rather put pickle as last resort or avoid at all. @Mateus please feel free to sync with me privately on missing actions items to make this feature a success.
>>> Just 1c,
>>> Best Regads,
>>> Giorgio
>>>
Re: [AIP-44] PoC JSON RPC
Posted by Mateusz Henc <mh...@google.com.INVALID>.
I just learned from https://docs.python.org/3/library/pickle.html
*Warning The pickle module is not secure. Only unpickle data you trust.It
is possible to construct malicious pickle data which will execute arbitrary
code during unpickling. Never unpickle data that could have come from an
untrusted source, or that could have been tampered with.*
So there we have a "trusted" component - Internal API, exposing an endpoint
that can be called from Worker, so from any arbitrary code. Unless there
are some ways to protect from this, it seems that JSON serialization is the
only option there.
Best regards,
Mateusz Henc
On Thu, Nov 3, 2022 at 9:45 AM Mateusz Henc <mh...@google.com> wrote:
> Thank you Giorgio.
>
> TBH I've never heard about FlatBuffer, but I will take a look.
> The big advantage I see for Pickle is seamless integration - no additional
> conversion code required for developers, especially that we get the
> arguments as a dictionary. Pickle handles it without any problem (at least
> in my tests). If FlatBuffer offers a similar experience then we definitely
> should consider it.
> The other question is if we should introduce yet another dependency to
> Airflow - the number of pypi packages is big anyway, which leads to many
> problems when users want to install their customer packages (dependency
> conflicts etc).
>
> Best regards,
> Mateusz Henc
>
>
> On Wed, Nov 2, 2022 at 6:59 PM Giorgio Zoppi <gi...@gmail.com>
> wrote:
>
>> Hello,
>> this is something i'd like to work too in my spare time but some i'd
>> rather use flatbuffers for the payload since its duality json/binary.
>> Flatbuffers have the nice feature that they're able to parse JSON
>> <http://json.org> files that conform to a schema into FlatBuffer binary
>> files, so you can have duality json -> binary_on_wire-> dataclasses at
>> receiver side. I'd rather put pickle as last resort or avoid at all.
>> @Mateus please feel free to sync with me privately on missing actions items
>> to make this feature a success.
>> Just 1c,
>> Best Regads,
>> Giorgio
>>
>>
Re: [AIP-44] PoC JSON RPC
Posted by Mateusz Henc <mh...@google.com.INVALID>.
Thank you Giorgio.
TBH I've never heard about FlatBuffer, but I will take a look.
The big advantage I see for Pickle is seamless integration - no additional
conversion code required for developers, especially that we get the
arguments as a dictionary. Pickle handles it without any problem (at least
in my tests). If FlatBuffer offers a similar experience then we definitely
should consider it.
The other question is if we should introduce yet another dependency to
Airflow - the number of pypi packages is big anyway, which leads to many
problems when users want to install their customer packages (dependency
conflicts etc).
Best regards,
Mateusz Henc
On Wed, Nov 2, 2022 at 6:59 PM Giorgio Zoppi <gi...@gmail.com>
wrote:
> Hello,
> this is something i'd like to work too in my spare time but some i'd
> rather use flatbuffers for the payload since its duality json/binary.
> Flatbuffers have the nice feature that they're able to parse JSON
> <http://json.org> files that conform to a schema into FlatBuffer binary
> files, so you can have duality json -> binary_on_wire-> dataclasses at
> receiver side. I'd rather put pickle as last resort or avoid at all.
> @Mateus please feel free to sync with me privately on missing actions items
> to make this feature a success.
> Just 1c,
> Best Regads,
> Giorgio
>
>
Re: [AIP-44] PoC JSON RPC
Posted by Giorgio Zoppi <gi...@gmail.com>.
Hello,
this is something i'd like to work too in my spare time but some i'd rather
use flatbuffers for the payload since its duality json/binary. Flatbuffers
have the nice feature that they're able to parse JSON <http://json.org>
files that conform to a schema into FlatBuffer binary files, so you can
have duality json -> binary_on_wire-> dataclasses at receiver side. I'd
rather put pickle as last resort or avoid at all. @Mateus please feel free
to sync with me privately on missing actions items to make this feature a
success.
Just 1c,
Best Regads,
Giorgio
Re: [AIP-44] PoC JSON RPC
Posted by Jarek Potiuk <ja...@potiuk.com>.
This is cool I will take a look shortly (sorry i have an urgent family
matter and i am not available).
I saw an initial version of Mateusz's POC and it completely addressed my
initial reservation about the 'development maintenance' effort - when it
comes to the interface we have for developers and the boilerplate code. We
can easily make it even better that gRPC approach , so I am cool to do it.
The decorator approach which I have not POCed before (but we wanted to do
it) looks cool.
I initially suggested to check pydantic but if it is not a good idea pickle
is good solution (all the bad things about pickle - security and potential
incompatibilities between python versions do not apply in this case (and we
already use it under the hood via Multiprocessing Ina a number of places).
But one other option we could use is to use our own Jason serializer - the
very same we use to serialize DAGs to DB. Since the overhead of JSON vs.
Binary seems to be small, we could do it both way. I believe Andrew's
comments were also about the wire format being binary. I wonder what others
think ?
Jm
śr., 2 lis 2022, 14:01 użytkownik Mateusz Henc <mh...@google.com> napisał:
> Hi,
> *JSON RPC API*
> Finally I had some time to work on an Internal API based on JSON RPC.
> I found that this solution can be really elegant and clean from a
> developer perspective.
> You can find the OpenAPi definition there
> <https://github.com/apache/airflow/commit/e86c389a004b26f22953e804e365ffaee200107c#diff-e454d231c105c349be8619b3112c8bfda74f0d45b6f9503e206fc0c8d5f15351> -
> I expects there 3 parameters:
>
> - jsonrpc - all examples had this, may be important for some
> versioning?
> - method_name -(string) name of the method to call
> - params - parameters for the method.
>
> Storing this file next to the existing Airflow APi allows it to run as
> part of Web Server. Of course it can still be run independently.
>
> To switch a method to make calls to the Internal API I implemented
> internal_api_call
> <https://github.com/apache/airflow/compare/main...mhenc:airflow:poc_44#diff-0f4f6450d07fffead241765439dcae32988ba1aceac81ed54c884505b813e882>
> decorator.
>
> Then we just add it to the method like
> @internal_api_call("sample_method")
> def sample_method(dag_id: string...):
>
> The decorator will either call the method body or send a request to the
> Internal APi endpoint (depending on the configuration).
> On the API-server side we have a rpc_api handler
> <https://github.com/apache/airflow/commit/eaf3f8e418d8bf00641f09837c4cf9286b1877ab#diff-024141cc0f52a9f8e84b56c9de06b0487b1bc0cd0def4749ac6490816bab3890> which
> contains the methods mapping:
> METHODS = {
> 'update_import_errors': DagFileProcessor.update_import_errors,
> 'process_file': processor.process_file
> }
> Note: `process_file`is just for testing purposes. All methods there
> should be static.
> Rpc endpoint handler then gets metod_name from the request and uses the
> proper function from METHODS. We may add some presubmit to make sure this
> array is properly populated.
>
> *Args/Results serialization*
> The main problem is parameters and results serialization and sending them
> over-the-wire. As this is JSON RPC then serialization to JSON was my first
> idea, however I found that it means we may need to define a custom
> serialization for many methods (example
> <https://github.com/apache/airflow/commit/e86c389a004b26f22953e804e365ffaee200107c#diff-024141cc0f52a9f8e84b56c9de06b0487b1bc0cd0def4749ac6490816bab3890R55>)
> - and not single but 4 of them (params_to_json, params_from_json,
> result_to_json, result_from_json).
>
> On the other side we could just use Pickle - serialize the args
> dictionary, convert to base64 (to send safely inside the JSON) then
> deserialize on the other side.
> This leads to the very elegant solution where all we need to do is change
> the method to static and add a decorator (compare rpc_api handler: pickle
> <https://github.com/apache/airflow/commit/eaf3f8e418d8bf00641f09837c4cf9286b1877ab#diff-024141cc0f52a9f8e84b56c9de06b0487b1bc0cd0def4749ac6490816bab3890>
> vs json
> <https://github.com/apache/airflow/commit/e86c389a004b26f22953e804e365ffaee200107c#diff-024141cc0f52a9f8e84b56c9de06b0487b1bc0cd0def4749ac6490816bab3890>
> )
> *Unless there are strong options not to use pickle, then I would
> recommend this solution.*
>
> There is also yet another idea: use Pydantic, however I found that it
> requires all objects to inherit from its base class - not sure if we should
> do that, but if you believe it would be useful, then I can check if it's
> feasible. The only problem I can see is K8s v1Pod objects that are used in
> multiple places (to be honest I didn't find a good solution to serialize
> and deserialize them properly to/from JSON).
>
> *Tests*
> I ran some tests to compare performance - using Jarek's testing scenarios
> <https://github.com/apache/airflow/pull/25094>:
> Tests were executed on localhost with
> time airflow internal-api test-client --num-repeats 20 --num-callbacks 20
> --test file_processor
> Results*:*
> - No Internal API : 104 s
> - JSON serialization: 119s
> - Pickle: 115s
>
> I exported my changes to PRs:
> https://github.com/apache/airflow/pull/27468/files
> https://github.com/apache/airflow/pull/27469/files
>
> Any comments appreciated.
>
> Best regards,
> Mateusz Henc
>