You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2023/12/04 19:09:10 UTC

[DISCUSS] Using serde for PyODBC case/other cases [was Serialization in Apache Airflow]

Hey here,

Also separating it out as another topic.

>
> It is a choice that needs to be made at the API level - here Common.sql. In
> this case the particular form of the tuple was, somewhat implicit, part of
> the spec.
>

Yep. I rectified the implicitness by storing this as an explicit decision
record
 in the ADR proposal. I think it's been - similarly as lack of Serialization
decision records - an omission that resulted in this, somewhat heated
debate :D


>
> The real question is, when you find something unserializable do you then
> provide a custom serializer for serde or do you convert your object to
> something that can be serialized by default. The first one will make it
> available to all other places where this objects needs to be serialized.
> The latter limits that scope to where the conversion happens - here
> common.sql


>
So to provide an example in context of our discussion of PyODBC.Row (see
> the discussion of the linked PR) only common.sql can now deal with
> PyODBC.Row. If for some reason another provider returns a PyODBC.Row that
> object remains unserializable.
>

Yes. That's fine for me. I think we do not always have to serialize things
the same way.
Additionally serializing something using serde uses a custom format that
requires
deserialization step for it to be usable. DBApiHook response can be used
"as is"
as tuple for any other operator used by Common SQL

I have explained what the downside is of the approach you took, although
> its impact in this case is probably low (not many other places will return
> PyODBC.Row, but consider DataFrame hypothetically).
>

Yes. For DataFrame I am fine. This is "common" format that it's worth to
have
serializer because it can be produced and consumed by many. It's quite
unlikely to happen to ODBC and simply DBApiHook has its own "standard"
being tuple that is the language DBAPIHook talks to the users. In a way
returning PyODBC.Row in this sense violates assumptions made a year
ago (now documented in the ADR proposal) - that this is the "API" of DB
Hook.
So while "make_serializable" was likely a wrong name, the goal of it was
"produce a tuple that is directly serializable".

Just to give another example - we do the same with Snowflake
that returns in some cases a Dictionary instead of tuples of values. In
this case
we cannot rely on Snowflake Dictionary - because we expect tuples as output
so we MUST convert (and we cannot use Serde here). Here it is mostly
accidental
that PyODBC.Row also conforms to Tuple semantics, but not fully. So
similarly to
Snowflake Dictionary we need to bring it to the semantics that users of the
DBAPIHook (Common SQL operators) will understand and expect. That's all.
Serializable property is one of those and what we've done is the same thing
we do for other DBApiHook implementations.


> I think you needed a serializable tuple so that it could go into XCom. This
> makes the need for an explicit choice imho.
>
> Yep. That's explicit choice we made.



> >
> >
> On speed. Afaik AIP-44 still makes use of the DAG Serialization code apart
> from Pydantic's rust implementation. The DAG serialization code is
> seriously
> slow. So, before claiming speed improvements here show the numbers please
> :-)


Yep. When we first added it we used Pydantic V1.And then it did not matter.
But
we have it using our serialization code - because at some point you
insisted on
making it part of "a serialization" - we have absolutely no need for that
in AIP-44
and I think when we go to prime time - we will just skip it entirely.

Going through either of the serializations of ours here is not needed
overhead at all - we can just purely rely on Rust and get ORM on one side
and Pydantic object on the other without touching any of our serialization
code.
It just work out-of-the box. without any of our "logic" to maintain except
explicitly
creating pure-python models that we are going to use (and which are
verified via MyPy).

More benchmarking and reasoning on Pydantic v2 speed improvements are here
https://thedataquarry.com/posts/why-pydantic-v2-matters/

While it's not the 16x I claimed (I mixed it with 16x improvements on
emulation of ARM processors), 5x it's still very impressive if you ask me.

We can of course keep both. However if you ever looked at the DAG
> serialization code you might want to reconsider ;-).


Yep. I am not at all strong on it. I do not intimately know that code
either :).


> Alternatively, you
> could build on serde and provide the guarantees that you require there.  It
> can be done. I have been able to serialize a DAG with serde (hence knowing
> the 1:10 ratio), but until now havent added it, because its schema looks
> different.
>

Yeah. I am good for that if we can pull it off with backwards compatibility
in place.


>
> BTW: There is no need from the XCom serializer / serde to deserialize into
> the original object. By convention we like to of course, but it is not
> required.
>

I understand, but the versioning feature for example assumes so. and adds
complexity if the serialized form is to be used. From what I understand
(and what some of the Serde tests
verify the API of the serde is that you do in-fact get the same, comparable
object
when you do the serialize/deserialize roundtrip. And well, the only way to
make use
of the serialized object is to deserialize it first. Because of
transparency, hiding the details
and design of the serde, you should not care or understand about the
"serialized" form so you
effectively MUST deserialize to use it.

So far - also looking at the tests - I was under the impression that it was
expected property of Serde to get the roundtrip to return an equivalent
object. But maybe I was mistaken here?

One more reason to have decision records captured somewhere if that was not
the intention.


> For the record: the XCom serializer lazy loads its dependencies. So it does
> not make
> core Airflow dependent on client libraries arguably.
>

But it means that if we have the same libraries used in providers and in
core - we have to declare the dependencies in two places - once for
provider extra and once for serde "extension" extra. You cannot use
provider dependencies if you want to have core use the same client library
because of two reasons:

1) they might depend on different versions
2) you would have to specify provider extra and install the provider (which
you might not need
   in this case

And in case 1) This might lead to the situation that if core and provider
depend on different, conflicting versions of the libraries, we are f*d -
because you might not be able to install provider
in certain version with core in certain version. if we move the
serialization code to the provider,
we have no such problem. It always depends on the version that the provider
requires.


The con is that the API becomes strict. We cannot change it easily anymore.
>

Of course we can extend that. But I believe you wanted to make ODBCRow
provider to depend on the very same API already? Isn't it. It seems I see
contradiction. It looks like it's ok for
the ODBCRow/provider to depend on serde and implement their serialization
code, but on the other hand it's not ok for (say) DeltaLake to do the same
and become Databricks provider?

Or maybe I missed something? Maybe your idea was to make the ODBC
serializer part of Airflow 2.8 and add >= airflow 2.8.0 for the ODBC
provider? That's what I call unnecessary coupling.


>
> > Maybe that's a bit of a different question for Pandas - as we have no
> > separate Pandas provider
> > and we use and link to it already from Airflow core. But... maybe it's
> also
> > a sign we need pandas
> > provider in this case? This question for me is open.
> >
> >
> To me this depends on what we want to do with data awareness (separate
> discussion). If you think:
>
> ObjectStoragePath : Table : Dataframe
>
> It is useful to have dataframe as part of core. This could be pandas or
> maybe polars which is so much faster :-).
>

Yeah - I am not too strong on that and I believe it has benefits to have
both Pandas and Polars. Especially if we can make sure that it's very much
loosely coupled (use public APIs ony and get very clear about SemVer/
Versioning for those to get as open as possible) with Pandas and polars
as optional extras, I am quite fine with it. I think we do not need a
provider in this case.

Case-by-case is fine to me as long as the choice is explicit and thus
> options are considered.
>
>
Yep. I think this discussion is mostly about that :). And hopefully will
result with some decisions recorded :)


>
> --
>
> --
> Bolke de Bruin
> bdbruin@gmail.com
>

Re: [DISCUSS] Using serde for PyODBC case/other cases [was Serialization in Apache Airflow]

Posted by Daniel Standish <da...@astronomer.io.INVALID>.
>
> Tried to get others some opportunity to comment, but I see it's mostly me
> <> Bolke.


It might help to start off this separate discussion thread with a simple /
concise problem statement.  (that might sound snarky but it ain't :) )

And I say *might help* cus you still might get crickets even if you
clarify...

But as for me, I didn't really follow the main thread and when I looked at
this one, which was a continuation of a convo, the context wasn't super
clear so I moved on.






On Mon, Dec 11, 2023 at 11:23 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> Tried to get others some opportunity to comment, but I see it's mostly me
> <> Bolke.
>
> >
> > > Yep. I rectified the implicitness by storing this as an explicit
> decision
> > > record
> > >  in the ADR proposal. I think it's been - similarly as lack of
> > >
> > <snip>
> >
>
> Happy to help here - since there is a good "vibe" on the ADR kind of
> approach, I am happy to  "lead by example" and propose a PR with ADRs for
> serialization (learning from the whole discussions and that's a good
> opportunity for me to learn some of the nitty-gritty details ).
>
> Would that work for you and can I count on some reviews ?
>
>
> > It was added to the DAG serializer at the time, I mentioned that that
> > looked weird
> > given the fact that it did not seem connected to DAG serialization (to
> me)
> > and serde
> > should be considered. So no, I wasn't insisting on making it part of
> > serialization imho,
> > it defacto was already.
> >
>
> Let's straighten it up after writing ADRs and rounding up the serialization
> case. We are aiming to enable AIP-44 and Pydantic in 2.9 so that's a good
> time to see and make some measurements there.
>
> >
> > >
> > > Going through either of the serializations of ours here is not needed
> > > overhead at all - we can just purely rely on Rust and get ORM on one
> side
> > > and Pydantic object on the other without touching any of our
> > serialization
> > > code.
> > >
> >
> > That's fine to me.
> >
>
> Yeah. We'll sort it out. Luckily it's a very localized place and we can
> simply open
> draft PR showing how it will look like and see if it looks simpler/whether
> we lose
> any of the properties that
>
> >
> >
> > I don't think you want to ;-).
> >
> >
> Happy also to take a look at helping to straighten up DAG serialization
> possibly
> once I get a bit more familiar with the context.
>
>
> > We could. It's probably just a three pass serializer then.
> >
>
> Interesting :).
>
>
> > >
> > Exactly not. By adding it to Airflow 2.8.0 it would have made every
> > previous version
> > of this provider suddenly work without any adjustments against Airflow
> > 2.8.0. Remember
> > that the original bug report said "cannot serialize value of type
> > PyODBC.Row".
> >
>
> I was more talking about compatibility of ODBC provider. If we want to
> directly
> return ODBC.Row, it will only be serializable in Airflow 2.8.0 (currently
> ODBC
> provider has >= Airflow 2.6.0. What I want to achieve is to implement a fix
> where you can upgrade just provider and get serializable output even if
> the user decides to stay with Airflow 2.8.0. This w
>
>
> > You can argue both ways here architecturally.
> >
> > 1) Airflow 2.8 with a new feature that allows PyODBC.Row to be
> serialized,
> > affecting past and future providers at large and (TaskFlow) returned
> > variables.
> >
>
> This requires users to upgrade to Airflow 2.8.0. Many of our users delay
> such upgrades for months or years. So we are talking about making it
> possible for them to upgrade "easily" where selective upgrades to
> providers is the easy way. This also works better for some
> Airflow-as-a-service
> where 2.8 might only be available months from now.
>
> Migration to a new Airflow version is way more involved and uptick of new
> versions (especially .0) is slow.
>
> We are basically talking about having something available now - for Airflow
> 2.6, 2.7, (and 2.8) now
> vs. many users having it months or years from now. And releasing it now
> fixes actual issue
> people have and raised us. And implementing it in provider has an added
> benefit that
> IF we also implement a generic serializer as part of ODBC and have a way
> for
> serde to discover serializers by providers, then we could implement it in
> the
> provider in a way that either uses internal "serialization" or allows serde
> to
> discover it.
>
> I see really no benefit of having Airflow know anything about ODBC. Are we
> going to
> implement all 80 provider's specific code and put it to airlfow and all
> dependencies (even
> optional) as airflow dependencies, now that we made enormous effort in
> separating these?
>
> Imagine ODBC code depends on odbc_library.
> What happens if Airflow has optional dependency on odbc_library <3 and the
> provider has dependency on odbc_library  >=3 ? Imagine that the provider
> returns
> ODBCRow from version 3 that serializer won't be able to handle.
>
> How do you install the two together? How do you install these two together?
> How do
> you make them work together?
>
> It really makes very little sense to have airflow depend on the library and
> code that
> provider already depends on, because this will create a whole new class of
> potential conflicts.
>
> I think the *only* possible way to make ODBC Row handled by serde is for
> serde to discover serializers from installed providers and use them from
> there.
>
> This solves all the "dependency conflict" cases nicely and is long term
> sustainable and maintainable.
>
>
> > This is exactly the intention with the serializers in
> > airflow.serialization.serializers,
> > however can be tough as not all expose the information in public APIs
> (e.g.
> > deltalake)
> > that allow to reinstate the object.
> >
> > But yes: loose, and pragmatic public API / SemVer.
> >
> > Yep.
>
> >
> >
>

RE: [DISCUSS] Using serde for PyODBC case/other cases [was Serialization in Apache Airflow]

Posted by Bienvenu Joffrey <jo...@infrabel.be>.
With new year's holiday, I didn't saw the discussion split. Sorry for the late catchup !

As maintainer of Airflow instances, it's great that I can bump any provider's version to get a bugfix quickly, without having to upgrade all the instances. But indeed, now, only ODBC provider can handle PyODBC.Row serialization.

> I think the *only* possible way to make ODBC Row handled by serde is for
> serde to discover serializers from installed providers and use them from
> there.

Sound like a nice solution ! But there, the ODBC hook would return a PyODBC.Row object, right ? Thus, defeating current initiative to have common tuples everywhere.

I'm going to expand a bit the scope, and do a bit of fiction:

IMO it would be great to have a real usable "DbApiHook": Not an abstract class, like now. But a Hook similar to the ODBC one, which allows to install any DB-API2.0 python libraries and setup a Connection with them - without subclassing anything. It could live outside of the common.sql provider, in a dedicated one. With this Hook, it will be possible to use the PyODBC python package directly. Thus, two providers will need pyodbc.Row serialization. Putting the serializer somewhere in a provider and discoverable for serde will avoid coupling with Airflow, while letting both Providers using it. Great !

Further fiction: Maybe a mechanism to discover those DB-API2.0 compliant packages ? Wrapping them into a thin provider. E.g.: We could have a 'PyODBC-provider' with just a serializer for serde, and PyODBC as dependency ?

Further further fiction: Would that open the door to dropping the ODBC Provider ? Operators needs to die after all😊 In this situation, do we care if we return a tuple, a Databricks.Row, or a PyODBC.Row ? Those objects are compliant with python DbAPI2.0 after all. Today, the user that add the 'ODBC provider' expect standard types, because he setup the 'Airflow standardized ODBC provider'. In the fiction world, the user that setup a 'PyODBC-provider' should expect to have PyODBC.Row object.


Again, sorry for catching up late. I'll open a PR with a proposal for pyodbc.row serializer in the odbc provider. The earlier the better if that might become a dependency to other providers in the future.


Joffrey Bienvenu

-----Original Message-----
From: Jarek Potiuk <ja...@potiuk.com> 
Sent: Monday, 11 December 2023 20:22
To: dev@airflow.apache.org
Subject: Re: [DISCUSS] Using serde for PyODBC case/other cases [was Serialization in Apache Airflow]

EXTERNAL MAIL: Ne cliquez pas sur les liens ou n'ouvrez pas les pièces jointes si vous ne connaissez pas l'expéditeur et ne lui faites pas confiance. En cas de doute, envoyez cet email en pièce jointe à abuse@infrabel.be<ma...@infrabel.be>.

Tried to get others some opportunity to comment, but I see it's mostly me <> Bolke.

>
> > Yep. I rectified the implicitness by storing this as an explicit 
> > decision record  in the ADR proposal. I think it's been - similarly 
> > as lack of
> >
> <snip>
>

Happy to help here - since there is a good "vibe" on the ADR kind of approach, I am happy to  "lead by example" and propose a PR with ADRs for serialization (learning from the whole discussions and that's a good opportunity for me to learn some of the nitty-gritty details ).

Would that work for you and can I count on some reviews ?


> It was added to the DAG serializer at the time, I mentioned that that 
> looked weird given the fact that it did not seem connected to DAG 
> serialization (to me) and serde should be considered. So no, I wasn't 
> insisting on making it part of serialization imho, it defacto was 
> already.
>

Let's straighten it up after writing ADRs and rounding up the serialization case. We are aiming to enable AIP-44 and Pydantic in 2.9 so that's a good time to see and make some measurements there.

>
> >
> > Going through either of the serializations of ours here is not 
> > needed overhead at all - we can just purely rely on Rust and get ORM 
> > on one side and Pydantic object on the other without touching any of 
> > our
> serialization
> > code.
> >
>
> That's fine to me.
>

Yeah. We'll sort it out. Luckily it's a very localized place and we can simply open draft PR showing how it will look like and see if it looks simpler/whether we lose any of the properties that

>
>
> I don't think you want to ;-).
>
>
Happy also to take a look at helping to straighten up DAG serialization possibly once I get a bit more familiar with the context.


> We could. It's probably just a three pass serializer then.
>

Interesting :).


> >
> Exactly not. By adding it to Airflow 2.8.0 it would have made every 
> previous version of this provider suddenly work without any 
> adjustments against Airflow 2.8.0. Remember that the original bug 
> report said "cannot serialize value of type PyODBC.Row".
>

I was more talking about compatibility of ODBC provider. If we want to
directly
return ODBC.Row, it will only be serializable in Airflow 2.8.0 (currently
ODBC
provider has >= Airflow 2.6.0. What I want to achieve is to implement a fix
where you can upgrade just provider and get serializable output even if
the user decides to stay with Airflow 2.8.0. This w


> You can argue both ways here architecturally.
>
> 1) Airflow 2.8 with a new feature that allows PyODBC.Row to be serialized,
> affecting past and future providers at large and (TaskFlow) returned
> variables.
>

This requires users to upgrade to Airflow 2.8.0. Many of our users delay
such upgrades for months or years. So we are talking about making it
possible for them to upgrade "easily" where selective upgrades to
providers is the easy way. This also works better for some
Airflow-as-a-service
where 2.8 might only be available months from now.

Migration to a new Airflow version is way more involved and uptick of new
versions (especially .0) is slow.

We are basically talking about having something available now - for Airflow
2.6, 2.7, (and 2.8) now
vs. many users having it months or years from now. And releasing it now
fixes actual issue
people have and raised us. And implementing it in provider has an added
benefit that
IF we also implement a generic serializer as part of ODBC and have a way for
serde to discover serializers by providers, then we could implement it in
the
provider in a way that either uses internal "serialization" or allows serde
to
discover it.

I see really no benefit of having Airflow know anything about ODBC. Are we
going to
implement all 80 provider's specific code and put it to airlfow and all
dependencies (even
optional) as airflow dependencies, now that we made enormous effort in
separating these?

Imagine ODBC code depends on odbc_library.
What happens if Airflow has optional dependency on odbc_library <3 and the
provider has dependency on odbc_library  >=3 ? Imagine that the provider
returns
ODBCRow from version 3 that serializer won't be able to handle.

How do you install the two together? How do you install these two together?
How do
you make them work together?

It really makes very little sense to have airflow depend on the library and
code that
provider already depends on, because this will create a whole new class of
potential conflicts.

I think the *only* possible way to make ODBC Row handled by serde is for
serde to discover serializers from installed providers and use them from
there.

This solves all the "dependency conflict" cases nicely and is long term
sustainable and maintainable.


> This is exactly the intention with the serializers in
> airflow.serialization.serializers,
> however can be tough as not all expose the information in public APIs (e.g.
> deltalake)
> that allow to reinstate the object.
>
> But yes: loose, and pragmatic public API / SemVer.
>
> Yep.

>
>

Re: [DISCUSS] Using serde for PyODBC case/other cases [was Serialization in Apache Airflow]

Posted by Jarek Potiuk <ja...@potiuk.com>.
Tried to get others some opportunity to comment, but I see it's mostly me
<> Bolke.

>
> > Yep. I rectified the implicitness by storing this as an explicit decision
> > record
> >  in the ADR proposal. I think it's been - similarly as lack of
> >
> <snip>
>

Happy to help here - since there is a good "vibe" on the ADR kind of
approach, I am happy to  "lead by example" and propose a PR with ADRs for
serialization (learning from the whole discussions and that's a good
opportunity for me to learn some of the nitty-gritty details ).

Would that work for you and can I count on some reviews ?


> It was added to the DAG serializer at the time, I mentioned that that
> looked weird
> given the fact that it did not seem connected to DAG serialization (to me)
> and serde
> should be considered. So no, I wasn't insisting on making it part of
> serialization imho,
> it defacto was already.
>

Let's straighten it up after writing ADRs and rounding up the serialization
case. We are aiming to enable AIP-44 and Pydantic in 2.9 so that's a good
time to see and make some measurements there.

>
> >
> > Going through either of the serializations of ours here is not needed
> > overhead at all - we can just purely rely on Rust and get ORM on one side
> > and Pydantic object on the other without touching any of our
> serialization
> > code.
> >
>
> That's fine to me.
>

Yeah. We'll sort it out. Luckily it's a very localized place and we can
simply open
draft PR showing how it will look like and see if it looks simpler/whether
we lose
any of the properties that

>
>
> I don't think you want to ;-).
>
>
Happy also to take a look at helping to straighten up DAG serialization
possibly
once I get a bit more familiar with the context.


> We could. It's probably just a three pass serializer then.
>

Interesting :).


> >
> Exactly not. By adding it to Airflow 2.8.0 it would have made every
> previous version
> of this provider suddenly work without any adjustments against Airflow
> 2.8.0. Remember
> that the original bug report said "cannot serialize value of type
> PyODBC.Row".
>

I was more talking about compatibility of ODBC provider. If we want to
directly
return ODBC.Row, it will only be serializable in Airflow 2.8.0 (currently
ODBC
provider has >= Airflow 2.6.0. What I want to achieve is to implement a fix
where you can upgrade just provider and get serializable output even if
the user decides to stay with Airflow 2.8.0. This w


> You can argue both ways here architecturally.
>
> 1) Airflow 2.8 with a new feature that allows PyODBC.Row to be serialized,
> affecting past and future providers at large and (TaskFlow) returned
> variables.
>

This requires users to upgrade to Airflow 2.8.0. Many of our users delay
such upgrades for months or years. So we are talking about making it
possible for them to upgrade "easily" where selective upgrades to
providers is the easy way. This also works better for some
Airflow-as-a-service
where 2.8 might only be available months from now.

Migration to a new Airflow version is way more involved and uptick of new
versions (especially .0) is slow.

We are basically talking about having something available now - for Airflow
2.6, 2.7, (and 2.8) now
vs. many users having it months or years from now. And releasing it now
fixes actual issue
people have and raised us. And implementing it in provider has an added
benefit that
IF we also implement a generic serializer as part of ODBC and have a way for
serde to discover serializers by providers, then we could implement it in
the
provider in a way that either uses internal "serialization" or allows serde
to
discover it.

I see really no benefit of having Airflow know anything about ODBC. Are we
going to
implement all 80 provider's specific code and put it to airlfow and all
dependencies (even
optional) as airflow dependencies, now that we made enormous effort in
separating these?

Imagine ODBC code depends on odbc_library.
What happens if Airflow has optional dependency on odbc_library <3 and the
provider has dependency on odbc_library  >=3 ? Imagine that the provider
returns
ODBCRow from version 3 that serializer won't be able to handle.

How do you install the two together? How do you install these two together?
How do
you make them work together?

It really makes very little sense to have airflow depend on the library and
code that
provider already depends on, because this will create a whole new class of
potential conflicts.

I think the *only* possible way to make ODBC Row handled by serde is for
serde to discover serializers from installed providers and use them from
there.

This solves all the "dependency conflict" cases nicely and is long term
sustainable and maintainable.


> This is exactly the intention with the serializers in
> airflow.serialization.serializers,
> however can be tough as not all expose the information in public APIs (e.g.
> deltalake)
> that allow to reinstate the object.
>
> But yes: loose, and pragmatic public API / SemVer.
>
> Yep.

>
>

Re: [DISCUSS] Using serde for PyODBC case/other cases [was Serialization in Apache Airflow]

Posted by Bolke de Bruin <bd...@gmail.com>.
On Mon, 4 Dec 2023 at 20:09, Jarek Potiuk <ja...@potiuk.com> wrote:

>
> >
> > It is a choice that needs to be made at the API level - here Common.sql.
> In
> > this case the particular form of the tuple was, somewhat implicit, part
> of
> > the spec.
> >
>
> Yep. I rectified the implicitness by storing this as an explicit decision
> record
>  in the ADR proposal. I think it's been - similarly as lack of
> Serialization
> decision records - an omission that resulted in this, somewhat heated
> debate :D
>
>
<snip>


> >
> So to provide an example in context of our discussion of PyODBC.Row (see
> > the discussion of the linked PR) only common.sql can now deal with
> > PyODBC.Row. If for some reason another provider returns a PyODBC.Row that
> > object remains unserializable.
> >
>
> Yes. That's fine for me. I think we do not always have to serialize things
> the same way.
> Additionally serializing something using serde uses a custom format that
> requires
> deserialization step for it to be usable. DBApiHook response can be used
> "as is"
> as tuple for any other operator used by Common SQL
>
> I have explained what the downside is of the approach you took, although
> > its impact in this case is probably low (not many other places will
> return
> > PyODBC.Row, but consider DataFrame hypothetically).
> >
>
> Yes. For DataFrame I am fine. This is "common" format that it's worth to
> have
> serializer because it can be produced and consumed by many. It's quite
> unlikely to happen to ODBC and simply DBApiHook has its own "standard"
> being tuple that is the language DBAPIHook talks to the users. In a way
> returning PyODBC.Row in this sense violates assumptions made a year
> ago (now documented in the ADR proposal) - that this is the "API" of DB
> Hook.
> So while "make_serializable" was likely a wrong name, the goal of it was
> "produce a tuple that is directly serializable".
>
>
Yep, understood.


>
>
> > I think you needed a serializable tuple so that it could go into XCom.
> This
> > makes the need for an explicit choice imho.
> >
> > Yep. That's explicit choice we made.
>
>
>
> > >
> > >
> > On speed. Afaik AIP-44 still makes use of the DAG Serialization code
> apart
> > from Pydantic's rust implementation. The DAG serialization code is
> > seriously
> > slow. So, before claiming speed improvements here show the numbers please
> > :-)
>
>
> Yep. When we first added it we used Pydantic V1.And then it did not matter.
> But
> we have it using our serialization code - because at some point you
> insisted on
> making it part of "a serialization" - we have absolutely no need for that
> in AIP-44
> and I think when we go to prime time - we will just skip it entirely.
>

It was added to the DAG serializer at the time, I mentioned that that
looked weird
given the fact that it did not seem connected to DAG serialization (to me)
and serde
should be considered. So no, I wasn't insisting on making it part of
serialization imho,
it defacto was already.


>
> Going through either of the serializations of ours here is not needed
> overhead at all - we can just purely rely on Rust and get ORM on one side
> and Pydantic object on the other without touching any of our serialization
> code.
>

That's fine to me.


>
> We can of course keep both. However if you ever looked at the DAG
> > serialization code you might want to reconsider ;-).
>
>
> Yep. I am not at all strong on it. I do not intimately know that code
> either :).
>
>
I don't think you want to ;-).


>
> > Alternatively, you
> > could build on serde and provide the guarantees that you require there.
> It
> > can be done. I have been able to serialize a DAG with serde (hence
> knowing
> > the 1:10 ratio), but until now havent added it, because its schema looks
> > different.
> >
>
> Yeah. I am good for that if we can pull it off with backwards compatibility
> in place.
>
>
We could. It's probably just a three pass serializer then.


>
> >
> > BTW: There is no need from the XCom serializer / serde to deserialize
> into
> > the original object. By convention we like to of course, but it is not
> > required.
> >
>
> I understand, but the versioning feature for example assumes so. and adds
> complexity if the serialized form is to be used. From what I understand
> (and what some of the Serde tests
> verify the API of the serde is that you do in-fact get the same, comparable
> object
> when you do the serialize/deserialize roundtrip. And well, the only way to
> make use
> of the serialized object is to deserialize it first. Because of
> transparency, hiding the details
> and design of the serde, you should not care or understand about the
> "serialized" form so you
> effectively MUST deserialize to use it.
>
> So far - also looking at the tests - I was under the impression that it was
> expected property of Serde to get the roundtrip to return an equivalent
> object. But maybe I was mistaken here?
>

By convention, it is not required. It is just good practice, because what
you return
from a TaskFlow task you expect to have the same properties in a downstream
task. Hence, we go through lengths to serialize `builtin.set`,
`builtin.tuple` and
`builtin.frozenset` as the JSONEncoder loses type information when using
them.

Nevertheless, this is python so duck typing...

> For the record: the XCom serializer lazy loads its dependencies. So it
> does
> > not make
> > core Airflow dependent on client libraries arguably.
> >
>
> But it means that if we have the same libraries used in providers and in
> core - we have to declare the dependencies in two places - once for
> provider extra and once for serde "extension" extra. You cannot use
> provider dependencies if you want to have core use the same client library
> because of two reasons:
>
> 1) they might depend on different versions
> 2) you would have to specify provider extra and install the provider (which
> you might not need
>    in this case
>
> And in case 1) This might lead to the situation that if core and provider
> depend on different, conflicting versions of the libraries, we are f*d -
> because you might not be able to install provider
> in certain version with core in certain version. if we move the
> serialization code to the provider,
> we have no such problem. It always depends on the version that the provider
> requires.
>
>
> The con is that the API becomes strict. We cannot change it easily anymore.
> >
>
> Of course we can extend that. But I believe you wanted to make ODBCRow
> provider to depend on the very same API already? Isn't it. It seems I see
> contradiction. It looks like it's ok for
> the ODBCRow/provider to depend on serde and implement their serialization
> code, but on the other hand it's not ok for (say) DeltaLake to do the same
> and become Databricks provider?
>
> Or maybe I missed something? Maybe your idea was to make the ODBC
> serializer part of Airflow 2.8 and add >= airflow 2.8.0 for the ODBC
> provider? That's what I call unnecessary coupling.
>
>
Exactly not. By adding it to Airflow 2.8.0 it would have made every
previous version
of this provider suddenly work without any adjustments against Airflow
2.8.0. Remember
that the original bug report said "cannot serialize value of type
PyODBC.Row".

So no coupling.

You can argue both ways here architecturally.

1) Airflow 2.8 with a new feature that allows PyODBC.Row to be serialized,
affecting past and future providers at large and (TaskFlow) returned
variables.
2) Update the provider so the provider does the serialization individually.

1. requires an update to Airflow
2. requires an update of the provider

The bug report was used as a trigger for changing / improving / applying
the intended architecture of common.sql, but for the bug report 1 was
sufficient.


>
> >
> > > Maybe that's a bit of a different question for Pandas - as we have no
> > > separate Pandas provider
> > > and we use and link to it already from Airflow core. But... maybe it's
> > also
> > > a sign we need pandas
> > > provider in this case? This question for me is open.
> > >
> > >
> > To me this depends on what we want to do with data awareness (separate
> > discussion). If you think:
> >
> > ObjectStoragePath : Table : Dataframe
> >
> > It is useful to have dataframe as part of core. This could be pandas or
> > maybe polars which is so much faster :-).
> >
>
> Yeah - I am not too strong on that and I believe it has benefits to have
> both Pandas and Polars. Especially if we can make sure that it's very much
> loosely coupled (use public APIs ony and get very clear about SemVer/
> Versioning for those to get as open as possible) with Pandas and polars
> as optional extras, I am quite fine with it. I think we do not need a
> provider in this case.
>

This is exactly the intention with the serializers in
airflow.serialization.serializers,
however can be tough as not all expose the information in public APIs (e.g.
deltalake)
that allow to reinstate the object.

But yes: loose, and pragmatic public API / SemVer.


>
> Case-by-case is fine to me as long as the choice is explicit and thus
> > options are considered.
> >
> >
> Yep. I think this discussion is mostly about that :). And hopefully will
> result with some decisions recorded :)
>
>
Another one for the record :-).

Bolke


-- 

--
Bolke de Bruin
bdbruin@gmail.com