You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Ash Berlin-Taylor <as...@apache.org> on 2025/07/02 12:35:20 UTC

Code sharing between Airflow Core and Task SDK - how do we achieve it

Hello everyone,

As we work on finishing off the code-level separation of Task SDK and Core (scheduler etc) we have come across some situations where we would like to share code between these.

However it’s not as straight forward of “just put it in a common dist they both depend upon” because one of the goals of the Task SDK separation was to have 100% complete version independence between the two, ideally even if they are built into the same image and venv. Most of the reason why this isn’t straight forward comes down to backwards compatibility - if we make an change to the common/shared distribution 


We’ve listed the options we have thought about in https://github.com/apache/airflow/issues/51545 (but that covers some more things that I don’t want to get in to in this discussion such as possibly separating operators and executors out of a single provider dist.)

To give a concrete example of some code I would like to share https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py — logging config. Another thing we will want to share will be the AirflowConfigParser class from airflow.configuration (but notably: only the parser class, _not_ the default config values, again, lets not dwell on the specifics of that) 

So to bring the options listed in the issue here for discussion, broadly speaking there are two high-level approaches: 

1. A single shared distribution
2. No shared package and copy/duplicate code

The advantage of Approach 1 is that we only have the code in one place. However for me, at least in this specific case of Logging config or AirflowConfigParser class is that backwards compatibility is much much harder.

The main advantage of Approach 2 is the the code is released with/embedded in the dist (i.e. apache-airflow-task-sdk would contain the right version of the logging config and ConfigParser etc). The downside is that either the code will need to be duplicated in the repo, or better yet it would live in a single place in the repo, but some tooling (TBD) will automatically handle the duplication, either at commit time, or my preference, at release time.

For this kind of shared “utility” code I am very strongly leaning towards option 2 with automation, as otherwise I think the backwards compatibility requirements would make it unworkable (very quickly over time the combinations we would have to test would just be unreasonable) and I don’t feel confident we can have things as stable as we need to really deliver the version separation/independency I want to delivery with AIP-72.

So unless someone feels very strongly about this, I will come up with a draft PR for further discussion that will implement code sharing via “vendoring” it at build time. I have an idea of how I can achieve this so we have a single version in the repo and it’ll work there, but at runtime we vendor it in to the shipped dist so it lives at something like `airflow.sdk._vendor` etc.

In terms of repo layout, this likely means we would end up with:

airflow-core/pyproject.toml
airflow-core/src/
airflow-core/tests/
task-sdk/pyproject.toml
task-sdk/src/
task-sdk/tests/
airflow-common/src
airflow-common/tests/
# Possibly no airflow-common/pyproject.toml, as deps would be included in the downstream projects. TBD.

Thoughts and feedback welcomed.

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Jarek Potiuk <ja...@potiuk.com>.
Yes =- we already use symlinking actually. For standard provider's examples
added to airflow-core. And yes that is a good option for anything shared in
2) mode. When packaging , by default such symlinks are stored as files they
point to.

J.


On Wed, Jul 2, 2025 at 9:33 PM Jens Scheffler <j_...@gmx.de.invalid>
wrote:

> I'd also rather prefer option 2 - reason here is it is rather pragmatic
> and we no not need to cut another package and have less package counts
> and dependencies.
>
> I remember some time ago I was checking (together with Jarek, I am not
> sure anymore...) if the usage of symlinks would be possible. To keep the
> source in one package but "symlink" it into another. If then at point of
> packaging/release the files are materialized we have 1 set of code.
>
> Otherwise if not possible still the redundancy could be solved by a
> pre-commit hook - and in Git the files are de-duplicated anyway based on
> content hash, so this does not hurt.
>
> On 02.07.25 18:49, Shahar Epstein wrote:
> > I support option 2 with proper automation & CI - the reasonings you've
> > shown for that make sense to me.
> >
> >
> > Shahar
> >
> >
> > On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> >> Hello everyone,
> >>
> >> As we work on finishing off the code-level separation of Task SDK and
> Core
> >> (scheduler etc) we have come across some situations where we would like
> to
> >> share code between these.
> >>
> >> However it’s not as straight forward of “just put it in a common dist
> they
> >> both depend upon” because one of the goals of the Task SDK separation
> was
> >> to have 100% complete version independence between the two, ideally
> even if
> >> they are built into the same image and venv. Most of the reason why this
> >> isn’t straight forward comes down to backwards compatibility - if we
> make
> >> an change to the common/shared distribution
> >>
> >>
> >> We’ve listed the options we have thought about in
> >> https://github.com/apache/airflow/issues/51545 (but that covers some
> more
> >> things that I don’t want to get in to in this discussion such as
> possibly
> >> separating operators and executors out of a single provider dist.)
> >>
> >> To give a concrete example of some code I would like to share
> >>
> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
> >> — logging config. Another thing we will want to share will be the
> >> AirflowConfigParser class from airflow.configuration (but notably: only
> the
> >> parser class, _not_ the default config values, again, lets not dwell on
> the
> >> specifics of that)
> >>
> >> So to bring the options listed in the issue here for discussion, broadly
> >> speaking there are two high-level approaches:
> >>
> >> 1. A single shared distribution
> >> 2. No shared package and copy/duplicate code
> >>
> >> The advantage of Approach 1 is that we only have the code in one place.
> >> However for me, at least in this specific case of Logging config or
> >> AirflowConfigParser class is that backwards compatibility is much much
> >> harder.
> >>
> >> The main advantage of Approach 2 is the the code is released
> with/embedded
> >> in the dist (i.e. apache-airflow-task-sdk would contain the right
> version
> >> of the logging config and ConfigParser etc). The downside is that either
> >> the code will need to be duplicated in the repo, or better yet it would
> >> live in a single place in the repo, but some tooling (TBD) will
> >> automatically handle the duplication, either at commit time, or my
> >> preference, at release time.
> >>
> >> For this kind of shared “utility” code I am very strongly leaning
> towards
> >> option 2 with automation, as otherwise I think the backwards
> compatibility
> >> requirements would make it unworkable (very quickly over time the
> >> combinations we would have to test would just be unreasonable) and I
> don’t
> >> feel confident we can have things as stable as we need to really deliver
> >> the version separation/independency I want to delivery with AIP-72.
> >>
> >> So unless someone feels very strongly about this, I will come up with a
> >> draft PR for further discussion that will implement code sharing via
> >> “vendoring” it at build time. I have an idea of how I can achieve this
> so
> >> we have a single version in the repo and it’ll work there, but at
> runtime
> >> we vendor it in to the shipped dist so it lives at something like
> >> `airflow.sdk._vendor` etc.
> >>
> >> In terms of repo layout, this likely means we would end up with:
> >>
> >> airflow-core/pyproject.toml
> >> airflow-core/src/
> >> airflow-core/tests/
> >> task-sdk/pyproject.toml
> >> task-sdk/src/
> >> task-sdk/tests/
> >> airflow-common/src
> >> airflow-common/tests/
> >> # Possibly no airflow-common/pyproject.toml, as deps would be included
> in
> >> the downstream projects. TBD.
> >>
> >> Thoughts and feedback welcomed.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Ash Berlin-Taylor <as...@apache.org>.
Oh one thing to note:

In order to get both mypy and IntelliJ working wit this, we needed to commit the typestubs, and in order to make Python not get confused by the src/airflow/_vendor/airflow_shared directory existing and treating it as a namespace package, so the loader installer code now looks like this:


https://github.com/apache/airflow/pull/53149/files#diff-734233b1301f78bec9035c8be3f55be15b9f6f1662da8e56897cb2e72b281a76R56-R71

```
def _install_loader():
    # Add a new module Finder to find the imports when the vendored sources aren't found. We only add it when
    # something that looks like the airflow repo layout is already in the sys path i.e. when we are running in
    # a repo checkout.
    #
    # We need to add this to the dir first as we now keep the `.pyi` files for the vendored libs, and
    for dir in sys.path:
        if dir.endswith("shared/core-and-tasksdk/src"):
            sys.meta_path.insert(0, _NonVendoredAirflowSharedFinder(dir))
            break


if __debug__:
    # i.e. we are not running with PYTHONOPTMIZE. This is almost never set, but if someone _does_ run it, we
    # def don't want to load code from the wrong place
    _install_loader() 
```

> On 10 Jul 2025, at 17:39, Ash Berlin-Taylor <as...@apache.org> wrote:
> 
> Not quite final PR, but good enough that I want to see how it behaves on CI, and other IDEs etc https://github.com/apache/airflow/pull/53149 (We updated the `setup_idea.py`, so either re-run that or add the new src root manually)
> 
> Let the naming discussions start!
> 
> -ash
> 
>> On 10 Jul 2025, at 12:51, Jarek Potiuk <ja...@potiuk.com> wrote:
>> 
>> Yeah. when we get the final PR I will also want to to test more scenarios -
>> with IDE/mypy integration switching branches, uv syncing etc. and will be
>> happy to help and document the contributor's doc to explain what and how to
>> work with it. This would be a super cool thing if we get it to
>> work seamlessly for everyone :)
>> 
>> On Thu, Jul 10, 2025 at 7:44 AM Amogh Desai <am...@gmail.com>
>> wrote:
>> 
>>> Agreed!
>>> 
>>> Once the PR is up, we can have these implementation level discussions
>>> over there. Good chat however!
>>> 
>>> Thanks & Regards,
>>> Amogh Desai
>>> 
>>> 
>>> On Wed, Jul 9, 2025 at 3:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>> 
>>>> Yeah. I think extracting one-by-one, feature-by-feature that we want to
>>>> share to a separate distribution is the best approach - it will actually
>>>> also help with the "__init__.py" cleanup - because almost by definition -
>>>> those distributions will not be able to "reach" outside - i.e. they only
>>>> can be "used" not "use" something else. Which means that - for example as
>>>> it is now - configuration using logging which is using configuration
>>>> (leading to circular dependencies and partially initialized modules) will
>>>> not happen and we will have to figure out rather how to inject
>>>> configuration into logging (from task-sdk, airflow-ctl, airflow-core) and
>>>> to get the right sequence of initialization - rather than have the
>>>> inter-feature-dependencies.
>>>> 
>>>> And that is precisely what low-cyclomatic complexity is about as well -
>>>> it generally leads to easier-to-maintain software that has well defined
>>>> functionality and does not have those weird circular dependencies we have
>>>> now. That's a kind-of side-effect of such a "per feature" split, but a very
>>>> desirable one.
>>>> 
>>>> J.
>>>> 
>>>> On Wed, Jul 9, 2025 at 10:17 AM Amogh Desai <am...@gmail.com>
>>>> wrote:
>>>> 
>>>>> Probably, you make a valid point.
>>>>> 
>>>>> Maybe this is an implementation detail, so we could figure it out as we
>>>>> start on a POC and factor in these things
>>>>> as we move along?
>>>>> 
>>>>> But from an initial guess, I would think that execution time related
>>>>> items (if we manage to enumerate them) would be something
>>>>> that would be better off in that "core_and_task_sdk" bundle.
>>>>> 
>>>>> Thanks & Regards,
>>>>> Amogh Desai
>>>>> 
>>>>> 
>>>>> On Tue, Jul 8, 2025 at 3:46 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>> 
>>>>>>> Not that I am against your idea and we can surely expand as we need
>>>>>> but we would not need to expand the
>>>>>> "core_and_task_sdk" if we put only the relevant items into it.
>>>>>> 
>>>>>> So if we move logging and config out, my question is what is really
>>>>>> relevant to "stay" in "core_and_task_sdk" ? And what we know will not be
>>>>>> needed by other distributions in the future ?
>>>>>> 
>>>>>> What would be the content of "core_and_task_sdk" ? Can we enumerate
>>>>>> what should go there now ?
>>>>>> 
>>>>>> My bet is that if we enumerate them, it will turn out that they are
>>>>>> good candidates to make separate "shared features" that can be logically
>>>>>> named and modularised, and that there is no real need to have shared
>>>>>> "core_and_task_sdk" which sounds like "bag of everything else".
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Tue, Jul 8, 2025 at 11:48 AM Amogh Desai <am...@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Yeah, I think what you are showcasing here is a step ahead of the
>>>>>>> initial proposal from Ash.
>>>>>>> 
>>>>>>> From the original proposal, the `core_and_task_sdk` *can* have the
>>>>>>> things relevant to just those two
>>>>>>> distros. Logging, Config are modules that might be needed by
>>>>>>> airflow-ctl for example, so ideally, those
>>>>>>> would not be good candidates to be put in there, ideally speaking.
>>>>>>> 
>>>>>>> The example of Kubernetes Utils sounds to be a good example, it will
>>>>>>> be used by KubeExecutor (lets say this is a
>>>>>>> module called "executors") and by KPO (providers), and the
>>>>>>> "shared/kubernetes" would probably be a good
>>>>>>> candidate for that.
>>>>>>> 
>>>>>>> Not that I am against your idea and we can surely expand as we need
>>>>>>> but we would not need to expand the
>>>>>>> "core_and_task_sdk" if we put only the relevant items into it.
>>>>>>> 
>>>>>>> Thanks & Regards,
>>>>>>> Amogh Desai
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, Jul 8, 2025 at 12:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>> 
>>>>>>>>> @Jarek Potiuk <ja...@potiuk.com> a little confused on what you
>>>>>>>> mean there, I am understanding the direction
>>>>>>>> but could you elaborate a bit more please?
>>>>>>>> 
>>>>>>>> Let me elaborate:
>>>>>>>> 
>>>>>>>> As I understand (maybe I am wrong?),  the proposal is that we have a
>>>>>>>> "core-and-task-sdk" folder which is a shared distribution that is
>>>>>>>> vendored-in into both "airflow-core" and "airflow-task-sdk". This
>>>>>>>> contains some shared code that we want to include in both distributions
>>>>>>>> (note that we never ever release "core-and-task-sdk"
>>>>>>>> distribution because it only contains code that is shared between the two
>>>>>>>> distributions we release.
>>>>>>>> 
>>>>>>>> That's fine and cool.
>>>>>>>> 
>>>>>>>> Imagine that this distribution contains "logging" (shared code
>>>>>>>> for logging) and "config" (shared code for configuration). - both needed in
>>>>>>>> "airflow-core" and "airflow-task-sdk". So far so good. But what happen if
>>>>>>>> we want to use logging in the same fashion in say "airflow-ctl" (that also
>>>>>>>> applies for other distributions that we might come up with) ? Are we going
>>>>>>>> to vendor in the whole "core-and-task-sdk" distribution in "airflow-ctl" ?
>>>>>>>> It would be far better if we just vendor in "logging" and do not vendor-in
>>>>>>>> "config".
>>>>>>>> 
>>>>>>>> And if we are going to have a mechanism to vendor-in "a distribution"
>>>>>>>> - there is nothing wrong with having the same way to vendor-in multiple
>>>>>>>> distributions - so we can easily do it this way (i added "orm" ,
>>>>>>>> "serialization". and "fast_api" as an example thing that we might want to
>>>>>>>> share - not sure if that is really something we want to do but it will
>>>>>>>> allow to illustrate my idea better)
>>>>>>>> 
>>>>>>>> /
>>>>>>>> airflow-ctl/
>>>>>>>> task-sdk/...
>>>>>>>> airflow-core/...
>>>>>>>> ....
>>>>>>>> shared/
>>>>>>>>   kubernetes/
>>>>>>>>     pyproject.toml
>>>>>>>>     src/
>>>>>>>>       airflow_shared_kubernetes/__init__.py
>>>>>>>>   logging/
>>>>>>>>     pyproject.toml
>>>>>>>>     src/
>>>>>>>>       airflow_shared_logging/__init__.py
>>>>>>>>   config/
>>>>>>>>     pyproject.toml
>>>>>>>>     src/
>>>>>>>>       airflow_shared_config/__init__.py
>>>>>>>>   orm/
>>>>>>>>     pyproject.toml
>>>>>>>>     src/
>>>>>>>>       airflow_shared_orm/__init__.py
>>>>>>>>    serialization/
>>>>>>>>     pyproject.toml
>>>>>>>>     src/
>>>>>>>>       airflow_shared_serialization/__init__.py
>>>>>>>>    fast_api/
>>>>>>>>     pyproject.toml
>>>>>>>>     src/
>>>>>>>>       airflow_shared_fast_api/__init__.py
>>>>>>>>   ...
>>>>>>>> 
>>>>>>>> This has multiple benefits (and I see no real drawbacks):
>>>>>>>> 
>>>>>>>> * the code can be really well modularised.  Those "things" we share
>>>>>>>> (and this also connects to the __init__.py discussion) - can be independent
>>>>>>>> - and (it follow Jens comment) it allows to keep low cyclomatic complexity
>>>>>>>> https://en.wikipedia.org/wiki/Cyclomatic_complexity . It will be way
>>>>>>>> easier to implement logging in the way that it does not import or use
>>>>>>>> config. This means for example that configuration for logging will need to
>>>>>>>> be injected when logging is initialized - and that's exactly what we want,
>>>>>>>> we do not want logging code to use configuration code directly - they
>>>>>>>> should be independent from each other and you should be free to vendor-in
>>>>>>>> either logging or config independently if you also vendored-in the other.
>>>>>>>> 
>>>>>>>> * it's much more logical. We split based on functionality we want to
>>>>>>>> share - not about the "distributions" we want to produce. That allows us -
>>>>>>>> in the future - to make different decisions on how we split our
>>>>>>>> distributions. For example (i do not tell we have to do it, or that we will
>>>>>>>> do it but we will have such a possibility) - we can add more shared
>>>>>>>> utilities we find useful in the same way and decide that "scheduler",
>>>>>>>> "api_server" or "scheduler" or "triggerer" or "dag processor" are split to
>>>>>>>> separate distributions - because for example we want to keep a number of
>>>>>>>> dependencies down. And for example "api_server" might use "fast_api",
>>>>>>>> "config", "logging", "orm" and "fast_api" and "serialization" , where the
>>>>>>>> scheduler should not need "fast_api". The "dag_processor" eventually might
>>>>>>>> not need "orm" nor "fast_api" and only use the other
>>>>>>>> 
>>>>>>>> This seems like a natural approach. If we have a mechanism to "share"
>>>>>>>> the code, it does not add complexity, but allows us to isolate independent
>>>>>>>> functionality into "isolated" boxes and use them
>>>>>>>> 
>>>>>>>> Also for cyclomatic complexity that is a complex word (badly chosen
>>>>>>>> as it scares people away) and has some math behind, but it really boils
>>>>>>>> down to very simple "rules of thumb" (and yes I am a big proponent of
>>>>>>>> having low cyclomatic complexity).
>>>>>>>> 
>>>>>>>> a) when you are building the "final" product (i.e. distribution you
>>>>>>>> want to release) - make sure that you only "use" things - that nothing else
>>>>>>>> is "using" you as a library.
>>>>>>>> b) when you are building a "shared" thing (a library) - make sure
>>>>>>>> that library is only "used" by others but it does not really "use" anything
>>>>>>>> else. For example in the case I explained above - we can achieve
>>>>>>>> low-cyclomatic complexity when:
>>>>>>>> 
>>>>>>>> * airflow-core uses: logging, config, orm, serialization, fast_api
>>>>>>>> * none of the "logging, config, orm, serialization, fast_api" use
>>>>>>>> each other - they are at the bottom of the "user -> used" tree
>>>>>>>> 
>>>>>>>> J.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Tue, Jul 8, 2025 at 8:16 AM Amogh Desai <am...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> I like the folder structure proposed by Ash and have no objections
>>>>>>>>> with it.
>>>>>>>>> 
>>>>>>>>> "core_and_task_sdk" sounds good to me and justifies what it should
>>>>>>>>> do pretty well.
>>>>>>>>> 
>>>>>>>>> @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
>>>>>>>>> there, I am understanding the direction
>>>>>>>>> but could you elaborate a bit more please?
>>>>>>>>> 
>>>>>>>>> Naming is REALLY hard!
>>>>>>>>> 
>>>>>>>>> Thanks & Regards,
>>>>>>>>> Amogh Desai
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Tue, Jul 8, 2025 at 2:52 AM Jarek Potiuk <ja...@potiuk.com>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> How about splitting it even more and having each shared "thing"
>>>>>>>>>> named?
>>>>>>>>>> "logging", "config" and sharing them explicitly and separately with
>>>>>>>>>> the
>>>>>>>>>> right "user" ?
>>>>>>>>>> That sounds way more modular and  we will be able to choose which
>>>>>>>>>> of the
>>>>>>>>>> shared "utils" we use where.
>>>>>>>>>> 
>>>>>>>>>> J.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Mon, Jul 7, 2025 at 11:13 PM Jens Scheffler
>>>>>>>>>> <j_...@gmx.de.invalid>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> I like "core_and_task_sdk the same like core-and-task-sdk - I
>>>>>>>>>> have no
>>>>>>>>>>> problem and it is a path only.
>>>>>>>>>>> 
>>>>>>>>>>> if we get to "dag-parser-scheduler-task-sdk-and-triggerer" which
>>>>>>>>>> is a
>>>>>>>>>>> bit bulky we then should name it "all-not-api-server" :-D
>>>>>>>>>>> 
>>>>>>>>>>> On 07.07.25 22:57, Ash Berlin-Taylor wrote:
>>>>>>>>>>>> In case I did a bad job explaining it, the “core and task sdk”
>>>>>>>>>> is not in
>>>>>>>>>>> the module name/import name, just in the file path.
>>>>>>>>>>>> 
>>>>>>>>>>>> Anyone have other ideas?
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 7 Jul 2025, at 21:37, Buğra Öztürk <oz...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks Ash! Looks cool! I like the structure. This will
>>>>>>>>>> enable all the
>>>>>>>>>>>>> combinations and structure looks easy to grasp. No strong
>>>>>>>>>> stance on the
>>>>>>>>>>>>> naming other than maybe it is a bit long with `and`,
>>>>>>>>>> `core_ctl` could be
>>>>>>>>>>>>> shorter, since no import path is defined like that, we can
>>>>>>>>>> give any name
>>>>>>>>>>>>> for sure.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Mon, 7 Jul 2025, 21:51 Jarek Potiuk, <ja...@potiuk.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Looks good but I think we should find some better logical
>>>>>>>>>> name for
>>>>>>>>>>>>>> core_and_sdk :)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
>>>>>>>>>>>>>> <j_...@gmx.de.invalid> napisał:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cool! Especially the "shared" folder with the ability to have
>>>>>>>>>>>>>>> N-combinations w/o exploding project repo root!
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 07.07.25 14:43, Ash Berlin-Taylor wrote:
>>>>>>>>>>>>>>>> Oh, and all of this will be explain in shared/README.md
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Okay, so it seems we have agreement on the approach here,
>>>>>>>>>> so I’ll
>>>>>>>>>>>>>>> continue with this, and on the dev call it was mentioned that
>>>>>>>>>>>>>>> “airflow-common” wasn’t a great name, so here is my proposal
>>>>>>>>>> for the
>>>>>>>>>>> file
>>>>>>>>>>>>>>> structure;
>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>> /
>>>>>>>>>>>>>>>>> task-sdk/...
>>>>>>>>>>>>>>>>> airflow-core/...
>>>>>>>>>>>>>>>>> shared/
>>>>>>>>>>>>>>>>>   kuberenetes/
>>>>>>>>>>>>>>>>>     pyproject.toml
>>>>>>>>>>>>>>>>>     src/
>>>>>>>>>>>>>>>>>       airflow_kube/__init__.py
>>>>>>>>>>>>>>>>>   core-and-tasksdk/
>>>>>>>>>>>>>>>>>     pyproject.toml
>>>>>>>>>>>>>>>>>     src/
>>>>>>>>>>>>>>>>>       airflow_shared/__init__.py
>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Things to note here: the “shared” folder has (the
>>>>>>>>>> possibility) of
>>>>>>>>>>>>>>> having multiple different shared “libraries” in it, in this
>>>>>>>>>> example I
>>>>>>>>>>> am
>>>>>>>>>>>>>>> supposing a hypothetical shared kuberenetes folder a world
>>>>>>>>>> in which we
>>>>>>>>>>>>>>> split the KubePodOperator and the KubeExecutor in to two
>>>>>>>>>> separate
>>>>>>>>>>>>>>> distributions (example only, not proposing we do that right
>>>>>>>>>> now, and
>>>>>>>>>>> that
>>>>>>>>>>>>>>> will be a separate discussion)
>>>>>>>>>>>>>>>>> The other things to note here:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> - the folder name in shared aims to be “self-documenting”,
>>>>>>>>>> hence the
>>>>>>>>>>>>>>> verbose “core-and-tasksdk” to say where the shared library is
>>>>>>>>>>> intended to
>>>>>>>>>>>>>>> be used.
>>>>>>>>>>>>>>>>> - the python module itself should almost always have an
>>>>>>>>>> `airflow_`
>>>>>>>>>>> (or
>>>>>>>>>>>>>>> maybe `_airflow_`?) prefix so that it does not conflict with
>>>>>>>>>> anything
>>>>>>>>>>>>>> else
>>>>>>>>>>>>>>> we might use. It won’t matter “in production” as those will
>>>>>>>>>> be
>>>>>>>>>>> vendored
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> to be imported as `airflow/_vendor/airflow_shared` etc, but
>>>>>>>>>> avoiding
>>>>>>>>>>>>>>> conflicts at dev time with the Finder approach is a good
>>>>>>>>>> safety
>>>>>>>>>>> measure.
>>>>>>>>>>>>>>>>> I will start making a real PR for this proposal now, but
>>>>>>>>>> I’m open to
>>>>>>>>>>>>>>> feedback (either here, or in the PR when I open it)
>>>>>>>>>>>>>>>>> -ash
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Yeah we have to try it and test - also building packages
>>>>>>>>>> happens
>>>>>>>>>>> semi
>>>>>>>>>>>>>>>>>> frequently when you run `uv sync` (they use some kind of
>>>>>>>>>> heuristics
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> decide when) and you can force it with `--reinstall` or
>>>>>>>>>>> `--refresh`.
>>>>>>>>>>>>>>>>>> Package build also happens every time when you run
>>>>>>>>>> "ci-image build`
>>>>>>>>>>>>>>> now in
>>>>>>>>>>>>>>>>>> breeze so it seems like it will nicely integrate in our
>>>>>>>>>> workflows.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Looks really cool Ash.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <
>>>>>>>>>> ash@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> It’s not just release time, but any time we build a
>>>>>>>>>> package which
>>>>>>>>>>>>>>> happens
>>>>>>>>>>>>>>>>>>> on “every” CI run. The normal unit tests will use code
>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>> airflow-common/src/airflow_common; the kube tests which
>>>>>>>>>> build an
>>>>>>>>>>>>>>> image will
>>>>>>>>>>>>>>>>>>> build the dists and vendor in the code from that commit.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> There is only a single copy of the shared code committed
>>>>>>>>>> to the
>>>>>>>>>>>>>> repo,
>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>> there is never anything to synchronise.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On 4 Jul 2025, at 15:53, Amogh Desai <
>>>>>>>>>> amoghdesai.oss@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> Thanks Ash.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> This is really cool and helpful that you were able to
>>>>>>>>>> test both
>>>>>>>>>>>>>>> scenarios
>>>>>>>>>>>>>>>>>>>> -- repo checkout
>>>>>>>>>>>>>>>>>>>> and also installing from the vendored package and the
>>>>>>>>>> resolution
>>>>>>>>>>>>>>> worked
>>>>>>>>>>>>>>>>>>>> fine too.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I like this idea compared the to relative import one
>>>>>>>>>> for few
>>>>>>>>>>>>>> reasons:
>>>>>>>>>>>>>>>>>>>> - It feels like it will take some time to adjust to the
>>>>>>>>>> new
>>>>>>>>>>> coding
>>>>>>>>>>>>>>>>>>> standard
>>>>>>>>>>>>>>>>>>>> that we will lay
>>>>>>>>>>>>>>>>>>>> if we impose relative imports in the shared dist
>>>>>>>>>>>>>>>>>>>> - We can continue using repo wise absolute import
>>>>>>>>>> standards, it
>>>>>>>>>>> is
>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>> much easier for situations
>>>>>>>>>>>>>>>>>>>> when we do global search in IDE to find + replace, this
>>>>>>>>>> could
>>>>>>>>>>> mean
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>> - The vendoring work is a proven and established
>>>>>>>>>> paradigm across
>>>>>>>>>>>>>>> projects
>>>>>>>>>>>>>>>>>>>> and would
>>>>>>>>>>>>>>>>>>>> out of box give us the build tooling we need also
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Nothing too against the relative import but with the
>>>>>>>>>> evidence
>>>>>>>>>>>>>>> provided
>>>>>>>>>>>>>>>>>>>> above, vendored approach
>>>>>>>>>>>>>>>>>>>> seems to only do us good.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Regarding synchronizing it, release time should be fine
>>>>>>>>>> as long
>>>>>>>>>>> as
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>> a good CI workflow to probably
>>>>>>>>>>>>>>>>>>>> catch such issues per PR if changes are made in shared
>>>>>>>>>> dist?
>>>>>>>>>>>>>>> (precommit
>>>>>>>>>>>>>>>>>>>> would make it really slow i guess)
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> If we can run our tests with vendored code we should be
>>>>>>>>>> mostly
>>>>>>>>>>>>>>> covered.
>>>>>>>>>>>>>>>>>>>> Good effort all!
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>>>>>>>>> Amogh Desai
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <
>>>>>>>>>>> ash@apache.org>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> Okay, I think I’ve got something that works and I’m
>>>>>>>>>> happy with.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>>>>>>>>>>>>>>>>>>>>> This produces the following from `uv build task-sdk`
>>>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>>>>>>>>>>>>>>>>>>>>> (`.whl.zip` as GH won't allow .whl upload, but will
>>>>>>>>>> .zip)
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>>> ❯ unzip -l
>>>>>>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
>>>>>>>>>>>>>>>>>>> grep
>>>>>>>>>>>>>>>>>>>>> _vendor
>>>>>>>>>>>>>>>>>>>>>    50  02-02-2020 00:00
>>>>>>>>>> airflow/sdk/_vendor/.gitignore
>>>>>>>>>>>>>>>>>>>>>  2082  02-02-2020 00:00
>>>>>>>>>> airflow/sdk/_vendor/__init__.py
>>>>>>>>>>>>>>>>>>>>>    28  02-02-2020 00:00
>>>>>>>>>>> airflow/sdk/_vendor/airflow_common.pyi
>>>>>>>>>>>>>>>>>>>>>    18  02-02-2020 00:00
>>>>>>>>>> airflow/sdk/_vendor/vendor.txt
>>>>>>>>>>>>>>>>>>>>>   785  02-02-2020 00:00
>>>>>>>>>>>>>>>>>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>>>>>>>>>>>>>>>>> 10628  02-02-2020 00:00
>>>>>>>>>>>>>>>>>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> And similarly in the .tar.gz, so our “sdist” is
>>>>>>>>>> complete too:
>>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>>>>>>> |grep
>>>>>>>>>>> _vendor
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> The plugin works at build time by including/copying
>>>>>>>>>> the libs
>>>>>>>>>>>>>>> specified
>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> vendor.txt into place (and let `vendoring` take care
>>>>>>>>>> of import
>>>>>>>>>>>>>>>>>>> rewrites.)
>>>>>>>>>>>>>>>>>>>>> For the imports to continue to work at “dev” time/from
>>>>>>>>>> a repo
>>>>>>>>>>>>>>> checkout,
>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>> have added a import finder to `sys.meta_path`, and
>>>>>>>>>> since it’s at
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> end of
>>>>>>>>>>>>>>>>>>>>> the list it will only be used if the normal import
>>>>>>>>>> can’t find
>>>>>>>>>>>>>>> things.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>>>>>>>>>>>>>>>>>>>>> This doesn’t quite give us the same runtime effect
>>>>>>>>>> “import
>>>>>>>>>>>>>>> rewriting”
>>>>>>>>>>>>>>>>>>>>> affect, as in this approach `airflow_common` is
>>>>>>>>>> directly loaded
>>>>>>>>>>>>>>> (i.e.
>>>>>>>>>>>>>>>>>>>>> airflow.sdk._vendor.airflow_common and airflow_common
>>>>>>>>>> exist in
>>>>>>>>>>>>>>>>>>>>> sys.modules), but it does work for everything that I
>>>>>>>>>> was able to
>>>>>>>>>>>>>>> test..
>>>>>>>>>>>>>>>>>>>>> I tested it with the diff at the end of this message.
>>>>>>>>>> My test
>>>>>>>>>>>>>>> ipython
>>>>>>>>>>>>>>>>>>>>> shell:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>>> In [1]: from
>>>>>>>>>> airflow.sdk._vendor.airflow_common.timezone import
>>>>>>>>>>>>>> foo
>>>>>>>>>>>>>>>>>>>>> In [2]: foo
>>>>>>>>>>>>>>>>>>>>> Out[2]: 1
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> In [3]: import airflow.sdk._vendor.airflow_common
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> In [4]: import
>>>>>>>>>> airflow.sdk._vendor.airflow_common.timezone
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>>>>>>>>>>>>>>>>>>>>> Out[5]:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>>>>>>>>>>>>>>>>>>>>> In [6]:
>>>>>>>>>> airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>>>>>>>>>>>>>>>>> Out[6]:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> And in an standalone environment with the SDK dist I
>>>>>>>>>> built (it
>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> matching airflow-core right now, but that is nothing
>>>>>>>>>> to do with
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> discussion):
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>>>>>>>>>>>>>>>>>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>>>>>>>>>>>>>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl
>>>>>>>>>> ipython
>>>>>>>>>>>>>>>>>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang
>>>>>>>>>> 18.1.8 ]
>>>>>>>>>>>>>>>>>>>>> Type 'copyright', 'credits' or 'license' for more
>>>>>>>>>> information
>>>>>>>>>>>>>>>>>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type
>>>>>>>>>> '?' for
>>>>>>>>>>>>>> help.
>>>>>>>>>>>>>>>>>>>>> Tip: You can use `%hist` to view history, see the
>>>>>>>>>> options with
>>>>>>>>>>>>>>>>>>> `%history?`
>>>>>>>>>>>>>>>>>>>>> In [1]: import
>>>>>>>>>> airflow.sdk._vendor.airflow_common.timezone
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> In [2]:
>>>>>>>>>> airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>>>>>>>>>>>>>>>>> Out[2]:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>>>>>>>>>>>>>>>>>>>>> ``
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> ```diff
>>>>>>>>>>>>>>>>>>>>> diff --git
>>>>>>>>>> a/airflow-common/src/airflow_common/__init__.py
>>>>>>>>>>>>>>>>>>>>> b/airflow-common/src/airflow_common/__init__.py
>>>>>>>>>>>>>>>>>>>>> index 13a83393a9..927b7c6b61 100644
>>>>>>>>>>>>>>>>>>>>> --- a/airflow-common/src/airflow_common/__init__.py
>>>>>>>>>>>>>>>>>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
>>>>>>>>>>>>>>>>>>>>> @@ -14,3 +14,5 @@
>>>>>>>>>>>>>>>>>>>>> # KIND, either express or implied.  See the License
>>>>>>>>>> for the
>>>>>>>>>>>>>>>>>>>>> # specific language governing permissions and
>>>>>>>>>> limitations
>>>>>>>>>>>>>>>>>>>>> # under the License.
>>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>>> +foo = 1
>>>>>>>>>>>>>>>>>>>>> diff --git
>>>>>>>>>> a/airflow-common/src/airflow_common/timezone.py
>>>>>>>>>>>>>>>>>>>>> b/airflow-common/src/airflow_common/timezone.py
>>>>>>>>>>>>>>>>>>>>> index 340b924c66..58384ef20f 100644
>>>>>>>>>>>>>>>>>>>>> --- a/airflow-common/src/airflow_common/timezone.py
>>>>>>>>>>>>>>>>>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
>>>>>>>>>>>>>>>>>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>>>>>>>>>>>>>>>>>>>>> version.parse(metadata.version("pendulum")).major == 3
>>>>>>>>>>>>>>>>>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
>>>>>>>>>>>>>>>>>>>>> utc = pendulum.UTC
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>>> +from airflow_common import foo
>>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>>> TIMEZONE: Timezone
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <
>>>>>>>>>> jarek@potiuk.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> I think both approaches are doable:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 1) -> We can very easily prevent bad imports by
>>>>>>>>>> pre-commit when
>>>>>>>>>>>>>>>>>>> importing
>>>>>>>>>>>>>>>>>>>>>> from different distributions and make sure we are
>>>>>>>>>> only doing
>>>>>>>>>>>>>>> relative
>>>>>>>>>>>>>>>>>>>>>> imports in the shared modules. We are doing plenty of
>>>>>>>>>> this
>>>>>>>>>>>>>>> already. And
>>>>>>>>>>>>>>>>>>>>> yes
>>>>>>>>>>>>>>>>>>>>>> it would require relative links we currently do not
>>>>>>>>>> allow.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 2) -> has one disadvantage that someone at some point
>>>>>>>>>> in time
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> decide to synchronize this and if it happens just
>>>>>>>>>> before
>>>>>>>>>>> release
>>>>>>>>>>>>>>> (I bet
>>>>>>>>>>>>>>>>>>>>>> this is going to happen) this will lead to solving
>>>>>>>>>> problems
>>>>>>>>>>> that
>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>> normally be solved during PR when you make a change
>>>>>>>>>> (i.e.
>>>>>>>>>>>>>> symbolic
>>>>>>>>>>>>>>> link
>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>> the advantage that whoever modifies shared code will
>>>>>>>>>> be
>>>>>>>>>>>>>> immediately
>>>>>>>>>>>>>>>>>>>>>> notified in their PR - that they broke something
>>>>>>>>>> because either
>>>>>>>>>>>>>>> static
>>>>>>>>>>>>>>>>>>>>>> checks or mypy or tests fail.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Ash, do you have an idea of a process (who and when)
>>>>>>>>>> does the
>>>>>>>>>>>>>>>>>>>>>> synchronisation in case of vendoring? Maybe we could
>>>>>>>>>> solve it
>>>>>>>>>>> if
>>>>>>>>>>>>>>> it is
>>>>>>>>>>>>>>>>>>>>> done
>>>>>>>>>>>>>>>>>>>>>> more frequently and with some regularity? We could
>>>>>>>>>> potentially
>>>>>>>>>>>>>>> force
>>>>>>>>>>>>>>>>>>>>>> re-vendoring at PR time as well any time shared code
>>>>>>>>>> changes
>>>>>>>>>>> (and
>>>>>>>>>>>>>>>>>>> prevent
>>>>>>>>>>>>>>>>>>>>>> it by pre-commit. And I can't think of some place
>>>>>>>>>> (other than
>>>>>>>>>>>>>>> releases)
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>> our development workflow and that seems to be a bit
>>>>>>>>>> too late as
>>>>>>>>>>>>>>> puts an
>>>>>>>>>>>>>>>>>>>>>> extra effort on fixing potential incompatibilities
>>>>>>>>>> introduced
>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>>>>> manager and delays the release. WDYT?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Re: relative links. I think for a shared library we
>>>>>>>>>> could
>>>>>>>>>>>>>>> potentially
>>>>>>>>>>>>>>>>>>>>> relax
>>>>>>>>>>>>>>>>>>>>>> this and allow them (and actually disallow absolute
>>>>>>>>>> links in
>>>>>>>>>>> the
>>>>>>>>>>>>>>> pieces
>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>> code that are shared - again, by pre-commit). As I
>>>>>>>>>> recall, the
>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>> reason
>>>>>>>>>>>>>>>>>>>>>> we forbade the relative links is because of how we
>>>>>>>>>> are (or
>>>>>>>>>>> maybe
>>>>>>>>>>>>>>> were)
>>>>>>>>>>>>>>>>>>>>>> doing DAG parsing and failures resulting from it. So
>>>>>>>>>> we decided
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>> allow it to keep consistency. The way how Dag parsing
>>>>>>>>>> works is
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>>>>> you are using importlib to read the Dag from a file,
>>>>>>>>>> the
>>>>>>>>>>> relative
>>>>>>>>>>>>>>>>>>> imports
>>>>>>>>>>>>>>>>>>>>>> do not work as it does not know what they should be
>>>>>>>>>> relative
>>>>>>>>>>> to.
>>>>>>>>>>>>>>> But if
>>>>>>>>>>>>>>>>>>>>>> relative import is done from an imported package, it
>>>>>>>>>> should be
>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>>>>> problem,
>>>>>>>>>>>>>>>>>>>>>> I think - otherwise our Dags would not be able to
>>>>>>>>>> import any
>>>>>>>>>>>>>>> library
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> uses relative imports.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Of course consistency might be the reason why we do
>>>>>>>>>> not want to
>>>>>>>>>>>>>>>>>>> introduce
>>>>>>>>>>>>>>>>>>>>>> relative imports. I don't see it as an issue if it is
>>>>>>>>>> guarded
>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>> pre-commit
>>>>>>>>>>>>>>>>>>>>>> though.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
>>>>>>>>>>>>>>> ash@apache.org>
>>>>>>>>>>>>>>>>>>>>>> napisał:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Oh yes, symlinks will work, with one big caveat: It
>>>>>>>>>> does mean
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>> can’t
>>>>>>>>>>>>>>>>>>>>>>> use absolute imports in one common module to another.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> For example
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>>>>>>>>>>>>>>>>>>>>>>> where we have
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>>>>> from airflow.utils.module_loading import
>>>>>>>>>> import_string
>>>>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> if we want to move serve_logs into this common lib
>>>>>>>>>> that is
>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>>> symlinked
>>>>>>>>>>>>>>>>>>>>>>> then we wouldn’t be able to have `from
>>>>>>>>>>>>>>> airflow_common.module_loading
>>>>>>>>>>>>>>>>>>>>> import
>>>>>>>>>>>>>>>>>>>>>>> import_string`.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I can think of two possible solutions here.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 1) is to allow/require relative imports in this
>>>>>>>>>> shared lib, so
>>>>>>>>>>>>>>> `from
>>>>>>>>>>>>>>>>>>>>>>> .module_loading import import_string`
>>>>>>>>>>>>>>>>>>>>>>> 2) is to use `vendoring`[1] (from the pip
>>>>>>>>>> maintainers) which
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>> handle
>>>>>>>>>>>>>>>>>>>>>>> import-rewriting for us.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I’d entirely forgot that symlinks in repos was a
>>>>>>>>>> thing, so I
>>>>>>>>>>>>>>> prepared
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>> minimal POC/demo of what vendoring approach could
>>>>>>>>>> look like
>>>>>>>>>>> here
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>>>>>>>>>>>>>>>>>>>>>>> Now personally I am more than happy with relative
>>>>>>>>>> imports, but
>>>>>>>>>>>>>>>>>>> generally
>>>>>>>>>>>>>>>>>>>>>>> as a project we have avoided them, so I think that
>>>>>>>>>> limits what
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>>>>>> with a symlink based approach.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> -ash
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> [1] https://github.com/pradyunsg/vendoring
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
>>>>>>>>>>>>>>>>>>> gopidesupavan@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> Thanks Ash
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Yes agree option 2 would be preferred for me.
>>>>>>>>>> Making sure we
>>>>>>>>>>>>>>> have all
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> gaurdriles to protect any unwanted behaviour in
>>>>>>>>>> code sharing
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> executing
>>>>>>>>>>>>>>>>>>>>>>>> right of tests between the packages.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Agree with others, option 2 would be
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
>>>>>>>>>>>>>>>>>>> amoghdesai.oss@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for starting this discussion, Ash.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> I would prefer option 2 here with proper tooling
>>>>>>>>>> to handle
>>>>>>>>>>> the
>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>> duplication at *release* time.
>>>>>>>>>>>>>>>>>>>>>>>>> It is best to have a dist that has all it needs in
>>>>>>>>>> itself.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Option 1 could very quickly get out of hand and if
>>>>>>>>>> we decide
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> separate
>>>>>>>>>>>>>>>>>>>>>>>>> triggerer /
>>>>>>>>>>>>>>>>>>>>>>>>> dag processor / config etc etc as separate
>>>>>>>>>> packages, back
>>>>>>>>>>>>>>> compat is
>>>>>>>>>>>>>>>>>>>>>>> going
>>>>>>>>>>>>>>>>>>>>>>>>> to be a nightmare
>>>>>>>>>>>>>>>>>>>>>>>>> and will bite us harder than we anticipate.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>>>>>>>>>>>>>> Amogh Desai
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <
>>>>>>>>>>>>>> kaxilnaik@gmail.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> I prefer Option 2 as well to avoid matrix of
>>>>>>>>>> dependencies
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>>>>>>>>>>>>>>>>>>>>> <j_scheffler@gmx.de.invalid
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd also rather prefer option 2 - reason here is
>>>>>>>>>> it is
>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>>>>>>> pragmatic
>>>>>>>>>>>>>>>>>>>>>>>>>>> and we no not need to cut another package and
>>>>>>>>>> have less
>>>>>>>>>>>>>>> package
>>>>>>>>>>>>>>>>>>>>> counts
>>>>>>>>>>>>>>>>>>>>>>>>>>> and dependencies.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> I remember some time ago I was checking
>>>>>>>>>> (together with
>>>>>>>>>>>>>> Jarek,
>>>>>>>>>>>>>>> I am
>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>> sure anymore...) if the usage of symlinks would
>>>>>>>>>> be
>>>>>>>>>>> possible.
>>>>>>>>>>>>>>> To
>>>>>>>>>>>>>>>>>>> keep
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> source in one package but "symlink" it into
>>>>>>>>>> another. If
>>>>>>>>>>> then
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>> packaging/release the files are materialized we
>>>>>>>>>> have 1 set
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> code.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise if not possible still the redundancy
>>>>>>>>>> could be
>>>>>>>>>>>>>>> solved by
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>> pre-commit hook - and in Git the files are
>>>>>>>>>> de-duplicated
>>>>>>>>>>>>>>> anyway
>>>>>>>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>> content hash, so this does not hurt.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I support option 2 with proper automation & CI
>>>>>>>>>> - the
>>>>>>>>>>>>>>> reasonings
>>>>>>>>>>>>>>>>>>>>>>>>> you've
>>>>>>>>>>>>>>>>>>>>>>>>>>>> shown for that make sense to me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Shahar
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash
>>>>>>>>>> Berlin-Taylor <
>>>>>>>>>>>>>>> ash@apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As we work on finishing off the code-level
>>>>>>>>>> separation of
>>>>>>>>>>>>>>> Task
>>>>>>>>>>>>>>>>>>> SDK
>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>> Core
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (scheduler etc) we have come across some
>>>>>>>>>> situations
>>>>>>>>>>> where
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> share code between these.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However it’s not as straight forward of “just
>>>>>>>>>> put it in
>>>>>>>>>>> a
>>>>>>>>>>>>>>> common
>>>>>>>>>>>>>>>>>>>>>>>>> dist
>>>>>>>>>>>>>>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> both depend upon” because one of the goals of
>>>>>>>>>> the Task
>>>>>>>>>>> SDK
>>>>>>>>>>>>>>>>>>>>>>>>> separation
>>>>>>>>>>>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to have 100% complete version independence
>>>>>>>>>> between the
>>>>>>>>>>>>>> two,
>>>>>>>>>>>>>>>>>>>>> ideally
>>>>>>>>>>>>>>>>>>>>>>>>>>> even if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> they are built into the same image and venv.
>>>>>>>>>> Most of the
>>>>>>>>>>>>>>> reason
>>>>>>>>>>>>>>>>>>>>> why
>>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> isn’t straight forward comes down to backwards
>>>>>>>>>>>>>>> compatibility -
>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an change to the common/shared distribution
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We’ve listed the options we have thought about
>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545
>>>>>>>>>> (but
>>>>>>>>>>> that
>>>>>>>>>>>>>>> covers
>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> things that I don’t want to get in to in this
>>>>>>>>>> discussion
>>>>>>>>>>>>>>> such as
>>>>>>>>>>>>>>>>>>>>>>>>>>> possibly
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> separating operators and executors out of a
>>>>>>>>>> single
>>>>>>>>>>>>>> provider
>>>>>>>>>>>>>>>>>>> dist.)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To give a concrete example of some code I
>>>>>>>>>> would like to
>>>>>>>>>>>>>>> share
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> — logging config. Another thing we will want
>>>>>>>>>> to share
>>>>>>>>>>> will
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AirflowConfigParser class from
>>>>>>>>>> airflow.configuration
>>>>>>>>>>> (but
>>>>>>>>>>>>>>>>>>> notably:
>>>>>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parser class, _not_ the default config values,
>>>>>>>>>> again,
>>>>>>>>>>> lets
>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> dwell
>>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specifics of that)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So to bring the options listed in the issue
>>>>>>>>>> here for
>>>>>>>>>>>>>>> discussion,
>>>>>>>>>>>>>>>>>>>>>>>>>> broadly
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> speaking there are two high-level approaches:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. A single shared distribution
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The advantage of Approach 1 is that we only
>>>>>>>>>> have the
>>>>>>>>>>> code
>>>>>>>>>>>>>>> in one
>>>>>>>>>>>>>>>>>>>>>>>>>> place.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However for me, at least in this specific case
>>>>>>>>>> of
>>>>>>>>>>> Logging
>>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> AirflowConfigParser class is that backwards
>>>>>>>>>>> compatibility
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> much
>>>>>>>>>>>>>>>>>>>>>>>>> much
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> harder.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main advantage of Approach 2 is the the
>>>>>>>>>> code is
>>>>>>>>>>>>>> released
>>>>>>>>>>>>>>>>>>>>>>>>>>> with/embedded
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk
>>>>>>>>>> would contain
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> right
>>>>>>>>>>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the logging config and ConfigParser etc).
>>>>>>>>>> The
>>>>>>>>>>> downside
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>> either
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the code will need to be duplicated in the
>>>>>>>>>> repo, or
>>>>>>>>>>> better
>>>>>>>>>>>>>>> yet
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> live in a single place in the repo, but some
>>>>>>>>>> tooling
>>>>>>>>>>> (TBD)
>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> automatically handle the duplication, either
>>>>>>>>>> at commit
>>>>>>>>>>>>>>> time, or
>>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> preference, at release time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For this kind of shared “utility” code I am
>>>>>>>>>> very
>>>>>>>>>>> strongly
>>>>>>>>>>>>>>>>>>> leaning
>>>>>>>>>>>>>>>>>>>>>>>>>>> towards
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> option 2 with automation, as otherwise I think
>>>>>>>>>> the
>>>>>>>>>>>>>> backwards
>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirements would make it unworkable (very
>>>>>>>>>> quickly over
>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> combinations we would have to test would just
>>>>>>>>>> be
>>>>>>>>>>>>>>> unreasonable)
>>>>>>>>>>>>>>>>>>>>> and I
>>>>>>>>>>>>>>>>>>>>>>>>>>> don’t
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feel confident we can have things as stable as
>>>>>>>>>> we need
>>>>>>>>>>> to
>>>>>>>>>>>>>>> really
>>>>>>>>>>>>>>>>>>>>>>>>>> deliver
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the version separation/independency I want to
>>>>>>>>>> delivery
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> AIP-72.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So unless someone feels very strongly about
>>>>>>>>>> this, I will
>>>>>>>>>>>>>>> come up
>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> draft PR for further discussion that will
>>>>>>>>>> implement code
>>>>>>>>>>>>>>> sharing
>>>>>>>>>>>>>>>>>>>>> via
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “vendoring” it at build time. I have an idea
>>>>>>>>>> of how I
>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> achieve
>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we have a single version in the repo and it’ll
>>>>>>>>>> work
>>>>>>>>>>> there,
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> runtime
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we vendor it in to the shipped dist so it
>>>>>>>>>> lives at
>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In terms of repo layout, this likely means we
>>>>>>>>>> would end
>>>>>>>>>>> up
>>>>>>>>>>>>>>> with:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> airflow-core/pyproject.toml
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> airflow-core/src/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> airflow-core/tests/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> task-sdk/pyproject.toml
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> task-sdk/src/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> task-sdk/tests/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> airflow-common/src
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> airflow-common/tests/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml,
>>>>>>>>>> as deps
>>>>>>>>>>> would
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>> included
>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the downstream projects. TBD.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thoughts and feedback welcomed.
>>>>>>>>>>>>>>> 
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>> dev-unsubscribe@airflow.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>> dev-help@airflow.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>> dev-unsubscribe@airflow.apache.org
>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>> dev-help@airflow.apache.org
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>> dev-help@airflow.apache.org
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>> dev-help@airflow.apache.org
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org


Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Ash Berlin-Taylor <as...@apache.org>.
Not quite final PR, but good enough that I want to see how it behaves on CI, and other IDEs etc https://github.com/apache/airflow/pull/53149 (We updated the `setup_idea.py`, so either re-run that or add the new src root manually)

Let the naming discussions start!

-ash

> On 10 Jul 2025, at 12:51, Jarek Potiuk <ja...@potiuk.com> wrote:
> 
> Yeah. when we get the final PR I will also want to to test more scenarios -
> with IDE/mypy integration switching branches, uv syncing etc. and will be
> happy to help and document the contributor's doc to explain what and how to
> work with it. This would be a super cool thing if we get it to
> work seamlessly for everyone :)
> 
> On Thu, Jul 10, 2025 at 7:44 AM Amogh Desai <am...@gmail.com>
> wrote:
> 
>> Agreed!
>> 
>> Once the PR is up, we can have these implementation level discussions
>> over there. Good chat however!
>> 
>> Thanks & Regards,
>> Amogh Desai
>> 
>> 
>> On Wed, Jul 9, 2025 at 3:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> 
>>> Yeah. I think extracting one-by-one, feature-by-feature that we want to
>>> share to a separate distribution is the best approach - it will actually
>>> also help with the "__init__.py" cleanup - because almost by definition -
>>> those distributions will not be able to "reach" outside - i.e. they only
>>> can be "used" not "use" something else. Which means that - for example as
>>> it is now - configuration using logging which is using configuration
>>> (leading to circular dependencies and partially initialized modules) will
>>> not happen and we will have to figure out rather how to inject
>>> configuration into logging (from task-sdk, airflow-ctl, airflow-core) and
>>> to get the right sequence of initialization - rather than have the
>>> inter-feature-dependencies.
>>> 
>>> And that is precisely what low-cyclomatic complexity is about as well -
>>> it generally leads to easier-to-maintain software that has well defined
>>> functionality and does not have those weird circular dependencies we have
>>> now. That's a kind-of side-effect of such a "per feature" split, but a very
>>> desirable one.
>>> 
>>> J.
>>> 
>>> On Wed, Jul 9, 2025 at 10:17 AM Amogh Desai <am...@gmail.com>
>>> wrote:
>>> 
>>>> Probably, you make a valid point.
>>>> 
>>>> Maybe this is an implementation detail, so we could figure it out as we
>>>> start on a POC and factor in these things
>>>> as we move along?
>>>> 
>>>> But from an initial guess, I would think that execution time related
>>>> items (if we manage to enumerate them) would be something
>>>> that would be better off in that "core_and_task_sdk" bundle.
>>>> 
>>>> Thanks & Regards,
>>>> Amogh Desai
>>>> 
>>>> 
>>>> On Tue, Jul 8, 2025 at 3:46 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>> 
>>>>>> Not that I am against your idea and we can surely expand as we need
>>>>> but we would not need to expand the
>>>>> "core_and_task_sdk" if we put only the relevant items into it.
>>>>> 
>>>>> So if we move logging and config out, my question is what is really
>>>>> relevant to "stay" in "core_and_task_sdk" ? And what we know will not be
>>>>> needed by other distributions in the future ?
>>>>> 
>>>>> What would be the content of "core_and_task_sdk" ? Can we enumerate
>>>>> what should go there now ?
>>>>> 
>>>>> My bet is that if we enumerate them, it will turn out that they are
>>>>> good candidates to make separate "shared features" that can be logically
>>>>> named and modularised, and that there is no real need to have shared
>>>>> "core_and_task_sdk" which sounds like "bag of everything else".
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, Jul 8, 2025 at 11:48 AM Amogh Desai <am...@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> Yeah, I think what you are showcasing here is a step ahead of the
>>>>>> initial proposal from Ash.
>>>>>> 
>>>>>> From the original proposal, the `core_and_task_sdk` *can* have the
>>>>>> things relevant to just those two
>>>>>> distros. Logging, Config are modules that might be needed by
>>>>>> airflow-ctl for example, so ideally, those
>>>>>> would not be good candidates to be put in there, ideally speaking.
>>>>>> 
>>>>>> The example of Kubernetes Utils sounds to be a good example, it will
>>>>>> be used by KubeExecutor (lets say this is a
>>>>>> module called "executors") and by KPO (providers), and the
>>>>>> "shared/kubernetes" would probably be a good
>>>>>> candidate for that.
>>>>>> 
>>>>>> Not that I am against your idea and we can surely expand as we need
>>>>>> but we would not need to expand the
>>>>>> "core_and_task_sdk" if we put only the relevant items into it.
>>>>>> 
>>>>>> Thanks & Regards,
>>>>>> Amogh Desai
>>>>>> 
>>>>>> 
>>>>>> On Tue, Jul 8, 2025 at 12:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>> 
>>>>>>>> @Jarek Potiuk <ja...@potiuk.com> a little confused on what you
>>>>>>> mean there, I am understanding the direction
>>>>>>> but could you elaborate a bit more please?
>>>>>>> 
>>>>>>> Let me elaborate:
>>>>>>> 
>>>>>>> As I understand (maybe I am wrong?),  the proposal is that we have a
>>>>>>> "core-and-task-sdk" folder which is a shared distribution that is
>>>>>>> vendored-in into both "airflow-core" and "airflow-task-sdk". This
>>>>>>> contains some shared code that we want to include in both distributions
>>>>>>> (note that we never ever release "core-and-task-sdk"
>>>>>>> distribution because it only contains code that is shared between the two
>>>>>>> distributions we release.
>>>>>>> 
>>>>>>> That's fine and cool.
>>>>>>> 
>>>>>>> Imagine that this distribution contains "logging" (shared code
>>>>>>> for logging) and "config" (shared code for configuration). - both needed in
>>>>>>> "airflow-core" and "airflow-task-sdk". So far so good. But what happen if
>>>>>>> we want to use logging in the same fashion in say "airflow-ctl" (that also
>>>>>>> applies for other distributions that we might come up with) ? Are we going
>>>>>>> to vendor in the whole "core-and-task-sdk" distribution in "airflow-ctl" ?
>>>>>>> It would be far better if we just vendor in "logging" and do not vendor-in
>>>>>>> "config".
>>>>>>> 
>>>>>>> And if we are going to have a mechanism to vendor-in "a distribution"
>>>>>>> - there is nothing wrong with having the same way to vendor-in multiple
>>>>>>> distributions - so we can easily do it this way (i added "orm" ,
>>>>>>> "serialization". and "fast_api" as an example thing that we might want to
>>>>>>> share - not sure if that is really something we want to do but it will
>>>>>>> allow to illustrate my idea better)
>>>>>>> 
>>>>>>> /
>>>>>>>  airflow-ctl/
>>>>>>>  task-sdk/...
>>>>>>>  airflow-core/...
>>>>>>>  ....
>>>>>>>  shared/
>>>>>>>    kubernetes/
>>>>>>>      pyproject.toml
>>>>>>>      src/
>>>>>>>        airflow_shared_kubernetes/__init__.py
>>>>>>>    logging/
>>>>>>>      pyproject.toml
>>>>>>>      src/
>>>>>>>        airflow_shared_logging/__init__.py
>>>>>>>    config/
>>>>>>>      pyproject.toml
>>>>>>>      src/
>>>>>>>        airflow_shared_config/__init__.py
>>>>>>>    orm/
>>>>>>>      pyproject.toml
>>>>>>>      src/
>>>>>>>        airflow_shared_orm/__init__.py
>>>>>>>     serialization/
>>>>>>>      pyproject.toml
>>>>>>>      src/
>>>>>>>        airflow_shared_serialization/__init__.py
>>>>>>>     fast_api/
>>>>>>>      pyproject.toml
>>>>>>>      src/
>>>>>>>        airflow_shared_fast_api/__init__.py
>>>>>>>    ...
>>>>>>> 
>>>>>>> This has multiple benefits (and I see no real drawbacks):
>>>>>>> 
>>>>>>> * the code can be really well modularised.  Those "things" we share
>>>>>>> (and this also connects to the __init__.py discussion) - can be independent
>>>>>>> - and (it follow Jens comment) it allows to keep low cyclomatic complexity
>>>>>>> https://en.wikipedia.org/wiki/Cyclomatic_complexity . It will be way
>>>>>>> easier to implement logging in the way that it does not import or use
>>>>>>> config. This means for example that configuration for logging will need to
>>>>>>> be injected when logging is initialized - and that's exactly what we want,
>>>>>>> we do not want logging code to use configuration code directly - they
>>>>>>> should be independent from each other and you should be free to vendor-in
>>>>>>> either logging or config independently if you also vendored-in the other.
>>>>>>> 
>>>>>>> * it's much more logical. We split based on functionality we want to
>>>>>>> share - not about the "distributions" we want to produce. That allows us -
>>>>>>> in the future - to make different decisions on how we split our
>>>>>>> distributions. For example (i do not tell we have to do it, or that we will
>>>>>>> do it but we will have such a possibility) - we can add more shared
>>>>>>> utilities we find useful in the same way and decide that "scheduler",
>>>>>>> "api_server" or "scheduler" or "triggerer" or "dag processor" are split to
>>>>>>> separate distributions - because for example we want to keep a number of
>>>>>>> dependencies down. And for example "api_server" might use "fast_api",
>>>>>>> "config", "logging", "orm" and "fast_api" and "serialization" , where the
>>>>>>> scheduler should not need "fast_api". The "dag_processor" eventually might
>>>>>>> not need "orm" nor "fast_api" and only use the other
>>>>>>> 
>>>>>>> This seems like a natural approach. If we have a mechanism to "share"
>>>>>>> the code, it does not add complexity, but allows us to isolate independent
>>>>>>> functionality into "isolated" boxes and use them
>>>>>>> 
>>>>>>> Also for cyclomatic complexity that is a complex word (badly chosen
>>>>>>> as it scares people away) and has some math behind, but it really boils
>>>>>>> down to very simple "rules of thumb" (and yes I am a big proponent of
>>>>>>> having low cyclomatic complexity).
>>>>>>> 
>>>>>>> a) when you are building the "final" product (i.e. distribution you
>>>>>>> want to release) - make sure that you only "use" things - that nothing else
>>>>>>> is "using" you as a library.
>>>>>>> b) when you are building a "shared" thing (a library) - make sure
>>>>>>> that library is only "used" by others but it does not really "use" anything
>>>>>>> else. For example in the case I explained above - we can achieve
>>>>>>> low-cyclomatic complexity when:
>>>>>>> 
>>>>>>> * airflow-core uses: logging, config, orm, serialization, fast_api
>>>>>>> * none of the "logging, config, orm, serialization, fast_api" use
>>>>>>> each other - they are at the bottom of the "user -> used" tree
>>>>>>> 
>>>>>>> J.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, Jul 8, 2025 at 8:16 AM Amogh Desai <am...@gmail.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> I like the folder structure proposed by Ash and have no objections
>>>>>>>> with it.
>>>>>>>> 
>>>>>>>> "core_and_task_sdk" sounds good to me and justifies what it should
>>>>>>>> do pretty well.
>>>>>>>> 
>>>>>>>> @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
>>>>>>>> there, I am understanding the direction
>>>>>>>> but could you elaborate a bit more please?
>>>>>>>> 
>>>>>>>> Naming is REALLY hard!
>>>>>>>> 
>>>>>>>> Thanks & Regards,
>>>>>>>> Amogh Desai
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Tue, Jul 8, 2025 at 2:52 AM Jarek Potiuk <ja...@potiuk.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> How about splitting it even more and having each shared "thing"
>>>>>>>>> named?
>>>>>>>>> "logging", "config" and sharing them explicitly and separately with
>>>>>>>>> the
>>>>>>>>> right "user" ?
>>>>>>>>> That sounds way more modular and  we will be able to choose which
>>>>>>>>> of the
>>>>>>>>> shared "utils" we use where.
>>>>>>>>> 
>>>>>>>>> J.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Mon, Jul 7, 2025 at 11:13 PM Jens Scheffler
>>>>>>>>> <j_...@gmx.de.invalid>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> I like "core_and_task_sdk the same like core-and-task-sdk - I
>>>>>>>>> have no
>>>>>>>>>> problem and it is a path only.
>>>>>>>>>> 
>>>>>>>>>> if we get to "dag-parser-scheduler-task-sdk-and-triggerer" which
>>>>>>>>> is a
>>>>>>>>>> bit bulky we then should name it "all-not-api-server" :-D
>>>>>>>>>> 
>>>>>>>>>> On 07.07.25 22:57, Ash Berlin-Taylor wrote:
>>>>>>>>>>> In case I did a bad job explaining it, the “core and task sdk”
>>>>>>>>> is not in
>>>>>>>>>> the module name/import name, just in the file path.
>>>>>>>>>>> 
>>>>>>>>>>> Anyone have other ideas?
>>>>>>>>>>> 
>>>>>>>>>>>> On 7 Jul 2025, at 21:37, Buğra Öztürk <oz...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks Ash! Looks cool! I like the structure. This will
>>>>>>>>> enable all the
>>>>>>>>>>>> combinations and structure looks easy to grasp. No strong
>>>>>>>>> stance on the
>>>>>>>>>>>> naming other than maybe it is a bit long with `and`,
>>>>>>>>> `core_ctl` could be
>>>>>>>>>>>> shorter, since no import path is defined like that, we can
>>>>>>>>> give any name
>>>>>>>>>>>> for sure.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Mon, 7 Jul 2025, 21:51 Jarek Potiuk, <ja...@potiuk.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Looks good but I think we should find some better logical
>>>>>>>>> name for
>>>>>>>>>>>>> core_and_sdk :)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
>>>>>>>>>>>>> <j_...@gmx.de.invalid> napisał:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cool! Especially the "shared" folder with the ability to have
>>>>>>>>>>>>>> N-combinations w/o exploding project repo root!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 07.07.25 14:43, Ash Berlin-Taylor wrote:
>>>>>>>>>>>>>>> Oh, and all of this will be explain in shared/README.md
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Okay, so it seems we have agreement on the approach here,
>>>>>>>>> so I’ll
>>>>>>>>>>>>>> continue with this, and on the dev call it was mentioned that
>>>>>>>>>>>>>> “airflow-common” wasn’t a great name, so here is my proposal
>>>>>>>>> for the
>>>>>>>>>> file
>>>>>>>>>>>>>> structure;
>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>> /
>>>>>>>>>>>>>>>>  task-sdk/...
>>>>>>>>>>>>>>>>  airflow-core/...
>>>>>>>>>>>>>>>>  shared/
>>>>>>>>>>>>>>>>    kuberenetes/
>>>>>>>>>>>>>>>>      pyproject.toml
>>>>>>>>>>>>>>>>      src/
>>>>>>>>>>>>>>>>        airflow_kube/__init__.py
>>>>>>>>>>>>>>>>    core-and-tasksdk/
>>>>>>>>>>>>>>>>      pyproject.toml
>>>>>>>>>>>>>>>>      src/
>>>>>>>>>>>>>>>>        airflow_shared/__init__.py
>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Things to note here: the “shared” folder has (the
>>>>>>>>> possibility) of
>>>>>>>>>>>>>> having multiple different shared “libraries” in it, in this
>>>>>>>>> example I
>>>>>>>>>> am
>>>>>>>>>>>>>> supposing a hypothetical shared kuberenetes folder a world
>>>>>>>>> in which we
>>>>>>>>>>>>>> split the KubePodOperator and the KubeExecutor in to two
>>>>>>>>> separate
>>>>>>>>>>>>>> distributions (example only, not proposing we do that right
>>>>>>>>> now, and
>>>>>>>>>> that
>>>>>>>>>>>>>> will be a separate discussion)
>>>>>>>>>>>>>>>> The other things to note here:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> - the folder name in shared aims to be “self-documenting”,
>>>>>>>>> hence the
>>>>>>>>>>>>>> verbose “core-and-tasksdk” to say where the shared library is
>>>>>>>>>> intended to
>>>>>>>>>>>>>> be used.
>>>>>>>>>>>>>>>> - the python module itself should almost always have an
>>>>>>>>> `airflow_`
>>>>>>>>>> (or
>>>>>>>>>>>>>> maybe `_airflow_`?) prefix so that it does not conflict with
>>>>>>>>> anything
>>>>>>>>>>>>> else
>>>>>>>>>>>>>> we might use. It won’t matter “in production” as those will
>>>>>>>>> be
>>>>>>>>>> vendored
>>>>>>>>>>>>> in
>>>>>>>>>>>>>> to be imported as `airflow/_vendor/airflow_shared` etc, but
>>>>>>>>> avoiding
>>>>>>>>>>>>>> conflicts at dev time with the Finder approach is a good
>>>>>>>>> safety
>>>>>>>>>> measure.
>>>>>>>>>>>>>>>> I will start making a real PR for this proposal now, but
>>>>>>>>> I’m open to
>>>>>>>>>>>>>> feedback (either here, or in the PR when I open it)
>>>>>>>>>>>>>>>> -ash
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Yeah we have to try it and test - also building packages
>>>>>>>>> happens
>>>>>>>>>> semi
>>>>>>>>>>>>>>>>> frequently when you run `uv sync` (they use some kind of
>>>>>>>>> heuristics
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> decide when) and you can force it with `--reinstall` or
>>>>>>>>>> `--refresh`.
>>>>>>>>>>>>>>>>> Package build also happens every time when you run
>>>>>>>>> "ci-image build`
>>>>>>>>>>>>>> now in
>>>>>>>>>>>>>>>>> breeze so it seems like it will nicely integrate in our
>>>>>>>>> workflows.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Looks really cool Ash.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <
>>>>>>>>> ash@apache.org>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> It’s not just release time, but any time we build a
>>>>>>>>> package which
>>>>>>>>>>>>>> happens
>>>>>>>>>>>>>>>>>> on “every” CI run. The normal unit tests will use code
>>>>>>>>> from
>>>>>>>>>>>>>>>>>> airflow-common/src/airflow_common; the kube tests which
>>>>>>>>> build an
>>>>>>>>>>>>>> image will
>>>>>>>>>>>>>>>>>> build the dists and vendor in the code from that commit.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> There is only a single copy of the shared code committed
>>>>>>>>> to the
>>>>>>>>>>>>> repo,
>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>> there is never anything to synchronise.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On 4 Jul 2025, at 15:53, Amogh Desai <
>>>>>>>>> amoghdesai.oss@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> Thanks Ash.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> This is really cool and helpful that you were able to
>>>>>>>>> test both
>>>>>>>>>>>>>> scenarios
>>>>>>>>>>>>>>>>>>> -- repo checkout
>>>>>>>>>>>>>>>>>>> and also installing from the vendored package and the
>>>>>>>>> resolution
>>>>>>>>>>>>>> worked
>>>>>>>>>>>>>>>>>>> fine too.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I like this idea compared the to relative import one
>>>>>>>>> for few
>>>>>>>>>>>>> reasons:
>>>>>>>>>>>>>>>>>>> - It feels like it will take some time to adjust to the
>>>>>>>>> new
>>>>>>>>>> coding
>>>>>>>>>>>>>>>>>> standard
>>>>>>>>>>>>>>>>>>> that we will lay
>>>>>>>>>>>>>>>>>>> if we impose relative imports in the shared dist
>>>>>>>>>>>>>>>>>>> - We can continue using repo wise absolute import
>>>>>>>>> standards, it
>>>>>>>>>> is
>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>> much easier for situations
>>>>>>>>>>>>>>>>>>> when we do global search in IDE to find + replace, this
>>>>>>>>> could
>>>>>>>>>> mean
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>> - The vendoring work is a proven and established
>>>>>>>>> paradigm across
>>>>>>>>>>>>>> projects
>>>>>>>>>>>>>>>>>>> and would
>>>>>>>>>>>>>>>>>>> out of box give us the build tooling we need also
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Nothing too against the relative import but with the
>>>>>>>>> evidence
>>>>>>>>>>>>>> provided
>>>>>>>>>>>>>>>>>>> above, vendored approach
>>>>>>>>>>>>>>>>>>> seems to only do us good.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Regarding synchronizing it, release time should be fine
>>>>>>>>> as long
>>>>>>>>>> as
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>> a good CI workflow to probably
>>>>>>>>>>>>>>>>>>> catch such issues per PR if changes are made in shared
>>>>>>>>> dist?
>>>>>>>>>>>>>> (precommit
>>>>>>>>>>>>>>>>>>> would make it really slow i guess)
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> If we can run our tests with vendored code we should be
>>>>>>>>> mostly
>>>>>>>>>>>>>> covered.
>>>>>>>>>>>>>>>>>>> Good effort all!
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>>>>>>>> Amogh Desai
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <
>>>>>>>>>> ash@apache.org>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> Okay, I think I’ve got something that works and I’m
>>>>>>>>> happy with.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>>>>>>>>>>>>>>>>>>>> This produces the following from `uv build task-sdk`
>>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>>>>>>>>>>>>>>>>>>>> (`.whl.zip` as GH won't allow .whl upload, but will
>>>>>>>>> .zip)
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>> ❯ unzip -l
>>>>>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
>>>>>>>>>>>>>>>>>> grep
>>>>>>>>>>>>>>>>>>>> _vendor
>>>>>>>>>>>>>>>>>>>>     50  02-02-2020 00:00
>>>>>>>>> airflow/sdk/_vendor/.gitignore
>>>>>>>>>>>>>>>>>>>>   2082  02-02-2020 00:00
>>>>>>>>> airflow/sdk/_vendor/__init__.py
>>>>>>>>>>>>>>>>>>>>     28  02-02-2020 00:00
>>>>>>>>>> airflow/sdk/_vendor/airflow_common.pyi
>>>>>>>>>>>>>>>>>>>>     18  02-02-2020 00:00
>>>>>>>>> airflow/sdk/_vendor/vendor.txt
>>>>>>>>>>>>>>>>>>>>    785  02-02-2020 00:00
>>>>>>>>>>>>>>>>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>>>>>>>>>>>>>>>>  10628  02-02-2020 00:00
>>>>>>>>>>>>>>>>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> And similarly in the .tar.gz, so our “sdist” is
>>>>>>>>> complete too:
>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>>>>>> |grep
>>>>>>>>>> _vendor
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> The plugin works at build time by including/copying
>>>>>>>>> the libs
>>>>>>>>>>>>>> specified
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> vendor.txt into place (and let `vendoring` take care
>>>>>>>>> of import
>>>>>>>>>>>>>>>>>> rewrites.)
>>>>>>>>>>>>>>>>>>>> For the imports to continue to work at “dev” time/from
>>>>>>>>> a repo
>>>>>>>>>>>>>> checkout,
>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>> have added a import finder to `sys.meta_path`, and
>>>>>>>>> since it’s at
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> end of
>>>>>>>>>>>>>>>>>>>> the list it will only be used if the normal import
>>>>>>>>> can’t find
>>>>>>>>>>>>>> things.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>>>>>>>>>>>>>>>>>>>> This doesn’t quite give us the same runtime effect
>>>>>>>>> “import
>>>>>>>>>>>>>> rewriting”
>>>>>>>>>>>>>>>>>>>> affect, as in this approach `airflow_common` is
>>>>>>>>> directly loaded
>>>>>>>>>>>>>> (i.e.
>>>>>>>>>>>>>>>>>>>> airflow.sdk._vendor.airflow_common and airflow_common
>>>>>>>>> exist in
>>>>>>>>>>>>>>>>>>>> sys.modules), but it does work for everything that I
>>>>>>>>> was able to
>>>>>>>>>>>>>> test..
>>>>>>>>>>>>>>>>>>>> I tested it with the diff at the end of this message.
>>>>>>>>> My test
>>>>>>>>>>>>>> ipython
>>>>>>>>>>>>>>>>>>>> shell:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>> In [1]: from
>>>>>>>>> airflow.sdk._vendor.airflow_common.timezone import
>>>>>>>>>>>>> foo
>>>>>>>>>>>>>>>>>>>> In [2]: foo
>>>>>>>>>>>>>>>>>>>> Out[2]: 1
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> In [3]: import airflow.sdk._vendor.airflow_common
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> In [4]: import
>>>>>>>>> airflow.sdk._vendor.airflow_common.timezone
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>>>>>>>>>>>>>>>>>>>> Out[5]:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>>>>>>>>>>>>>>>>>>>> In [6]:
>>>>>>>>> airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>>>>>>>>>>>>>>>> Out[6]:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> And in an standalone environment with the SDK dist I
>>>>>>>>> built (it
>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> matching airflow-core right now, but that is nothing
>>>>>>>>> to do with
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> discussion):
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>>>>>>>>>>>>>>>>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>>>>>>>>>>>>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl
>>>>>>>>> ipython
>>>>>>>>>>>>>>>>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang
>>>>>>>>> 18.1.8 ]
>>>>>>>>>>>>>>>>>>>> Type 'copyright', 'credits' or 'license' for more
>>>>>>>>> information
>>>>>>>>>>>>>>>>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type
>>>>>>>>> '?' for
>>>>>>>>>>>>> help.
>>>>>>>>>>>>>>>>>>>> Tip: You can use `%hist` to view history, see the
>>>>>>>>> options with
>>>>>>>>>>>>>>>>>> `%history?`
>>>>>>>>>>>>>>>>>>>> In [1]: import
>>>>>>>>> airflow.sdk._vendor.airflow_common.timezone
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> In [2]:
>>>>>>>>> airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>>>>>>>>>>>>>>>> Out[2]:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>>>>>>>>>>>>>>>>>>>> ``
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> ```diff
>>>>>>>>>>>>>>>>>>>> diff --git
>>>>>>>>> a/airflow-common/src/airflow_common/__init__.py
>>>>>>>>>>>>>>>>>>>> b/airflow-common/src/airflow_common/__init__.py
>>>>>>>>>>>>>>>>>>>> index 13a83393a9..927b7c6b61 100644
>>>>>>>>>>>>>>>>>>>> --- a/airflow-common/src/airflow_common/__init__.py
>>>>>>>>>>>>>>>>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
>>>>>>>>>>>>>>>>>>>> @@ -14,3 +14,5 @@
>>>>>>>>>>>>>>>>>>>> # KIND, either express or implied.  See the License
>>>>>>>>> for the
>>>>>>>>>>>>>>>>>>>> # specific language governing permissions and
>>>>>>>>> limitations
>>>>>>>>>>>>>>>>>>>> # under the License.
>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>> +foo = 1
>>>>>>>>>>>>>>>>>>>> diff --git
>>>>>>>>> a/airflow-common/src/airflow_common/timezone.py
>>>>>>>>>>>>>>>>>>>> b/airflow-common/src/airflow_common/timezone.py
>>>>>>>>>>>>>>>>>>>> index 340b924c66..58384ef20f 100644
>>>>>>>>>>>>>>>>>>>> --- a/airflow-common/src/airflow_common/timezone.py
>>>>>>>>>>>>>>>>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
>>>>>>>>>>>>>>>>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>>>>>>>>>>>>>>>>>>>> version.parse(metadata.version("pendulum")).major == 3
>>>>>>>>>>>>>>>>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
>>>>>>>>>>>>>>>>>>>> utc = pendulum.UTC
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>> +from airflow_common import foo
>>>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>>>> TIMEZONE: Timezone
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <
>>>>>>>>> jarek@potiuk.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> I think both approaches are doable:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 1) -> We can very easily prevent bad imports by
>>>>>>>>> pre-commit when
>>>>>>>>>>>>>>>>>> importing
>>>>>>>>>>>>>>>>>>>>> from different distributions and make sure we are
>>>>>>>>> only doing
>>>>>>>>>>>>>> relative
>>>>>>>>>>>>>>>>>>>>> imports in the shared modules. We are doing plenty of
>>>>>>>>> this
>>>>>>>>>>>>>> already. And
>>>>>>>>>>>>>>>>>>>> yes
>>>>>>>>>>>>>>>>>>>>> it would require relative links we currently do not
>>>>>>>>> allow.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 2) -> has one disadvantage that someone at some point
>>>>>>>>> in time
>>>>>>>>>>>>> will
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> decide to synchronize this and if it happens just
>>>>>>>>> before
>>>>>>>>>> release
>>>>>>>>>>>>>> (I bet
>>>>>>>>>>>>>>>>>>>>> this is going to happen) this will lead to solving
>>>>>>>>> problems
>>>>>>>>>> that
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>> normally be solved during PR when you make a change
>>>>>>>>> (i.e.
>>>>>>>>>>>>> symbolic
>>>>>>>>>>>>>> link
>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>> the advantage that whoever modifies shared code will
>>>>>>>>> be
>>>>>>>>>>>>> immediately
>>>>>>>>>>>>>>>>>>>>> notified in their PR - that they broke something
>>>>>>>>> because either
>>>>>>>>>>>>>> static
>>>>>>>>>>>>>>>>>>>>> checks or mypy or tests fail.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Ash, do you have an idea of a process (who and when)
>>>>>>>>> does the
>>>>>>>>>>>>>>>>>>>>> synchronisation in case of vendoring? Maybe we could
>>>>>>>>> solve it
>>>>>>>>>> if
>>>>>>>>>>>>>> it is
>>>>>>>>>>>>>>>>>>>> done
>>>>>>>>>>>>>>>>>>>>> more frequently and with some regularity? We could
>>>>>>>>> potentially
>>>>>>>>>>>>>> force
>>>>>>>>>>>>>>>>>>>>> re-vendoring at PR time as well any time shared code
>>>>>>>>> changes
>>>>>>>>>> (and
>>>>>>>>>>>>>>>>>> prevent
>>>>>>>>>>>>>>>>>>>>> it by pre-commit. And I can't think of some place
>>>>>>>>> (other than
>>>>>>>>>>>>>> releases)
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> our development workflow and that seems to be a bit
>>>>>>>>> too late as
>>>>>>>>>>>>>> puts an
>>>>>>>>>>>>>>>>>>>>> extra effort on fixing potential incompatibilities
>>>>>>>>> introduced
>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>>>> manager and delays the release. WDYT?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Re: relative links. I think for a shared library we
>>>>>>>>> could
>>>>>>>>>>>>>> potentially
>>>>>>>>>>>>>>>>>>>> relax
>>>>>>>>>>>>>>>>>>>>> this and allow them (and actually disallow absolute
>>>>>>>>> links in
>>>>>>>>>> the
>>>>>>>>>>>>>> pieces
>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>> code that are shared - again, by pre-commit). As I
>>>>>>>>> recall, the
>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>> reason
>>>>>>>>>>>>>>>>>>>>> we forbade the relative links is because of how we
>>>>>>>>> are (or
>>>>>>>>>> maybe
>>>>>>>>>>>>>> were)
>>>>>>>>>>>>>>>>>>>>> doing DAG parsing and failures resulting from it. So
>>>>>>>>> we decided
>>>>>>>>>>>>> to
>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> allow it to keep consistency. The way how Dag parsing
>>>>>>>>> works is
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>>>> you are using importlib to read the Dag from a file,
>>>>>>>>> the
>>>>>>>>>> relative
>>>>>>>>>>>>>>>>>> imports
>>>>>>>>>>>>>>>>>>>>> do not work as it does not know what they should be
>>>>>>>>> relative
>>>>>>>>>> to.
>>>>>>>>>>>>>> But if
>>>>>>>>>>>>>>>>>>>>> relative import is done from an imported package, it
>>>>>>>>> should be
>>>>>>>>>> no
>>>>>>>>>>>>>>>>>>>> problem,
>>>>>>>>>>>>>>>>>>>>> I think - otherwise our Dags would not be able to
>>>>>>>>> import any
>>>>>>>>>>>>>> library
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> uses relative imports.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Of course consistency might be the reason why we do
>>>>>>>>> not want to
>>>>>>>>>>>>>>>>>> introduce
>>>>>>>>>>>>>>>>>>>>> relative imports. I don't see it as an issue if it is
>>>>>>>>> guarded
>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>> pre-commit
>>>>>>>>>>>>>>>>>>>>> though.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
>>>>>>>>>>>>>> ash@apache.org>
>>>>>>>>>>>>>>>>>>>>> napisał:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Oh yes, symlinks will work, with one big caveat: It
>>>>>>>>> does mean
>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>> can’t
>>>>>>>>>>>>>>>>>>>>>> use absolute imports in one common module to another.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> For example
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>>>>>>>>>>>>>>>>>>>>>> where we have
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>>>> from airflow.utils.module_loading import
>>>>>>>>> import_string
>>>>>>>>>>>>>>>>>>>>>> ```
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> if we want to move serve_logs into this common lib
>>>>>>>>> that is
>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>>> symlinked
>>>>>>>>>>>>>>>>>>>>>> then we wouldn’t be able to have `from
>>>>>>>>>>>>>> airflow_common.module_loading
>>>>>>>>>>>>>>>>>>>> import
>>>>>>>>>>>>>>>>>>>>>> import_string`.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I can think of two possible solutions here.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 1) is to allow/require relative imports in this
>>>>>>>>> shared lib, so
>>>>>>>>>>>>>> `from
>>>>>>>>>>>>>>>>>>>>>> .module_loading import import_string`
>>>>>>>>>>>>>>>>>>>>>> 2) is to use `vendoring`[1] (from the pip
>>>>>>>>> maintainers) which
>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>> handle
>>>>>>>>>>>>>>>>>>>>>> import-rewriting for us.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I’d entirely forgot that symlinks in repos was a
>>>>>>>>> thing, so I
>>>>>>>>>>>>>> prepared
>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>> minimal POC/demo of what vendoring approach could
>>>>>>>>> look like
>>>>>>>>>> here
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>>>>>>>>>>>>>>>>>>>>>> Now personally I am more than happy with relative
>>>>>>>>> imports, but
>>>>>>>>>>>>>>>>>> generally
>>>>>>>>>>>>>>>>>>>>>> as a project we have avoided them, so I think that
>>>>>>>>> limits what
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>>>>> with a symlink based approach.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> -ash
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> [1] https://github.com/pradyunsg/vendoring
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
>>>>>>>>>>>>>>>>>> gopidesupavan@gmail.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> Thanks Ash
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Yes agree option 2 would be preferred for me.
>>>>>>>>> Making sure we
>>>>>>>>>>>>>> have all
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> gaurdriles to protect any unwanted behaviour in
>>>>>>>>> code sharing
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> executing
>>>>>>>>>>>>>>>>>>>>>>> right of tests between the packages.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Agree with others, option 2 would be
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
>>>>>>>>>>>>>>>>>> amoghdesai.oss@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for starting this discussion, Ash.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I would prefer option 2 here with proper tooling
>>>>>>>>> to handle
>>>>>>>>>> the
>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>> duplication at *release* time.
>>>>>>>>>>>>>>>>>>>>>>>> It is best to have a dist that has all it needs in
>>>>>>>>> itself.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Option 1 could very quickly get out of hand and if
>>>>>>>>> we decide
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> separate
>>>>>>>>>>>>>>>>>>>>>>>> triggerer /
>>>>>>>>>>>>>>>>>>>>>>>> dag processor / config etc etc as separate
>>>>>>>>> packages, back
>>>>>>>>>>>>>> compat is
>>>>>>>>>>>>>>>>>>>>>> going
>>>>>>>>>>>>>>>>>>>>>>>> to be a nightmare
>>>>>>>>>>>>>>>>>>>>>>>> and will bite us harder than we anticipate.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>>>>>>>>>>>>> Amogh Desai
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <
>>>>>>>>>>>>> kaxilnaik@gmail.com>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> I prefer Option 2 as well to avoid matrix of
>>>>>>>>> dependencies
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>>>>>>>>>>>>>>>>>>>> <j_scheffler@gmx.de.invalid
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> I'd also rather prefer option 2 - reason here is
>>>>>>>>> it is
>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>>>>>> pragmatic
>>>>>>>>>>>>>>>>>>>>>>>>>> and we no not need to cut another package and
>>>>>>>>> have less
>>>>>>>>>>>>>> package
>>>>>>>>>>>>>>>>>>>> counts
>>>>>>>>>>>>>>>>>>>>>>>>>> and dependencies.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> I remember some time ago I was checking
>>>>>>>>> (together with
>>>>>>>>>>>>> Jarek,
>>>>>>>>>>>>>> I am
>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>> sure anymore...) if the usage of symlinks would
>>>>>>>>> be
>>>>>>>>>> possible.
>>>>>>>>>>>>>> To
>>>>>>>>>>>>>>>>>> keep
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> source in one package but "symlink" it into
>>>>>>>>> another. If
>>>>>>>>>> then
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>> packaging/release the files are materialized we
>>>>>>>>> have 1 set
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> code.
>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise if not possible still the redundancy
>>>>>>>>> could be
>>>>>>>>>>>>>> solved by
>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>> pre-commit hook - and in Git the files are
>>>>>>>>> de-duplicated
>>>>>>>>>>>>>> anyway
>>>>>>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>> content hash, so this does not hurt.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> I support option 2 with proper automation & CI
>>>>>>>>> - the
>>>>>>>>>>>>>> reasonings
>>>>>>>>>>>>>>>>>>>>>>>> you've
>>>>>>>>>>>>>>>>>>>>>>>>>>> shown for that make sense to me.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Shahar
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash
>>>>>>>>> Berlin-Taylor <
>>>>>>>>>>>>>> ash@apache.org
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> As we work on finishing off the code-level
>>>>>>>>> separation of
>>>>>>>>>>>>>> Task
>>>>>>>>>>>>>>>>>> SDK
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> Core
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (scheduler etc) we have come across some
>>>>>>>>> situations
>>>>>>>>>> where
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> share code between these.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> However it’s not as straight forward of “just
>>>>>>>>> put it in
>>>>>>>>>> a
>>>>>>>>>>>>>> common
>>>>>>>>>>>>>>>>>>>>>>>> dist
>>>>>>>>>>>>>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>>>>>>>>>>>>>> both depend upon” because one of the goals of
>>>>>>>>> the Task
>>>>>>>>>> SDK
>>>>>>>>>>>>>>>>>>>>>>>> separation
>>>>>>>>>>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to have 100% complete version independence
>>>>>>>>> between the
>>>>>>>>>>>>> two,
>>>>>>>>>>>>>>>>>>>> ideally
>>>>>>>>>>>>>>>>>>>>>>>>>> even if
>>>>>>>>>>>>>>>>>>>>>>>>>>>> they are built into the same image and venv.
>>>>>>>>> Most of the
>>>>>>>>>>>>>> reason
>>>>>>>>>>>>>>>>>>>> why
>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>> isn’t straight forward comes down to backwards
>>>>>>>>>>>>>> compatibility -
>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>>>>>>>>>>>>> an change to the common/shared distribution
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> We’ve listed the options we have thought about
>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545
>>>>>>>>> (but
>>>>>>>>>> that
>>>>>>>>>>>>>> covers
>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>>>>>>> things that I don’t want to get in to in this
>>>>>>>>> discussion
>>>>>>>>>>>>>> such as
>>>>>>>>>>>>>>>>>>>>>>>>>> possibly
>>>>>>>>>>>>>>>>>>>>>>>>>>>> separating operators and executors out of a
>>>>>>>>> single
>>>>>>>>>>>>> provider
>>>>>>>>>>>>>>>>>> dist.)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> To give a concrete example of some code I
>>>>>>>>> would like to
>>>>>>>>>>>>>> share
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>>>>>>>>>>>>>>>>>>>>>>>>>> — logging config. Another thing we will want
>>>>>>>>> to share
>>>>>>>>>> will
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> AirflowConfigParser class from
>>>>>>>>> airflow.configuration
>>>>>>>>>> (but
>>>>>>>>>>>>>>>>>> notably:
>>>>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> parser class, _not_ the default config values,
>>>>>>>>> again,
>>>>>>>>>> lets
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> dwell
>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> specifics of that)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> So to bring the options listed in the issue
>>>>>>>>> here for
>>>>>>>>>>>>>> discussion,
>>>>>>>>>>>>>>>>>>>>>>>>> broadly
>>>>>>>>>>>>>>>>>>>>>>>>>>>> speaking there are two high-level approaches:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. A single shared distribution
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The advantage of Approach 1 is that we only
>>>>>>>>> have the
>>>>>>>>>> code
>>>>>>>>>>>>>> in one
>>>>>>>>>>>>>>>>>>>>>>>>> place.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> However for me, at least in this specific case
>>>>>>>>> of
>>>>>>>>>> Logging
>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>>> AirflowConfigParser class is that backwards
>>>>>>>>>> compatibility
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> much
>>>>>>>>>>>>>>>>>>>>>>>> much
>>>>>>>>>>>>>>>>>>>>>>>>>>>> harder.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main advantage of Approach 2 is the the
>>>>>>>>> code is
>>>>>>>>>>>>> released
>>>>>>>>>>>>>>>>>>>>>>>>>> with/embedded
>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk
>>>>>>>>> would contain
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> right
>>>>>>>>>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the logging config and ConfigParser etc).
>>>>>>>>> The
>>>>>>>>>> downside
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> either
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the code will need to be duplicated in the
>>>>>>>>> repo, or
>>>>>>>>>> better
>>>>>>>>>>>>>> yet
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>>>>>>>> live in a single place in the repo, but some
>>>>>>>>> tooling
>>>>>>>>>> (TBD)
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>> automatically handle the duplication, either
>>>>>>>>> at commit
>>>>>>>>>>>>>> time, or
>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>>>>>>>>> preference, at release time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> For this kind of shared “utility” code I am
>>>>>>>>> very
>>>>>>>>>> strongly
>>>>>>>>>>>>>>>>>> leaning
>>>>>>>>>>>>>>>>>>>>>>>>>> towards
>>>>>>>>>>>>>>>>>>>>>>>>>>>> option 2 with automation, as otherwise I think
>>>>>>>>> the
>>>>>>>>>>>>> backwards
>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirements would make it unworkable (very
>>>>>>>>> quickly over
>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> combinations we would have to test would just
>>>>>>>>> be
>>>>>>>>>>>>>> unreasonable)
>>>>>>>>>>>>>>>>>>>> and I
>>>>>>>>>>>>>>>>>>>>>>>>>> don’t
>>>>>>>>>>>>>>>>>>>>>>>>>>>> feel confident we can have things as stable as
>>>>>>>>> we need
>>>>>>>>>> to
>>>>>>>>>>>>>> really
>>>>>>>>>>>>>>>>>>>>>>>>> deliver
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the version separation/independency I want to
>>>>>>>>> delivery
>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> AIP-72.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> So unless someone feels very strongly about
>>>>>>>>> this, I will
>>>>>>>>>>>>>> come up
>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> draft PR for further discussion that will
>>>>>>>>> implement code
>>>>>>>>>>>>>> sharing
>>>>>>>>>>>>>>>>>>>> via
>>>>>>>>>>>>>>>>>>>>>>>>>>>> “vendoring” it at build time. I have an idea
>>>>>>>>> of how I
>>>>>>>>>> can
>>>>>>>>>>>>>>>>>> achieve
>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>>>>>>>>> we have a single version in the repo and it’ll
>>>>>>>>> work
>>>>>>>>>> there,
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> runtime
>>>>>>>>>>>>>>>>>>>>>>>>>>>> we vendor it in to the shipped dist so it
>>>>>>>>> lives at
>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> In terms of repo layout, this likely means we
>>>>>>>>> would end
>>>>>>>>>> up
>>>>>>>>>>>>>> with:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> airflow-core/pyproject.toml
>>>>>>>>>>>>>>>>>>>>>>>>>>>> airflow-core/src/
>>>>>>>>>>>>>>>>>>>>>>>>>>>> airflow-core/tests/
>>>>>>>>>>>>>>>>>>>>>>>>>>>> task-sdk/pyproject.toml
>>>>>>>>>>>>>>>>>>>>>>>>>>>> task-sdk/src/
>>>>>>>>>>>>>>>>>>>>>>>>>>>> task-sdk/tests/
>>>>>>>>>>>>>>>>>>>>>>>>>>>> airflow-common/src
>>>>>>>>>>>>>>>>>>>>>>>>>>>> airflow-common/tests/
>>>>>>>>>>>>>>>>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml,
>>>>>>>>> as deps
>>>>>>>>>> would
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>> included
>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the downstream projects. TBD.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thoughts and feedback welcomed.
>>>>>>>>>>>>>> 
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>> dev-unsubscribe@airflow.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>> dev-help@airflow.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>> dev-unsubscribe@airflow.apache.org
>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>> dev-help@airflow.apache.org
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>> dev-help@airflow.apache.org
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>> dev-help@airflow.apache.org
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 


Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Jarek Potiuk <ja...@potiuk.com>.
Yeah. when we get the final PR I will also want to to test more scenarios -
with IDE/mypy integration switching branches, uv syncing etc. and will be
happy to help and document the contributor's doc to explain what and how to
work with it. This would be a super cool thing if we get it to
work seamlessly for everyone :)

On Thu, Jul 10, 2025 at 7:44 AM Amogh Desai <am...@gmail.com>
wrote:

> Agreed!
>
> Once the PR is up, we can have these implementation level discussions
> over there. Good chat however!
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Wed, Jul 9, 2025 at 3:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Yeah. I think extracting one-by-one, feature-by-feature that we want to
>> share to a separate distribution is the best approach - it will actually
>> also help with the "__init__.py" cleanup - because almost by definition -
>> those distributions will not be able to "reach" outside - i.e. they only
>> can be "used" not "use" something else. Which means that - for example as
>> it is now - configuration using logging which is using configuration
>> (leading to circular dependencies and partially initialized modules) will
>> not happen and we will have to figure out rather how to inject
>> configuration into logging (from task-sdk, airflow-ctl, airflow-core) and
>> to get the right sequence of initialization - rather than have the
>> inter-feature-dependencies.
>>
>> And that is precisely what low-cyclomatic complexity is about as well -
>> it generally leads to easier-to-maintain software that has well defined
>> functionality and does not have those weird circular dependencies we have
>> now. That's a kind-of side-effect of such a "per feature" split, but a very
>> desirable one.
>>
>> J.
>>
>> On Wed, Jul 9, 2025 at 10:17 AM Amogh Desai <am...@gmail.com>
>> wrote:
>>
>>> Probably, you make a valid point.
>>>
>>> Maybe this is an implementation detail, so we could figure it out as we
>>> start on a POC and factor in these things
>>> as we move along?
>>>
>>> But from an initial guess, I would think that execution time related
>>> items (if we manage to enumerate them) would be something
>>> that would be better off in that "core_and_task_sdk" bundle.
>>>
>>> Thanks & Regards,
>>> Amogh Desai
>>>
>>>
>>> On Tue, Jul 8, 2025 at 3:46 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>> > Not that I am against your idea and we can surely expand as we need
>>>> but we would not need to expand the
>>>> "core_and_task_sdk" if we put only the relevant items into it.
>>>>
>>>> So if we move logging and config out, my question is what is really
>>>> relevant to "stay" in "core_and_task_sdk" ? And what we know will not be
>>>> needed by other distributions in the future ?
>>>>
>>>> What would be the content of "core_and_task_sdk" ? Can we enumerate
>>>> what should go there now ?
>>>>
>>>> My bet is that if we enumerate them, it will turn out that they are
>>>> good candidates to make separate "shared features" that can be logically
>>>> named and modularised, and that there is no real need to have shared
>>>> "core_and_task_sdk" which sounds like "bag of everything else".
>>>>
>>>>
>>>>
>>>> On Tue, Jul 8, 2025 at 11:48 AM Amogh Desai <am...@gmail.com>
>>>> wrote:
>>>>
>>>>> Yeah, I think what you are showcasing here is a step ahead of the
>>>>> initial proposal from Ash.
>>>>>
>>>>> From the original proposal, the `core_and_task_sdk` *can* have the
>>>>> things relevant to just those two
>>>>> distros. Logging, Config are modules that might be needed by
>>>>> airflow-ctl for example, so ideally, those
>>>>> would not be good candidates to be put in there, ideally speaking.
>>>>>
>>>>> The example of Kubernetes Utils sounds to be a good example, it will
>>>>> be used by KubeExecutor (lets say this is a
>>>>> module called "executors") and by KPO (providers), and the
>>>>> "shared/kubernetes" would probably be a good
>>>>> candidate for that.
>>>>>
>>>>> Not that I am against your idea and we can surely expand as we need
>>>>> but we would not need to expand the
>>>>> "core_and_task_sdk" if we put only the relevant items into it.
>>>>>
>>>>> Thanks & Regards,
>>>>> Amogh Desai
>>>>>
>>>>>
>>>>> On Tue, Jul 8, 2025 at 12:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>
>>>>>> > @Jarek Potiuk <ja...@potiuk.com> a little confused on what you
>>>>>> mean there, I am understanding the direction
>>>>>> but could you elaborate a bit more please?
>>>>>>
>>>>>> Let me elaborate:
>>>>>>
>>>>>> As I understand (maybe I am wrong?),  the proposal is that we have a
>>>>>> "core-and-task-sdk" folder which is a shared distribution that is
>>>>>> vendored-in into both "airflow-core" and "airflow-task-sdk". This
>>>>>> contains some shared code that we want to include in both distributions
>>>>>> (note that we never ever release "core-and-task-sdk"
>>>>>> distribution because it only contains code that is shared between the two
>>>>>> distributions we release.
>>>>>>
>>>>>> That's fine and cool.
>>>>>>
>>>>>> Imagine that this distribution contains "logging" (shared code
>>>>>> for logging) and "config" (shared code for configuration). - both needed in
>>>>>> "airflow-core" and "airflow-task-sdk". So far so good. But what happen if
>>>>>> we want to use logging in the same fashion in say "airflow-ctl" (that also
>>>>>> applies for other distributions that we might come up with) ? Are we going
>>>>>> to vendor in the whole "core-and-task-sdk" distribution in "airflow-ctl" ?
>>>>>> It would be far better if we just vendor in "logging" and do not vendor-in
>>>>>> "config".
>>>>>>
>>>>>> And if we are going to have a mechanism to vendor-in "a distribution"
>>>>>> - there is nothing wrong with having the same way to vendor-in multiple
>>>>>> distributions - so we can easily do it this way (i added "orm" ,
>>>>>> "serialization". and "fast_api" as an example thing that we might want to
>>>>>> share - not sure if that is really something we want to do but it will
>>>>>> allow to illustrate my idea better)
>>>>>>
>>>>>> /
>>>>>>   airflow-ctl/
>>>>>>   task-sdk/...
>>>>>>   airflow-core/...
>>>>>>   ....
>>>>>>   shared/
>>>>>>     kubernetes/
>>>>>>       pyproject.toml
>>>>>>       src/
>>>>>>         airflow_shared_kubernetes/__init__.py
>>>>>>     logging/
>>>>>>       pyproject.toml
>>>>>>       src/
>>>>>>         airflow_shared_logging/__init__.py
>>>>>>     config/
>>>>>>       pyproject.toml
>>>>>>       src/
>>>>>>         airflow_shared_config/__init__.py
>>>>>>     orm/
>>>>>>       pyproject.toml
>>>>>>       src/
>>>>>>         airflow_shared_orm/__init__.py
>>>>>>      serialization/
>>>>>>       pyproject.toml
>>>>>>       src/
>>>>>>         airflow_shared_serialization/__init__.py
>>>>>>      fast_api/
>>>>>>       pyproject.toml
>>>>>>       src/
>>>>>>         airflow_shared_fast_api/__init__.py
>>>>>>     ...
>>>>>>
>>>>>> This has multiple benefits (and I see no real drawbacks):
>>>>>>
>>>>>> * the code can be really well modularised.  Those "things" we share
>>>>>> (and this also connects to the __init__.py discussion) - can be independent
>>>>>> - and (it follow Jens comment) it allows to keep low cyclomatic complexity
>>>>>> https://en.wikipedia.org/wiki/Cyclomatic_complexity . It will be way
>>>>>> easier to implement logging in the way that it does not import or use
>>>>>> config. This means for example that configuration for logging will need to
>>>>>> be injected when logging is initialized - and that's exactly what we want,
>>>>>> we do not want logging code to use configuration code directly - they
>>>>>> should be independent from each other and you should be free to vendor-in
>>>>>> either logging or config independently if you also vendored-in the other.
>>>>>>
>>>>>> * it's much more logical. We split based on functionality we want to
>>>>>> share - not about the "distributions" we want to produce. That allows us -
>>>>>> in the future - to make different decisions on how we split our
>>>>>> distributions. For example (i do not tell we have to do it, or that we will
>>>>>> do it but we will have such a possibility) - we can add more shared
>>>>>> utilities we find useful in the same way and decide that "scheduler",
>>>>>> "api_server" or "scheduler" or "triggerer" or "dag processor" are split to
>>>>>> separate distributions - because for example we want to keep a number of
>>>>>> dependencies down. And for example "api_server" might use "fast_api",
>>>>>> "config", "logging", "orm" and "fast_api" and "serialization" , where the
>>>>>> scheduler should not need "fast_api". The "dag_processor" eventually might
>>>>>> not need "orm" nor "fast_api" and only use the other
>>>>>>
>>>>>> This seems like a natural approach. If we have a mechanism to "share"
>>>>>> the code, it does not add complexity, but allows us to isolate independent
>>>>>> functionality into "isolated" boxes and use them
>>>>>>
>>>>>> Also for cyclomatic complexity that is a complex word (badly chosen
>>>>>> as it scares people away) and has some math behind, but it really boils
>>>>>> down to very simple "rules of thumb" (and yes I am a big proponent of
>>>>>> having low cyclomatic complexity).
>>>>>>
>>>>>> a) when you are building the "final" product (i.e. distribution you
>>>>>> want to release) - make sure that you only "use" things - that nothing else
>>>>>> is "using" you as a library.
>>>>>> b) when you are building a "shared" thing (a library) - make sure
>>>>>> that library is only "used" by others but it does not really "use" anything
>>>>>> else. For example in the case I explained above - we can achieve
>>>>>> low-cyclomatic complexity when:
>>>>>>
>>>>>> * airflow-core uses: logging, config, orm, serialization, fast_api
>>>>>> * none of the "logging, config, orm, serialization, fast_api" use
>>>>>> each other - they are at the bottom of the "user -> used" tree
>>>>>>
>>>>>> J.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 8, 2025 at 8:16 AM Amogh Desai <am...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I like the folder structure proposed by Ash and have no objections
>>>>>>> with it.
>>>>>>>
>>>>>>> "core_and_task_sdk" sounds good to me and justifies what it should
>>>>>>> do pretty well.
>>>>>>>
>>>>>>> @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
>>>>>>> there, I am understanding the direction
>>>>>>> but could you elaborate a bit more please?
>>>>>>>
>>>>>>> Naming is REALLY hard!
>>>>>>>
>>>>>>> Thanks & Regards,
>>>>>>> Amogh Desai
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jul 8, 2025 at 2:52 AM Jarek Potiuk <ja...@potiuk.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> How about splitting it even more and having each shared "thing"
>>>>>>>> named?
>>>>>>>> "logging", "config" and sharing them explicitly and separately with
>>>>>>>> the
>>>>>>>> right "user" ?
>>>>>>>> That sounds way more modular and  we will be able to choose which
>>>>>>>> of the
>>>>>>>> shared "utils" we use where.
>>>>>>>>
>>>>>>>> J.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jul 7, 2025 at 11:13 PM Jens Scheffler
>>>>>>>> <j_...@gmx.de.invalid>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> > I like "core_and_task_sdk the same like core-and-task-sdk - I
>>>>>>>> have no
>>>>>>>> > problem and it is a path only.
>>>>>>>> >
>>>>>>>> > if we get to "dag-parser-scheduler-task-sdk-and-triggerer" which
>>>>>>>> is a
>>>>>>>> > bit bulky we then should name it "all-not-api-server" :-D
>>>>>>>> >
>>>>>>>> > On 07.07.25 22:57, Ash Berlin-Taylor wrote:
>>>>>>>> > > In case I did a bad job explaining it, the “core and task sdk”
>>>>>>>> is not in
>>>>>>>> > the module name/import name, just in the file path.
>>>>>>>> > >
>>>>>>>> > > Anyone have other ideas?
>>>>>>>> > >
>>>>>>>> > >> On 7 Jul 2025, at 21:37, Buğra Öztürk <oz...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> > >>
>>>>>>>> > >> Thanks Ash! Looks cool! I like the structure. This will
>>>>>>>> enable all the
>>>>>>>> > >> combinations and structure looks easy to grasp. No strong
>>>>>>>> stance on the
>>>>>>>> > >> naming other than maybe it is a bit long with `and`,
>>>>>>>> `core_ctl` could be
>>>>>>>> > >> shorter, since no import path is defined like that, we can
>>>>>>>> give any name
>>>>>>>> > >> for sure.
>>>>>>>> > >>
>>>>>>>> > >> Best regards,
>>>>>>>> > >>
>>>>>>>> > >>> On Mon, 7 Jul 2025, 21:51 Jarek Potiuk, <ja...@potiuk.com>
>>>>>>>> wrote:
>>>>>>>> > >>>
>>>>>>>> > >>> Looks good but I think we should find some better logical
>>>>>>>> name for
>>>>>>>> > >>> core_and_sdk :)
>>>>>>>> > >>>
>>>>>>>> > >>> pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
>>>>>>>> > >>> <j_...@gmx.de.invalid> napisał:
>>>>>>>> > >>>
>>>>>>>> > >>>> Cool! Especially the "shared" folder with the ability to have
>>>>>>>> > >>>> N-combinations w/o exploding project repo root!
>>>>>>>> > >>>>
>>>>>>>> > >>>> On 07.07.25 14:43, Ash Berlin-Taylor wrote:
>>>>>>>> > >>>>> Oh, and all of this will be explain in shared/README.md
>>>>>>>> > >>>>>
>>>>>>>> > >>>>>> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org>
>>>>>>>> wrote:
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>> Okay, so it seems we have agreement on the approach here,
>>>>>>>> so I’ll
>>>>>>>> > >>>> continue with this, and on the dev call it was mentioned that
>>>>>>>> > >>>> “airflow-common” wasn’t a great name, so here is my proposal
>>>>>>>> for the
>>>>>>>> > file
>>>>>>>> > >>>> structure;
>>>>>>>> > >>>>>> ```
>>>>>>>> > >>>>>> /
>>>>>>>> > >>>>>>   task-sdk/...
>>>>>>>> > >>>>>>   airflow-core/...
>>>>>>>> > >>>>>>   shared/
>>>>>>>> > >>>>>>     kuberenetes/
>>>>>>>> > >>>>>>       pyproject.toml
>>>>>>>> > >>>>>>       src/
>>>>>>>> > >>>>>>         airflow_kube/__init__.py
>>>>>>>> > >>>>>>     core-and-tasksdk/
>>>>>>>> > >>>>>>       pyproject.toml
>>>>>>>> > >>>>>>       src/
>>>>>>>> > >>>>>>         airflow_shared/__init__.py
>>>>>>>> > >>>>>> ```
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>> Things to note here: the “shared” folder has (the
>>>>>>>> possibility) of
>>>>>>>> > >>>> having multiple different shared “libraries” in it, in this
>>>>>>>> example I
>>>>>>>> > am
>>>>>>>> > >>>> supposing a hypothetical shared kuberenetes folder a world
>>>>>>>> in which we
>>>>>>>> > >>>> split the KubePodOperator and the KubeExecutor in to two
>>>>>>>> separate
>>>>>>>> > >>>> distributions (example only, not proposing we do that right
>>>>>>>> now, and
>>>>>>>> > that
>>>>>>>> > >>>> will be a separate discussion)
>>>>>>>> > >>>>>> The other things to note here:
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>> - the folder name in shared aims to be “self-documenting”,
>>>>>>>> hence the
>>>>>>>> > >>>> verbose “core-and-tasksdk” to say where the shared library is
>>>>>>>> > intended to
>>>>>>>> > >>>> be used.
>>>>>>>> > >>>>>> - the python module itself should almost always have an
>>>>>>>> `airflow_`
>>>>>>>> > (or
>>>>>>>> > >>>> maybe `_airflow_`?) prefix so that it does not conflict with
>>>>>>>> anything
>>>>>>>> > >>> else
>>>>>>>> > >>>> we might use. It won’t matter “in production” as those will
>>>>>>>> be
>>>>>>>> > vendored
>>>>>>>> > >>> in
>>>>>>>> > >>>> to be imported as `airflow/_vendor/airflow_shared` etc, but
>>>>>>>> avoiding
>>>>>>>> > >>>> conflicts at dev time with the Finder approach is a good
>>>>>>>> safety
>>>>>>>> > measure.
>>>>>>>> > >>>>>> I will start making a real PR for this proposal now, but
>>>>>>>> I’m open to
>>>>>>>> > >>>> feedback (either here, or in the PR when I open it)
>>>>>>>> > >>>>>> -ash
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com>
>>>>>>>> wrote:
>>>>>>>> > >>>>>>>
>>>>>>>> > >>>>>>> Yeah we have to try it and test - also building packages
>>>>>>>> happens
>>>>>>>> > semi
>>>>>>>> > >>>>>>> frequently when you run `uv sync` (they use some kind of
>>>>>>>> heuristics
>>>>>>>> > >>> to
>>>>>>>> > >>>>>>> decide when) and you can force it with `--reinstall` or
>>>>>>>> > `--refresh`.
>>>>>>>> > >>>>>>> Package build also happens every time when you run
>>>>>>>> "ci-image build`
>>>>>>>> > >>>> now in
>>>>>>>> > >>>>>>> breeze so it seems like it will nicely integrate in our
>>>>>>>> workflows.
>>>>>>>> > >>>>>>>
>>>>>>>> > >>>>>>> Looks really cool Ash.
>>>>>>>> > >>>>>>>
>>>>>>>> > >>>>>>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <
>>>>>>>> ash@apache.org>
>>>>>>>> > >>>> wrote:
>>>>>>>> > >>>>>>>> It’s not just release time, but any time we build a
>>>>>>>> package which
>>>>>>>> > >>>> happens
>>>>>>>> > >>>>>>>> on “every” CI run. The normal unit tests will use code
>>>>>>>> from
>>>>>>>> > >>>>>>>> airflow-common/src/airflow_common; the kube tests which
>>>>>>>> build an
>>>>>>>> > >>>> image will
>>>>>>>> > >>>>>>>> build the dists and vendor in the code from that commit.
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>> There is only a single copy of the shared code committed
>>>>>>>> to the
>>>>>>>> > >>> repo,
>>>>>>>> > >>>> so
>>>>>>>> > >>>>>>>> there is never anything to synchronise.
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>>> On 4 Jul 2025, at 15:53, Amogh Desai <
>>>>>>>> amoghdesai.oss@gmail.com>
>>>>>>>> > >>>> wrote:
>>>>>>>> > >>>>>>>>> Thanks Ash.
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>> This is really cool and helpful that you were able to
>>>>>>>> test both
>>>>>>>> > >>>> scenarios
>>>>>>>> > >>>>>>>>> -- repo checkout
>>>>>>>> > >>>>>>>>> and also installing from the vendored package and the
>>>>>>>> resolution
>>>>>>>> > >>>> worked
>>>>>>>> > >>>>>>>>> fine too.
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>> I like this idea compared the to relative import one
>>>>>>>> for few
>>>>>>>> > >>> reasons:
>>>>>>>> > >>>>>>>>> - It feels like it will take some time to adjust to the
>>>>>>>> new
>>>>>>>> > coding
>>>>>>>> > >>>>>>>> standard
>>>>>>>> > >>>>>>>>> that we will lay
>>>>>>>> > >>>>>>>>> if we impose relative imports in the shared dist
>>>>>>>> > >>>>>>>>> - We can continue using repo wise absolute import
>>>>>>>> standards, it
>>>>>>>> > is
>>>>>>>> > >>>> also
>>>>>>>> > >>>>>>>>> much easier for situations
>>>>>>>> > >>>>>>>>> when we do global search in IDE to find + replace, this
>>>>>>>> could
>>>>>>>> > mean
>>>>>>>> > >>> a
>>>>>>>> > >>>>>>>> change
>>>>>>>> > >>>>>>>>> there
>>>>>>>> > >>>>>>>>> - The vendoring work is a proven and established
>>>>>>>> paradigm across
>>>>>>>> > >>>> projects
>>>>>>>> > >>>>>>>>> and would
>>>>>>>> > >>>>>>>>> out of box give us the build tooling we need also
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>> Nothing too against the relative import but with the
>>>>>>>> evidence
>>>>>>>> > >>>> provided
>>>>>>>> > >>>>>>>>> above, vendored approach
>>>>>>>> > >>>>>>>>> seems to only do us good.
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>> Regarding synchronizing it, release time should be fine
>>>>>>>> as long
>>>>>>>> > as
>>>>>>>> > >>> we
>>>>>>>> > >>>>>>>> have
>>>>>>>> > >>>>>>>>> a good CI workflow to probably
>>>>>>>> > >>>>>>>>> catch such issues per PR if changes are made in shared
>>>>>>>> dist?
>>>>>>>> > >>>> (precommit
>>>>>>>> > >>>>>>>>> would make it really slow i guess)
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>> If we can run our tests with vendored code we should be
>>>>>>>> mostly
>>>>>>>> > >>>> covered.
>>>>>>>> > >>>>>>>>> Good effort all!
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>> Thanks & Regards,
>>>>>>>> > >>>>>>>>> Amogh Desai
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>>
>>>>>>>> > >>>>>>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <
>>>>>>>> > ash@apache.org>
>>>>>>>> > >>>>>>>> wrote:
>>>>>>>> > >>>>>>>>>> Okay, I think I’ve got something that works and I’m
>>>>>>>> happy with.
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>
>>>>>>>> >
>>>>>>>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>>>>>>>> > >>>>>>>>>> This produces the following from `uv build task-sdk`
>>>>>>>> > >>>>>>>>>> -
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>
>>>>>>>> >
>>>>>>>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>>>>> > >>>>>>>>>> -
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>
>>>>>>>> >
>>>>>>>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>>>>>>>> > >>>>>>>>>> (`.whl.zip` as GH won't allow .whl upload, but will
>>>>>>>> .zip)
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> ```
>>>>>>>> > >>>>>>>>>> ❯ unzip -l
>>>>>>>> > >>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
>>>>>>>> > >>>>>>>> grep
>>>>>>>> > >>>>>>>>>> _vendor
>>>>>>>> > >>>>>>>>>>      50  02-02-2020 00:00
>>>>>>>>  airflow/sdk/_vendor/.gitignore
>>>>>>>> > >>>>>>>>>>    2082  02-02-2020 00:00
>>>>>>>>  airflow/sdk/_vendor/__init__.py
>>>>>>>> > >>>>>>>>>>      28  02-02-2020 00:00
>>>>>>>> >  airflow/sdk/_vendor/airflow_common.pyi
>>>>>>>> > >>>>>>>>>>      18  02-02-2020 00:00
>>>>>>>>  airflow/sdk/_vendor/vendor.txt
>>>>>>>> > >>>>>>>>>>     785  02-02-2020 00:00
>>>>>>>> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>>>> > >>>>>>>>>>   10628  02-02-2020 00:00
>>>>>>>> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>>>> > >>>>>>>>>> ```
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> And similarly in the .tar.gz, so our “sdist” is
>>>>>>>> complete too:
>>>>>>>> > >>>>>>>>>> ```
>>>>>>>> > >>>>>>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>>>>> |grep
>>>>>>>> > _vendor
>>>>>>>> > >>>>>>>>>>
>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>
>>>>>>>> >
>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>>>>>>>> > >>>>>>>>>>
>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>
>>>>>>>> >
>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>>>> > >>>
>>>>>>>> >
>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>>>> > >>>>>>>>>> ```
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> The plugin works at build time by including/copying
>>>>>>>> the libs
>>>>>>>> > >>>> specified
>>>>>>>> > >>>>>>>> in
>>>>>>>> > >>>>>>>>>> vendor.txt into place (and let `vendoring` take care
>>>>>>>> of import
>>>>>>>> > >>>>>>>> rewrites.)
>>>>>>>> > >>>>>>>>>> For the imports to continue to work at “dev” time/from
>>>>>>>> a repo
>>>>>>>> > >>>> checkout,
>>>>>>>> > >>>>>>>> I
>>>>>>>> > >>>>>>>>>> have added a import finder to `sys.meta_path`, and
>>>>>>>> since it’s at
>>>>>>>> > >>> the
>>>>>>>> > >>>>>>>> end of
>>>>>>>> > >>>>>>>>>> the list it will only be used if the normal import
>>>>>>>> can’t find
>>>>>>>> > >>>> things.
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>
>>>>>>>> >
>>>>>>>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>>>>>>>> > >>>>>>>>>> This doesn’t quite give us the same runtime effect
>>>>>>>> “import
>>>>>>>> > >>>> rewriting”
>>>>>>>> > >>>>>>>>>> affect, as in this approach `airflow_common` is
>>>>>>>> directly loaded
>>>>>>>> > >>>> (i.e.
>>>>>>>> > >>>>>>>>>> airflow.sdk._vendor.airflow_common and airflow_common
>>>>>>>> exist in
>>>>>>>> > >>>>>>>>>> sys.modules), but it does work for everything that I
>>>>>>>> was able to
>>>>>>>> > >>>> test..
>>>>>>>> > >>>>>>>>>> I tested it with the diff at the end of this message.
>>>>>>>> My test
>>>>>>>> > >>>> ipython
>>>>>>>> > >>>>>>>>>> shell:
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> ```
>>>>>>>> > >>>>>>>>>> In [1]: from
>>>>>>>> airflow.sdk._vendor.airflow_common.timezone import
>>>>>>>> > >>> foo
>>>>>>>> > >>>>>>>>>> In [2]: foo
>>>>>>>> > >>>>>>>>>> Out[2]: 1
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> In [3]: import airflow.sdk._vendor.airflow_common
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> In [4]: import
>>>>>>>> airflow.sdk._vendor.airflow_common.timezone
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>>>>>>>> > >>>>>>>>>> Out[5]:
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>
>>>>>>>> >
>>>>>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>>>>>>>> > >>>>>>>>>> In [6]:
>>>>>>>> airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>>>> > >>>>>>>>>> Out[6]:
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>
>>>>>>>> >
>>>>>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>>>>>>>> > >>>>>>>>>> ```
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> And in an standalone environment with the SDK dist I
>>>>>>>> built (it
>>>>>>>> > >>>> needed
>>>>>>>> > >>>>>>>> the
>>>>>>>> > >>>>>>>>>> matching airflow-core right now, but that is nothing
>>>>>>>> to do with
>>>>>>>> > >>> this
>>>>>>>> > >>>>>>>>>> discussion):
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> ```
>>>>>>>> > >>>>>>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>>>>>>>> > >>>>>>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>>>>>>>> > >>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl
>>>>>>>> ipython
>>>>>>>> > >>>>>>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang
>>>>>>>> 18.1.8 ]
>>>>>>>> > >>>>>>>>>> Type 'copyright', 'credits' or 'license' for more
>>>>>>>> information
>>>>>>>> > >>>>>>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type
>>>>>>>> '?' for
>>>>>>>> > >>> help.
>>>>>>>> > >>>>>>>>>> Tip: You can use `%hist` to view history, see the
>>>>>>>> options with
>>>>>>>> > >>>>>>>> `%history?`
>>>>>>>> > >>>>>>>>>> In [1]: import
>>>>>>>> airflow.sdk._vendor.airflow_common.timezone
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> In [2]:
>>>>>>>> airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>>>> > >>>>>>>>>> Out[2]:
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>
>>>>>>>> >
>>>>>>>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>>>>>>>> > >>>>>>>>>> ``
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> ```diff
>>>>>>>> > >>>>>>>>>> diff --git
>>>>>>>> a/airflow-common/src/airflow_common/__init__.py
>>>>>>>> > >>>>>>>>>> b/airflow-common/src/airflow_common/__init__.py
>>>>>>>> > >>>>>>>>>> index 13a83393a9..927b7c6b61 100644
>>>>>>>> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/__init__.py
>>>>>>>> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
>>>>>>>> > >>>>>>>>>> @@ -14,3 +14,5 @@
>>>>>>>> > >>>>>>>>>> # KIND, either express or implied.  See the License
>>>>>>>> for the
>>>>>>>> > >>>>>>>>>> # specific language governing permissions and
>>>>>>>> limitations
>>>>>>>> > >>>>>>>>>> # under the License.
>>>>>>>> > >>>>>>>>>> +
>>>>>>>> > >>>>>>>>>> +foo = 1
>>>>>>>> > >>>>>>>>>> diff --git
>>>>>>>> a/airflow-common/src/airflow_common/timezone.py
>>>>>>>> > >>>>>>>>>> b/airflow-common/src/airflow_common/timezone.py
>>>>>>>> > >>>>>>>>>> index 340b924c66..58384ef20f 100644
>>>>>>>> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/timezone.py
>>>>>>>> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
>>>>>>>> > >>>>>>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>>>>>>>> > >>>>>>>>>> version.parse(metadata.version("pendulum")).major == 3
>>>>>>>> > >>>>>>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
>>>>>>>> > >>>>>>>>>> utc = pendulum.UTC
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> +
>>>>>>>> > >>>>>>>>>> +from airflow_common import foo
>>>>>>>> > >>>>>>>>>> +
>>>>>>>> > >>>>>>>>>> TIMEZONE: Timezone
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>> ```
>>>>>>>> > >>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <
>>>>>>>> jarek@potiuk.com>
>>>>>>>> > wrote:
>>>>>>>> > >>>>>>>>>>> I think both approaches are doable:
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> 1) -> We can very easily prevent bad imports by
>>>>>>>> pre-commit when
>>>>>>>> > >>>>>>>> importing
>>>>>>>> > >>>>>>>>>>> from different distributions and make sure we are
>>>>>>>> only doing
>>>>>>>> > >>>> relative
>>>>>>>> > >>>>>>>>>>> imports in the shared modules. We are doing plenty of
>>>>>>>> this
>>>>>>>> > >>>> already. And
>>>>>>>> > >>>>>>>>>> yes
>>>>>>>> > >>>>>>>>>>> it would require relative links we currently do not
>>>>>>>> allow.
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> 2) -> has one disadvantage that someone at some point
>>>>>>>> in time
>>>>>>>> > >>> will
>>>>>>>> > >>>> have
>>>>>>>> > >>>>>>>>>> to
>>>>>>>> > >>>>>>>>>>> decide to synchronize this and if it happens just
>>>>>>>> before
>>>>>>>> > release
>>>>>>>> > >>>> (I bet
>>>>>>>> > >>>>>>>>>>> this is going to happen) this will lead to solving
>>>>>>>> problems
>>>>>>>> > that
>>>>>>>> > >>>> would
>>>>>>>> > >>>>>>>>>>> normally be solved during PR when you make a change
>>>>>>>> (i.e.
>>>>>>>> > >>> symbolic
>>>>>>>> > >>>> link
>>>>>>>> > >>>>>>>>>> has
>>>>>>>> > >>>>>>>>>>> the advantage that whoever modifies shared code will
>>>>>>>> be
>>>>>>>> > >>> immediately
>>>>>>>> > >>>>>>>>>>> notified in their PR - that they broke something
>>>>>>>> because either
>>>>>>>> > >>>> static
>>>>>>>> > >>>>>>>>>>> checks or mypy or tests fail.
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> Ash, do you have an idea of a process (who and when)
>>>>>>>> does the
>>>>>>>> > >>>>>>>>>>> synchronisation in case of vendoring? Maybe we could
>>>>>>>> solve it
>>>>>>>> > if
>>>>>>>> > >>>> it is
>>>>>>>> > >>>>>>>>>> done
>>>>>>>> > >>>>>>>>>>> more frequently and with some regularity? We could
>>>>>>>> potentially
>>>>>>>> > >>>> force
>>>>>>>> > >>>>>>>>>>> re-vendoring at PR time as well any time shared code
>>>>>>>> changes
>>>>>>>> > (and
>>>>>>>> > >>>>>>>> prevent
>>>>>>>> > >>>>>>>>>>> it by pre-commit. And I can't think of some place
>>>>>>>> (other than
>>>>>>>> > >>>> releases)
>>>>>>>> > >>>>>>>>>> in
>>>>>>>> > >>>>>>>>>>> our development workflow and that seems to be a bit
>>>>>>>> too late as
>>>>>>>> > >>>> puts an
>>>>>>>> > >>>>>>>>>>> extra effort on fixing potential incompatibilities
>>>>>>>> introduced
>>>>>>>> > on
>>>>>>>> > >>>>>>>> release
>>>>>>>> > >>>>>>>>>>> manager and delays the release. WDYT?
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> Re: relative links. I think for a shared library we
>>>>>>>> could
>>>>>>>> > >>>> potentially
>>>>>>>> > >>>>>>>>>> relax
>>>>>>>> > >>>>>>>>>>> this and allow them (and actually disallow absolute
>>>>>>>> links in
>>>>>>>> > the
>>>>>>>> > >>>> pieces
>>>>>>>> > >>>>>>>>>> of
>>>>>>>> > >>>>>>>>>>> code that are shared - again, by pre-commit). As I
>>>>>>>> recall, the
>>>>>>>> > >>> only
>>>>>>>> > >>>>>>>>>> reason
>>>>>>>> > >>>>>>>>>>> we forbade the relative links is because of how we
>>>>>>>> are (or
>>>>>>>> > maybe
>>>>>>>> > >>>> were)
>>>>>>>> > >>>>>>>>>>> doing DAG parsing and failures resulting from it. So
>>>>>>>> we decided
>>>>>>>> > >>> to
>>>>>>>> > >>>> just
>>>>>>>> > >>>>>>>>>> not
>>>>>>>> > >>>>>>>>>>> allow it to keep consistency. The way how Dag parsing
>>>>>>>> works is
>>>>>>>> > >>> that
>>>>>>>> > >>>>>>>> when
>>>>>>>> > >>>>>>>>>>> you are using importlib to read the Dag from a file,
>>>>>>>> the
>>>>>>>> > relative
>>>>>>>> > >>>>>>>> imports
>>>>>>>> > >>>>>>>>>>> do not work as it does not know what they should be
>>>>>>>> relative
>>>>>>>> > to.
>>>>>>>> > >>>> But if
>>>>>>>> > >>>>>>>>>>> relative import is done from an imported package, it
>>>>>>>> should be
>>>>>>>> > no
>>>>>>>> > >>>>>>>>>> problem,
>>>>>>>> > >>>>>>>>>>> I think - otherwise our Dags would not be able to
>>>>>>>> import any
>>>>>>>> > >>>> library
>>>>>>>> > >>>>>>>> that
>>>>>>>> > >>>>>>>>>>> uses relative imports.
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> Of course consistency might be the reason why we do
>>>>>>>> not want to
>>>>>>>> > >>>>>>>> introduce
>>>>>>>> > >>>>>>>>>>> relative imports. I don't see it as an issue if it is
>>>>>>>> guarded
>>>>>>>> > by
>>>>>>>> > >>>>>>>>>> pre-commit
>>>>>>>> > >>>>>>>>>>> though.
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> J.
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> J.
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
>>>>>>>> > >>>> ash@apache.org>
>>>>>>>> > >>>>>>>>>>> napisał:
>>>>>>>> > >>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> Oh yes, symlinks will work, with one big caveat: It
>>>>>>>> does mean
>>>>>>>> > >>> you
>>>>>>>> > >>>>>>>> can’t
>>>>>>>> > >>>>>>>>>>>> use absolute imports in one common module to another.
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> For example
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>
>>>>>>>> >
>>>>>>>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>>>>>>>> > >>>>>>>>>>>> where we have
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> ```
>>>>>>>> > >>>>>>>>>>>> from airflow.utils.module_loading import
>>>>>>>> import_string
>>>>>>>> > >>>>>>>>>>>> ```
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> if we want to move serve_logs into this common lib
>>>>>>>> that is
>>>>>>>> > then
>>>>>>>> > >>>>>>>>>> symlinked
>>>>>>>> > >>>>>>>>>>>> then we wouldn’t be able to have `from
>>>>>>>> > >>>> airflow_common.module_loading
>>>>>>>> > >>>>>>>>>> import
>>>>>>>> > >>>>>>>>>>>> import_string`.
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> I can think of two possible solutions here.
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> 1) is to allow/require relative imports in this
>>>>>>>> shared lib, so
>>>>>>>> > >>>> `from
>>>>>>>> > >>>>>>>>>>>> .module_loading import import_string`
>>>>>>>> > >>>>>>>>>>>> 2) is to use `vendoring`[1] (from the pip
>>>>>>>> maintainers) which
>>>>>>>> > >>> will
>>>>>>>> > >>>>>>>> handle
>>>>>>>> > >>>>>>>>>>>> import-rewriting for us.
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> I’d entirely forgot that symlinks in repos was a
>>>>>>>> thing, so I
>>>>>>>> > >>>> prepared
>>>>>>>> > >>>>>>>> a
>>>>>>>> > >>>>>>>>>>>> minimal POC/demo of what vendoring approach could
>>>>>>>> look like
>>>>>>>> > here
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>
>>>>>>>> >
>>>>>>>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>>>>>>>> > >>>>>>>>>>>> Now personally I am more than happy with relative
>>>>>>>> imports, but
>>>>>>>> > >>>>>>>> generally
>>>>>>>> > >>>>>>>>>>>> as a project we have avoided them, so I think that
>>>>>>>> limits what
>>>>>>>> > >>> we
>>>>>>>> > >>>>>>>> could
>>>>>>>> > >>>>>>>>>> do
>>>>>>>> > >>>>>>>>>>>> with a symlink based approach.
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> -ash
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>> [1] https://github.com/pradyunsg/vendoring
>>>>>>>> > >>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
>>>>>>>> > >>>>>>>> gopidesupavan@gmail.com>
>>>>>>>> > >>>>>>>>>>>> wrote:
>>>>>>>> > >>>>>>>>>>>>> Thanks Ash
>>>>>>>> > >>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>> Yes agree option 2 would be preferred for me.
>>>>>>>> Making sure we
>>>>>>>> > >>>> have all
>>>>>>>> > >>>>>>>>>> the
>>>>>>>> > >>>>>>>>>>>>> gaurdriles to protect any unwanted behaviour in
>>>>>>>> code sharing
>>>>>>>> > >>> and
>>>>>>>> > >>>>>>>>>>>> executing
>>>>>>>> > >>>>>>>>>>>>> right of tests between the packages.
>>>>>>>> > >>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>> Agree with others, option 2 would be
>>>>>>>> > >>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
>>>>>>>> > >>>>>>>> amoghdesai.oss@gmail.com>
>>>>>>>> > >>>>>>>>>>>>> wrote:
>>>>>>>> > >>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>> Thanks for starting this discussion, Ash.
>>>>>>>> > >>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>> I would prefer option 2 here with proper tooling
>>>>>>>> to handle
>>>>>>>> > the
>>>>>>>> > >>>> code
>>>>>>>> > >>>>>>>>>>>>>> duplication at *release* time.
>>>>>>>> > >>>>>>>>>>>>>> It is best to have a dist that has all it needs in
>>>>>>>> itself.
>>>>>>>> > >>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>> Option 1 could very quickly get out of hand and if
>>>>>>>> we decide
>>>>>>>> > >>> to
>>>>>>>> > >>>>>>>>>> separate
>>>>>>>> > >>>>>>>>>>>>>> triggerer /
>>>>>>>> > >>>>>>>>>>>>>> dag processor / config etc etc as separate
>>>>>>>> packages, back
>>>>>>>> > >>>> compat is
>>>>>>>> > >>>>>>>>>>>> going
>>>>>>>> > >>>>>>>>>>>>>> to be a nightmare
>>>>>>>> > >>>>>>>>>>>>>> and will bite us harder than we anticipate.
>>>>>>>> > >>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>> > >>>>>>>>>>>>>> Amogh Desai
>>>>>>>> > >>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <
>>>>>>>> > >>> kaxilnaik@gmail.com>
>>>>>>>> > >>>>>>>>>> wrote:
>>>>>>>> > >>>>>>>>>>>>>>> I prefer Option 2 as well to avoid matrix of
>>>>>>>> dependencies
>>>>>>>> > >>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>>>>>>>> > >>>>>>>>>> <j_scheffler@gmx.de.invalid
>>>>>>>> > >>>>>>>>>>>>>>> wrote:
>>>>>>>> > >>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>> I'd also rather prefer option 2 - reason here is
>>>>>>>> it is
>>>>>>>> > >>> rather
>>>>>>>> > >>>>>>>>>>>> pragmatic
>>>>>>>> > >>>>>>>>>>>>>>>> and we no not need to cut another package and
>>>>>>>> have less
>>>>>>>> > >>>> package
>>>>>>>> > >>>>>>>>>> counts
>>>>>>>> > >>>>>>>>>>>>>>>> and dependencies.
>>>>>>>> > >>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>> I remember some time ago I was checking
>>>>>>>> (together with
>>>>>>>> > >>> Jarek,
>>>>>>>> > >>>> I am
>>>>>>>> > >>>>>>>>>> not
>>>>>>>> > >>>>>>>>>>>>>>>> sure anymore...) if the usage of symlinks would
>>>>>>>> be
>>>>>>>> > possible.
>>>>>>>> > >>>> To
>>>>>>>> > >>>>>>>> keep
>>>>>>>> > >>>>>>>>>>>>>> the
>>>>>>>> > >>>>>>>>>>>>>>>> source in one package but "symlink" it into
>>>>>>>> another. If
>>>>>>>> > then
>>>>>>>> > >>>> at
>>>>>>>> > >>>>>>>>>> point
>>>>>>>> > >>>>>>>>>>>>>> of
>>>>>>>> > >>>>>>>>>>>>>>>> packaging/release the files are materialized we
>>>>>>>> have 1 set
>>>>>>>> > >>> of
>>>>>>>> > >>>>>>>> code.
>>>>>>>> > >>>>>>>>>>>>>>>> Otherwise if not possible still the redundancy
>>>>>>>> could be
>>>>>>>> > >>>> solved by
>>>>>>>> > >>>>>>>> a
>>>>>>>> > >>>>>>>>>>>>>>>> pre-commit hook - and in Git the files are
>>>>>>>> de-duplicated
>>>>>>>> > >>>> anyway
>>>>>>>> > >>>>>>>>>> based
>>>>>>>> > >>>>>>>>>>>>>> on
>>>>>>>> > >>>>>>>>>>>>>>>> content hash, so this does not hurt.
>>>>>>>> > >>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>>>>>> > >>>>>>>>>>>>>>>>> I support option 2 with proper automation & CI
>>>>>>>> - the
>>>>>>>> > >>>> reasonings
>>>>>>>> > >>>>>>>>>>>>>> you've
>>>>>>>> > >>>>>>>>>>>>>>>>> shown for that make sense to me.
>>>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>> Shahar
>>>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash
>>>>>>>> Berlin-Taylor <
>>>>>>>> > >>>> ash@apache.org
>>>>>>>> > >>>>>>>>>>>>>>> wrote:
>>>>>>>> > >>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>>> As we work on finishing off the code-level
>>>>>>>> separation of
>>>>>>>> > >>>> Task
>>>>>>>> > >>>>>>>> SDK
>>>>>>>> > >>>>>>>>>>>>>> and
>>>>>>>> > >>>>>>>>>>>>>>>> Core
>>>>>>>> > >>>>>>>>>>>>>>>>>> (scheduler etc) we have come across some
>>>>>>>> situations
>>>>>>>> > where
>>>>>>>> > >>> we
>>>>>>>> > >>>>>>>> would
>>>>>>>> > >>>>>>>>>>>>>>> like
>>>>>>>> > >>>>>>>>>>>>>>>> to
>>>>>>>> > >>>>>>>>>>>>>>>>>> share code between these.
>>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>>> However it’s not as straight forward of “just
>>>>>>>> put it in
>>>>>>>> > a
>>>>>>>> > >>>> common
>>>>>>>> > >>>>>>>>>>>>>> dist
>>>>>>>> > >>>>>>>>>>>>>>>> they
>>>>>>>> > >>>>>>>>>>>>>>>>>> both depend upon” because one of the goals of
>>>>>>>> the Task
>>>>>>>> > SDK
>>>>>>>> > >>>>>>>>>>>>>> separation
>>>>>>>> > >>>>>>>>>>>>>>>> was
>>>>>>>> > >>>>>>>>>>>>>>>>>> to have 100% complete version independence
>>>>>>>> between the
>>>>>>>> > >>> two,
>>>>>>>> > >>>>>>>>>> ideally
>>>>>>>> > >>>>>>>>>>>>>>>> even if
>>>>>>>> > >>>>>>>>>>>>>>>>>> they are built into the same image and venv.
>>>>>>>> Most of the
>>>>>>>> > >>>> reason
>>>>>>>> > >>>>>>>>>> why
>>>>>>>> > >>>>>>>>>>>>>>> this
>>>>>>>> > >>>>>>>>>>>>>>>>>> isn’t straight forward comes down to backwards
>>>>>>>> > >>>> compatibility -
>>>>>>>> > >>>>>>>> if
>>>>>>>> > >>>>>>>>>> we
>>>>>>>> > >>>>>>>>>>>>>>>> make
>>>>>>>> > >>>>>>>>>>>>>>>>>> an change to the common/shared distribution
>>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>>> We’ve listed the options we have thought about
>>>>>>>> in
>>>>>>>> > >>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545
>>>>>>>> (but
>>>>>>>> > that
>>>>>>>> > >>>> covers
>>>>>>>> > >>>>>>>>>>>>>> some
>>>>>>>> > >>>>>>>>>>>>>>>> more
>>>>>>>> > >>>>>>>>>>>>>>>>>> things that I don’t want to get in to in this
>>>>>>>> discussion
>>>>>>>> > >>>> such as
>>>>>>>> > >>>>>>>>>>>>>>>> possibly
>>>>>>>> > >>>>>>>>>>>>>>>>>> separating operators and executors out of a
>>>>>>>> single
>>>>>>>> > >>> provider
>>>>>>>> > >>>>>>>> dist.)
>>>>>>>> > >>>>>>>>>>>>>>>>>> To give a concrete example of some code I
>>>>>>>> would like to
>>>>>>>> > >>>> share
>>>>>>>> > >>>
>>>>>>>> >
>>>>>>>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>>>>>> > >>>>>>>>>>>>>>>>>> — logging config. Another thing we will want
>>>>>>>> to share
>>>>>>>> > will
>>>>>>>> > >>>> be
>>>>>>>> > >>>>>>>> the
>>>>>>>> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class from
>>>>>>>> airflow.configuration
>>>>>>>> > (but
>>>>>>>> > >>>>>>>> notably:
>>>>>>>> > >>>>>>>>>>>>>>> only
>>>>>>>> > >>>>>>>>>>>>>>>> the
>>>>>>>> > >>>>>>>>>>>>>>>>>> parser class, _not_ the default config values,
>>>>>>>> again,
>>>>>>>> > lets
>>>>>>>> > >>>> not
>>>>>>>> > >>>>>>>>>> dwell
>>>>>>>> > >>>>>>>>>>>>>>> on
>>>>>>>> > >>>>>>>>>>>>>>>> the
>>>>>>>> > >>>>>>>>>>>>>>>>>> specifics of that)
>>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>>> So to bring the options listed in the issue
>>>>>>>> here for
>>>>>>>> > >>>> discussion,
>>>>>>>> > >>>>>>>>>>>>>>> broadly
>>>>>>>> > >>>>>>>>>>>>>>>>>> speaking there are two high-level approaches:
>>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>>> 1. A single shared distribution
>>>>>>>> > >>>>>>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
>>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>>> The advantage of Approach 1 is that we only
>>>>>>>> have the
>>>>>>>> > code
>>>>>>>> > >>>> in one
>>>>>>>> > >>>>>>>>>>>>>>> place.
>>>>>>>> > >>>>>>>>>>>>>>>>>> However for me, at least in this specific case
>>>>>>>> of
>>>>>>>> > Logging
>>>>>>>> > >>>> config
>>>>>>>> > >>>>>>>>>> or
>>>>>>>> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class is that backwards
>>>>>>>> > compatibility
>>>>>>>> > >>> is
>>>>>>>> > >>>>>>>> much
>>>>>>>> > >>>>>>>>>>>>>> much
>>>>>>>> > >>>>>>>>>>>>>>>>>> harder.
>>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>>> The main advantage of Approach 2 is the the
>>>>>>>> code is
>>>>>>>> > >>> released
>>>>>>>> > >>>>>>>>>>>>>>>> with/embedded
>>>>>>>> > >>>>>>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk
>>>>>>>> would contain
>>>>>>>> > >>> the
>>>>>>>> > >>>>>>>> right
>>>>>>>> > >>>>>>>>>>>>>>>> version
>>>>>>>> > >>>>>>>>>>>>>>>>>> of the logging config and ConfigParser etc).
>>>>>>>> The
>>>>>>>> > downside
>>>>>>>> > >>> is
>>>>>>>> > >>>>>>>> that
>>>>>>>> > >>>>>>>>>>>>>>> either
>>>>>>>> > >>>>>>>>>>>>>>>>>> the code will need to be duplicated in the
>>>>>>>> repo, or
>>>>>>>> > better
>>>>>>>> > >>>> yet
>>>>>>>> > >>>>>>>> it
>>>>>>>> > >>>>>>>>>>>>>>> would
>>>>>>>> > >>>>>>>>>>>>>>>>>> live in a single place in the repo, but some
>>>>>>>> tooling
>>>>>>>> > (TBD)
>>>>>>>> > >>>> will
>>>>>>>> > >>>>>>>>>>>>>>>>>> automatically handle the duplication, either
>>>>>>>> at commit
>>>>>>>> > >>>> time, or
>>>>>>>> > >>>>>>>> my
>>>>>>>> > >>>>>>>>>>>>>>>>>> preference, at release time.
>>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>>> For this kind of shared “utility” code I am
>>>>>>>> very
>>>>>>>> > strongly
>>>>>>>> > >>>>>>>> leaning
>>>>>>>> > >>>>>>>>>>>>>>>> towards
>>>>>>>> > >>>>>>>>>>>>>>>>>> option 2 with automation, as otherwise I think
>>>>>>>> the
>>>>>>>> > >>> backwards
>>>>>>>> > >>>>>>>>>>>>>>>> compatibility
>>>>>>>> > >>>>>>>>>>>>>>>>>> requirements would make it unworkable (very
>>>>>>>> quickly over
>>>>>>>> > >>>> time
>>>>>>>> > >>>>>>>> the
>>>>>>>> > >>>>>>>>>>>>>>>>>> combinations we would have to test would just
>>>>>>>> be
>>>>>>>> > >>>> unreasonable)
>>>>>>>> > >>>>>>>>>> and I
>>>>>>>> > >>>>>>>>>>>>>>>> don’t
>>>>>>>> > >>>>>>>>>>>>>>>>>> feel confident we can have things as stable as
>>>>>>>> we need
>>>>>>>> > to
>>>>>>>> > >>>> really
>>>>>>>> > >>>>>>>>>>>>>>> deliver
>>>>>>>> > >>>>>>>>>>>>>>>>>> the version separation/independency I want to
>>>>>>>> delivery
>>>>>>>> > >>> with
>>>>>>>> > >>>>>>>>>> AIP-72.
>>>>>>>> > >>>>>>>>>>>>>>>>>> So unless someone feels very strongly about
>>>>>>>> this, I will
>>>>>>>> > >>>> come up
>>>>>>>> > >>>>>>>>>>>>>> with
>>>>>>>> > >>>>>>>>>>>>>>> a
>>>>>>>> > >>>>>>>>>>>>>>>>>> draft PR for further discussion that will
>>>>>>>> implement code
>>>>>>>> > >>>> sharing
>>>>>>>> > >>>>>>>>>> via
>>>>>>>> > >>>>>>>>>>>>>>>>>> “vendoring” it at build time. I have an idea
>>>>>>>> of how I
>>>>>>>> > can
>>>>>>>> > >>>>>>>> achieve
>>>>>>>> > >>>>>>>>>>>>>> this
>>>>>>>> > >>>>>>>>>>>>>>>> so
>>>>>>>> > >>>>>>>>>>>>>>>>>> we have a single version in the repo and it’ll
>>>>>>>> work
>>>>>>>> > there,
>>>>>>>> > >>>> but
>>>>>>>> > >>>>>>>> at
>>>>>>>> > >>>>>>>>>>>>>>>> runtime
>>>>>>>> > >>>>>>>>>>>>>>>>>> we vendor it in to the shipped dist so it
>>>>>>>> lives at
>>>>>>>> > >>> something
>>>>>>>> > >>>>>>>> like
>>>>>>>> > >>>>>>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
>>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>>> In terms of repo layout, this likely means we
>>>>>>>> would end
>>>>>>>> > up
>>>>>>>> > >>>> with:
>>>>>>>> > >>>>>>>>>>>>>>>>>> airflow-core/pyproject.toml
>>>>>>>> > >>>>>>>>>>>>>>>>>> airflow-core/src/
>>>>>>>> > >>>>>>>>>>>>>>>>>> airflow-core/tests/
>>>>>>>> > >>>>>>>>>>>>>>>>>> task-sdk/pyproject.toml
>>>>>>>> > >>>>>>>>>>>>>>>>>> task-sdk/src/
>>>>>>>> > >>>>>>>>>>>>>>>>>> task-sdk/tests/
>>>>>>>> > >>>>>>>>>>>>>>>>>> airflow-common/src
>>>>>>>> > >>>>>>>>>>>>>>>>>> airflow-common/tests/
>>>>>>>> > >>>>>>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml,
>>>>>>>> as deps
>>>>>>>> > would
>>>>>>>> > >>>> be
>>>>>>>> > >>>>>>>>>>>>>> included
>>>>>>>> > >>>>>>>>>>>>>>>> in
>>>>>>>> > >>>>>>>>>>>>>>>>>> the downstream projects. TBD.
>>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>>>>>>>>>>> Thoughts and feedback welcomed.
>>>>>>>> > >>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>> > dev-unsubscribe@airflow.apache.org
>>>>>>>> > >>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>> > >>> dev-help@airflow.apache.org
>>>>>>>> > >>>>>>>>>>>>>>>>
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> > >>>>>>>> To unsubscribe, e-mail:
>>>>>>>> dev-unsubscribe@airflow.apache.org
>>>>>>>> > >>>>>>>> For additional commands, e-mail:
>>>>>>>> dev-help@airflow.apache.org
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>>>
>>>>>>>> > >>>>>>
>>>>>>>> >
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>> > >>>>>> For additional commands, e-mail:
>>>>>>>> dev-help@airflow.apache.org
>>>>>>>> > >>>>>>
>>>>>>>> > >>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> > >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>> > >>>>> For additional commands, e-mail:
>>>>>>>> dev-help@airflow.apache.org
>>>>>>>> > >>>>>
>>>>>>>> > >>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> > >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>> > >>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>> > >>>>
>>>>>>>> > >>>>
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>> > > For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>> > >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>> > For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>> >
>>>>>>>> >
>>>>>>>>
>>>>>>>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Amogh Desai <am...@gmail.com>.
Agreed!

Once the PR is up, we can have these implementation level discussions
over there. Good chat however!

Thanks & Regards,
Amogh Desai


On Wed, Jul 9, 2025 at 3:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> Yeah. I think extracting one-by-one, feature-by-feature that we want to
> share to a separate distribution is the best approach - it will actually
> also help with the "__init__.py" cleanup - because almost by definition -
> those distributions will not be able to "reach" outside - i.e. they only
> can be "used" not "use" something else. Which means that - for example as
> it is now - configuration using logging which is using configuration
> (leading to circular dependencies and partially initialized modules) will
> not happen and we will have to figure out rather how to inject
> configuration into logging (from task-sdk, airflow-ctl, airflow-core) and
> to get the right sequence of initialization - rather than have the
> inter-feature-dependencies.
>
> And that is precisely what low-cyclomatic complexity is about as well - it
> generally leads to easier-to-maintain software that has well defined
> functionality and does not have those weird circular dependencies we have
> now. That's a kind-of side-effect of such a "per feature" split, but a very
> desirable one.
>
> J.
>
> On Wed, Jul 9, 2025 at 10:17 AM Amogh Desai <am...@gmail.com>
> wrote:
>
>> Probably, you make a valid point.
>>
>> Maybe this is an implementation detail, so we could figure it out as we
>> start on a POC and factor in these things
>> as we move along?
>>
>> But from an initial guess, I would think that execution time related
>> items (if we manage to enumerate them) would be something
>> that would be better off in that "core_and_task_sdk" bundle.
>>
>> Thanks & Regards,
>> Amogh Desai
>>
>>
>> On Tue, Jul 8, 2025 at 3:46 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> > Not that I am against your idea and we can surely expand as we need
>>> but we would not need to expand the
>>> "core_and_task_sdk" if we put only the relevant items into it.
>>>
>>> So if we move logging and config out, my question is what is really
>>> relevant to "stay" in "core_and_task_sdk" ? And what we know will not be
>>> needed by other distributions in the future ?
>>>
>>> What would be the content of "core_and_task_sdk" ? Can we enumerate what
>>> should go there now ?
>>>
>>> My bet is that if we enumerate them, it will turn out that they are good
>>> candidates to make separate "shared features" that can be logically named
>>> and modularised, and that there is no real need to have shared
>>> "core_and_task_sdk" which sounds like "bag of everything else".
>>>
>>>
>>>
>>> On Tue, Jul 8, 2025 at 11:48 AM Amogh Desai <am...@gmail.com>
>>> wrote:
>>>
>>>> Yeah, I think what you are showcasing here is a step ahead of the
>>>> initial proposal from Ash.
>>>>
>>>> From the original proposal, the `core_and_task_sdk` *can* have the
>>>> things relevant to just those two
>>>> distros. Logging, Config are modules that might be needed by
>>>> airflow-ctl for example, so ideally, those
>>>> would not be good candidates to be put in there, ideally speaking.
>>>>
>>>> The example of Kubernetes Utils sounds to be a good example, it will be
>>>> used by KubeExecutor (lets say this is a
>>>> module called "executors") and by KPO (providers), and the
>>>> "shared/kubernetes" would probably be a good
>>>> candidate for that.
>>>>
>>>> Not that I am against your idea and we can surely expand as we need but
>>>> we would not need to expand the
>>>> "core_and_task_sdk" if we put only the relevant items into it.
>>>>
>>>> Thanks & Regards,
>>>> Amogh Desai
>>>>
>>>>
>>>> On Tue, Jul 8, 2025 at 12:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>>> > @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
>>>>> there, I am understanding the direction
>>>>> but could you elaborate a bit more please?
>>>>>
>>>>> Let me elaborate:
>>>>>
>>>>> As I understand (maybe I am wrong?),  the proposal is that we have a
>>>>> "core-and-task-sdk" folder which is a shared distribution that is
>>>>> vendored-in into both "airflow-core" and "airflow-task-sdk". This
>>>>> contains some shared code that we want to include in both distributions
>>>>> (note that we never ever release "core-and-task-sdk"
>>>>> distribution because it only contains code that is shared between the two
>>>>> distributions we release.
>>>>>
>>>>> That's fine and cool.
>>>>>
>>>>> Imagine that this distribution contains "logging" (shared code
>>>>> for logging) and "config" (shared code for configuration). - both needed in
>>>>> "airflow-core" and "airflow-task-sdk". So far so good. But what happen if
>>>>> we want to use logging in the same fashion in say "airflow-ctl" (that also
>>>>> applies for other distributions that we might come up with) ? Are we going
>>>>> to vendor in the whole "core-and-task-sdk" distribution in "airflow-ctl" ?
>>>>> It would be far better if we just vendor in "logging" and do not vendor-in
>>>>> "config".
>>>>>
>>>>> And if we are going to have a mechanism to vendor-in "a distribution"
>>>>> - there is nothing wrong with having the same way to vendor-in multiple
>>>>> distributions - so we can easily do it this way (i added "orm" ,
>>>>> "serialization". and "fast_api" as an example thing that we might want to
>>>>> share - not sure if that is really something we want to do but it will
>>>>> allow to illustrate my idea better)
>>>>>
>>>>> /
>>>>>   airflow-ctl/
>>>>>   task-sdk/...
>>>>>   airflow-core/...
>>>>>   ....
>>>>>   shared/
>>>>>     kubernetes/
>>>>>       pyproject.toml
>>>>>       src/
>>>>>         airflow_shared_kubernetes/__init__.py
>>>>>     logging/
>>>>>       pyproject.toml
>>>>>       src/
>>>>>         airflow_shared_logging/__init__.py
>>>>>     config/
>>>>>       pyproject.toml
>>>>>       src/
>>>>>         airflow_shared_config/__init__.py
>>>>>     orm/
>>>>>       pyproject.toml
>>>>>       src/
>>>>>         airflow_shared_orm/__init__.py
>>>>>      serialization/
>>>>>       pyproject.toml
>>>>>       src/
>>>>>         airflow_shared_serialization/__init__.py
>>>>>      fast_api/
>>>>>       pyproject.toml
>>>>>       src/
>>>>>         airflow_shared_fast_api/__init__.py
>>>>>     ...
>>>>>
>>>>> This has multiple benefits (and I see no real drawbacks):
>>>>>
>>>>> * the code can be really well modularised.  Those "things" we share
>>>>> (and this also connects to the __init__.py discussion) - can be independent
>>>>> - and (it follow Jens comment) it allows to keep low cyclomatic complexity
>>>>> https://en.wikipedia.org/wiki/Cyclomatic_complexity . It will be way
>>>>> easier to implement logging in the way that it does not import or use
>>>>> config. This means for example that configuration for logging will need to
>>>>> be injected when logging is initialized - and that's exactly what we want,
>>>>> we do not want logging code to use configuration code directly - they
>>>>> should be independent from each other and you should be free to vendor-in
>>>>> either logging or config independently if you also vendored-in the other.
>>>>>
>>>>> * it's much more logical. We split based on functionality we want to
>>>>> share - not about the "distributions" we want to produce. That allows us -
>>>>> in the future - to make different decisions on how we split our
>>>>> distributions. For example (i do not tell we have to do it, or that we will
>>>>> do it but we will have such a possibility) - we can add more shared
>>>>> utilities we find useful in the same way and decide that "scheduler",
>>>>> "api_server" or "scheduler" or "triggerer" or "dag processor" are split to
>>>>> separate distributions - because for example we want to keep a number of
>>>>> dependencies down. And for example "api_server" might use "fast_api",
>>>>> "config", "logging", "orm" and "fast_api" and "serialization" , where the
>>>>> scheduler should not need "fast_api". The "dag_processor" eventually might
>>>>> not need "orm" nor "fast_api" and only use the other
>>>>>
>>>>> This seems like a natural approach. If we have a mechanism to "share"
>>>>> the code, it does not add complexity, but allows us to isolate independent
>>>>> functionality into "isolated" boxes and use them
>>>>>
>>>>> Also for cyclomatic complexity that is a complex word (badly chosen as
>>>>> it scares people away) and has some math behind, but it really boils down
>>>>> to very simple "rules of thumb" (and yes I am a big proponent of having low
>>>>> cyclomatic complexity).
>>>>>
>>>>> a) when you are building the "final" product (i.e. distribution you
>>>>> want to release) - make sure that you only "use" things - that nothing else
>>>>> is "using" you as a library.
>>>>> b) when you are building a "shared" thing (a library) - make sure that
>>>>> library is only "used" by others but it does not really "use" anything
>>>>> else. For example in the case I explained above - we can achieve
>>>>> low-cyclomatic complexity when:
>>>>>
>>>>> * airflow-core uses: logging, config, orm, serialization, fast_api
>>>>> * none of the "logging, config, orm, serialization, fast_api" use each
>>>>> other - they are at the bottom of the "user -> used" tree
>>>>>
>>>>> J.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jul 8, 2025 at 8:16 AM Amogh Desai <am...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I like the folder structure proposed by Ash and have no objections
>>>>>> with it.
>>>>>>
>>>>>> "core_and_task_sdk" sounds good to me and justifies what it should do
>>>>>> pretty well.
>>>>>>
>>>>>> @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
>>>>>> there, I am understanding the direction
>>>>>> but could you elaborate a bit more please?
>>>>>>
>>>>>> Naming is REALLY hard!
>>>>>>
>>>>>> Thanks & Regards,
>>>>>> Amogh Desai
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 8, 2025 at 2:52 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>
>>>>>>> How about splitting it even more and having each shared "thing"
>>>>>>> named?
>>>>>>> "logging", "config" and sharing them explicitly and separately with
>>>>>>> the
>>>>>>> right "user" ?
>>>>>>> That sounds way more modular and  we will be able to choose which of
>>>>>>> the
>>>>>>> shared "utils" we use where.
>>>>>>>
>>>>>>> J.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jul 7, 2025 at 11:13 PM Jens Scheffler
>>>>>>> <j_...@gmx.de.invalid>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > I like "core_and_task_sdk the same like core-and-task-sdk - I have
>>>>>>> no
>>>>>>> > problem and it is a path only.
>>>>>>> >
>>>>>>> > if we get to "dag-parser-scheduler-task-sdk-and-triggerer" which
>>>>>>> is a
>>>>>>> > bit bulky we then should name it "all-not-api-server" :-D
>>>>>>> >
>>>>>>> > On 07.07.25 22:57, Ash Berlin-Taylor wrote:
>>>>>>> > > In case I did a bad job explaining it, the “core and task sdk”
>>>>>>> is not in
>>>>>>> > the module name/import name, just in the file path.
>>>>>>> > >
>>>>>>> > > Anyone have other ideas?
>>>>>>> > >
>>>>>>> > >> On 7 Jul 2025, at 21:37, Buğra Öztürk <oz...@gmail.com>
>>>>>>> wrote:
>>>>>>> > >>
>>>>>>> > >> Thanks Ash! Looks cool! I like the structure. This will enable
>>>>>>> all the
>>>>>>> > >> combinations and structure looks easy to grasp. No strong
>>>>>>> stance on the
>>>>>>> > >> naming other than maybe it is a bit long with `and`, `core_ctl`
>>>>>>> could be
>>>>>>> > >> shorter, since no import path is defined like that, we can give
>>>>>>> any name
>>>>>>> > >> for sure.
>>>>>>> > >>
>>>>>>> > >> Best regards,
>>>>>>> > >>
>>>>>>> > >>> On Mon, 7 Jul 2025, 21:51 Jarek Potiuk, <ja...@potiuk.com>
>>>>>>> wrote:
>>>>>>> > >>>
>>>>>>> > >>> Looks good but I think we should find some better logical name
>>>>>>> for
>>>>>>> > >>> core_and_sdk :)
>>>>>>> > >>>
>>>>>>> > >>> pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
>>>>>>> > >>> <j_...@gmx.de.invalid> napisał:
>>>>>>> > >>>
>>>>>>> > >>>> Cool! Especially the "shared" folder with the ability to have
>>>>>>> > >>>> N-combinations w/o exploding project repo root!
>>>>>>> > >>>>
>>>>>>> > >>>> On 07.07.25 14:43, Ash Berlin-Taylor wrote:
>>>>>>> > >>>>> Oh, and all of this will be explain in shared/README.md
>>>>>>> > >>>>>
>>>>>>> > >>>>>> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org>
>>>>>>> wrote:
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> Okay, so it seems we have agreement on the approach here,
>>>>>>> so I’ll
>>>>>>> > >>>> continue with this, and on the dev call it was mentioned that
>>>>>>> > >>>> “airflow-common” wasn’t a great name, so here is my proposal
>>>>>>> for the
>>>>>>> > file
>>>>>>> > >>>> structure;
>>>>>>> > >>>>>> ```
>>>>>>> > >>>>>> /
>>>>>>> > >>>>>>   task-sdk/...
>>>>>>> > >>>>>>   airflow-core/...
>>>>>>> > >>>>>>   shared/
>>>>>>> > >>>>>>     kuberenetes/
>>>>>>> > >>>>>>       pyproject.toml
>>>>>>> > >>>>>>       src/
>>>>>>> > >>>>>>         airflow_kube/__init__.py
>>>>>>> > >>>>>>     core-and-tasksdk/
>>>>>>> > >>>>>>       pyproject.toml
>>>>>>> > >>>>>>       src/
>>>>>>> > >>>>>>         airflow_shared/__init__.py
>>>>>>> > >>>>>> ```
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> Things to note here: the “shared” folder has (the
>>>>>>> possibility) of
>>>>>>> > >>>> having multiple different shared “libraries” in it, in this
>>>>>>> example I
>>>>>>> > am
>>>>>>> > >>>> supposing a hypothetical shared kuberenetes folder a world in
>>>>>>> which we
>>>>>>> > >>>> split the KubePodOperator and the KubeExecutor in to two
>>>>>>> separate
>>>>>>> > >>>> distributions (example only, not proposing we do that right
>>>>>>> now, and
>>>>>>> > that
>>>>>>> > >>>> will be a separate discussion)
>>>>>>> > >>>>>> The other things to note here:
>>>>>>> > >>>>>>
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> - the folder name in shared aims to be “self-documenting”,
>>>>>>> hence the
>>>>>>> > >>>> verbose “core-and-tasksdk” to say where the shared library is
>>>>>>> > intended to
>>>>>>> > >>>> be used.
>>>>>>> > >>>>>> - the python module itself should almost always have an
>>>>>>> `airflow_`
>>>>>>> > (or
>>>>>>> > >>>> maybe `_airflow_`?) prefix so that it does not conflict with
>>>>>>> anything
>>>>>>> > >>> else
>>>>>>> > >>>> we might use. It won’t matter “in production” as those will be
>>>>>>> > vendored
>>>>>>> > >>> in
>>>>>>> > >>>> to be imported as `airflow/_vendor/airflow_shared` etc, but
>>>>>>> avoiding
>>>>>>> > >>>> conflicts at dev time with the Finder approach is a good
>>>>>>> safety
>>>>>>> > measure.
>>>>>>> > >>>>>> I will start making a real PR for this proposal now, but
>>>>>>> I’m open to
>>>>>>> > >>>> feedback (either here, or in the PR when I open it)
>>>>>>> > >>>>>> -ash
>>>>>>> > >>>>>>
>>>>>>> > >>>>>>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com>
>>>>>>> wrote:
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>> Yeah we have to try it and test - also building packages
>>>>>>> happens
>>>>>>> > semi
>>>>>>> > >>>>>>> frequently when you run `uv sync` (they use some kind of
>>>>>>> heuristics
>>>>>>> > >>> to
>>>>>>> > >>>>>>> decide when) and you can force it with `--reinstall` or
>>>>>>> > `--refresh`.
>>>>>>> > >>>>>>> Package build also happens every time when you run
>>>>>>> "ci-image build`
>>>>>>> > >>>> now in
>>>>>>> > >>>>>>> breeze so it seems like it will nicely integrate in our
>>>>>>> workflows.
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>> Looks really cool Ash.
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <
>>>>>>> ash@apache.org>
>>>>>>> > >>>> wrote:
>>>>>>> > >>>>>>>> It’s not just release time, but any time we build a
>>>>>>> package which
>>>>>>> > >>>> happens
>>>>>>> > >>>>>>>> on “every” CI run. The normal unit tests will use code
>>>>>>> from
>>>>>>> > >>>>>>>> airflow-common/src/airflow_common; the kube tests which
>>>>>>> build an
>>>>>>> > >>>> image will
>>>>>>> > >>>>>>>> build the dists and vendor in the code from that commit.
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>> There is only a single copy of the shared code committed
>>>>>>> to the
>>>>>>> > >>> repo,
>>>>>>> > >>>> so
>>>>>>> > >>>>>>>> there is never anything to synchronise.
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>>> On 4 Jul 2025, at 15:53, Amogh Desai <
>>>>>>> amoghdesai.oss@gmail.com>
>>>>>>> > >>>> wrote:
>>>>>>> > >>>>>>>>> Thanks Ash.
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> This is really cool and helpful that you were able to
>>>>>>> test both
>>>>>>> > >>>> scenarios
>>>>>>> > >>>>>>>>> -- repo checkout
>>>>>>> > >>>>>>>>> and also installing from the vendored package and the
>>>>>>> resolution
>>>>>>> > >>>> worked
>>>>>>> > >>>>>>>>> fine too.
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> I like this idea compared the to relative import one for
>>>>>>> few
>>>>>>> > >>> reasons:
>>>>>>> > >>>>>>>>> - It feels like it will take some time to adjust to the
>>>>>>> new
>>>>>>> > coding
>>>>>>> > >>>>>>>> standard
>>>>>>> > >>>>>>>>> that we will lay
>>>>>>> > >>>>>>>>> if we impose relative imports in the shared dist
>>>>>>> > >>>>>>>>> - We can continue using repo wise absolute import
>>>>>>> standards, it
>>>>>>> > is
>>>>>>> > >>>> also
>>>>>>> > >>>>>>>>> much easier for situations
>>>>>>> > >>>>>>>>> when we do global search in IDE to find + replace, this
>>>>>>> could
>>>>>>> > mean
>>>>>>> > >>> a
>>>>>>> > >>>>>>>> change
>>>>>>> > >>>>>>>>> there
>>>>>>> > >>>>>>>>> - The vendoring work is a proven and established
>>>>>>> paradigm across
>>>>>>> > >>>> projects
>>>>>>> > >>>>>>>>> and would
>>>>>>> > >>>>>>>>> out of box give us the build tooling we need also
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> Nothing too against the relative import but with the
>>>>>>> evidence
>>>>>>> > >>>> provided
>>>>>>> > >>>>>>>>> above, vendored approach
>>>>>>> > >>>>>>>>> seems to only do us good.
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> Regarding synchronizing it, release time should be fine
>>>>>>> as long
>>>>>>> > as
>>>>>>> > >>> we
>>>>>>> > >>>>>>>> have
>>>>>>> > >>>>>>>>> a good CI workflow to probably
>>>>>>> > >>>>>>>>> catch such issues per PR if changes are made in shared
>>>>>>> dist?
>>>>>>> > >>>> (precommit
>>>>>>> > >>>>>>>>> would make it really slow i guess)
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> If we can run our tests with vendored code we should be
>>>>>>> mostly
>>>>>>> > >>>> covered.
>>>>>>> > >>>>>>>>> Good effort all!
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> Thanks & Regards,
>>>>>>> > >>>>>>>>> Amogh Desai
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <
>>>>>>> > ash@apache.org>
>>>>>>> > >>>>>>>> wrote:
>>>>>>> > >>>>>>>>>> Okay, I think I’ve got something that works and I’m
>>>>>>> happy with.
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>
>>>>>>> >
>>>>>>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>>>>>>> > >>>>>>>>>> This produces the following from `uv build task-sdk`
>>>>>>> > >>>>>>>>>> -
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>
>>>>>>> >
>>>>>>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>>>> > >>>>>>>>>> -
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>
>>>>>>> >
>>>>>>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>>>>>>> > >>>>>>>>>> (`.whl.zip` as GH won't allow .whl upload, but will
>>>>>>> .zip)
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> ```
>>>>>>> > >>>>>>>>>> ❯ unzip -l
>>>>>>> > >>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
>>>>>>> > >>>>>>>> grep
>>>>>>> > >>>>>>>>>> _vendor
>>>>>>> > >>>>>>>>>>      50  02-02-2020 00:00
>>>>>>>  airflow/sdk/_vendor/.gitignore
>>>>>>> > >>>>>>>>>>    2082  02-02-2020 00:00
>>>>>>>  airflow/sdk/_vendor/__init__.py
>>>>>>> > >>>>>>>>>>      28  02-02-2020 00:00
>>>>>>> >  airflow/sdk/_vendor/airflow_common.pyi
>>>>>>> > >>>>>>>>>>      18  02-02-2020 00:00
>>>>>>>  airflow/sdk/_vendor/vendor.txt
>>>>>>> > >>>>>>>>>>     785  02-02-2020 00:00
>>>>>>> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>>> > >>>>>>>>>>   10628  02-02-2020 00:00
>>>>>>> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>>> > >>>>>>>>>> ```
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> And similarly in the .tar.gz, so our “sdist” is
>>>>>>> complete too:
>>>>>>> > >>>>>>>>>> ```
>>>>>>> > >>>>>>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>>>> |grep
>>>>>>> > _vendor
>>>>>>> > >>>>>>>>>>
>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>>>>>>> > >>>>>>>>>>
>>>>>>> > apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>
>>>>>>> >
>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>>>>>>> > >>>>>>>>>>
>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>
>>>>>>> >
>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>>> > >>>
>>>>>>> >
>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>>> > >>>>>>>>>> ```
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> The plugin works at build time by including/copying the
>>>>>>> libs
>>>>>>> > >>>> specified
>>>>>>> > >>>>>>>> in
>>>>>>> > >>>>>>>>>> vendor.txt into place (and let `vendoring` take care of
>>>>>>> import
>>>>>>> > >>>>>>>> rewrites.)
>>>>>>> > >>>>>>>>>> For the imports to continue to work at “dev” time/from
>>>>>>> a repo
>>>>>>> > >>>> checkout,
>>>>>>> > >>>>>>>> I
>>>>>>> > >>>>>>>>>> have added a import finder to `sys.meta_path`, and
>>>>>>> since it’s at
>>>>>>> > >>> the
>>>>>>> > >>>>>>>> end of
>>>>>>> > >>>>>>>>>> the list it will only be used if the normal import
>>>>>>> can’t find
>>>>>>> > >>>> things.
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>
>>>>>>> >
>>>>>>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>>>>>>> > >>>>>>>>>> This doesn’t quite give us the same runtime effect
>>>>>>> “import
>>>>>>> > >>>> rewriting”
>>>>>>> > >>>>>>>>>> affect, as in this approach `airflow_common` is
>>>>>>> directly loaded
>>>>>>> > >>>> (i.e.
>>>>>>> > >>>>>>>>>> airflow.sdk._vendor.airflow_common and airflow_common
>>>>>>> exist in
>>>>>>> > >>>>>>>>>> sys.modules), but it does work for everything that I
>>>>>>> was able to
>>>>>>> > >>>> test..
>>>>>>> > >>>>>>>>>> I tested it with the diff at the end of this message.
>>>>>>> My test
>>>>>>> > >>>> ipython
>>>>>>> > >>>>>>>>>> shell:
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> ```
>>>>>>> > >>>>>>>>>> In [1]: from
>>>>>>> airflow.sdk._vendor.airflow_common.timezone import
>>>>>>> > >>> foo
>>>>>>> > >>>>>>>>>> In [2]: foo
>>>>>>> > >>>>>>>>>> Out[2]: 1
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> In [3]: import airflow.sdk._vendor.airflow_common
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> In [4]: import
>>>>>>> airflow.sdk._vendor.airflow_common.timezone
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>>>>>>> > >>>>>>>>>> Out[5]:
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>
>>>>>>> >
>>>>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>>>>>>> > >>>>>>>>>> In [6]:
>>>>>>> airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>>> > >>>>>>>>>> Out[6]:
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>
>>>>>>> >
>>>>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>>>>>>> > >>>>>>>>>> ```
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> And in an standalone environment with the SDK dist I
>>>>>>> built (it
>>>>>>> > >>>> needed
>>>>>>> > >>>>>>>> the
>>>>>>> > >>>>>>>>>> matching airflow-core right now, but that is nothing to
>>>>>>> do with
>>>>>>> > >>> this
>>>>>>> > >>>>>>>>>> discussion):
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> ```
>>>>>>> > >>>>>>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>>>>>>> > >>>>>>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>>>>>>> > >>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl
>>>>>>> ipython
>>>>>>> > >>>>>>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang
>>>>>>> 18.1.8 ]
>>>>>>> > >>>>>>>>>> Type 'copyright', 'credits' or 'license' for more
>>>>>>> information
>>>>>>> > >>>>>>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type
>>>>>>> '?' for
>>>>>>> > >>> help.
>>>>>>> > >>>>>>>>>> Tip: You can use `%hist` to view history, see the
>>>>>>> options with
>>>>>>> > >>>>>>>> `%history?`
>>>>>>> > >>>>>>>>>> In [1]: import
>>>>>>> airflow.sdk._vendor.airflow_common.timezone
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> In [2]:
>>>>>>> airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>>> > >>>>>>>>>> Out[2]:
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>
>>>>>>> >
>>>>>>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>>>>>>> > >>>>>>>>>> ``
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> ```diff
>>>>>>> > >>>>>>>>>> diff --git
>>>>>>> a/airflow-common/src/airflow_common/__init__.py
>>>>>>> > >>>>>>>>>> b/airflow-common/src/airflow_common/__init__.py
>>>>>>> > >>>>>>>>>> index 13a83393a9..927b7c6b61 100644
>>>>>>> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/__init__.py
>>>>>>> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
>>>>>>> > >>>>>>>>>> @@ -14,3 +14,5 @@
>>>>>>> > >>>>>>>>>> # KIND, either express or implied.  See the License for
>>>>>>> the
>>>>>>> > >>>>>>>>>> # specific language governing permissions and
>>>>>>> limitations
>>>>>>> > >>>>>>>>>> # under the License.
>>>>>>> > >>>>>>>>>> +
>>>>>>> > >>>>>>>>>> +foo = 1
>>>>>>> > >>>>>>>>>> diff --git
>>>>>>> a/airflow-common/src/airflow_common/timezone.py
>>>>>>> > >>>>>>>>>> b/airflow-common/src/airflow_common/timezone.py
>>>>>>> > >>>>>>>>>> index 340b924c66..58384ef20f 100644
>>>>>>> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/timezone.py
>>>>>>> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
>>>>>>> > >>>>>>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>>>>>>> > >>>>>>>>>> version.parse(metadata.version("pendulum")).major == 3
>>>>>>> > >>>>>>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
>>>>>>> > >>>>>>>>>> utc = pendulum.UTC
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> +
>>>>>>> > >>>>>>>>>> +from airflow_common import foo
>>>>>>> > >>>>>>>>>> +
>>>>>>> > >>>>>>>>>> TIMEZONE: Timezone
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> ```
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <
>>>>>>> jarek@potiuk.com>
>>>>>>> > wrote:
>>>>>>> > >>>>>>>>>>> I think both approaches are doable:
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>> 1) -> We can very easily prevent bad imports by
>>>>>>> pre-commit when
>>>>>>> > >>>>>>>> importing
>>>>>>> > >>>>>>>>>>> from different distributions and make sure we are only
>>>>>>> doing
>>>>>>> > >>>> relative
>>>>>>> > >>>>>>>>>>> imports in the shared modules. We are doing plenty of
>>>>>>> this
>>>>>>> > >>>> already. And
>>>>>>> > >>>>>>>>>> yes
>>>>>>> > >>>>>>>>>>> it would require relative links we currently do not
>>>>>>> allow.
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>> 2) -> has one disadvantage that someone at some point
>>>>>>> in time
>>>>>>> > >>> will
>>>>>>> > >>>> have
>>>>>>> > >>>>>>>>>> to
>>>>>>> > >>>>>>>>>>> decide to synchronize this and if it happens just
>>>>>>> before
>>>>>>> > release
>>>>>>> > >>>> (I bet
>>>>>>> > >>>>>>>>>>> this is going to happen) this will lead to solving
>>>>>>> problems
>>>>>>> > that
>>>>>>> > >>>> would
>>>>>>> > >>>>>>>>>>> normally be solved during PR when you make a change
>>>>>>> (i.e.
>>>>>>> > >>> symbolic
>>>>>>> > >>>> link
>>>>>>> > >>>>>>>>>> has
>>>>>>> > >>>>>>>>>>> the advantage that whoever modifies shared code will be
>>>>>>> > >>> immediately
>>>>>>> > >>>>>>>>>>> notified in their PR - that they broke something
>>>>>>> because either
>>>>>>> > >>>> static
>>>>>>> > >>>>>>>>>>> checks or mypy or tests fail.
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>> Ash, do you have an idea of a process (who and when)
>>>>>>> does the
>>>>>>> > >>>>>>>>>>> synchronisation in case of vendoring? Maybe we could
>>>>>>> solve it
>>>>>>> > if
>>>>>>> > >>>> it is
>>>>>>> > >>>>>>>>>> done
>>>>>>> > >>>>>>>>>>> more frequently and with some regularity? We could
>>>>>>> potentially
>>>>>>> > >>>> force
>>>>>>> > >>>>>>>>>>> re-vendoring at PR time as well any time shared code
>>>>>>> changes
>>>>>>> > (and
>>>>>>> > >>>>>>>> prevent
>>>>>>> > >>>>>>>>>>> it by pre-commit. And I can't think of some place
>>>>>>> (other than
>>>>>>> > >>>> releases)
>>>>>>> > >>>>>>>>>> in
>>>>>>> > >>>>>>>>>>> our development workflow and that seems to be a bit
>>>>>>> too late as
>>>>>>> > >>>> puts an
>>>>>>> > >>>>>>>>>>> extra effort on fixing potential incompatibilities
>>>>>>> introduced
>>>>>>> > on
>>>>>>> > >>>>>>>> release
>>>>>>> > >>>>>>>>>>> manager and delays the release. WDYT?
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>> Re: relative links. I think for a shared library we
>>>>>>> could
>>>>>>> > >>>> potentially
>>>>>>> > >>>>>>>>>> relax
>>>>>>> > >>>>>>>>>>> this and allow them (and actually disallow absolute
>>>>>>> links in
>>>>>>> > the
>>>>>>> > >>>> pieces
>>>>>>> > >>>>>>>>>> of
>>>>>>> > >>>>>>>>>>> code that are shared - again, by pre-commit). As I
>>>>>>> recall, the
>>>>>>> > >>> only
>>>>>>> > >>>>>>>>>> reason
>>>>>>> > >>>>>>>>>>> we forbade the relative links is because of how we are
>>>>>>> (or
>>>>>>> > maybe
>>>>>>> > >>>> were)
>>>>>>> > >>>>>>>>>>> doing DAG parsing and failures resulting from it. So
>>>>>>> we decided
>>>>>>> > >>> to
>>>>>>> > >>>> just
>>>>>>> > >>>>>>>>>> not
>>>>>>> > >>>>>>>>>>> allow it to keep consistency. The way how Dag parsing
>>>>>>> works is
>>>>>>> > >>> that
>>>>>>> > >>>>>>>> when
>>>>>>> > >>>>>>>>>>> you are using importlib to read the Dag from a file,
>>>>>>> the
>>>>>>> > relative
>>>>>>> > >>>>>>>> imports
>>>>>>> > >>>>>>>>>>> do not work as it does not know what they should be
>>>>>>> relative
>>>>>>> > to.
>>>>>>> > >>>> But if
>>>>>>> > >>>>>>>>>>> relative import is done from an imported package, it
>>>>>>> should be
>>>>>>> > no
>>>>>>> > >>>>>>>>>> problem,
>>>>>>> > >>>>>>>>>>> I think - otherwise our Dags would not be able to
>>>>>>> import any
>>>>>>> > >>>> library
>>>>>>> > >>>>>>>> that
>>>>>>> > >>>>>>>>>>> uses relative imports.
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>> Of course consistency might be the reason why we do
>>>>>>> not want to
>>>>>>> > >>>>>>>> introduce
>>>>>>> > >>>>>>>>>>> relative imports. I don't see it as an issue if it is
>>>>>>> guarded
>>>>>>> > by
>>>>>>> > >>>>>>>>>> pre-commit
>>>>>>> > >>>>>>>>>>> though.
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>> J.
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>> J.
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
>>>>>>> > >>>> ash@apache.org>
>>>>>>> > >>>>>>>>>>> napisał:
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> Oh yes, symlinks will work, with one big caveat: It
>>>>>>> does mean
>>>>>>> > >>> you
>>>>>>> > >>>>>>>> can’t
>>>>>>> > >>>>>>>>>>>> use absolute imports in one common module to another.
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> For example
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>
>>>>>>> >
>>>>>>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>>>>>>> > >>>>>>>>>>>> where we have
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> ```
>>>>>>> > >>>>>>>>>>>> from airflow.utils.module_loading import import_string
>>>>>>> > >>>>>>>>>>>> ```
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> if we want to move serve_logs into this common lib
>>>>>>> that is
>>>>>>> > then
>>>>>>> > >>>>>>>>>> symlinked
>>>>>>> > >>>>>>>>>>>> then we wouldn’t be able to have `from
>>>>>>> > >>>> airflow_common.module_loading
>>>>>>> > >>>>>>>>>> import
>>>>>>> > >>>>>>>>>>>> import_string`.
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> I can think of two possible solutions here.
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> 1) is to allow/require relative imports in this
>>>>>>> shared lib, so
>>>>>>> > >>>> `from
>>>>>>> > >>>>>>>>>>>> .module_loading import import_string`
>>>>>>> > >>>>>>>>>>>> 2) is to use `vendoring`[1] (from the pip
>>>>>>> maintainers) which
>>>>>>> > >>> will
>>>>>>> > >>>>>>>> handle
>>>>>>> > >>>>>>>>>>>> import-rewriting for us.
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> I’d entirely forgot that symlinks in repos was a
>>>>>>> thing, so I
>>>>>>> > >>>> prepared
>>>>>>> > >>>>>>>> a
>>>>>>> > >>>>>>>>>>>> minimal POC/demo of what vendoring approach could
>>>>>>> look like
>>>>>>> > here
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>
>>>>>>> >
>>>>>>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>>>>>>> > >>>>>>>>>>>> Now personally I am more than happy with relative
>>>>>>> imports, but
>>>>>>> > >>>>>>>> generally
>>>>>>> > >>>>>>>>>>>> as a project we have avoided them, so I think that
>>>>>>> limits what
>>>>>>> > >>> we
>>>>>>> > >>>>>>>> could
>>>>>>> > >>>>>>>>>> do
>>>>>>> > >>>>>>>>>>>> with a symlink based approach.
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> -ash
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> [1] https://github.com/pradyunsg/vendoring
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
>>>>>>> > >>>>>>>> gopidesupavan@gmail.com>
>>>>>>> > >>>>>>>>>>>> wrote:
>>>>>>> > >>>>>>>>>>>>> Thanks Ash
>>>>>>> > >>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>> Yes agree option 2 would be preferred for me. Making
>>>>>>> sure we
>>>>>>> > >>>> have all
>>>>>>> > >>>>>>>>>> the
>>>>>>> > >>>>>>>>>>>>> gaurdriles to protect any unwanted behaviour in code
>>>>>>> sharing
>>>>>>> > >>> and
>>>>>>> > >>>>>>>>>>>> executing
>>>>>>> > >>>>>>>>>>>>> right of tests between the packages.
>>>>>>> > >>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>> Agree with others, option 2 would be
>>>>>>> > >>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
>>>>>>> > >>>>>>>> amoghdesai.oss@gmail.com>
>>>>>>> > >>>>>>>>>>>>> wrote:
>>>>>>> > >>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>> Thanks for starting this discussion, Ash.
>>>>>>> > >>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>> I would prefer option 2 here with proper tooling to
>>>>>>> handle
>>>>>>> > the
>>>>>>> > >>>> code
>>>>>>> > >>>>>>>>>>>>>> duplication at *release* time.
>>>>>>> > >>>>>>>>>>>>>> It is best to have a dist that has all it needs in
>>>>>>> itself.
>>>>>>> > >>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>> Option 1 could very quickly get out of hand and if
>>>>>>> we decide
>>>>>>> > >>> to
>>>>>>> > >>>>>>>>>> separate
>>>>>>> > >>>>>>>>>>>>>> triggerer /
>>>>>>> > >>>>>>>>>>>>>> dag processor / config etc etc as separate
>>>>>>> packages, back
>>>>>>> > >>>> compat is
>>>>>>> > >>>>>>>>>>>> going
>>>>>>> > >>>>>>>>>>>>>> to be a nightmare
>>>>>>> > >>>>>>>>>>>>>> and will bite us harder than we anticipate.
>>>>>>> > >>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>> Thanks & Regards,
>>>>>>> > >>>>>>>>>>>>>> Amogh Desai
>>>>>>> > >>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <
>>>>>>> > >>> kaxilnaik@gmail.com>
>>>>>>> > >>>>>>>>>> wrote:
>>>>>>> > >>>>>>>>>>>>>>> I prefer Option 2 as well to avoid matrix of
>>>>>>> dependencies
>>>>>>> > >>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>>>>>>> > >>>>>>>>>> <j_scheffler@gmx.de.invalid
>>>>>>> > >>>>>>>>>>>>>>> wrote:
>>>>>>> > >>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>> I'd also rather prefer option 2 - reason here is
>>>>>>> it is
>>>>>>> > >>> rather
>>>>>>> > >>>>>>>>>>>> pragmatic
>>>>>>> > >>>>>>>>>>>>>>>> and we no not need to cut another package and
>>>>>>> have less
>>>>>>> > >>>> package
>>>>>>> > >>>>>>>>>> counts
>>>>>>> > >>>>>>>>>>>>>>>> and dependencies.
>>>>>>> > >>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>> I remember some time ago I was checking (together
>>>>>>> with
>>>>>>> > >>> Jarek,
>>>>>>> > >>>> I am
>>>>>>> > >>>>>>>>>> not
>>>>>>> > >>>>>>>>>>>>>>>> sure anymore...) if the usage of symlinks would be
>>>>>>> > possible.
>>>>>>> > >>>> To
>>>>>>> > >>>>>>>> keep
>>>>>>> > >>>>>>>>>>>>>> the
>>>>>>> > >>>>>>>>>>>>>>>> source in one package but "symlink" it into
>>>>>>> another. If
>>>>>>> > then
>>>>>>> > >>>> at
>>>>>>> > >>>>>>>>>> point
>>>>>>> > >>>>>>>>>>>>>> of
>>>>>>> > >>>>>>>>>>>>>>>> packaging/release the files are materialized we
>>>>>>> have 1 set
>>>>>>> > >>> of
>>>>>>> > >>>>>>>> code.
>>>>>>> > >>>>>>>>>>>>>>>> Otherwise if not possible still the redundancy
>>>>>>> could be
>>>>>>> > >>>> solved by
>>>>>>> > >>>>>>>> a
>>>>>>> > >>>>>>>>>>>>>>>> pre-commit hook - and in Git the files are
>>>>>>> de-duplicated
>>>>>>> > >>>> anyway
>>>>>>> > >>>>>>>>>> based
>>>>>>> > >>>>>>>>>>>>>> on
>>>>>>> > >>>>>>>>>>>>>>>> content hash, so this does not hurt.
>>>>>>> > >>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>>>>> > >>>>>>>>>>>>>>>>> I support option 2 with proper automation & CI -
>>>>>>> the
>>>>>>> > >>>> reasonings
>>>>>>> > >>>>>>>>>>>>>> you've
>>>>>>> > >>>>>>>>>>>>>>>>> shown for that make sense to me.
>>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>> Shahar
>>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor
>>>>>>> <
>>>>>>> > >>>> ash@apache.org
>>>>>>> > >>>>>>>>>>>>>>> wrote:
>>>>>>> > >>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> As we work on finishing off the code-level
>>>>>>> separation of
>>>>>>> > >>>> Task
>>>>>>> > >>>>>>>> SDK
>>>>>>> > >>>>>>>>>>>>>> and
>>>>>>> > >>>>>>>>>>>>>>>> Core
>>>>>>> > >>>>>>>>>>>>>>>>>> (scheduler etc) we have come across some
>>>>>>> situations
>>>>>>> > where
>>>>>>> > >>> we
>>>>>>> > >>>>>>>> would
>>>>>>> > >>>>>>>>>>>>>>> like
>>>>>>> > >>>>>>>>>>>>>>>> to
>>>>>>> > >>>>>>>>>>>>>>>>>> share code between these.
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> However it’s not as straight forward of “just
>>>>>>> put it in
>>>>>>> > a
>>>>>>> > >>>> common
>>>>>>> > >>>>>>>>>>>>>> dist
>>>>>>> > >>>>>>>>>>>>>>>> they
>>>>>>> > >>>>>>>>>>>>>>>>>> both depend upon” because one of the goals of
>>>>>>> the Task
>>>>>>> > SDK
>>>>>>> > >>>>>>>>>>>>>> separation
>>>>>>> > >>>>>>>>>>>>>>>> was
>>>>>>> > >>>>>>>>>>>>>>>>>> to have 100% complete version independence
>>>>>>> between the
>>>>>>> > >>> two,
>>>>>>> > >>>>>>>>>> ideally
>>>>>>> > >>>>>>>>>>>>>>>> even if
>>>>>>> > >>>>>>>>>>>>>>>>>> they are built into the same image and venv.
>>>>>>> Most of the
>>>>>>> > >>>> reason
>>>>>>> > >>>>>>>>>> why
>>>>>>> > >>>>>>>>>>>>>>> this
>>>>>>> > >>>>>>>>>>>>>>>>>> isn’t straight forward comes down to backwards
>>>>>>> > >>>> compatibility -
>>>>>>> > >>>>>>>> if
>>>>>>> > >>>>>>>>>> we
>>>>>>> > >>>>>>>>>>>>>>>> make
>>>>>>> > >>>>>>>>>>>>>>>>>> an change to the common/shared distribution
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> We’ve listed the options we have thought about
>>>>>>> in
>>>>>>> > >>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545
>>>>>>> (but
>>>>>>> > that
>>>>>>> > >>>> covers
>>>>>>> > >>>>>>>>>>>>>> some
>>>>>>> > >>>>>>>>>>>>>>>> more
>>>>>>> > >>>>>>>>>>>>>>>>>> things that I don’t want to get in to in this
>>>>>>> discussion
>>>>>>> > >>>> such as
>>>>>>> > >>>>>>>>>>>>>>>> possibly
>>>>>>> > >>>>>>>>>>>>>>>>>> separating operators and executors out of a
>>>>>>> single
>>>>>>> > >>> provider
>>>>>>> > >>>>>>>> dist.)
>>>>>>> > >>>>>>>>>>>>>>>>>> To give a concrete example of some code I would
>>>>>>> like to
>>>>>>> > >>>> share
>>>>>>> > >>>
>>>>>>> >
>>>>>>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>>>>> > >>>>>>>>>>>>>>>>>> — logging config. Another thing we will want to
>>>>>>> share
>>>>>>> > will
>>>>>>> > >>>> be
>>>>>>> > >>>>>>>> the
>>>>>>> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class from
>>>>>>> airflow.configuration
>>>>>>> > (but
>>>>>>> > >>>>>>>> notably:
>>>>>>> > >>>>>>>>>>>>>>> only
>>>>>>> > >>>>>>>>>>>>>>>> the
>>>>>>> > >>>>>>>>>>>>>>>>>> parser class, _not_ the default config values,
>>>>>>> again,
>>>>>>> > lets
>>>>>>> > >>>> not
>>>>>>> > >>>>>>>>>> dwell
>>>>>>> > >>>>>>>>>>>>>>> on
>>>>>>> > >>>>>>>>>>>>>>>> the
>>>>>>> > >>>>>>>>>>>>>>>>>> specifics of that)
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> So to bring the options listed in the issue
>>>>>>> here for
>>>>>>> > >>>> discussion,
>>>>>>> > >>>>>>>>>>>>>>> broadly
>>>>>>> > >>>>>>>>>>>>>>>>>> speaking there are two high-level approaches:
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> 1. A single shared distribution
>>>>>>> > >>>>>>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> The advantage of Approach 1 is that we only
>>>>>>> have the
>>>>>>> > code
>>>>>>> > >>>> in one
>>>>>>> > >>>>>>>>>>>>>>> place.
>>>>>>> > >>>>>>>>>>>>>>>>>> However for me, at least in this specific case
>>>>>>> of
>>>>>>> > Logging
>>>>>>> > >>>> config
>>>>>>> > >>>>>>>>>> or
>>>>>>> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class is that backwards
>>>>>>> > compatibility
>>>>>>> > >>> is
>>>>>>> > >>>>>>>> much
>>>>>>> > >>>>>>>>>>>>>> much
>>>>>>> > >>>>>>>>>>>>>>>>>> harder.
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> The main advantage of Approach 2 is the the
>>>>>>> code is
>>>>>>> > >>> released
>>>>>>> > >>>>>>>>>>>>>>>> with/embedded
>>>>>>> > >>>>>>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would
>>>>>>> contain
>>>>>>> > >>> the
>>>>>>> > >>>>>>>> right
>>>>>>> > >>>>>>>>>>>>>>>> version
>>>>>>> > >>>>>>>>>>>>>>>>>> of the logging config and ConfigParser etc). The
>>>>>>> > downside
>>>>>>> > >>> is
>>>>>>> > >>>>>>>> that
>>>>>>> > >>>>>>>>>>>>>>> either
>>>>>>> > >>>>>>>>>>>>>>>>>> the code will need to be duplicated in the
>>>>>>> repo, or
>>>>>>> > better
>>>>>>> > >>>> yet
>>>>>>> > >>>>>>>> it
>>>>>>> > >>>>>>>>>>>>>>> would
>>>>>>> > >>>>>>>>>>>>>>>>>> live in a single place in the repo, but some
>>>>>>> tooling
>>>>>>> > (TBD)
>>>>>>> > >>>> will
>>>>>>> > >>>>>>>>>>>>>>>>>> automatically handle the duplication, either at
>>>>>>> commit
>>>>>>> > >>>> time, or
>>>>>>> > >>>>>>>> my
>>>>>>> > >>>>>>>>>>>>>>>>>> preference, at release time.
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> For this kind of shared “utility” code I am very
>>>>>>> > strongly
>>>>>>> > >>>>>>>> leaning
>>>>>>> > >>>>>>>>>>>>>>>> towards
>>>>>>> > >>>>>>>>>>>>>>>>>> option 2 with automation, as otherwise I think
>>>>>>> the
>>>>>>> > >>> backwards
>>>>>>> > >>>>>>>>>>>>>>>> compatibility
>>>>>>> > >>>>>>>>>>>>>>>>>> requirements would make it unworkable (very
>>>>>>> quickly over
>>>>>>> > >>>> time
>>>>>>> > >>>>>>>> the
>>>>>>> > >>>>>>>>>>>>>>>>>> combinations we would have to test would just be
>>>>>>> > >>>> unreasonable)
>>>>>>> > >>>>>>>>>> and I
>>>>>>> > >>>>>>>>>>>>>>>> don’t
>>>>>>> > >>>>>>>>>>>>>>>>>> feel confident we can have things as stable as
>>>>>>> we need
>>>>>>> > to
>>>>>>> > >>>> really
>>>>>>> > >>>>>>>>>>>>>>> deliver
>>>>>>> > >>>>>>>>>>>>>>>>>> the version separation/independency I want to
>>>>>>> delivery
>>>>>>> > >>> with
>>>>>>> > >>>>>>>>>> AIP-72.
>>>>>>> > >>>>>>>>>>>>>>>>>> So unless someone feels very strongly about
>>>>>>> this, I will
>>>>>>> > >>>> come up
>>>>>>> > >>>>>>>>>>>>>> with
>>>>>>> > >>>>>>>>>>>>>>> a
>>>>>>> > >>>>>>>>>>>>>>>>>> draft PR for further discussion that will
>>>>>>> implement code
>>>>>>> > >>>> sharing
>>>>>>> > >>>>>>>>>> via
>>>>>>> > >>>>>>>>>>>>>>>>>> “vendoring” it at build time. I have an idea of
>>>>>>> how I
>>>>>>> > can
>>>>>>> > >>>>>>>> achieve
>>>>>>> > >>>>>>>>>>>>>> this
>>>>>>> > >>>>>>>>>>>>>>>> so
>>>>>>> > >>>>>>>>>>>>>>>>>> we have a single version in the repo and it’ll
>>>>>>> work
>>>>>>> > there,
>>>>>>> > >>>> but
>>>>>>> > >>>>>>>> at
>>>>>>> > >>>>>>>>>>>>>>>> runtime
>>>>>>> > >>>>>>>>>>>>>>>>>> we vendor it in to the shipped dist so it lives
>>>>>>> at
>>>>>>> > >>> something
>>>>>>> > >>>>>>>> like
>>>>>>> > >>>>>>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> In terms of repo layout, this likely means we
>>>>>>> would end
>>>>>>> > up
>>>>>>> > >>>> with:
>>>>>>> > >>>>>>>>>>>>>>>>>> airflow-core/pyproject.toml
>>>>>>> > >>>>>>>>>>>>>>>>>> airflow-core/src/
>>>>>>> > >>>>>>>>>>>>>>>>>> airflow-core/tests/
>>>>>>> > >>>>>>>>>>>>>>>>>> task-sdk/pyproject.toml
>>>>>>> > >>>>>>>>>>>>>>>>>> task-sdk/src/
>>>>>>> > >>>>>>>>>>>>>>>>>> task-sdk/tests/
>>>>>>> > >>>>>>>>>>>>>>>>>> airflow-common/src
>>>>>>> > >>>>>>>>>>>>>>>>>> airflow-common/tests/
>>>>>>> > >>>>>>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as
>>>>>>> deps
>>>>>>> > would
>>>>>>> > >>>> be
>>>>>>> > >>>>>>>>>>>>>> included
>>>>>>> > >>>>>>>>>>>>>>>> in
>>>>>>> > >>>>>>>>>>>>>>>>>> the downstream projects. TBD.
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> Thoughts and feedback welcomed.
>>>>>>> > >>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>> > dev-unsubscribe@airflow.apache.org
>>>>>>> > >>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>> > >>> dev-help@airflow.apache.org
>>>>>>> > >>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>
>>>>>>> > >>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> > >>>>>>>> To unsubscribe, e-mail:
>>>>>>> dev-unsubscribe@airflow.apache.org
>>>>>>> > >>>>>>>> For additional commands, e-mail:
>>>>>>> dev-help@airflow.apache.org
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>
>>>>>>> >
>>>>>>> ---------------------------------------------------------------------
>>>>>>> > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>> > >>>>>> For additional commands, e-mail:
>>>>>>> dev-help@airflow.apache.org
>>>>>>> > >>>>>>
>>>>>>> > >>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> > >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>> > >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>> > >>>>>
>>>>>>> > >>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> > >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>> > >>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>> > >>>>
>>>>>>> > >>>>
>>>>>>> > >
>>>>>>> > >
>>>>>>> ---------------------------------------------------------------------
>>>>>>> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>> > > For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>> > >
>>>>>>> >
>>>>>>> >
>>>>>>> ---------------------------------------------------------------------
>>>>>>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>> > For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Jarek Potiuk <ja...@potiuk.com>.
Yeah. I think extracting one-by-one, feature-by-feature that we want to
share to a separate distribution is the best approach - it will actually
also help with the "__init__.py" cleanup - because almost by definition -
those distributions will not be able to "reach" outside - i.e. they only
can be "used" not "use" something else. Which means that - for example as
it is now - configuration using logging which is using configuration
(leading to circular dependencies and partially initialized modules) will
not happen and we will have to figure out rather how to inject
configuration into logging (from task-sdk, airflow-ctl, airflow-core) and
to get the right sequence of initialization - rather than have the
inter-feature-dependencies.

And that is precisely what low-cyclomatic complexity is about as well - it
generally leads to easier-to-maintain software that has well defined
functionality and does not have those weird circular dependencies we have
now. That's a kind-of side-effect of such a "per feature" split, but a very
desirable one.

J.

On Wed, Jul 9, 2025 at 10:17 AM Amogh Desai <am...@gmail.com>
wrote:

> Probably, you make a valid point.
>
> Maybe this is an implementation detail, so we could figure it out as we
> start on a POC and factor in these things
> as we move along?
>
> But from an initial guess, I would think that execution time related items
> (if we manage to enumerate them) would be something
> that would be better off in that "core_and_task_sdk" bundle.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Tue, Jul 8, 2025 at 3:46 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> > Not that I am against your idea and we can surely expand as we need but
>> we would not need to expand the
>> "core_and_task_sdk" if we put only the relevant items into it.
>>
>> So if we move logging and config out, my question is what is really
>> relevant to "stay" in "core_and_task_sdk" ? And what we know will not be
>> needed by other distributions in the future ?
>>
>> What would be the content of "core_and_task_sdk" ? Can we enumerate what
>> should go there now ?
>>
>> My bet is that if we enumerate them, it will turn out that they are good
>> candidates to make separate "shared features" that can be logically named
>> and modularised, and that there is no real need to have shared
>> "core_and_task_sdk" which sounds like "bag of everything else".
>>
>>
>>
>> On Tue, Jul 8, 2025 at 11:48 AM Amogh Desai <am...@gmail.com>
>> wrote:
>>
>>> Yeah, I think what you are showcasing here is a step ahead of the
>>> initial proposal from Ash.
>>>
>>> From the original proposal, the `core_and_task_sdk` *can* have the
>>> things relevant to just those two
>>> distros. Logging, Config are modules that might be needed by airflow-ctl
>>> for example, so ideally, those
>>> would not be good candidates to be put in there, ideally speaking.
>>>
>>> The example of Kubernetes Utils sounds to be a good example, it will be
>>> used by KubeExecutor (lets say this is a
>>> module called "executors") and by KPO (providers), and the
>>> "shared/kubernetes" would probably be a good
>>> candidate for that.
>>>
>>> Not that I am against your idea and we can surely expand as we need but
>>> we would not need to expand the
>>> "core_and_task_sdk" if we put only the relevant items into it.
>>>
>>> Thanks & Regards,
>>> Amogh Desai
>>>
>>>
>>> On Tue, Jul 8, 2025 at 12:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>> > @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
>>>> there, I am understanding the direction
>>>> but could you elaborate a bit more please?
>>>>
>>>> Let me elaborate:
>>>>
>>>> As I understand (maybe I am wrong?),  the proposal is that we have a
>>>> "core-and-task-sdk" folder which is a shared distribution that is
>>>> vendored-in into both "airflow-core" and "airflow-task-sdk". This
>>>> contains some shared code that we want to include in both distributions
>>>> (note that we never ever release "core-and-task-sdk"
>>>> distribution because it only contains code that is shared between the two
>>>> distributions we release.
>>>>
>>>> That's fine and cool.
>>>>
>>>> Imagine that this distribution contains "logging" (shared code
>>>> for logging) and "config" (shared code for configuration). - both needed in
>>>> "airflow-core" and "airflow-task-sdk". So far so good. But what happen if
>>>> we want to use logging in the same fashion in say "airflow-ctl" (that also
>>>> applies for other distributions that we might come up with) ? Are we going
>>>> to vendor in the whole "core-and-task-sdk" distribution in "airflow-ctl" ?
>>>> It would be far better if we just vendor in "logging" and do not vendor-in
>>>> "config".
>>>>
>>>> And if we are going to have a mechanism to vendor-in "a distribution" -
>>>> there is nothing wrong with having the same way to vendor-in multiple
>>>> distributions - so we can easily do it this way (i added "orm" ,
>>>> "serialization". and "fast_api" as an example thing that we might want to
>>>> share - not sure if that is really something we want to do but it will
>>>> allow to illustrate my idea better)
>>>>
>>>> /
>>>>   airflow-ctl/
>>>>   task-sdk/...
>>>>   airflow-core/...
>>>>   ....
>>>>   shared/
>>>>     kubernetes/
>>>>       pyproject.toml
>>>>       src/
>>>>         airflow_shared_kubernetes/__init__.py
>>>>     logging/
>>>>       pyproject.toml
>>>>       src/
>>>>         airflow_shared_logging/__init__.py
>>>>     config/
>>>>       pyproject.toml
>>>>       src/
>>>>         airflow_shared_config/__init__.py
>>>>     orm/
>>>>       pyproject.toml
>>>>       src/
>>>>         airflow_shared_orm/__init__.py
>>>>      serialization/
>>>>       pyproject.toml
>>>>       src/
>>>>         airflow_shared_serialization/__init__.py
>>>>      fast_api/
>>>>       pyproject.toml
>>>>       src/
>>>>         airflow_shared_fast_api/__init__.py
>>>>     ...
>>>>
>>>> This has multiple benefits (and I see no real drawbacks):
>>>>
>>>> * the code can be really well modularised.  Those "things" we share
>>>> (and this also connects to the __init__.py discussion) - can be independent
>>>> - and (it follow Jens comment) it allows to keep low cyclomatic complexity
>>>> https://en.wikipedia.org/wiki/Cyclomatic_complexity . It will be way
>>>> easier to implement logging in the way that it does not import or use
>>>> config. This means for example that configuration for logging will need to
>>>> be injected when logging is initialized - and that's exactly what we want,
>>>> we do not want logging code to use configuration code directly - they
>>>> should be independent from each other and you should be free to vendor-in
>>>> either logging or config independently if you also vendored-in the other.
>>>>
>>>> * it's much more logical. We split based on functionality we want to
>>>> share - not about the "distributions" we want to produce. That allows us -
>>>> in the future - to make different decisions on how we split our
>>>> distributions. For example (i do not tell we have to do it, or that we will
>>>> do it but we will have such a possibility) - we can add more shared
>>>> utilities we find useful in the same way and decide that "scheduler",
>>>> "api_server" or "scheduler" or "triggerer" or "dag processor" are split to
>>>> separate distributions - because for example we want to keep a number of
>>>> dependencies down. And for example "api_server" might use "fast_api",
>>>> "config", "logging", "orm" and "fast_api" and "serialization" , where the
>>>> scheduler should not need "fast_api". The "dag_processor" eventually might
>>>> not need "orm" nor "fast_api" and only use the other
>>>>
>>>> This seems like a natural approach. If we have a mechanism to "share"
>>>> the code, it does not add complexity, but allows us to isolate independent
>>>> functionality into "isolated" boxes and use them
>>>>
>>>> Also for cyclomatic complexity that is a complex word (badly chosen as
>>>> it scares people away) and has some math behind, but it really boils down
>>>> to very simple "rules of thumb" (and yes I am a big proponent of having low
>>>> cyclomatic complexity).
>>>>
>>>> a) when you are building the "final" product (i.e. distribution you
>>>> want to release) - make sure that you only "use" things - that nothing else
>>>> is "using" you as a library.
>>>> b) when you are building a "shared" thing (a library) - make sure that
>>>> library is only "used" by others but it does not really "use" anything
>>>> else. For example in the case I explained above - we can achieve
>>>> low-cyclomatic complexity when:
>>>>
>>>> * airflow-core uses: logging, config, orm, serialization, fast_api
>>>> * none of the "logging, config, orm, serialization, fast_api" use each
>>>> other - they are at the bottom of the "user -> used" tree
>>>>
>>>> J.
>>>>
>>>>
>>>>
>>>> On Tue, Jul 8, 2025 at 8:16 AM Amogh Desai <am...@gmail.com>
>>>> wrote:
>>>>
>>>>> I like the folder structure proposed by Ash and have no objections
>>>>> with it.
>>>>>
>>>>> "core_and_task_sdk" sounds good to me and justifies what it should do
>>>>> pretty well.
>>>>>
>>>>> @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
>>>>> there, I am understanding the direction
>>>>> but could you elaborate a bit more please?
>>>>>
>>>>> Naming is REALLY hard!
>>>>>
>>>>> Thanks & Regards,
>>>>> Amogh Desai
>>>>>
>>>>>
>>>>> On Tue, Jul 8, 2025 at 2:52 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>
>>>>>> How about splitting it even more and having each shared "thing" named?
>>>>>> "logging", "config" and sharing them explicitly and separately with
>>>>>> the
>>>>>> right "user" ?
>>>>>> That sounds way more modular and  we will be able to choose which of
>>>>>> the
>>>>>> shared "utils" we use where.
>>>>>>
>>>>>> J.
>>>>>>
>>>>>>
>>>>>> On Mon, Jul 7, 2025 at 11:13 PM Jens Scheffler
>>>>>> <j_...@gmx.de.invalid>
>>>>>> wrote:
>>>>>>
>>>>>> > I like "core_and_task_sdk the same like core-and-task-sdk - I have
>>>>>> no
>>>>>> > problem and it is a path only.
>>>>>> >
>>>>>> > if we get to "dag-parser-scheduler-task-sdk-and-triggerer" which is
>>>>>> a
>>>>>> > bit bulky we then should name it "all-not-api-server" :-D
>>>>>> >
>>>>>> > On 07.07.25 22:57, Ash Berlin-Taylor wrote:
>>>>>> > > In case I did a bad job explaining it, the “core and task sdk” is
>>>>>> not in
>>>>>> > the module name/import name, just in the file path.
>>>>>> > >
>>>>>> > > Anyone have other ideas?
>>>>>> > >
>>>>>> > >> On 7 Jul 2025, at 21:37, Buğra Öztürk <oz...@gmail.com>
>>>>>> wrote:
>>>>>> > >>
>>>>>> > >> Thanks Ash! Looks cool! I like the structure. This will enable
>>>>>> all the
>>>>>> > >> combinations and structure looks easy to grasp. No strong stance
>>>>>> on the
>>>>>> > >> naming other than maybe it is a bit long with `and`, `core_ctl`
>>>>>> could be
>>>>>> > >> shorter, since no import path is defined like that, we can give
>>>>>> any name
>>>>>> > >> for sure.
>>>>>> > >>
>>>>>> > >> Best regards,
>>>>>> > >>
>>>>>> > >>> On Mon, 7 Jul 2025, 21:51 Jarek Potiuk, <ja...@potiuk.com>
>>>>>> wrote:
>>>>>> > >>>
>>>>>> > >>> Looks good but I think we should find some better logical name
>>>>>> for
>>>>>> > >>> core_and_sdk :)
>>>>>> > >>>
>>>>>> > >>> pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
>>>>>> > >>> <j_...@gmx.de.invalid> napisał:
>>>>>> > >>>
>>>>>> > >>>> Cool! Especially the "shared" folder with the ability to have
>>>>>> > >>>> N-combinations w/o exploding project repo root!
>>>>>> > >>>>
>>>>>> > >>>> On 07.07.25 14:43, Ash Berlin-Taylor wrote:
>>>>>> > >>>>> Oh, and all of this will be explain in shared/README.md
>>>>>> > >>>>>
>>>>>> > >>>>>> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org>
>>>>>> wrote:
>>>>>> > >>>>>>
>>>>>> > >>>>>> Okay, so it seems we have agreement on the approach here, so
>>>>>> I’ll
>>>>>> > >>>> continue with this, and on the dev call it was mentioned that
>>>>>> > >>>> “airflow-common” wasn’t a great name, so here is my proposal
>>>>>> for the
>>>>>> > file
>>>>>> > >>>> structure;
>>>>>> > >>>>>> ```
>>>>>> > >>>>>> /
>>>>>> > >>>>>>   task-sdk/...
>>>>>> > >>>>>>   airflow-core/...
>>>>>> > >>>>>>   shared/
>>>>>> > >>>>>>     kuberenetes/
>>>>>> > >>>>>>       pyproject.toml
>>>>>> > >>>>>>       src/
>>>>>> > >>>>>>         airflow_kube/__init__.py
>>>>>> > >>>>>>     core-and-tasksdk/
>>>>>> > >>>>>>       pyproject.toml
>>>>>> > >>>>>>       src/
>>>>>> > >>>>>>         airflow_shared/__init__.py
>>>>>> > >>>>>> ```
>>>>>> > >>>>>>
>>>>>> > >>>>>> Things to note here: the “shared” folder has (the
>>>>>> possibility) of
>>>>>> > >>>> having multiple different shared “libraries” in it, in this
>>>>>> example I
>>>>>> > am
>>>>>> > >>>> supposing a hypothetical shared kuberenetes folder a world in
>>>>>> which we
>>>>>> > >>>> split the KubePodOperator and the KubeExecutor in to two
>>>>>> separate
>>>>>> > >>>> distributions (example only, not proposing we do that right
>>>>>> now, and
>>>>>> > that
>>>>>> > >>>> will be a separate discussion)
>>>>>> > >>>>>> The other things to note here:
>>>>>> > >>>>>>
>>>>>> > >>>>>>
>>>>>> > >>>>>> - the folder name in shared aims to be “self-documenting”,
>>>>>> hence the
>>>>>> > >>>> verbose “core-and-tasksdk” to say where the shared library is
>>>>>> > intended to
>>>>>> > >>>> be used.
>>>>>> > >>>>>> - the python module itself should almost always have an
>>>>>> `airflow_`
>>>>>> > (or
>>>>>> > >>>> maybe `_airflow_`?) prefix so that it does not conflict with
>>>>>> anything
>>>>>> > >>> else
>>>>>> > >>>> we might use. It won’t matter “in production” as those will be
>>>>>> > vendored
>>>>>> > >>> in
>>>>>> > >>>> to be imported as `airflow/_vendor/airflow_shared` etc, but
>>>>>> avoiding
>>>>>> > >>>> conflicts at dev time with the Finder approach is a good safety
>>>>>> > measure.
>>>>>> > >>>>>> I will start making a real PR for this proposal now, but I’m
>>>>>> open to
>>>>>> > >>>> feedback (either here, or in the PR when I open it)
>>>>>> > >>>>>> -ash
>>>>>> > >>>>>>
>>>>>> > >>>>>>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com>
>>>>>> wrote:
>>>>>> > >>>>>>>
>>>>>> > >>>>>>> Yeah we have to try it and test - also building packages
>>>>>> happens
>>>>>> > semi
>>>>>> > >>>>>>> frequently when you run `uv sync` (they use some kind of
>>>>>> heuristics
>>>>>> > >>> to
>>>>>> > >>>>>>> decide when) and you can force it with `--reinstall` or
>>>>>> > `--refresh`.
>>>>>> > >>>>>>> Package build also happens every time when you run
>>>>>> "ci-image build`
>>>>>> > >>>> now in
>>>>>> > >>>>>>> breeze so it seems like it will nicely integrate in our
>>>>>> workflows.
>>>>>> > >>>>>>>
>>>>>> > >>>>>>> Looks really cool Ash.
>>>>>> > >>>>>>>
>>>>>> > >>>>>>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <
>>>>>> ash@apache.org>
>>>>>> > >>>> wrote:
>>>>>> > >>>>>>>> It’s not just release time, but any time we build a
>>>>>> package which
>>>>>> > >>>> happens
>>>>>> > >>>>>>>> on “every” CI run. The normal unit tests will use code from
>>>>>> > >>>>>>>> airflow-common/src/airflow_common; the kube tests which
>>>>>> build an
>>>>>> > >>>> image will
>>>>>> > >>>>>>>> build the dists and vendor in the code from that commit.
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>> There is only a single copy of the shared code committed
>>>>>> to the
>>>>>> > >>> repo,
>>>>>> > >>>> so
>>>>>> > >>>>>>>> there is never anything to synchronise.
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>>> On 4 Jul 2025, at 15:53, Amogh Desai <
>>>>>> amoghdesai.oss@gmail.com>
>>>>>> > >>>> wrote:
>>>>>> > >>>>>>>>> Thanks Ash.
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>> This is really cool and helpful that you were able to
>>>>>> test both
>>>>>> > >>>> scenarios
>>>>>> > >>>>>>>>> -- repo checkout
>>>>>> > >>>>>>>>> and also installing from the vendored package and the
>>>>>> resolution
>>>>>> > >>>> worked
>>>>>> > >>>>>>>>> fine too.
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>> I like this idea compared the to relative import one for
>>>>>> few
>>>>>> > >>> reasons:
>>>>>> > >>>>>>>>> - It feels like it will take some time to adjust to the
>>>>>> new
>>>>>> > coding
>>>>>> > >>>>>>>> standard
>>>>>> > >>>>>>>>> that we will lay
>>>>>> > >>>>>>>>> if we impose relative imports in the shared dist
>>>>>> > >>>>>>>>> - We can continue using repo wise absolute import
>>>>>> standards, it
>>>>>> > is
>>>>>> > >>>> also
>>>>>> > >>>>>>>>> much easier for situations
>>>>>> > >>>>>>>>> when we do global search in IDE to find + replace, this
>>>>>> could
>>>>>> > mean
>>>>>> > >>> a
>>>>>> > >>>>>>>> change
>>>>>> > >>>>>>>>> there
>>>>>> > >>>>>>>>> - The vendoring work is a proven and established paradigm
>>>>>> across
>>>>>> > >>>> projects
>>>>>> > >>>>>>>>> and would
>>>>>> > >>>>>>>>> out of box give us the build tooling we need also
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>> Nothing too against the relative import but with the
>>>>>> evidence
>>>>>> > >>>> provided
>>>>>> > >>>>>>>>> above, vendored approach
>>>>>> > >>>>>>>>> seems to only do us good.
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>> Regarding synchronizing it, release time should be fine
>>>>>> as long
>>>>>> > as
>>>>>> > >>> we
>>>>>> > >>>>>>>> have
>>>>>> > >>>>>>>>> a good CI workflow to probably
>>>>>> > >>>>>>>>> catch such issues per PR if changes are made in shared
>>>>>> dist?
>>>>>> > >>>> (precommit
>>>>>> > >>>>>>>>> would make it really slow i guess)
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>> If we can run our tests with vendored code we should be
>>>>>> mostly
>>>>>> > >>>> covered.
>>>>>> > >>>>>>>>> Good effort all!
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>> Thanks & Regards,
>>>>>> > >>>>>>>>> Amogh Desai
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>>
>>>>>> > >>>>>>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <
>>>>>> > ash@apache.org>
>>>>>> > >>>>>>>> wrote:
>>>>>> > >>>>>>>>>> Okay, I think I’ve got something that works and I’m
>>>>>> happy with.
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>
>>>>>> >
>>>>>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>>>>>> > >>>>>>>>>> This produces the following from `uv build task-sdk`
>>>>>> > >>>>>>>>>> -
>>>>>> > >>>>>>>>>>
>>>>>> > >>>
>>>>>> >
>>>>>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>>> > >>>>>>>>>> -
>>>>>> > >>>>>>>>>>
>>>>>> > >>>
>>>>>> >
>>>>>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>>>>>> > >>>>>>>>>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> ```
>>>>>> > >>>>>>>>>> ❯ unzip -l
>>>>>> > >>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
>>>>>> > >>>>>>>> grep
>>>>>> > >>>>>>>>>> _vendor
>>>>>> > >>>>>>>>>>      50  02-02-2020 00:00
>>>>>>  airflow/sdk/_vendor/.gitignore
>>>>>> > >>>>>>>>>>    2082  02-02-2020 00:00
>>>>>>  airflow/sdk/_vendor/__init__.py
>>>>>> > >>>>>>>>>>      28  02-02-2020 00:00
>>>>>> >  airflow/sdk/_vendor/airflow_common.pyi
>>>>>> > >>>>>>>>>>      18  02-02-2020 00:00
>>>>>>  airflow/sdk/_vendor/vendor.txt
>>>>>> > >>>>>>>>>>     785  02-02-2020 00:00
>>>>>> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>> > >>>>>>>>>>   10628  02-02-2020 00:00
>>>>>> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>> > >>>>>>>>>> ```
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> And similarly in the .tar.gz, so our “sdist” is complete
>>>>>> too:
>>>>>> > >>>>>>>>>> ```
>>>>>> > >>>>>>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>>> |grep
>>>>>> > _vendor
>>>>>> > >>>>>>>>>>
>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>>>>>> > >>>>>>>>>>
>>>>>> > apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>
>>>>>> >
>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>>>>>> > >>>>>>>>>>
>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>
>>>>>> >
>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>> > >>>
>>>>>> >
>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>> > >>>>>>>>>> ```
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> The plugin works at build time by including/copying the
>>>>>> libs
>>>>>> > >>>> specified
>>>>>> > >>>>>>>> in
>>>>>> > >>>>>>>>>> vendor.txt into place (and let `vendoring` take care of
>>>>>> import
>>>>>> > >>>>>>>> rewrites.)
>>>>>> > >>>>>>>>>> For the imports to continue to work at “dev” time/from a
>>>>>> repo
>>>>>> > >>>> checkout,
>>>>>> > >>>>>>>> I
>>>>>> > >>>>>>>>>> have added a import finder to `sys.meta_path`, and since
>>>>>> it’s at
>>>>>> > >>> the
>>>>>> > >>>>>>>> end of
>>>>>> > >>>>>>>>>> the list it will only be used if the normal import can’t
>>>>>> find
>>>>>> > >>>> things.
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>
>>>>>> >
>>>>>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>>>>>> > >>>>>>>>>> This doesn’t quite give us the same runtime effect
>>>>>> “import
>>>>>> > >>>> rewriting”
>>>>>> > >>>>>>>>>> affect, as in this approach `airflow_common` is directly
>>>>>> loaded
>>>>>> > >>>> (i.e.
>>>>>> > >>>>>>>>>> airflow.sdk._vendor.airflow_common and airflow_common
>>>>>> exist in
>>>>>> > >>>>>>>>>> sys.modules), but it does work for everything that I was
>>>>>> able to
>>>>>> > >>>> test..
>>>>>> > >>>>>>>>>> I tested it with the diff at the end of this message. My
>>>>>> test
>>>>>> > >>>> ipython
>>>>>> > >>>>>>>>>> shell:
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> ```
>>>>>> > >>>>>>>>>> In [1]: from airflow.sdk._vendor.airflow_common.timezone
>>>>>> import
>>>>>> > >>> foo
>>>>>> > >>>>>>>>>> In [2]: foo
>>>>>> > >>>>>>>>>> Out[2]: 1
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> In [3]: import airflow.sdk._vendor.airflow_common
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> In [4]: import
>>>>>> airflow.sdk._vendor.airflow_common.timezone
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>>>>>> > >>>>>>>>>> Out[5]:
>>>>>> > >>>>>>>>>>
>>>>>> > >>>
>>>>>> >
>>>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>>>>>> > >>>>>>>>>> In [6]:
>>>>>> airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>> > >>>>>>>>>> Out[6]:
>>>>>> > >>>>>>>>>>
>>>>>> > >>>
>>>>>> >
>>>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>>>>>> > >>>>>>>>>> ```
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> And in an standalone environment with the SDK dist I
>>>>>> built (it
>>>>>> > >>>> needed
>>>>>> > >>>>>>>> the
>>>>>> > >>>>>>>>>> matching airflow-core right now, but that is nothing to
>>>>>> do with
>>>>>> > >>> this
>>>>>> > >>>>>>>>>> discussion):
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> ```
>>>>>> > >>>>>>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>>>>>> > >>>>>>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>>>>>> > >>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl
>>>>>> ipython
>>>>>> > >>>>>>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang
>>>>>> 18.1.8 ]
>>>>>> > >>>>>>>>>> Type 'copyright', 'credits' or 'license' for more
>>>>>> information
>>>>>> > >>>>>>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type
>>>>>> '?' for
>>>>>> > >>> help.
>>>>>> > >>>>>>>>>> Tip: You can use `%hist` to view history, see the
>>>>>> options with
>>>>>> > >>>>>>>> `%history?`
>>>>>> > >>>>>>>>>> In [1]: import
>>>>>> airflow.sdk._vendor.airflow_common.timezone
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> In [2]:
>>>>>> airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>> > >>>>>>>>>> Out[2]:
>>>>>> > >>>>>>>>>>
>>>>>> > >>>
>>>>>> >
>>>>>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>>>>>> > >>>>>>>>>> ``
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> ```diff
>>>>>> > >>>>>>>>>> diff --git
>>>>>> a/airflow-common/src/airflow_common/__init__.py
>>>>>> > >>>>>>>>>> b/airflow-common/src/airflow_common/__init__.py
>>>>>> > >>>>>>>>>> index 13a83393a9..927b7c6b61 100644
>>>>>> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/__init__.py
>>>>>> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
>>>>>> > >>>>>>>>>> @@ -14,3 +14,5 @@
>>>>>> > >>>>>>>>>> # KIND, either express or implied.  See the License for
>>>>>> the
>>>>>> > >>>>>>>>>> # specific language governing permissions and limitations
>>>>>> > >>>>>>>>>> # under the License.
>>>>>> > >>>>>>>>>> +
>>>>>> > >>>>>>>>>> +foo = 1
>>>>>> > >>>>>>>>>> diff --git
>>>>>> a/airflow-common/src/airflow_common/timezone.py
>>>>>> > >>>>>>>>>> b/airflow-common/src/airflow_common/timezone.py
>>>>>> > >>>>>>>>>> index 340b924c66..58384ef20f 100644
>>>>>> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/timezone.py
>>>>>> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
>>>>>> > >>>>>>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>>>>>> > >>>>>>>>>> version.parse(metadata.version("pendulum")).major == 3
>>>>>> > >>>>>>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
>>>>>> > >>>>>>>>>> utc = pendulum.UTC
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> +
>>>>>> > >>>>>>>>>> +from airflow_common import foo
>>>>>> > >>>>>>>>>> +
>>>>>> > >>>>>>>>>> TIMEZONE: Timezone
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>> ```
>>>>>> > >>>>>>>>>>
>>>>>> > >>>>>>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <
>>>>>> jarek@potiuk.com>
>>>>>> > wrote:
>>>>>> > >>>>>>>>>>> I think both approaches are doable:
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>> 1) -> We can very easily prevent bad imports by
>>>>>> pre-commit when
>>>>>> > >>>>>>>> importing
>>>>>> > >>>>>>>>>>> from different distributions and make sure we are only
>>>>>> doing
>>>>>> > >>>> relative
>>>>>> > >>>>>>>>>>> imports in the shared modules. We are doing plenty of
>>>>>> this
>>>>>> > >>>> already. And
>>>>>> > >>>>>>>>>> yes
>>>>>> > >>>>>>>>>>> it would require relative links we currently do not
>>>>>> allow.
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>> 2) -> has one disadvantage that someone at some point
>>>>>> in time
>>>>>> > >>> will
>>>>>> > >>>> have
>>>>>> > >>>>>>>>>> to
>>>>>> > >>>>>>>>>>> decide to synchronize this and if it happens just before
>>>>>> > release
>>>>>> > >>>> (I bet
>>>>>> > >>>>>>>>>>> this is going to happen) this will lead to solving
>>>>>> problems
>>>>>> > that
>>>>>> > >>>> would
>>>>>> > >>>>>>>>>>> normally be solved during PR when you make a change
>>>>>> (i.e.
>>>>>> > >>> symbolic
>>>>>> > >>>> link
>>>>>> > >>>>>>>>>> has
>>>>>> > >>>>>>>>>>> the advantage that whoever modifies shared code will be
>>>>>> > >>> immediately
>>>>>> > >>>>>>>>>>> notified in their PR - that they broke something
>>>>>> because either
>>>>>> > >>>> static
>>>>>> > >>>>>>>>>>> checks or mypy or tests fail.
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>> Ash, do you have an idea of a process (who and when)
>>>>>> does the
>>>>>> > >>>>>>>>>>> synchronisation in case of vendoring? Maybe we could
>>>>>> solve it
>>>>>> > if
>>>>>> > >>>> it is
>>>>>> > >>>>>>>>>> done
>>>>>> > >>>>>>>>>>> more frequently and with some regularity? We could
>>>>>> potentially
>>>>>> > >>>> force
>>>>>> > >>>>>>>>>>> re-vendoring at PR time as well any time shared code
>>>>>> changes
>>>>>> > (and
>>>>>> > >>>>>>>> prevent
>>>>>> > >>>>>>>>>>> it by pre-commit. And I can't think of some place
>>>>>> (other than
>>>>>> > >>>> releases)
>>>>>> > >>>>>>>>>> in
>>>>>> > >>>>>>>>>>> our development workflow and that seems to be a bit too
>>>>>> late as
>>>>>> > >>>> puts an
>>>>>> > >>>>>>>>>>> extra effort on fixing potential incompatibilities
>>>>>> introduced
>>>>>> > on
>>>>>> > >>>>>>>> release
>>>>>> > >>>>>>>>>>> manager and delays the release. WDYT?
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>> Re: relative links. I think for a shared library we
>>>>>> could
>>>>>> > >>>> potentially
>>>>>> > >>>>>>>>>> relax
>>>>>> > >>>>>>>>>>> this and allow them (and actually disallow absolute
>>>>>> links in
>>>>>> > the
>>>>>> > >>>> pieces
>>>>>> > >>>>>>>>>> of
>>>>>> > >>>>>>>>>>> code that are shared - again, by pre-commit). As I
>>>>>> recall, the
>>>>>> > >>> only
>>>>>> > >>>>>>>>>> reason
>>>>>> > >>>>>>>>>>> we forbade the relative links is because of how we are
>>>>>> (or
>>>>>> > maybe
>>>>>> > >>>> were)
>>>>>> > >>>>>>>>>>> doing DAG parsing and failures resulting from it. So we
>>>>>> decided
>>>>>> > >>> to
>>>>>> > >>>> just
>>>>>> > >>>>>>>>>> not
>>>>>> > >>>>>>>>>>> allow it to keep consistency. The way how Dag parsing
>>>>>> works is
>>>>>> > >>> that
>>>>>> > >>>>>>>> when
>>>>>> > >>>>>>>>>>> you are using importlib to read the Dag from a file, the
>>>>>> > relative
>>>>>> > >>>>>>>> imports
>>>>>> > >>>>>>>>>>> do not work as it does not know what they should be
>>>>>> relative
>>>>>> > to.
>>>>>> > >>>> But if
>>>>>> > >>>>>>>>>>> relative import is done from an imported package, it
>>>>>> should be
>>>>>> > no
>>>>>> > >>>>>>>>>> problem,
>>>>>> > >>>>>>>>>>> I think - otherwise our Dags would not be able to
>>>>>> import any
>>>>>> > >>>> library
>>>>>> > >>>>>>>> that
>>>>>> > >>>>>>>>>>> uses relative imports.
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>> Of course consistency might be the reason why we do not
>>>>>> want to
>>>>>> > >>>>>>>> introduce
>>>>>> > >>>>>>>>>>> relative imports. I don't see it as an issue if it is
>>>>>> guarded
>>>>>> > by
>>>>>> > >>>>>>>>>> pre-commit
>>>>>> > >>>>>>>>>>> though.
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>> J.
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>> J.
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
>>>>>> > >>>> ash@apache.org>
>>>>>> > >>>>>>>>>>> napisał:
>>>>>> > >>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> Oh yes, symlinks will work, with one big caveat: It
>>>>>> does mean
>>>>>> > >>> you
>>>>>> > >>>>>>>> can’t
>>>>>> > >>>>>>>>>>>> use absolute imports in one common module to another.
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> For example
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>
>>>>>> >
>>>>>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>>>>>> > >>>>>>>>>>>> where we have
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> ```
>>>>>> > >>>>>>>>>>>> from airflow.utils.module_loading import import_string
>>>>>> > >>>>>>>>>>>> ```
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> if we want to move serve_logs into this common lib
>>>>>> that is
>>>>>> > then
>>>>>> > >>>>>>>>>> symlinked
>>>>>> > >>>>>>>>>>>> then we wouldn’t be able to have `from
>>>>>> > >>>> airflow_common.module_loading
>>>>>> > >>>>>>>>>> import
>>>>>> > >>>>>>>>>>>> import_string`.
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> I can think of two possible solutions here.
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> 1) is to allow/require relative imports in this shared
>>>>>> lib, so
>>>>>> > >>>> `from
>>>>>> > >>>>>>>>>>>> .module_loading import import_string`
>>>>>> > >>>>>>>>>>>> 2) is to use `vendoring`[1] (from the pip maintainers)
>>>>>> which
>>>>>> > >>> will
>>>>>> > >>>>>>>> handle
>>>>>> > >>>>>>>>>>>> import-rewriting for us.
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> I’d entirely forgot that symlinks in repos was a
>>>>>> thing, so I
>>>>>> > >>>> prepared
>>>>>> > >>>>>>>> a
>>>>>> > >>>>>>>>>>>> minimal POC/demo of what vendoring approach could look
>>>>>> like
>>>>>> > here
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>
>>>>>> >
>>>>>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>>>>>> > >>>>>>>>>>>> Now personally I am more than happy with relative
>>>>>> imports, but
>>>>>> > >>>>>>>> generally
>>>>>> > >>>>>>>>>>>> as a project we have avoided them, so I think that
>>>>>> limits what
>>>>>> > >>> we
>>>>>> > >>>>>>>> could
>>>>>> > >>>>>>>>>> do
>>>>>> > >>>>>>>>>>>> with a symlink based approach.
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> -ash
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>> [1] https://github.com/pradyunsg/vendoring
>>>>>> > >>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
>>>>>> > >>>>>>>> gopidesupavan@gmail.com>
>>>>>> > >>>>>>>>>>>> wrote:
>>>>>> > >>>>>>>>>>>>> Thanks Ash
>>>>>> > >>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>> Yes agree option 2 would be preferred for me. Making
>>>>>> sure we
>>>>>> > >>>> have all
>>>>>> > >>>>>>>>>> the
>>>>>> > >>>>>>>>>>>>> gaurdriles to protect any unwanted behaviour in code
>>>>>> sharing
>>>>>> > >>> and
>>>>>> > >>>>>>>>>>>> executing
>>>>>> > >>>>>>>>>>>>> right of tests between the packages.
>>>>>> > >>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>> Agree with others, option 2 would be
>>>>>> > >>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
>>>>>> > >>>>>>>> amoghdesai.oss@gmail.com>
>>>>>> > >>>>>>>>>>>>> wrote:
>>>>>> > >>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>> Thanks for starting this discussion, Ash.
>>>>>> > >>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>> I would prefer option 2 here with proper tooling to
>>>>>> handle
>>>>>> > the
>>>>>> > >>>> code
>>>>>> > >>>>>>>>>>>>>> duplication at *release* time.
>>>>>> > >>>>>>>>>>>>>> It is best to have a dist that has all it needs in
>>>>>> itself.
>>>>>> > >>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>> Option 1 could very quickly get out of hand and if
>>>>>> we decide
>>>>>> > >>> to
>>>>>> > >>>>>>>>>> separate
>>>>>> > >>>>>>>>>>>>>> triggerer /
>>>>>> > >>>>>>>>>>>>>> dag processor / config etc etc as separate packages,
>>>>>> back
>>>>>> > >>>> compat is
>>>>>> > >>>>>>>>>>>> going
>>>>>> > >>>>>>>>>>>>>> to be a nightmare
>>>>>> > >>>>>>>>>>>>>> and will bite us harder than we anticipate.
>>>>>> > >>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>> Thanks & Regards,
>>>>>> > >>>>>>>>>>>>>> Amogh Desai
>>>>>> > >>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <
>>>>>> > >>> kaxilnaik@gmail.com>
>>>>>> > >>>>>>>>>> wrote:
>>>>>> > >>>>>>>>>>>>>>> I prefer Option 2 as well to avoid matrix of
>>>>>> dependencies
>>>>>> > >>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>>>>>> > >>>>>>>>>> <j_scheffler@gmx.de.invalid
>>>>>> > >>>>>>>>>>>>>>> wrote:
>>>>>> > >>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>> I'd also rather prefer option 2 - reason here is
>>>>>> it is
>>>>>> > >>> rather
>>>>>> > >>>>>>>>>>>> pragmatic
>>>>>> > >>>>>>>>>>>>>>>> and we no not need to cut another package and have
>>>>>> less
>>>>>> > >>>> package
>>>>>> > >>>>>>>>>> counts
>>>>>> > >>>>>>>>>>>>>>>> and dependencies.
>>>>>> > >>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>> I remember some time ago I was checking (together
>>>>>> with
>>>>>> > >>> Jarek,
>>>>>> > >>>> I am
>>>>>> > >>>>>>>>>> not
>>>>>> > >>>>>>>>>>>>>>>> sure anymore...) if the usage of symlinks would be
>>>>>> > possible.
>>>>>> > >>>> To
>>>>>> > >>>>>>>> keep
>>>>>> > >>>>>>>>>>>>>> the
>>>>>> > >>>>>>>>>>>>>>>> source in one package but "symlink" it into
>>>>>> another. If
>>>>>> > then
>>>>>> > >>>> at
>>>>>> > >>>>>>>>>> point
>>>>>> > >>>>>>>>>>>>>> of
>>>>>> > >>>>>>>>>>>>>>>> packaging/release the files are materialized we
>>>>>> have 1 set
>>>>>> > >>> of
>>>>>> > >>>>>>>> code.
>>>>>> > >>>>>>>>>>>>>>>> Otherwise if not possible still the redundancy
>>>>>> could be
>>>>>> > >>>> solved by
>>>>>> > >>>>>>>> a
>>>>>> > >>>>>>>>>>>>>>>> pre-commit hook - and in Git the files are
>>>>>> de-duplicated
>>>>>> > >>>> anyway
>>>>>> > >>>>>>>>>> based
>>>>>> > >>>>>>>>>>>>>> on
>>>>>> > >>>>>>>>>>>>>>>> content hash, so this does not hurt.
>>>>>> > >>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>>>> > >>>>>>>>>>>>>>>>> I support option 2 with proper automation & CI -
>>>>>> the
>>>>>> > >>>> reasonings
>>>>>> > >>>>>>>>>>>>>> you've
>>>>>> > >>>>>>>>>>>>>>>>> shown for that make sense to me.
>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>> Shahar
>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <
>>>>>> > >>>> ash@apache.org
>>>>>> > >>>>>>>>>>>>>>> wrote:
>>>>>> > >>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>>> As we work on finishing off the code-level
>>>>>> separation of
>>>>>> > >>>> Task
>>>>>> > >>>>>>>> SDK
>>>>>> > >>>>>>>>>>>>>> and
>>>>>> > >>>>>>>>>>>>>>>> Core
>>>>>> > >>>>>>>>>>>>>>>>>> (scheduler etc) we have come across some
>>>>>> situations
>>>>>> > where
>>>>>> > >>> we
>>>>>> > >>>>>>>> would
>>>>>> > >>>>>>>>>>>>>>> like
>>>>>> > >>>>>>>>>>>>>>>> to
>>>>>> > >>>>>>>>>>>>>>>>>> share code between these.
>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>>> However it’s not as straight forward of “just
>>>>>> put it in
>>>>>> > a
>>>>>> > >>>> common
>>>>>> > >>>>>>>>>>>>>> dist
>>>>>> > >>>>>>>>>>>>>>>> they
>>>>>> > >>>>>>>>>>>>>>>>>> both depend upon” because one of the goals of
>>>>>> the Task
>>>>>> > SDK
>>>>>> > >>>>>>>>>>>>>> separation
>>>>>> > >>>>>>>>>>>>>>>> was
>>>>>> > >>>>>>>>>>>>>>>>>> to have 100% complete version independence
>>>>>> between the
>>>>>> > >>> two,
>>>>>> > >>>>>>>>>> ideally
>>>>>> > >>>>>>>>>>>>>>>> even if
>>>>>> > >>>>>>>>>>>>>>>>>> they are built into the same image and venv.
>>>>>> Most of the
>>>>>> > >>>> reason
>>>>>> > >>>>>>>>>> why
>>>>>> > >>>>>>>>>>>>>>> this
>>>>>> > >>>>>>>>>>>>>>>>>> isn’t straight forward comes down to backwards
>>>>>> > >>>> compatibility -
>>>>>> > >>>>>>>> if
>>>>>> > >>>>>>>>>> we
>>>>>> > >>>>>>>>>>>>>>>> make
>>>>>> > >>>>>>>>>>>>>>>>>> an change to the common/shared distribution
>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>>> We’ve listed the options we have thought about in
>>>>>> > >>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545
>>>>>> (but
>>>>>> > that
>>>>>> > >>>> covers
>>>>>> > >>>>>>>>>>>>>> some
>>>>>> > >>>>>>>>>>>>>>>> more
>>>>>> > >>>>>>>>>>>>>>>>>> things that I don’t want to get in to in this
>>>>>> discussion
>>>>>> > >>>> such as
>>>>>> > >>>>>>>>>>>>>>>> possibly
>>>>>> > >>>>>>>>>>>>>>>>>> separating operators and executors out of a
>>>>>> single
>>>>>> > >>> provider
>>>>>> > >>>>>>>> dist.)
>>>>>> > >>>>>>>>>>>>>>>>>> To give a concrete example of some code I would
>>>>>> like to
>>>>>> > >>>> share
>>>>>> > >>>
>>>>>> >
>>>>>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>>>> > >>>>>>>>>>>>>>>>>> — logging config. Another thing we will want to
>>>>>> share
>>>>>> > will
>>>>>> > >>>> be
>>>>>> > >>>>>>>> the
>>>>>> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class from
>>>>>> airflow.configuration
>>>>>> > (but
>>>>>> > >>>>>>>> notably:
>>>>>> > >>>>>>>>>>>>>>> only
>>>>>> > >>>>>>>>>>>>>>>> the
>>>>>> > >>>>>>>>>>>>>>>>>> parser class, _not_ the default config values,
>>>>>> again,
>>>>>> > lets
>>>>>> > >>>> not
>>>>>> > >>>>>>>>>> dwell
>>>>>> > >>>>>>>>>>>>>>> on
>>>>>> > >>>>>>>>>>>>>>>> the
>>>>>> > >>>>>>>>>>>>>>>>>> specifics of that)
>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>>> So to bring the options listed in the issue here
>>>>>> for
>>>>>> > >>>> discussion,
>>>>>> > >>>>>>>>>>>>>>> broadly
>>>>>> > >>>>>>>>>>>>>>>>>> speaking there are two high-level approaches:
>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>>> 1. A single shared distribution
>>>>>> > >>>>>>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>>> The advantage of Approach 1 is that we only have
>>>>>> the
>>>>>> > code
>>>>>> > >>>> in one
>>>>>> > >>>>>>>>>>>>>>> place.
>>>>>> > >>>>>>>>>>>>>>>>>> However for me, at least in this specific case of
>>>>>> > Logging
>>>>>> > >>>> config
>>>>>> > >>>>>>>>>> or
>>>>>> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class is that backwards
>>>>>> > compatibility
>>>>>> > >>> is
>>>>>> > >>>>>>>> much
>>>>>> > >>>>>>>>>>>>>> much
>>>>>> > >>>>>>>>>>>>>>>>>> harder.
>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>>> The main advantage of Approach 2 is the the code
>>>>>> is
>>>>>> > >>> released
>>>>>> > >>>>>>>>>>>>>>>> with/embedded
>>>>>> > >>>>>>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would
>>>>>> contain
>>>>>> > >>> the
>>>>>> > >>>>>>>> right
>>>>>> > >>>>>>>>>>>>>>>> version
>>>>>> > >>>>>>>>>>>>>>>>>> of the logging config and ConfigParser etc). The
>>>>>> > downside
>>>>>> > >>> is
>>>>>> > >>>>>>>> that
>>>>>> > >>>>>>>>>>>>>>> either
>>>>>> > >>>>>>>>>>>>>>>>>> the code will need to be duplicated in the repo,
>>>>>> or
>>>>>> > better
>>>>>> > >>>> yet
>>>>>> > >>>>>>>> it
>>>>>> > >>>>>>>>>>>>>>> would
>>>>>> > >>>>>>>>>>>>>>>>>> live in a single place in the repo, but some
>>>>>> tooling
>>>>>> > (TBD)
>>>>>> > >>>> will
>>>>>> > >>>>>>>>>>>>>>>>>> automatically handle the duplication, either at
>>>>>> commit
>>>>>> > >>>> time, or
>>>>>> > >>>>>>>> my
>>>>>> > >>>>>>>>>>>>>>>>>> preference, at release time.
>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>>> For this kind of shared “utility” code I am very
>>>>>> > strongly
>>>>>> > >>>>>>>> leaning
>>>>>> > >>>>>>>>>>>>>>>> towards
>>>>>> > >>>>>>>>>>>>>>>>>> option 2 with automation, as otherwise I think
>>>>>> the
>>>>>> > >>> backwards
>>>>>> > >>>>>>>>>>>>>>>> compatibility
>>>>>> > >>>>>>>>>>>>>>>>>> requirements would make it unworkable (very
>>>>>> quickly over
>>>>>> > >>>> time
>>>>>> > >>>>>>>> the
>>>>>> > >>>>>>>>>>>>>>>>>> combinations we would have to test would just be
>>>>>> > >>>> unreasonable)
>>>>>> > >>>>>>>>>> and I
>>>>>> > >>>>>>>>>>>>>>>> don’t
>>>>>> > >>>>>>>>>>>>>>>>>> feel confident we can have things as stable as
>>>>>> we need
>>>>>> > to
>>>>>> > >>>> really
>>>>>> > >>>>>>>>>>>>>>> deliver
>>>>>> > >>>>>>>>>>>>>>>>>> the version separation/independency I want to
>>>>>> delivery
>>>>>> > >>> with
>>>>>> > >>>>>>>>>> AIP-72.
>>>>>> > >>>>>>>>>>>>>>>>>> So unless someone feels very strongly about
>>>>>> this, I will
>>>>>> > >>>> come up
>>>>>> > >>>>>>>>>>>>>> with
>>>>>> > >>>>>>>>>>>>>>> a
>>>>>> > >>>>>>>>>>>>>>>>>> draft PR for further discussion that will
>>>>>> implement code
>>>>>> > >>>> sharing
>>>>>> > >>>>>>>>>> via
>>>>>> > >>>>>>>>>>>>>>>>>> “vendoring” it at build time. I have an idea of
>>>>>> how I
>>>>>> > can
>>>>>> > >>>>>>>> achieve
>>>>>> > >>>>>>>>>>>>>> this
>>>>>> > >>>>>>>>>>>>>>>> so
>>>>>> > >>>>>>>>>>>>>>>>>> we have a single version in the repo and it’ll
>>>>>> work
>>>>>> > there,
>>>>>> > >>>> but
>>>>>> > >>>>>>>> at
>>>>>> > >>>>>>>>>>>>>>>> runtime
>>>>>> > >>>>>>>>>>>>>>>>>> we vendor it in to the shipped dist so it lives
>>>>>> at
>>>>>> > >>> something
>>>>>> > >>>>>>>> like
>>>>>> > >>>>>>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>>> In terms of repo layout, this likely means we
>>>>>> would end
>>>>>> > up
>>>>>> > >>>> with:
>>>>>> > >>>>>>>>>>>>>>>>>> airflow-core/pyproject.toml
>>>>>> > >>>>>>>>>>>>>>>>>> airflow-core/src/
>>>>>> > >>>>>>>>>>>>>>>>>> airflow-core/tests/
>>>>>> > >>>>>>>>>>>>>>>>>> task-sdk/pyproject.toml
>>>>>> > >>>>>>>>>>>>>>>>>> task-sdk/src/
>>>>>> > >>>>>>>>>>>>>>>>>> task-sdk/tests/
>>>>>> > >>>>>>>>>>>>>>>>>> airflow-common/src
>>>>>> > >>>>>>>>>>>>>>>>>> airflow-common/tests/
>>>>>> > >>>>>>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as
>>>>>> deps
>>>>>> > would
>>>>>> > >>>> be
>>>>>> > >>>>>>>>>>>>>> included
>>>>>> > >>>>>>>>>>>>>>>> in
>>>>>> > >>>>>>>>>>>>>>>>>> the downstream projects. TBD.
>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>>>>>>>>>>> Thoughts and feedback welcomed.
>>>>>> > >>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>> > dev-unsubscribe@airflow.apache.org
>>>>>> > >>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>> > >>> dev-help@airflow.apache.org
>>>>>> > >>>>>>>>>>>>>>>>
>>>>>> > >>>>>>>>
>>>>>> > >>>
>>>>>> ---------------------------------------------------------------------
>>>>>> > >>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>> > >>>>>>>> For additional commands, e-mail:
>>>>>> dev-help@airflow.apache.org
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>
>>>>>> >
>>>>>> ---------------------------------------------------------------------
>>>>>> > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>> > >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>> > >>>>>>
>>>>>> > >>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> > >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>> > >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>> > >>>>>
>>>>>> > >>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> > >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>> > >>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>> > >>>>
>>>>>> > >>>>
>>>>>> > >
>>>>>> > >
>>>>>> ---------------------------------------------------------------------
>>>>>> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>> > > For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>> > >
>>>>>> >
>>>>>> >
>>>>>> ---------------------------------------------------------------------
>>>>>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>> > For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>> >
>>>>>> >
>>>>>>
>>>>>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Amogh Desai <am...@gmail.com>.
Probably, you make a valid point.

Maybe this is an implementation detail, so we could figure it out as we
start on a POC and factor in these things
as we move along?

But from an initial guess, I would think that execution time related items
(if we manage to enumerate them) would be something
that would be better off in that "core_and_task_sdk" bundle.

Thanks & Regards,
Amogh Desai


On Tue, Jul 8, 2025 at 3:46 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> > Not that I am against your idea and we can surely expand as we need but
> we would not need to expand the
> "core_and_task_sdk" if we put only the relevant items into it.
>
> So if we move logging and config out, my question is what is really
> relevant to "stay" in "core_and_task_sdk" ? And what we know will not be
> needed by other distributions in the future ?
>
> What would be the content of "core_and_task_sdk" ? Can we enumerate what
> should go there now ?
>
> My bet is that if we enumerate them, it will turn out that they are good
> candidates to make separate "shared features" that can be logically named
> and modularised, and that there is no real need to have shared
> "core_and_task_sdk" which sounds like "bag of everything else".
>
>
>
> On Tue, Jul 8, 2025 at 11:48 AM Amogh Desai <am...@gmail.com>
> wrote:
>
>> Yeah, I think what you are showcasing here is a step ahead of the
>> initial proposal from Ash.
>>
>> From the original proposal, the `core_and_task_sdk` *can* have the
>> things relevant to just those two
>> distros. Logging, Config are modules that might be needed by airflow-ctl
>> for example, so ideally, those
>> would not be good candidates to be put in there, ideally speaking.
>>
>> The example of Kubernetes Utils sounds to be a good example, it will be
>> used by KubeExecutor (lets say this is a
>> module called "executors") and by KPO (providers), and the
>> "shared/kubernetes" would probably be a good
>> candidate for that.
>>
>> Not that I am against your idea and we can surely expand as we need but
>> we would not need to expand the
>> "core_and_task_sdk" if we put only the relevant items into it.
>>
>> Thanks & Regards,
>> Amogh Desai
>>
>>
>> On Tue, Jul 8, 2025 at 12:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> > @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
>>> there, I am understanding the direction
>>> but could you elaborate a bit more please?
>>>
>>> Let me elaborate:
>>>
>>> As I understand (maybe I am wrong?),  the proposal is that we have a
>>> "core-and-task-sdk" folder which is a shared distribution that is
>>> vendored-in into both "airflow-core" and "airflow-task-sdk". This
>>> contains some shared code that we want to include in both distributions
>>> (note that we never ever release "core-and-task-sdk"
>>> distribution because it only contains code that is shared between the two
>>> distributions we release.
>>>
>>> That's fine and cool.
>>>
>>> Imagine that this distribution contains "logging" (shared code
>>> for logging) and "config" (shared code for configuration). - both needed in
>>> "airflow-core" and "airflow-task-sdk". So far so good. But what happen if
>>> we want to use logging in the same fashion in say "airflow-ctl" (that also
>>> applies for other distributions that we might come up with) ? Are we going
>>> to vendor in the whole "core-and-task-sdk" distribution in "airflow-ctl" ?
>>> It would be far better if we just vendor in "logging" and do not vendor-in
>>> "config".
>>>
>>> And if we are going to have a mechanism to vendor-in "a distribution" -
>>> there is nothing wrong with having the same way to vendor-in multiple
>>> distributions - so we can easily do it this way (i added "orm" ,
>>> "serialization". and "fast_api" as an example thing that we might want to
>>> share - not sure if that is really something we want to do but it will
>>> allow to illustrate my idea better)
>>>
>>> /
>>>   airflow-ctl/
>>>   task-sdk/...
>>>   airflow-core/...
>>>   ....
>>>   shared/
>>>     kubernetes/
>>>       pyproject.toml
>>>       src/
>>>         airflow_shared_kubernetes/__init__.py
>>>     logging/
>>>       pyproject.toml
>>>       src/
>>>         airflow_shared_logging/__init__.py
>>>     config/
>>>       pyproject.toml
>>>       src/
>>>         airflow_shared_config/__init__.py
>>>     orm/
>>>       pyproject.toml
>>>       src/
>>>         airflow_shared_orm/__init__.py
>>>      serialization/
>>>       pyproject.toml
>>>       src/
>>>         airflow_shared_serialization/__init__.py
>>>      fast_api/
>>>       pyproject.toml
>>>       src/
>>>         airflow_shared_fast_api/__init__.py
>>>     ...
>>>
>>> This has multiple benefits (and I see no real drawbacks):
>>>
>>> * the code can be really well modularised.  Those "things" we share (and
>>> this also connects to the __init__.py discussion) - can be independent -
>>> and (it follow Jens comment) it allows to keep low cyclomatic complexity
>>> https://en.wikipedia.org/wiki/Cyclomatic_complexity . It will be way
>>> easier to implement logging in the way that it does not import or use
>>> config. This means for example that configuration for logging will need to
>>> be injected when logging is initialized - and that's exactly what we want,
>>> we do not want logging code to use configuration code directly - they
>>> should be independent from each other and you should be free to vendor-in
>>> either logging or config independently if you also vendored-in the other.
>>>
>>> * it's much more logical. We split based on functionality we want to
>>> share - not about the "distributions" we want to produce. That allows us -
>>> in the future - to make different decisions on how we split our
>>> distributions. For example (i do not tell we have to do it, or that we will
>>> do it but we will have such a possibility) - we can add more shared
>>> utilities we find useful in the same way and decide that "scheduler",
>>> "api_server" or "scheduler" or "triggerer" or "dag processor" are split to
>>> separate distributions - because for example we want to keep a number of
>>> dependencies down. And for example "api_server" might use "fast_api",
>>> "config", "logging", "orm" and "fast_api" and "serialization" , where the
>>> scheduler should not need "fast_api". The "dag_processor" eventually might
>>> not need "orm" nor "fast_api" and only use the other
>>>
>>> This seems like a natural approach. If we have a mechanism to "share"
>>> the code, it does not add complexity, but allows us to isolate independent
>>> functionality into "isolated" boxes and use them
>>>
>>> Also for cyclomatic complexity that is a complex word (badly chosen as
>>> it scares people away) and has some math behind, but it really boils down
>>> to very simple "rules of thumb" (and yes I am a big proponent of having low
>>> cyclomatic complexity).
>>>
>>> a) when you are building the "final" product (i.e. distribution you want
>>> to release) - make sure that you only "use" things - that nothing else is
>>> "using" you as a library.
>>> b) when you are building a "shared" thing (a library) - make sure that
>>> library is only "used" by others but it does not really "use" anything
>>> else. For example in the case I explained above - we can achieve
>>> low-cyclomatic complexity when:
>>>
>>> * airflow-core uses: logging, config, orm, serialization, fast_api
>>> * none of the "logging, config, orm, serialization, fast_api" use each
>>> other - they are at the bottom of the "user -> used" tree
>>>
>>> J.
>>>
>>>
>>>
>>> On Tue, Jul 8, 2025 at 8:16 AM Amogh Desai <am...@gmail.com>
>>> wrote:
>>>
>>>> I like the folder structure proposed by Ash and have no objections with
>>>> it.
>>>>
>>>> "core_and_task_sdk" sounds good to me and justifies what it should do
>>>> pretty well.
>>>>
>>>> @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
>>>> there, I am understanding the direction
>>>> but could you elaborate a bit more please?
>>>>
>>>> Naming is REALLY hard!
>>>>
>>>> Thanks & Regards,
>>>> Amogh Desai
>>>>
>>>>
>>>> On Tue, Jul 8, 2025 at 2:52 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>>> How about splitting it even more and having each shared "thing" named?
>>>>> "logging", "config" and sharing them explicitly and separately with the
>>>>> right "user" ?
>>>>> That sounds way more modular and  we will be able to choose which of
>>>>> the
>>>>> shared "utils" we use where.
>>>>>
>>>>> J.
>>>>>
>>>>>
>>>>> On Mon, Jul 7, 2025 at 11:13 PM Jens Scheffler
>>>>> <j_...@gmx.de.invalid>
>>>>> wrote:
>>>>>
>>>>> > I like "core_and_task_sdk the same like core-and-task-sdk - I have no
>>>>> > problem and it is a path only.
>>>>> >
>>>>> > if we get to "dag-parser-scheduler-task-sdk-and-triggerer" which is a
>>>>> > bit bulky we then should name it "all-not-api-server" :-D
>>>>> >
>>>>> > On 07.07.25 22:57, Ash Berlin-Taylor wrote:
>>>>> > > In case I did a bad job explaining it, the “core and task sdk” is
>>>>> not in
>>>>> > the module name/import name, just in the file path.
>>>>> > >
>>>>> > > Anyone have other ideas?
>>>>> > >
>>>>> > >> On 7 Jul 2025, at 21:37, Buğra Öztürk <oz...@gmail.com>
>>>>> wrote:
>>>>> > >>
>>>>> > >> Thanks Ash! Looks cool! I like the structure. This will enable
>>>>> all the
>>>>> > >> combinations and structure looks easy to grasp. No strong stance
>>>>> on the
>>>>> > >> naming other than maybe it is a bit long with `and`, `core_ctl`
>>>>> could be
>>>>> > >> shorter, since no import path is defined like that, we can give
>>>>> any name
>>>>> > >> for sure.
>>>>> > >>
>>>>> > >> Best regards,
>>>>> > >>
>>>>> > >>> On Mon, 7 Jul 2025, 21:51 Jarek Potiuk, <ja...@potiuk.com>
>>>>> wrote:
>>>>> > >>>
>>>>> > >>> Looks good but I think we should find some better logical name
>>>>> for
>>>>> > >>> core_and_sdk :)
>>>>> > >>>
>>>>> > >>> pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
>>>>> > >>> <j_...@gmx.de.invalid> napisał:
>>>>> > >>>
>>>>> > >>>> Cool! Especially the "shared" folder with the ability to have
>>>>> > >>>> N-combinations w/o exploding project repo root!
>>>>> > >>>>
>>>>> > >>>> On 07.07.25 14:43, Ash Berlin-Taylor wrote:
>>>>> > >>>>> Oh, and all of this will be explain in shared/README.md
>>>>> > >>>>>
>>>>> > >>>>>> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org>
>>>>> wrote:
>>>>> > >>>>>>
>>>>> > >>>>>> Okay, so it seems we have agreement on the approach here, so
>>>>> I’ll
>>>>> > >>>> continue with this, and on the dev call it was mentioned that
>>>>> > >>>> “airflow-common” wasn’t a great name, so here is my proposal
>>>>> for the
>>>>> > file
>>>>> > >>>> structure;
>>>>> > >>>>>> ```
>>>>> > >>>>>> /
>>>>> > >>>>>>   task-sdk/...
>>>>> > >>>>>>   airflow-core/...
>>>>> > >>>>>>   shared/
>>>>> > >>>>>>     kuberenetes/
>>>>> > >>>>>>       pyproject.toml
>>>>> > >>>>>>       src/
>>>>> > >>>>>>         airflow_kube/__init__.py
>>>>> > >>>>>>     core-and-tasksdk/
>>>>> > >>>>>>       pyproject.toml
>>>>> > >>>>>>       src/
>>>>> > >>>>>>         airflow_shared/__init__.py
>>>>> > >>>>>> ```
>>>>> > >>>>>>
>>>>> > >>>>>> Things to note here: the “shared” folder has (the
>>>>> possibility) of
>>>>> > >>>> having multiple different shared “libraries” in it, in this
>>>>> example I
>>>>> > am
>>>>> > >>>> supposing a hypothetical shared kuberenetes folder a world in
>>>>> which we
>>>>> > >>>> split the KubePodOperator and the KubeExecutor in to two
>>>>> separate
>>>>> > >>>> distributions (example only, not proposing we do that right
>>>>> now, and
>>>>> > that
>>>>> > >>>> will be a separate discussion)
>>>>> > >>>>>> The other things to note here:
>>>>> > >>>>>>
>>>>> > >>>>>>
>>>>> > >>>>>> - the folder name in shared aims to be “self-documenting”,
>>>>> hence the
>>>>> > >>>> verbose “core-and-tasksdk” to say where the shared library is
>>>>> > intended to
>>>>> > >>>> be used.
>>>>> > >>>>>> - the python module itself should almost always have an
>>>>> `airflow_`
>>>>> > (or
>>>>> > >>>> maybe `_airflow_`?) prefix so that it does not conflict with
>>>>> anything
>>>>> > >>> else
>>>>> > >>>> we might use. It won’t matter “in production” as those will be
>>>>> > vendored
>>>>> > >>> in
>>>>> > >>>> to be imported as `airflow/_vendor/airflow_shared` etc, but
>>>>> avoiding
>>>>> > >>>> conflicts at dev time with the Finder approach is a good safety
>>>>> > measure.
>>>>> > >>>>>> I will start making a real PR for this proposal now, but I’m
>>>>> open to
>>>>> > >>>> feedback (either here, or in the PR when I open it)
>>>>> > >>>>>> -ash
>>>>> > >>>>>>
>>>>> > >>>>>>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com>
>>>>> wrote:
>>>>> > >>>>>>>
>>>>> > >>>>>>> Yeah we have to try it and test - also building packages
>>>>> happens
>>>>> > semi
>>>>> > >>>>>>> frequently when you run `uv sync` (they use some kind of
>>>>> heuristics
>>>>> > >>> to
>>>>> > >>>>>>> decide when) and you can force it with `--reinstall` or
>>>>> > `--refresh`.
>>>>> > >>>>>>> Package build also happens every time when you run "ci-image
>>>>> build`
>>>>> > >>>> now in
>>>>> > >>>>>>> breeze so it seems like it will nicely integrate in our
>>>>> workflows.
>>>>> > >>>>>>>
>>>>> > >>>>>>> Looks really cool Ash.
>>>>> > >>>>>>>
>>>>> > >>>>>>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <
>>>>> ash@apache.org>
>>>>> > >>>> wrote:
>>>>> > >>>>>>>> It’s not just release time, but any time we build a package
>>>>> which
>>>>> > >>>> happens
>>>>> > >>>>>>>> on “every” CI run. The normal unit tests will use code from
>>>>> > >>>>>>>> airflow-common/src/airflow_common; the kube tests which
>>>>> build an
>>>>> > >>>> image will
>>>>> > >>>>>>>> build the dists and vendor in the code from that commit.
>>>>> > >>>>>>>>
>>>>> > >>>>>>>> There is only a single copy of the shared code committed to
>>>>> the
>>>>> > >>> repo,
>>>>> > >>>> so
>>>>> > >>>>>>>> there is never anything to synchronise.
>>>>> > >>>>>>>>
>>>>> > >>>>>>>>> On 4 Jul 2025, at 15:53, Amogh Desai <
>>>>> amoghdesai.oss@gmail.com>
>>>>> > >>>> wrote:
>>>>> > >>>>>>>>> Thanks Ash.
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> This is really cool and helpful that you were able to test
>>>>> both
>>>>> > >>>> scenarios
>>>>> > >>>>>>>>> -- repo checkout
>>>>> > >>>>>>>>> and also installing from the vendored package and the
>>>>> resolution
>>>>> > >>>> worked
>>>>> > >>>>>>>>> fine too.
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> I like this idea compared the to relative import one for
>>>>> few
>>>>> > >>> reasons:
>>>>> > >>>>>>>>> - It feels like it will take some time to adjust to the new
>>>>> > coding
>>>>> > >>>>>>>> standard
>>>>> > >>>>>>>>> that we will lay
>>>>> > >>>>>>>>> if we impose relative imports in the shared dist
>>>>> > >>>>>>>>> - We can continue using repo wise absolute import
>>>>> standards, it
>>>>> > is
>>>>> > >>>> also
>>>>> > >>>>>>>>> much easier for situations
>>>>> > >>>>>>>>> when we do global search in IDE to find + replace, this
>>>>> could
>>>>> > mean
>>>>> > >>> a
>>>>> > >>>>>>>> change
>>>>> > >>>>>>>>> there
>>>>> > >>>>>>>>> - The vendoring work is a proven and established paradigm
>>>>> across
>>>>> > >>>> projects
>>>>> > >>>>>>>>> and would
>>>>> > >>>>>>>>> out of box give us the build tooling we need also
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> Nothing too against the relative import but with the
>>>>> evidence
>>>>> > >>>> provided
>>>>> > >>>>>>>>> above, vendored approach
>>>>> > >>>>>>>>> seems to only do us good.
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> Regarding synchronizing it, release time should be fine as
>>>>> long
>>>>> > as
>>>>> > >>> we
>>>>> > >>>>>>>> have
>>>>> > >>>>>>>>> a good CI workflow to probably
>>>>> > >>>>>>>>> catch such issues per PR if changes are made in shared
>>>>> dist?
>>>>> > >>>> (precommit
>>>>> > >>>>>>>>> would make it really slow i guess)
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> If we can run our tests with vendored code we should be
>>>>> mostly
>>>>> > >>>> covered.
>>>>> > >>>>>>>>> Good effort all!
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>> Thanks & Regards,
>>>>> > >>>>>>>>> Amogh Desai
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>>
>>>>> > >>>>>>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <
>>>>> > ash@apache.org>
>>>>> > >>>>>>>> wrote:
>>>>> > >>>>>>>>>> Okay, I think I’ve got something that works and I’m happy
>>>>> with.
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>
>>>>> >
>>>>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>>>>> > >>>>>>>>>> This produces the following from `uv build task-sdk`
>>>>> > >>>>>>>>>> -
>>>>> > >>>>>>>>>>
>>>>> > >>>
>>>>> >
>>>>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>> > >>>>>>>>>> -
>>>>> > >>>>>>>>>>
>>>>> > >>>
>>>>> >
>>>>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>>>>> > >>>>>>>>>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> ```
>>>>> > >>>>>>>>>> ❯ unzip -l
>>>>> > >>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
>>>>> > >>>>>>>> grep
>>>>> > >>>>>>>>>> _vendor
>>>>> > >>>>>>>>>>      50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
>>>>> > >>>>>>>>>>    2082  02-02-2020 00:00
>>>>>  airflow/sdk/_vendor/__init__.py
>>>>> > >>>>>>>>>>      28  02-02-2020 00:00
>>>>> >  airflow/sdk/_vendor/airflow_common.pyi
>>>>> > >>>>>>>>>>      18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
>>>>> > >>>>>>>>>>     785  02-02-2020 00:00
>>>>> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
>>>>> > >>>>>>>>>>   10628  02-02-2020 00:00
>>>>> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
>>>>> > >>>>>>>>>> ```
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> And similarly in the .tar.gz, so our “sdist” is complete
>>>>> too:
>>>>> > >>>>>>>>>> ```
>>>>> > >>>>>>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep
>>>>> > _vendor
>>>>> > >>>>>>>>>>
>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>>>>> > >>>>>>>>>>
>>>>> > apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>>>>> > >>>>>>>>>>
>>>>> > >>>>
>>>>> >
>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>>>>> > >>>>>>>>>>
>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>
>>>>> >
>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>>>>> > >>>
>>>>> >
>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>>>>> > >>>>>>>>>> ```
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> The plugin works at build time by including/copying the
>>>>> libs
>>>>> > >>>> specified
>>>>> > >>>>>>>> in
>>>>> > >>>>>>>>>> vendor.txt into place (and let `vendoring` take care of
>>>>> import
>>>>> > >>>>>>>> rewrites.)
>>>>> > >>>>>>>>>> For the imports to continue to work at “dev” time/from a
>>>>> repo
>>>>> > >>>> checkout,
>>>>> > >>>>>>>> I
>>>>> > >>>>>>>>>> have added a import finder to `sys.meta_path`, and since
>>>>> it’s at
>>>>> > >>> the
>>>>> > >>>>>>>> end of
>>>>> > >>>>>>>>>> the list it will only be used if the normal import can’t
>>>>> find
>>>>> > >>>> things.
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>
>>>>> >
>>>>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>>>>> > >>>>>>>>>> This doesn’t quite give us the same runtime effect “import
>>>>> > >>>> rewriting”
>>>>> > >>>>>>>>>> affect, as in this approach `airflow_common` is directly
>>>>> loaded
>>>>> > >>>> (i.e.
>>>>> > >>>>>>>>>> airflow.sdk._vendor.airflow_common and airflow_common
>>>>> exist in
>>>>> > >>>>>>>>>> sys.modules), but it does work for everything that I was
>>>>> able to
>>>>> > >>>> test..
>>>>> > >>>>>>>>>> I tested it with the diff at the end of this message. My
>>>>> test
>>>>> > >>>> ipython
>>>>> > >>>>>>>>>> shell:
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> ```
>>>>> > >>>>>>>>>> In [1]: from airflow.sdk._vendor.airflow_common.timezone
>>>>> import
>>>>> > >>> foo
>>>>> > >>>>>>>>>> In [2]: foo
>>>>> > >>>>>>>>>> Out[2]: 1
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> In [3]: import airflow.sdk._vendor.airflow_common
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> In [4]: import airflow.sdk._vendor.airflow_common.timezone
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>>>>> > >>>>>>>>>> Out[5]:
>>>>> > >>>>>>>>>>
>>>>> > >>>
>>>>> >
>>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>>>>> > >>>>>>>>>> In [6]:
>>>>> airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>> > >>>>>>>>>> Out[6]:
>>>>> > >>>>>>>>>>
>>>>> > >>>
>>>>> >
>>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>>>>> > >>>>>>>>>> ```
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> And in an standalone environment with the SDK dist I
>>>>> built (it
>>>>> > >>>> needed
>>>>> > >>>>>>>> the
>>>>> > >>>>>>>>>> matching airflow-core right now, but that is nothing to
>>>>> do with
>>>>> > >>> this
>>>>> > >>>>>>>>>> discussion):
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> ```
>>>>> > >>>>>>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>>>>> > >>>>>>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>>>>> > >>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl
>>>>> ipython
>>>>> > >>>>>>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8
>>>>> ]
>>>>> > >>>>>>>>>> Type 'copyright', 'credits' or 'license' for more
>>>>> information
>>>>> > >>>>>>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type '?'
>>>>> for
>>>>> > >>> help.
>>>>> > >>>>>>>>>> Tip: You can use `%hist` to view history, see the options
>>>>> with
>>>>> > >>>>>>>> `%history?`
>>>>> > >>>>>>>>>> In [1]: import airflow.sdk._vendor.airflow_common.timezone
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> In [2]:
>>>>> airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>> > >>>>>>>>>> Out[2]:
>>>>> > >>>>>>>>>>
>>>>> > >>>
>>>>> >
>>>>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>>>>> > >>>>>>>>>> ``
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> ```diff
>>>>> > >>>>>>>>>> diff --git a/airflow-common/src/airflow_common/__init__.py
>>>>> > >>>>>>>>>> b/airflow-common/src/airflow_common/__init__.py
>>>>> > >>>>>>>>>> index 13a83393a9..927b7c6b61 100644
>>>>> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/__init__.py
>>>>> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
>>>>> > >>>>>>>>>> @@ -14,3 +14,5 @@
>>>>> > >>>>>>>>>> # KIND, either express or implied.  See the License for
>>>>> the
>>>>> > >>>>>>>>>> # specific language governing permissions and limitations
>>>>> > >>>>>>>>>> # under the License.
>>>>> > >>>>>>>>>> +
>>>>> > >>>>>>>>>> +foo = 1
>>>>> > >>>>>>>>>> diff --git a/airflow-common/src/airflow_common/timezone.py
>>>>> > >>>>>>>>>> b/airflow-common/src/airflow_common/timezone.py
>>>>> > >>>>>>>>>> index 340b924c66..58384ef20f 100644
>>>>> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/timezone.py
>>>>> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
>>>>> > >>>>>>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>>>>> > >>>>>>>>>> version.parse(metadata.version("pendulum")).major == 3
>>>>> > >>>>>>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
>>>>> > >>>>>>>>>> utc = pendulum.UTC
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> +
>>>>> > >>>>>>>>>> +from airflow_common import foo
>>>>> > >>>>>>>>>> +
>>>>> > >>>>>>>>>> TIMEZONE: Timezone
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>> ```
>>>>> > >>>>>>>>>>
>>>>> > >>>>>>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <jarek@potiuk.com
>>>>> >
>>>>> > wrote:
>>>>> > >>>>>>>>>>> I think both approaches are doable:
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> 1) -> We can very easily prevent bad imports by
>>>>> pre-commit when
>>>>> > >>>>>>>> importing
>>>>> > >>>>>>>>>>> from different distributions and make sure we are only
>>>>> doing
>>>>> > >>>> relative
>>>>> > >>>>>>>>>>> imports in the shared modules. We are doing plenty of
>>>>> this
>>>>> > >>>> already. And
>>>>> > >>>>>>>>>> yes
>>>>> > >>>>>>>>>>> it would require relative links we currently do not
>>>>> allow.
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> 2) -> has one disadvantage that someone at some point in
>>>>> time
>>>>> > >>> will
>>>>> > >>>> have
>>>>> > >>>>>>>>>> to
>>>>> > >>>>>>>>>>> decide to synchronize this and if it happens just before
>>>>> > release
>>>>> > >>>> (I bet
>>>>> > >>>>>>>>>>> this is going to happen) this will lead to solving
>>>>> problems
>>>>> > that
>>>>> > >>>> would
>>>>> > >>>>>>>>>>> normally be solved during PR when you make a change (i.e.
>>>>> > >>> symbolic
>>>>> > >>>> link
>>>>> > >>>>>>>>>> has
>>>>> > >>>>>>>>>>> the advantage that whoever modifies shared code will be
>>>>> > >>> immediately
>>>>> > >>>>>>>>>>> notified in their PR - that they broke something because
>>>>> either
>>>>> > >>>> static
>>>>> > >>>>>>>>>>> checks or mypy or tests fail.
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> Ash, do you have an idea of a process (who and when)
>>>>> does the
>>>>> > >>>>>>>>>>> synchronisation in case of vendoring? Maybe we could
>>>>> solve it
>>>>> > if
>>>>> > >>>> it is
>>>>> > >>>>>>>>>> done
>>>>> > >>>>>>>>>>> more frequently and with some regularity? We could
>>>>> potentially
>>>>> > >>>> force
>>>>> > >>>>>>>>>>> re-vendoring at PR time as well any time shared code
>>>>> changes
>>>>> > (and
>>>>> > >>>>>>>> prevent
>>>>> > >>>>>>>>>>> it by pre-commit. And I can't think of some place (other
>>>>> than
>>>>> > >>>> releases)
>>>>> > >>>>>>>>>> in
>>>>> > >>>>>>>>>>> our development workflow and that seems to be a bit too
>>>>> late as
>>>>> > >>>> puts an
>>>>> > >>>>>>>>>>> extra effort on fixing potential incompatibilities
>>>>> introduced
>>>>> > on
>>>>> > >>>>>>>> release
>>>>> > >>>>>>>>>>> manager and delays the release. WDYT?
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> Re: relative links. I think for a shared library we could
>>>>> > >>>> potentially
>>>>> > >>>>>>>>>> relax
>>>>> > >>>>>>>>>>> this and allow them (and actually disallow absolute
>>>>> links in
>>>>> > the
>>>>> > >>>> pieces
>>>>> > >>>>>>>>>> of
>>>>> > >>>>>>>>>>> code that are shared - again, by pre-commit). As I
>>>>> recall, the
>>>>> > >>> only
>>>>> > >>>>>>>>>> reason
>>>>> > >>>>>>>>>>> we forbade the relative links is because of how we are
>>>>> (or
>>>>> > maybe
>>>>> > >>>> were)
>>>>> > >>>>>>>>>>> doing DAG parsing and failures resulting from it. So we
>>>>> decided
>>>>> > >>> to
>>>>> > >>>> just
>>>>> > >>>>>>>>>> not
>>>>> > >>>>>>>>>>> allow it to keep consistency. The way how Dag parsing
>>>>> works is
>>>>> > >>> that
>>>>> > >>>>>>>> when
>>>>> > >>>>>>>>>>> you are using importlib to read the Dag from a file, the
>>>>> > relative
>>>>> > >>>>>>>> imports
>>>>> > >>>>>>>>>>> do not work as it does not know what they should be
>>>>> relative
>>>>> > to.
>>>>> > >>>> But if
>>>>> > >>>>>>>>>>> relative import is done from an imported package, it
>>>>> should be
>>>>> > no
>>>>> > >>>>>>>>>> problem,
>>>>> > >>>>>>>>>>> I think - otherwise our Dags would not be able to import
>>>>> any
>>>>> > >>>> library
>>>>> > >>>>>>>> that
>>>>> > >>>>>>>>>>> uses relative imports.
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> Of course consistency might be the reason why we do not
>>>>> want to
>>>>> > >>>>>>>> introduce
>>>>> > >>>>>>>>>>> relative imports. I don't see it as an issue if it is
>>>>> guarded
>>>>> > by
>>>>> > >>>>>>>>>> pre-commit
>>>>> > >>>>>>>>>>> though.
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> J.
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> J.
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
>>>>> > >>>> ash@apache.org>
>>>>> > >>>>>>>>>>> napisał:
>>>>> > >>>>>>>>>>>
>>>>> > >>>>>>>>>>>> Oh yes, symlinks will work, with one big caveat: It
>>>>> does mean
>>>>> > >>> you
>>>>> > >>>>>>>> can’t
>>>>> > >>>>>>>>>>>> use absolute imports in one common module to another.
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> For example
>>>>> > >>>>>>>>>>>>
>>>>> > >>>
>>>>> >
>>>>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>>>>> > >>>>>>>>>>>> where we have
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> ```
>>>>> > >>>>>>>>>>>> from airflow.utils.module_loading import import_string
>>>>> > >>>>>>>>>>>> ```
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> if we want to move serve_logs into this common lib that
>>>>> is
>>>>> > then
>>>>> > >>>>>>>>>> symlinked
>>>>> > >>>>>>>>>>>> then we wouldn’t be able to have `from
>>>>> > >>>> airflow_common.module_loading
>>>>> > >>>>>>>>>> import
>>>>> > >>>>>>>>>>>> import_string`.
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> I can think of two possible solutions here.
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> 1) is to allow/require relative imports in this shared
>>>>> lib, so
>>>>> > >>>> `from
>>>>> > >>>>>>>>>>>> .module_loading import import_string`
>>>>> > >>>>>>>>>>>> 2) is to use `vendoring`[1] (from the pip maintainers)
>>>>> which
>>>>> > >>> will
>>>>> > >>>>>>>> handle
>>>>> > >>>>>>>>>>>> import-rewriting for us.
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> I’d entirely forgot that symlinks in repos was a thing,
>>>>> so I
>>>>> > >>>> prepared
>>>>> > >>>>>>>> a
>>>>> > >>>>>>>>>>>> minimal POC/demo of what vendoring approach could look
>>>>> like
>>>>> > here
>>>>> > >>>>>>>>>>>>
>>>>> > >>>
>>>>> >
>>>>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>>>>> > >>>>>>>>>>>> Now personally I am more than happy with relative
>>>>> imports, but
>>>>> > >>>>>>>> generally
>>>>> > >>>>>>>>>>>> as a project we have avoided them, so I think that
>>>>> limits what
>>>>> > >>> we
>>>>> > >>>>>>>> could
>>>>> > >>>>>>>>>> do
>>>>> > >>>>>>>>>>>> with a symlink based approach.
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> -ash
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>> [1] https://github.com/pradyunsg/vendoring
>>>>> > >>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
>>>>> > >>>>>>>> gopidesupavan@gmail.com>
>>>>> > >>>>>>>>>>>> wrote:
>>>>> > >>>>>>>>>>>>> Thanks Ash
>>>>> > >>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>> Yes agree option 2 would be preferred for me. Making
>>>>> sure we
>>>>> > >>>> have all
>>>>> > >>>>>>>>>> the
>>>>> > >>>>>>>>>>>>> gaurdriles to protect any unwanted behaviour in code
>>>>> sharing
>>>>> > >>> and
>>>>> > >>>>>>>>>>>> executing
>>>>> > >>>>>>>>>>>>> right of tests between the packages.
>>>>> > >>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>> Agree with others, option 2 would be
>>>>> > >>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
>>>>> > >>>>>>>> amoghdesai.oss@gmail.com>
>>>>> > >>>>>>>>>>>>> wrote:
>>>>> > >>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>> Thanks for starting this discussion, Ash.
>>>>> > >>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>> I would prefer option 2 here with proper tooling to
>>>>> handle
>>>>> > the
>>>>> > >>>> code
>>>>> > >>>>>>>>>>>>>> duplication at *release* time.
>>>>> > >>>>>>>>>>>>>> It is best to have a dist that has all it needs in
>>>>> itself.
>>>>> > >>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>> Option 1 could very quickly get out of hand and if we
>>>>> decide
>>>>> > >>> to
>>>>> > >>>>>>>>>> separate
>>>>> > >>>>>>>>>>>>>> triggerer /
>>>>> > >>>>>>>>>>>>>> dag processor / config etc etc as separate packages,
>>>>> back
>>>>> > >>>> compat is
>>>>> > >>>>>>>>>>>> going
>>>>> > >>>>>>>>>>>>>> to be a nightmare
>>>>> > >>>>>>>>>>>>>> and will bite us harder than we anticipate.
>>>>> > >>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>> Thanks & Regards,
>>>>> > >>>>>>>>>>>>>> Amogh Desai
>>>>> > >>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <
>>>>> > >>> kaxilnaik@gmail.com>
>>>>> > >>>>>>>>>> wrote:
>>>>> > >>>>>>>>>>>>>>> I prefer Option 2 as well to avoid matrix of
>>>>> dependencies
>>>>> > >>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>>>>> > >>>>>>>>>> <j_scheffler@gmx.de.invalid
>>>>> > >>>>>>>>>>>>>>> wrote:
>>>>> > >>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>> I'd also rather prefer option 2 - reason here is it
>>>>> is
>>>>> > >>> rather
>>>>> > >>>>>>>>>>>> pragmatic
>>>>> > >>>>>>>>>>>>>>>> and we no not need to cut another package and have
>>>>> less
>>>>> > >>>> package
>>>>> > >>>>>>>>>> counts
>>>>> > >>>>>>>>>>>>>>>> and dependencies.
>>>>> > >>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>> I remember some time ago I was checking (together
>>>>> with
>>>>> > >>> Jarek,
>>>>> > >>>> I am
>>>>> > >>>>>>>>>> not
>>>>> > >>>>>>>>>>>>>>>> sure anymore...) if the usage of symlinks would be
>>>>> > possible.
>>>>> > >>>> To
>>>>> > >>>>>>>> keep
>>>>> > >>>>>>>>>>>>>> the
>>>>> > >>>>>>>>>>>>>>>> source in one package but "symlink" it into
>>>>> another. If
>>>>> > then
>>>>> > >>>> at
>>>>> > >>>>>>>>>> point
>>>>> > >>>>>>>>>>>>>> of
>>>>> > >>>>>>>>>>>>>>>> packaging/release the files are materialized we
>>>>> have 1 set
>>>>> > >>> of
>>>>> > >>>>>>>> code.
>>>>> > >>>>>>>>>>>>>>>> Otherwise if not possible still the redundancy
>>>>> could be
>>>>> > >>>> solved by
>>>>> > >>>>>>>> a
>>>>> > >>>>>>>>>>>>>>>> pre-commit hook - and in Git the files are
>>>>> de-duplicated
>>>>> > >>>> anyway
>>>>> > >>>>>>>>>> based
>>>>> > >>>>>>>>>>>>>> on
>>>>> > >>>>>>>>>>>>>>>> content hash, so this does not hurt.
>>>>> > >>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>>> > >>>>>>>>>>>>>>>>> I support option 2 with proper automation & CI -
>>>>> the
>>>>> > >>>> reasonings
>>>>> > >>>>>>>>>>>>>> you've
>>>>> > >>>>>>>>>>>>>>>>> shown for that make sense to me.
>>>>> > >>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>> Shahar
>>>>> > >>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <
>>>>> > >>>> ash@apache.org
>>>>> > >>>>>>>>>>>>>>> wrote:
>>>>> > >>>>>>>>>>>>>>>>>> Hello everyone,
>>>>> > >>>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>>> As we work on finishing off the code-level
>>>>> separation of
>>>>> > >>>> Task
>>>>> > >>>>>>>> SDK
>>>>> > >>>>>>>>>>>>>> and
>>>>> > >>>>>>>>>>>>>>>> Core
>>>>> > >>>>>>>>>>>>>>>>>> (scheduler etc) we have come across some
>>>>> situations
>>>>> > where
>>>>> > >>> we
>>>>> > >>>>>>>> would
>>>>> > >>>>>>>>>>>>>>> like
>>>>> > >>>>>>>>>>>>>>>> to
>>>>> > >>>>>>>>>>>>>>>>>> share code between these.
>>>>> > >>>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>>> However it’s not as straight forward of “just put
>>>>> it in
>>>>> > a
>>>>> > >>>> common
>>>>> > >>>>>>>>>>>>>> dist
>>>>> > >>>>>>>>>>>>>>>> they
>>>>> > >>>>>>>>>>>>>>>>>> both depend upon” because one of the goals of the
>>>>> Task
>>>>> > SDK
>>>>> > >>>>>>>>>>>>>> separation
>>>>> > >>>>>>>>>>>>>>>> was
>>>>> > >>>>>>>>>>>>>>>>>> to have 100% complete version independence
>>>>> between the
>>>>> > >>> two,
>>>>> > >>>>>>>>>> ideally
>>>>> > >>>>>>>>>>>>>>>> even if
>>>>> > >>>>>>>>>>>>>>>>>> they are built into the same image and venv. Most
>>>>> of the
>>>>> > >>>> reason
>>>>> > >>>>>>>>>> why
>>>>> > >>>>>>>>>>>>>>> this
>>>>> > >>>>>>>>>>>>>>>>>> isn’t straight forward comes down to backwards
>>>>> > >>>> compatibility -
>>>>> > >>>>>>>> if
>>>>> > >>>>>>>>>> we
>>>>> > >>>>>>>>>>>>>>>> make
>>>>> > >>>>>>>>>>>>>>>>>> an change to the common/shared distribution
>>>>> > >>>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>>> We’ve listed the options we have thought about in
>>>>> > >>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545
>>>>> (but
>>>>> > that
>>>>> > >>>> covers
>>>>> > >>>>>>>>>>>>>> some
>>>>> > >>>>>>>>>>>>>>>> more
>>>>> > >>>>>>>>>>>>>>>>>> things that I don’t want to get in to in this
>>>>> discussion
>>>>> > >>>> such as
>>>>> > >>>>>>>>>>>>>>>> possibly
>>>>> > >>>>>>>>>>>>>>>>>> separating operators and executors out of a single
>>>>> > >>> provider
>>>>> > >>>>>>>> dist.)
>>>>> > >>>>>>>>>>>>>>>>>> To give a concrete example of some code I would
>>>>> like to
>>>>> > >>>> share
>>>>> > >>>
>>>>> >
>>>>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>>> > >>>>>>>>>>>>>>>>>> — logging config. Another thing we will want to
>>>>> share
>>>>> > will
>>>>> > >>>> be
>>>>> > >>>>>>>> the
>>>>> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class from
>>>>> airflow.configuration
>>>>> > (but
>>>>> > >>>>>>>> notably:
>>>>> > >>>>>>>>>>>>>>> only
>>>>> > >>>>>>>>>>>>>>>> the
>>>>> > >>>>>>>>>>>>>>>>>> parser class, _not_ the default config values,
>>>>> again,
>>>>> > lets
>>>>> > >>>> not
>>>>> > >>>>>>>>>> dwell
>>>>> > >>>>>>>>>>>>>>> on
>>>>> > >>>>>>>>>>>>>>>> the
>>>>> > >>>>>>>>>>>>>>>>>> specifics of that)
>>>>> > >>>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>>> So to bring the options listed in the issue here
>>>>> for
>>>>> > >>>> discussion,
>>>>> > >>>>>>>>>>>>>>> broadly
>>>>> > >>>>>>>>>>>>>>>>>> speaking there are two high-level approaches:
>>>>> > >>>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>>> 1. A single shared distribution
>>>>> > >>>>>>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
>>>>> > >>>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>>> The advantage of Approach 1 is that we only have
>>>>> the
>>>>> > code
>>>>> > >>>> in one
>>>>> > >>>>>>>>>>>>>>> place.
>>>>> > >>>>>>>>>>>>>>>>>> However for me, at least in this specific case of
>>>>> > Logging
>>>>> > >>>> config
>>>>> > >>>>>>>>>> or
>>>>> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class is that backwards
>>>>> > compatibility
>>>>> > >>> is
>>>>> > >>>>>>>> much
>>>>> > >>>>>>>>>>>>>> much
>>>>> > >>>>>>>>>>>>>>>>>> harder.
>>>>> > >>>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>>> The main advantage of Approach 2 is the the code
>>>>> is
>>>>> > >>> released
>>>>> > >>>>>>>>>>>>>>>> with/embedded
>>>>> > >>>>>>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would
>>>>> contain
>>>>> > >>> the
>>>>> > >>>>>>>> right
>>>>> > >>>>>>>>>>>>>>>> version
>>>>> > >>>>>>>>>>>>>>>>>> of the logging config and ConfigParser etc). The
>>>>> > downside
>>>>> > >>> is
>>>>> > >>>>>>>> that
>>>>> > >>>>>>>>>>>>>>> either
>>>>> > >>>>>>>>>>>>>>>>>> the code will need to be duplicated in the repo,
>>>>> or
>>>>> > better
>>>>> > >>>> yet
>>>>> > >>>>>>>> it
>>>>> > >>>>>>>>>>>>>>> would
>>>>> > >>>>>>>>>>>>>>>>>> live in a single place in the repo, but some
>>>>> tooling
>>>>> > (TBD)
>>>>> > >>>> will
>>>>> > >>>>>>>>>>>>>>>>>> automatically handle the duplication, either at
>>>>> commit
>>>>> > >>>> time, or
>>>>> > >>>>>>>> my
>>>>> > >>>>>>>>>>>>>>>>>> preference, at release time.
>>>>> > >>>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>>> For this kind of shared “utility” code I am very
>>>>> > strongly
>>>>> > >>>>>>>> leaning
>>>>> > >>>>>>>>>>>>>>>> towards
>>>>> > >>>>>>>>>>>>>>>>>> option 2 with automation, as otherwise I think the
>>>>> > >>> backwards
>>>>> > >>>>>>>>>>>>>>>> compatibility
>>>>> > >>>>>>>>>>>>>>>>>> requirements would make it unworkable (very
>>>>> quickly over
>>>>> > >>>> time
>>>>> > >>>>>>>> the
>>>>> > >>>>>>>>>>>>>>>>>> combinations we would have to test would just be
>>>>> > >>>> unreasonable)
>>>>> > >>>>>>>>>> and I
>>>>> > >>>>>>>>>>>>>>>> don’t
>>>>> > >>>>>>>>>>>>>>>>>> feel confident we can have things as stable as we
>>>>> need
>>>>> > to
>>>>> > >>>> really
>>>>> > >>>>>>>>>>>>>>> deliver
>>>>> > >>>>>>>>>>>>>>>>>> the version separation/independency I want to
>>>>> delivery
>>>>> > >>> with
>>>>> > >>>>>>>>>> AIP-72.
>>>>> > >>>>>>>>>>>>>>>>>> So unless someone feels very strongly about this,
>>>>> I will
>>>>> > >>>> come up
>>>>> > >>>>>>>>>>>>>> with
>>>>> > >>>>>>>>>>>>>>> a
>>>>> > >>>>>>>>>>>>>>>>>> draft PR for further discussion that will
>>>>> implement code
>>>>> > >>>> sharing
>>>>> > >>>>>>>>>> via
>>>>> > >>>>>>>>>>>>>>>>>> “vendoring” it at build time. I have an idea of
>>>>> how I
>>>>> > can
>>>>> > >>>>>>>> achieve
>>>>> > >>>>>>>>>>>>>> this
>>>>> > >>>>>>>>>>>>>>>> so
>>>>> > >>>>>>>>>>>>>>>>>> we have a single version in the repo and it’ll
>>>>> work
>>>>> > there,
>>>>> > >>>> but
>>>>> > >>>>>>>> at
>>>>> > >>>>>>>>>>>>>>>> runtime
>>>>> > >>>>>>>>>>>>>>>>>> we vendor it in to the shipped dist so it lives at
>>>>> > >>> something
>>>>> > >>>>>>>> like
>>>>> > >>>>>>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
>>>>> > >>>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>>> In terms of repo layout, this likely means we
>>>>> would end
>>>>> > up
>>>>> > >>>> with:
>>>>> > >>>>>>>>>>>>>>>>>> airflow-core/pyproject.toml
>>>>> > >>>>>>>>>>>>>>>>>> airflow-core/src/
>>>>> > >>>>>>>>>>>>>>>>>> airflow-core/tests/
>>>>> > >>>>>>>>>>>>>>>>>> task-sdk/pyproject.toml
>>>>> > >>>>>>>>>>>>>>>>>> task-sdk/src/
>>>>> > >>>>>>>>>>>>>>>>>> task-sdk/tests/
>>>>> > >>>>>>>>>>>>>>>>>> airflow-common/src
>>>>> > >>>>>>>>>>>>>>>>>> airflow-common/tests/
>>>>> > >>>>>>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as
>>>>> deps
>>>>> > would
>>>>> > >>>> be
>>>>> > >>>>>>>>>>>>>> included
>>>>> > >>>>>>>>>>>>>>>> in
>>>>> > >>>>>>>>>>>>>>>>>> the downstream projects. TBD.
>>>>> > >>>>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>>>>>>>>>>> Thoughts and feedback welcomed.
>>>>> > >>>>
>>>>> ---------------------------------------------------------------------
>>>>> > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>> > dev-unsubscribe@airflow.apache.org
>>>>> > >>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>> > >>> dev-help@airflow.apache.org
>>>>> > >>>>>>>>>>>>>>>>
>>>>> > >>>>>>>>
>>>>> > >>>
>>>>> ---------------------------------------------------------------------
>>>>> > >>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>> > >>>>>>>> For additional commands, e-mail:
>>>>> dev-help@airflow.apache.org
>>>>> > >>>>>>>>
>>>>> > >>>>>>>>
>>>>> > >>>>>>
>>>>> > ---------------------------------------------------------------------
>>>>> > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>> > >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>> > >>>>>>
>>>>> > >>>>>
>>>>> ---------------------------------------------------------------------
>>>>> > >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>> > >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>> > >>>>>
>>>>> > >>>>
>>>>> ---------------------------------------------------------------------
>>>>> > >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>> > >>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>> > >>>>
>>>>> > >>>>
>>>>> > >
>>>>> > >
>>>>> ---------------------------------------------------------------------
>>>>> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>> > > For additional commands, e-mail: dev-help@airflow.apache.org
>>>>> > >
>>>>> >
>>>>> > ---------------------------------------------------------------------
>>>>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>> > For additional commands, e-mail: dev-help@airflow.apache.org
>>>>> >
>>>>> >
>>>>>
>>>>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Jarek Potiuk <ja...@potiuk.com>.
> Not that I am against your idea and we can surely expand as we need but
we would not need to expand the
"core_and_task_sdk" if we put only the relevant items into it.

So if we move logging and config out, my question is what is really
relevant to "stay" in "core_and_task_sdk" ? And what we know will not be
needed by other distributions in the future ?

What would be the content of "core_and_task_sdk" ? Can we enumerate what
should go there now ?

My bet is that if we enumerate them, it will turn out that they are good
candidates to make separate "shared features" that can be logically named
and modularised, and that there is no real need to have shared
"core_and_task_sdk" which sounds like "bag of everything else".



On Tue, Jul 8, 2025 at 11:48 AM Amogh Desai <am...@gmail.com>
wrote:

> Yeah, I think what you are showcasing here is a step ahead of the
> initial proposal from Ash.
>
> From the original proposal, the `core_and_task_sdk` *can* have the things
> relevant to just those two
> distros. Logging, Config are modules that might be needed by airflow-ctl
> for example, so ideally, those
> would not be good candidates to be put in there, ideally speaking.
>
> The example of Kubernetes Utils sounds to be a good example, it will be
> used by KubeExecutor (lets say this is a
> module called "executors") and by KPO (providers), and the
> "shared/kubernetes" would probably be a good
> candidate for that.
>
> Not that I am against your idea and we can surely expand as we need but we
> would not need to expand the
> "core_and_task_sdk" if we put only the relevant items into it.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Tue, Jul 8, 2025 at 12:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> > @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
>> there, I am understanding the direction
>> but could you elaborate a bit more please?
>>
>> Let me elaborate:
>>
>> As I understand (maybe I am wrong?),  the proposal is that we have a
>> "core-and-task-sdk" folder which is a shared distribution that is
>> vendored-in into both "airflow-core" and "airflow-task-sdk". This
>> contains some shared code that we want to include in both distributions
>> (note that we never ever release "core-and-task-sdk"
>> distribution because it only contains code that is shared between the two
>> distributions we release.
>>
>> That's fine and cool.
>>
>> Imagine that this distribution contains "logging" (shared code
>> for logging) and "config" (shared code for configuration). - both needed in
>> "airflow-core" and "airflow-task-sdk". So far so good. But what happen if
>> we want to use logging in the same fashion in say "airflow-ctl" (that also
>> applies for other distributions that we might come up with) ? Are we going
>> to vendor in the whole "core-and-task-sdk" distribution in "airflow-ctl" ?
>> It would be far better if we just vendor in "logging" and do not vendor-in
>> "config".
>>
>> And if we are going to have a mechanism to vendor-in "a distribution" -
>> there is nothing wrong with having the same way to vendor-in multiple
>> distributions - so we can easily do it this way (i added "orm" ,
>> "serialization". and "fast_api" as an example thing that we might want to
>> share - not sure if that is really something we want to do but it will
>> allow to illustrate my idea better)
>>
>> /
>>   airflow-ctl/
>>   task-sdk/...
>>   airflow-core/...
>>   ....
>>   shared/
>>     kubernetes/
>>       pyproject.toml
>>       src/
>>         airflow_shared_kubernetes/__init__.py
>>     logging/
>>       pyproject.toml
>>       src/
>>         airflow_shared_logging/__init__.py
>>     config/
>>       pyproject.toml
>>       src/
>>         airflow_shared_config/__init__.py
>>     orm/
>>       pyproject.toml
>>       src/
>>         airflow_shared_orm/__init__.py
>>      serialization/
>>       pyproject.toml
>>       src/
>>         airflow_shared_serialization/__init__.py
>>      fast_api/
>>       pyproject.toml
>>       src/
>>         airflow_shared_fast_api/__init__.py
>>     ...
>>
>> This has multiple benefits (and I see no real drawbacks):
>>
>> * the code can be really well modularised.  Those "things" we share (and
>> this also connects to the __init__.py discussion) - can be independent -
>> and (it follow Jens comment) it allows to keep low cyclomatic complexity
>> https://en.wikipedia.org/wiki/Cyclomatic_complexity . It will be way
>> easier to implement logging in the way that it does not import or use
>> config. This means for example that configuration for logging will need to
>> be injected when logging is initialized - and that's exactly what we want,
>> we do not want logging code to use configuration code directly - they
>> should be independent from each other and you should be free to vendor-in
>> either logging or config independently if you also vendored-in the other.
>>
>> * it's much more logical. We split based on functionality we want to
>> share - not about the "distributions" we want to produce. That allows us -
>> in the future - to make different decisions on how we split our
>> distributions. For example (i do not tell we have to do it, or that we will
>> do it but we will have such a possibility) - we can add more shared
>> utilities we find useful in the same way and decide that "scheduler",
>> "api_server" or "scheduler" or "triggerer" or "dag processor" are split to
>> separate distributions - because for example we want to keep a number of
>> dependencies down. And for example "api_server" might use "fast_api",
>> "config", "logging", "orm" and "fast_api" and "serialization" , where the
>> scheduler should not need "fast_api". The "dag_processor" eventually might
>> not need "orm" nor "fast_api" and only use the other
>>
>> This seems like a natural approach. If we have a mechanism to "share" the
>> code, it does not add complexity, but allows us to isolate independent
>> functionality into "isolated" boxes and use them
>>
>> Also for cyclomatic complexity that is a complex word (badly chosen as it
>> scares people away) and has some math behind, but it really boils down to
>> very simple "rules of thumb" (and yes I am a big proponent of having low
>> cyclomatic complexity).
>>
>> a) when you are building the "final" product (i.e. distribution you want
>> to release) - make sure that you only "use" things - that nothing else is
>> "using" you as a library.
>> b) when you are building a "shared" thing (a library) - make sure that
>> library is only "used" by others but it does not really "use" anything
>> else. For example in the case I explained above - we can achieve
>> low-cyclomatic complexity when:
>>
>> * airflow-core uses: logging, config, orm, serialization, fast_api
>> * none of the "logging, config, orm, serialization, fast_api" use each
>> other - they are at the bottom of the "user -> used" tree
>>
>> J.
>>
>>
>>
>> On Tue, Jul 8, 2025 at 8:16 AM Amogh Desai <am...@gmail.com>
>> wrote:
>>
>>> I like the folder structure proposed by Ash and have no objections with
>>> it.
>>>
>>> "core_and_task_sdk" sounds good to me and justifies what it should do
>>> pretty well.
>>>
>>> @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
>>> there, I am understanding the direction
>>> but could you elaborate a bit more please?
>>>
>>> Naming is REALLY hard!
>>>
>>> Thanks & Regards,
>>> Amogh Desai
>>>
>>>
>>> On Tue, Jul 8, 2025 at 2:52 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>> How about splitting it even more and having each shared "thing" named?
>>>> "logging", "config" and sharing them explicitly and separately with the
>>>> right "user" ?
>>>> That sounds way more modular and  we will be able to choose which of the
>>>> shared "utils" we use where.
>>>>
>>>> J.
>>>>
>>>>
>>>> On Mon, Jul 7, 2025 at 11:13 PM Jens Scheffler
>>>> <j_...@gmx.de.invalid>
>>>> wrote:
>>>>
>>>> > I like "core_and_task_sdk the same like core-and-task-sdk - I have no
>>>> > problem and it is a path only.
>>>> >
>>>> > if we get to "dag-parser-scheduler-task-sdk-and-triggerer" which is a
>>>> > bit bulky we then should name it "all-not-api-server" :-D
>>>> >
>>>> > On 07.07.25 22:57, Ash Berlin-Taylor wrote:
>>>> > > In case I did a bad job explaining it, the “core and task sdk” is
>>>> not in
>>>> > the module name/import name, just in the file path.
>>>> > >
>>>> > > Anyone have other ideas?
>>>> > >
>>>> > >> On 7 Jul 2025, at 21:37, Buğra Öztürk <oz...@gmail.com>
>>>> wrote:
>>>> > >>
>>>> > >> Thanks Ash! Looks cool! I like the structure. This will enable
>>>> all the
>>>> > >> combinations and structure looks easy to grasp. No strong stance
>>>> on the
>>>> > >> naming other than maybe it is a bit long with `and`, `core_ctl`
>>>> could be
>>>> > >> shorter, since no import path is defined like that, we can give
>>>> any name
>>>> > >> for sure.
>>>> > >>
>>>> > >> Best regards,
>>>> > >>
>>>> > >>> On Mon, 7 Jul 2025, 21:51 Jarek Potiuk, <ja...@potiuk.com> wrote:
>>>> > >>>
>>>> > >>> Looks good but I think we should find some better logical name for
>>>> > >>> core_and_sdk :)
>>>> > >>>
>>>> > >>> pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
>>>> > >>> <j_...@gmx.de.invalid> napisał:
>>>> > >>>
>>>> > >>>> Cool! Especially the "shared" folder with the ability to have
>>>> > >>>> N-combinations w/o exploding project repo root!
>>>> > >>>>
>>>> > >>>> On 07.07.25 14:43, Ash Berlin-Taylor wrote:
>>>> > >>>>> Oh, and all of this will be explain in shared/README.md
>>>> > >>>>>
>>>> > >>>>>> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org>
>>>> wrote:
>>>> > >>>>>>
>>>> > >>>>>> Okay, so it seems we have agreement on the approach here, so
>>>> I’ll
>>>> > >>>> continue with this, and on the dev call it was mentioned that
>>>> > >>>> “airflow-common” wasn’t a great name, so here is my proposal for
>>>> the
>>>> > file
>>>> > >>>> structure;
>>>> > >>>>>> ```
>>>> > >>>>>> /
>>>> > >>>>>>   task-sdk/...
>>>> > >>>>>>   airflow-core/...
>>>> > >>>>>>   shared/
>>>> > >>>>>>     kuberenetes/
>>>> > >>>>>>       pyproject.toml
>>>> > >>>>>>       src/
>>>> > >>>>>>         airflow_kube/__init__.py
>>>> > >>>>>>     core-and-tasksdk/
>>>> > >>>>>>       pyproject.toml
>>>> > >>>>>>       src/
>>>> > >>>>>>         airflow_shared/__init__.py
>>>> > >>>>>> ```
>>>> > >>>>>>
>>>> > >>>>>> Things to note here: the “shared” folder has (the possibility)
>>>> of
>>>> > >>>> having multiple different shared “libraries” in it, in this
>>>> example I
>>>> > am
>>>> > >>>> supposing a hypothetical shared kuberenetes folder a world in
>>>> which we
>>>> > >>>> split the KubePodOperator and the KubeExecutor in to two separate
>>>> > >>>> distributions (example only, not proposing we do that right now,
>>>> and
>>>> > that
>>>> > >>>> will be a separate discussion)
>>>> > >>>>>> The other things to note here:
>>>> > >>>>>>
>>>> > >>>>>>
>>>> > >>>>>> - the folder name in shared aims to be “self-documenting”,
>>>> hence the
>>>> > >>>> verbose “core-and-tasksdk” to say where the shared library is
>>>> > intended to
>>>> > >>>> be used.
>>>> > >>>>>> - the python module itself should almost always have an
>>>> `airflow_`
>>>> > (or
>>>> > >>>> maybe `_airflow_`?) prefix so that it does not conflict with
>>>> anything
>>>> > >>> else
>>>> > >>>> we might use. It won’t matter “in production” as those will be
>>>> > vendored
>>>> > >>> in
>>>> > >>>> to be imported as `airflow/_vendor/airflow_shared` etc, but
>>>> avoiding
>>>> > >>>> conflicts at dev time with the Finder approach is a good safety
>>>> > measure.
>>>> > >>>>>> I will start making a real PR for this proposal now, but I’m
>>>> open to
>>>> > >>>> feedback (either here, or in the PR when I open it)
>>>> > >>>>>> -ash
>>>> > >>>>>>
>>>> > >>>>>>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com>
>>>> wrote:
>>>> > >>>>>>>
>>>> > >>>>>>> Yeah we have to try it and test - also building packages
>>>> happens
>>>> > semi
>>>> > >>>>>>> frequently when you run `uv sync` (they use some kind of
>>>> heuristics
>>>> > >>> to
>>>> > >>>>>>> decide when) and you can force it with `--reinstall` or
>>>> > `--refresh`.
>>>> > >>>>>>> Package build also happens every time when you run "ci-image
>>>> build`
>>>> > >>>> now in
>>>> > >>>>>>> breeze so it seems like it will nicely integrate in our
>>>> workflows.
>>>> > >>>>>>>
>>>> > >>>>>>> Looks really cool Ash.
>>>> > >>>>>>>
>>>> > >>>>>>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <
>>>> ash@apache.org>
>>>> > >>>> wrote:
>>>> > >>>>>>>> It’s not just release time, but any time we build a package
>>>> which
>>>> > >>>> happens
>>>> > >>>>>>>> on “every” CI run. The normal unit tests will use code from
>>>> > >>>>>>>> airflow-common/src/airflow_common; the kube tests which
>>>> build an
>>>> > >>>> image will
>>>> > >>>>>>>> build the dists and vendor in the code from that commit.
>>>> > >>>>>>>>
>>>> > >>>>>>>> There is only a single copy of the shared code committed to
>>>> the
>>>> > >>> repo,
>>>> > >>>> so
>>>> > >>>>>>>> there is never anything to synchronise.
>>>> > >>>>>>>>
>>>> > >>>>>>>>> On 4 Jul 2025, at 15:53, Amogh Desai <
>>>> amoghdesai.oss@gmail.com>
>>>> > >>>> wrote:
>>>> > >>>>>>>>> Thanks Ash.
>>>> > >>>>>>>>>
>>>> > >>>>>>>>> This is really cool and helpful that you were able to test
>>>> both
>>>> > >>>> scenarios
>>>> > >>>>>>>>> -- repo checkout
>>>> > >>>>>>>>> and also installing from the vendored package and the
>>>> resolution
>>>> > >>>> worked
>>>> > >>>>>>>>> fine too.
>>>> > >>>>>>>>>
>>>> > >>>>>>>>> I like this idea compared the to relative import one for few
>>>> > >>> reasons:
>>>> > >>>>>>>>> - It feels like it will take some time to adjust to the new
>>>> > coding
>>>> > >>>>>>>> standard
>>>> > >>>>>>>>> that we will lay
>>>> > >>>>>>>>> if we impose relative imports in the shared dist
>>>> > >>>>>>>>> - We can continue using repo wise absolute import
>>>> standards, it
>>>> > is
>>>> > >>>> also
>>>> > >>>>>>>>> much easier for situations
>>>> > >>>>>>>>> when we do global search in IDE to find + replace, this
>>>> could
>>>> > mean
>>>> > >>> a
>>>> > >>>>>>>> change
>>>> > >>>>>>>>> there
>>>> > >>>>>>>>> - The vendoring work is a proven and established paradigm
>>>> across
>>>> > >>>> projects
>>>> > >>>>>>>>> and would
>>>> > >>>>>>>>> out of box give us the build tooling we need also
>>>> > >>>>>>>>>
>>>> > >>>>>>>>> Nothing too against the relative import but with the
>>>> evidence
>>>> > >>>> provided
>>>> > >>>>>>>>> above, vendored approach
>>>> > >>>>>>>>> seems to only do us good.
>>>> > >>>>>>>>>
>>>> > >>>>>>>>> Regarding synchronizing it, release time should be fine as
>>>> long
>>>> > as
>>>> > >>> we
>>>> > >>>>>>>> have
>>>> > >>>>>>>>> a good CI workflow to probably
>>>> > >>>>>>>>> catch such issues per PR if changes are made in shared dist?
>>>> > >>>> (precommit
>>>> > >>>>>>>>> would make it really slow i guess)
>>>> > >>>>>>>>>
>>>> > >>>>>>>>> If we can run our tests with vendored code we should be
>>>> mostly
>>>> > >>>> covered.
>>>> > >>>>>>>>> Good effort all!
>>>> > >>>>>>>>>
>>>> > >>>>>>>>> Thanks & Regards,
>>>> > >>>>>>>>> Amogh Desai
>>>> > >>>>>>>>>
>>>> > >>>>>>>>>
>>>> > >>>>>>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <
>>>> > ash@apache.org>
>>>> > >>>>>>>> wrote:
>>>> > >>>>>>>>>> Okay, I think I’ve got something that works and I’m happy
>>>> with.
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>
>>>> >
>>>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>>>> > >>>>>>>>>> This produces the following from `uv build task-sdk`
>>>> > >>>>>>>>>> -
>>>> > >>>>>>>>>>
>>>> > >>>
>>>> >
>>>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>>>> > >>>>>>>>>> -
>>>> > >>>>>>>>>>
>>>> > >>>
>>>> >
>>>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>>>> > >>>>>>>>>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> ```
>>>> > >>>>>>>>>> ❯ unzip -l
>>>> > >>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
>>>> > >>>>>>>> grep
>>>> > >>>>>>>>>> _vendor
>>>> > >>>>>>>>>>      50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
>>>> > >>>>>>>>>>    2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
>>>> > >>>>>>>>>>      28  02-02-2020 00:00
>>>> >  airflow/sdk/_vendor/airflow_common.pyi
>>>> > >>>>>>>>>>      18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
>>>> > >>>>>>>>>>     785  02-02-2020 00:00
>>>> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
>>>> > >>>>>>>>>>   10628  02-02-2020 00:00
>>>> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
>>>> > >>>>>>>>>> ```
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> And similarly in the .tar.gz, so our “sdist” is complete
>>>> too:
>>>> > >>>>>>>>>> ```
>>>> > >>>>>>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep
>>>> > _vendor
>>>> > >>>>>>>>>>
>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>>>> > >>>>>>>>>>
>>>> > apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>>>> > >>>>>>>>>>
>>>> > >>>>
>>>> >
>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>>>> > >>>>>>>>>>
>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>
>>>> >
>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>>>> > >>>
>>>> >
>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>>>> > >>>>>>>>>> ```
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> The plugin works at build time by including/copying the
>>>> libs
>>>> > >>>> specified
>>>> > >>>>>>>> in
>>>> > >>>>>>>>>> vendor.txt into place (and let `vendoring` take care of
>>>> import
>>>> > >>>>>>>> rewrites.)
>>>> > >>>>>>>>>> For the imports to continue to work at “dev” time/from a
>>>> repo
>>>> > >>>> checkout,
>>>> > >>>>>>>> I
>>>> > >>>>>>>>>> have added a import finder to `sys.meta_path`, and since
>>>> it’s at
>>>> > >>> the
>>>> > >>>>>>>> end of
>>>> > >>>>>>>>>> the list it will only be used if the normal import can’t
>>>> find
>>>> > >>>> things.
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>
>>>> >
>>>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>>>> > >>>>>>>>>> This doesn’t quite give us the same runtime effect “import
>>>> > >>>> rewriting”
>>>> > >>>>>>>>>> affect, as in this approach `airflow_common` is directly
>>>> loaded
>>>> > >>>> (i.e.
>>>> > >>>>>>>>>> airflow.sdk._vendor.airflow_common and airflow_common
>>>> exist in
>>>> > >>>>>>>>>> sys.modules), but it does work for everything that I was
>>>> able to
>>>> > >>>> test..
>>>> > >>>>>>>>>> I tested it with the diff at the end of this message. My
>>>> test
>>>> > >>>> ipython
>>>> > >>>>>>>>>> shell:
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> ```
>>>> > >>>>>>>>>> In [1]: from airflow.sdk._vendor.airflow_common.timezone
>>>> import
>>>> > >>> foo
>>>> > >>>>>>>>>> In [2]: foo
>>>> > >>>>>>>>>> Out[2]: 1
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> In [3]: import airflow.sdk._vendor.airflow_common
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> In [4]: import airflow.sdk._vendor.airflow_common.timezone
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>>>> > >>>>>>>>>> Out[5]:
>>>> > >>>>>>>>>>
>>>> > >>>
>>>> >
>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>>>> > >>>>>>>>>> In [6]:
>>>> airflow.sdk._vendor.airflow_common.timezone.__file__
>>>> > >>>>>>>>>> Out[6]:
>>>> > >>>>>>>>>>
>>>> > >>>
>>>> >
>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>>>> > >>>>>>>>>> ```
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> And in an standalone environment with the SDK dist I built
>>>> (it
>>>> > >>>> needed
>>>> > >>>>>>>> the
>>>> > >>>>>>>>>> matching airflow-core right now, but that is nothing to do
>>>> with
>>>> > >>> this
>>>> > >>>>>>>>>> discussion):
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> ```
>>>> > >>>>>>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>>>> > >>>>>>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>>>> > >>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
>>>> > >>>>>>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
>>>> > >>>>>>>>>> Type 'copyright', 'credits' or 'license' for more
>>>> information
>>>> > >>>>>>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type '?'
>>>> for
>>>> > >>> help.
>>>> > >>>>>>>>>> Tip: You can use `%hist` to view history, see the options
>>>> with
>>>> > >>>>>>>> `%history?`
>>>> > >>>>>>>>>> In [1]: import airflow.sdk._vendor.airflow_common.timezone
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> In [2]:
>>>> airflow.sdk._vendor.airflow_common.timezone.__file__
>>>> > >>>>>>>>>> Out[2]:
>>>> > >>>>>>>>>>
>>>> > >>>
>>>> >
>>>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>>>> > >>>>>>>>>> ``
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> ```diff
>>>> > >>>>>>>>>> diff --git a/airflow-common/src/airflow_common/__init__.py
>>>> > >>>>>>>>>> b/airflow-common/src/airflow_common/__init__.py
>>>> > >>>>>>>>>> index 13a83393a9..927b7c6b61 100644
>>>> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/__init__.py
>>>> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
>>>> > >>>>>>>>>> @@ -14,3 +14,5 @@
>>>> > >>>>>>>>>> # KIND, either express or implied.  See the License for the
>>>> > >>>>>>>>>> # specific language governing permissions and limitations
>>>> > >>>>>>>>>> # under the License.
>>>> > >>>>>>>>>> +
>>>> > >>>>>>>>>> +foo = 1
>>>> > >>>>>>>>>> diff --git a/airflow-common/src/airflow_common/timezone.py
>>>> > >>>>>>>>>> b/airflow-common/src/airflow_common/timezone.py
>>>> > >>>>>>>>>> index 340b924c66..58384ef20f 100644
>>>> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/timezone.py
>>>> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
>>>> > >>>>>>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>>>> > >>>>>>>>>> version.parse(metadata.version("pendulum")).major == 3
>>>> > >>>>>>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
>>>> > >>>>>>>>>> utc = pendulum.UTC
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> +
>>>> > >>>>>>>>>> +from airflow_common import foo
>>>> > >>>>>>>>>> +
>>>> > >>>>>>>>>> TIMEZONE: Timezone
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>> ```
>>>> > >>>>>>>>>>
>>>> > >>>>>>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com>
>>>> > wrote:
>>>> > >>>>>>>>>>> I think both approaches are doable:
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>> 1) -> We can very easily prevent bad imports by
>>>> pre-commit when
>>>> > >>>>>>>> importing
>>>> > >>>>>>>>>>> from different distributions and make sure we are only
>>>> doing
>>>> > >>>> relative
>>>> > >>>>>>>>>>> imports in the shared modules. We are doing plenty of this
>>>> > >>>> already. And
>>>> > >>>>>>>>>> yes
>>>> > >>>>>>>>>>> it would require relative links we currently do not allow.
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>> 2) -> has one disadvantage that someone at some point in
>>>> time
>>>> > >>> will
>>>> > >>>> have
>>>> > >>>>>>>>>> to
>>>> > >>>>>>>>>>> decide to synchronize this and if it happens just before
>>>> > release
>>>> > >>>> (I bet
>>>> > >>>>>>>>>>> this is going to happen) this will lead to solving
>>>> problems
>>>> > that
>>>> > >>>> would
>>>> > >>>>>>>>>>> normally be solved during PR when you make a change (i.e.
>>>> > >>> symbolic
>>>> > >>>> link
>>>> > >>>>>>>>>> has
>>>> > >>>>>>>>>>> the advantage that whoever modifies shared code will be
>>>> > >>> immediately
>>>> > >>>>>>>>>>> notified in their PR - that they broke something because
>>>> either
>>>> > >>>> static
>>>> > >>>>>>>>>>> checks or mypy or tests fail.
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>> Ash, do you have an idea of a process (who and when) does
>>>> the
>>>> > >>>>>>>>>>> synchronisation in case of vendoring? Maybe we could
>>>> solve it
>>>> > if
>>>> > >>>> it is
>>>> > >>>>>>>>>> done
>>>> > >>>>>>>>>>> more frequently and with some regularity? We could
>>>> potentially
>>>> > >>>> force
>>>> > >>>>>>>>>>> re-vendoring at PR time as well any time shared code
>>>> changes
>>>> > (and
>>>> > >>>>>>>> prevent
>>>> > >>>>>>>>>>> it by pre-commit. And I can't think of some place (other
>>>> than
>>>> > >>>> releases)
>>>> > >>>>>>>>>> in
>>>> > >>>>>>>>>>> our development workflow and that seems to be a bit too
>>>> late as
>>>> > >>>> puts an
>>>> > >>>>>>>>>>> extra effort on fixing potential incompatibilities
>>>> introduced
>>>> > on
>>>> > >>>>>>>> release
>>>> > >>>>>>>>>>> manager and delays the release. WDYT?
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>> Re: relative links. I think for a shared library we could
>>>> > >>>> potentially
>>>> > >>>>>>>>>> relax
>>>> > >>>>>>>>>>> this and allow them (and actually disallow absolute links
>>>> in
>>>> > the
>>>> > >>>> pieces
>>>> > >>>>>>>>>> of
>>>> > >>>>>>>>>>> code that are shared - again, by pre-commit). As I
>>>> recall, the
>>>> > >>> only
>>>> > >>>>>>>>>> reason
>>>> > >>>>>>>>>>> we forbade the relative links is because of how we are (or
>>>> > maybe
>>>> > >>>> were)
>>>> > >>>>>>>>>>> doing DAG parsing and failures resulting from it. So we
>>>> decided
>>>> > >>> to
>>>> > >>>> just
>>>> > >>>>>>>>>> not
>>>> > >>>>>>>>>>> allow it to keep consistency. The way how Dag parsing
>>>> works is
>>>> > >>> that
>>>> > >>>>>>>> when
>>>> > >>>>>>>>>>> you are using importlib to read the Dag from a file, the
>>>> > relative
>>>> > >>>>>>>> imports
>>>> > >>>>>>>>>>> do not work as it does not know what they should be
>>>> relative
>>>> > to.
>>>> > >>>> But if
>>>> > >>>>>>>>>>> relative import is done from an imported package, it
>>>> should be
>>>> > no
>>>> > >>>>>>>>>> problem,
>>>> > >>>>>>>>>>> I think - otherwise our Dags would not be able to import
>>>> any
>>>> > >>>> library
>>>> > >>>>>>>> that
>>>> > >>>>>>>>>>> uses relative imports.
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>> Of course consistency might be the reason why we do not
>>>> want to
>>>> > >>>>>>>> introduce
>>>> > >>>>>>>>>>> relative imports. I don't see it as an issue if it is
>>>> guarded
>>>> > by
>>>> > >>>>>>>>>> pre-commit
>>>> > >>>>>>>>>>> though.
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>> J.
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>> J.
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
>>>> > >>>> ash@apache.org>
>>>> > >>>>>>>>>>> napisał:
>>>> > >>>>>>>>>>>
>>>> > >>>>>>>>>>>> Oh yes, symlinks will work, with one big caveat: It does
>>>> mean
>>>> > >>> you
>>>> > >>>>>>>> can’t
>>>> > >>>>>>>>>>>> use absolute imports in one common module to another.
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>> For example
>>>> > >>>>>>>>>>>>
>>>> > >>>
>>>> >
>>>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>>>> > >>>>>>>>>>>> where we have
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>> ```
>>>> > >>>>>>>>>>>> from airflow.utils.module_loading import import_string
>>>> > >>>>>>>>>>>> ```
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>> if we want to move serve_logs into this common lib that
>>>> is
>>>> > then
>>>> > >>>>>>>>>> symlinked
>>>> > >>>>>>>>>>>> then we wouldn’t be able to have `from
>>>> > >>>> airflow_common.module_loading
>>>> > >>>>>>>>>> import
>>>> > >>>>>>>>>>>> import_string`.
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>> I can think of two possible solutions here.
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>> 1) is to allow/require relative imports in this shared
>>>> lib, so
>>>> > >>>> `from
>>>> > >>>>>>>>>>>> .module_loading import import_string`
>>>> > >>>>>>>>>>>> 2) is to use `vendoring`[1] (from the pip maintainers)
>>>> which
>>>> > >>> will
>>>> > >>>>>>>> handle
>>>> > >>>>>>>>>>>> import-rewriting for us.
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>> I’d entirely forgot that symlinks in repos was a thing,
>>>> so I
>>>> > >>>> prepared
>>>> > >>>>>>>> a
>>>> > >>>>>>>>>>>> minimal POC/demo of what vendoring approach could look
>>>> like
>>>> > here
>>>> > >>>>>>>>>>>>
>>>> > >>>
>>>> >
>>>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>>>> > >>>>>>>>>>>> Now personally I am more than happy with relative
>>>> imports, but
>>>> > >>>>>>>> generally
>>>> > >>>>>>>>>>>> as a project we have avoided them, so I think that
>>>> limits what
>>>> > >>> we
>>>> > >>>>>>>> could
>>>> > >>>>>>>>>> do
>>>> > >>>>>>>>>>>> with a symlink based approach.
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>> -ash
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>> [1] https://github.com/pradyunsg/vendoring
>>>> > >>>>>>>>>>>>
>>>> > >>>>>>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
>>>> > >>>>>>>> gopidesupavan@gmail.com>
>>>> > >>>>>>>>>>>> wrote:
>>>> > >>>>>>>>>>>>> Thanks Ash
>>>> > >>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>> Yes agree option 2 would be preferred for me. Making
>>>> sure we
>>>> > >>>> have all
>>>> > >>>>>>>>>> the
>>>> > >>>>>>>>>>>>> gaurdriles to protect any unwanted behaviour in code
>>>> sharing
>>>> > >>> and
>>>> > >>>>>>>>>>>> executing
>>>> > >>>>>>>>>>>>> right of tests between the packages.
>>>> > >>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>> Agree with others, option 2 would be
>>>> > >>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
>>>> > >>>>>>>> amoghdesai.oss@gmail.com>
>>>> > >>>>>>>>>>>>> wrote:
>>>> > >>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>> Thanks for starting this discussion, Ash.
>>>> > >>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>> I would prefer option 2 here with proper tooling to
>>>> handle
>>>> > the
>>>> > >>>> code
>>>> > >>>>>>>>>>>>>> duplication at *release* time.
>>>> > >>>>>>>>>>>>>> It is best to have a dist that has all it needs in
>>>> itself.
>>>> > >>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>> Option 1 could very quickly get out of hand and if we
>>>> decide
>>>> > >>> to
>>>> > >>>>>>>>>> separate
>>>> > >>>>>>>>>>>>>> triggerer /
>>>> > >>>>>>>>>>>>>> dag processor / config etc etc as separate packages,
>>>> back
>>>> > >>>> compat is
>>>> > >>>>>>>>>>>> going
>>>> > >>>>>>>>>>>>>> to be a nightmare
>>>> > >>>>>>>>>>>>>> and will bite us harder than we anticipate.
>>>> > >>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>> Thanks & Regards,
>>>> > >>>>>>>>>>>>>> Amogh Desai
>>>> > >>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <
>>>> > >>> kaxilnaik@gmail.com>
>>>> > >>>>>>>>>> wrote:
>>>> > >>>>>>>>>>>>>>> I prefer Option 2 as well to avoid matrix of
>>>> dependencies
>>>> > >>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>>>> > >>>>>>>>>> <j_scheffler@gmx.de.invalid
>>>> > >>>>>>>>>>>>>>> wrote:
>>>> > >>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>> I'd also rather prefer option 2 - reason here is it
>>>> is
>>>> > >>> rather
>>>> > >>>>>>>>>>>> pragmatic
>>>> > >>>>>>>>>>>>>>>> and we no not need to cut another package and have
>>>> less
>>>> > >>>> package
>>>> > >>>>>>>>>> counts
>>>> > >>>>>>>>>>>>>>>> and dependencies.
>>>> > >>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>> I remember some time ago I was checking (together
>>>> with
>>>> > >>> Jarek,
>>>> > >>>> I am
>>>> > >>>>>>>>>> not
>>>> > >>>>>>>>>>>>>>>> sure anymore...) if the usage of symlinks would be
>>>> > possible.
>>>> > >>>> To
>>>> > >>>>>>>> keep
>>>> > >>>>>>>>>>>>>> the
>>>> > >>>>>>>>>>>>>>>> source in one package but "symlink" it into another.
>>>> If
>>>> > then
>>>> > >>>> at
>>>> > >>>>>>>>>> point
>>>> > >>>>>>>>>>>>>> of
>>>> > >>>>>>>>>>>>>>>> packaging/release the files are materialized we have
>>>> 1 set
>>>> > >>> of
>>>> > >>>>>>>> code.
>>>> > >>>>>>>>>>>>>>>> Otherwise if not possible still the redundancy could
>>>> be
>>>> > >>>> solved by
>>>> > >>>>>>>> a
>>>> > >>>>>>>>>>>>>>>> pre-commit hook - and in Git the files are
>>>> de-duplicated
>>>> > >>>> anyway
>>>> > >>>>>>>>>> based
>>>> > >>>>>>>>>>>>>> on
>>>> > >>>>>>>>>>>>>>>> content hash, so this does not hurt.
>>>> > >>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>> > >>>>>>>>>>>>>>>>> I support option 2 with proper automation & CI - the
>>>> > >>>> reasonings
>>>> > >>>>>>>>>>>>>> you've
>>>> > >>>>>>>>>>>>>>>>> shown for that make sense to me.
>>>> > >>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>> Shahar
>>>> > >>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <
>>>> > >>>> ash@apache.org
>>>> > >>>>>>>>>>>>>>> wrote:
>>>> > >>>>>>>>>>>>>>>>>> Hello everyone,
>>>> > >>>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>>> As we work on finishing off the code-level
>>>> separation of
>>>> > >>>> Task
>>>> > >>>>>>>> SDK
>>>> > >>>>>>>>>>>>>> and
>>>> > >>>>>>>>>>>>>>>> Core
>>>> > >>>>>>>>>>>>>>>>>> (scheduler etc) we have come across some situations
>>>> > where
>>>> > >>> we
>>>> > >>>>>>>> would
>>>> > >>>>>>>>>>>>>>> like
>>>> > >>>>>>>>>>>>>>>> to
>>>> > >>>>>>>>>>>>>>>>>> share code between these.
>>>> > >>>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>>> However it’s not as straight forward of “just put
>>>> it in
>>>> > a
>>>> > >>>> common
>>>> > >>>>>>>>>>>>>> dist
>>>> > >>>>>>>>>>>>>>>> they
>>>> > >>>>>>>>>>>>>>>>>> both depend upon” because one of the goals of the
>>>> Task
>>>> > SDK
>>>> > >>>>>>>>>>>>>> separation
>>>> > >>>>>>>>>>>>>>>> was
>>>> > >>>>>>>>>>>>>>>>>> to have 100% complete version independence between
>>>> the
>>>> > >>> two,
>>>> > >>>>>>>>>> ideally
>>>> > >>>>>>>>>>>>>>>> even if
>>>> > >>>>>>>>>>>>>>>>>> they are built into the same image and venv. Most
>>>> of the
>>>> > >>>> reason
>>>> > >>>>>>>>>> why
>>>> > >>>>>>>>>>>>>>> this
>>>> > >>>>>>>>>>>>>>>>>> isn’t straight forward comes down to backwards
>>>> > >>>> compatibility -
>>>> > >>>>>>>> if
>>>> > >>>>>>>>>> we
>>>> > >>>>>>>>>>>>>>>> make
>>>> > >>>>>>>>>>>>>>>>>> an change to the common/shared distribution
>>>> > >>>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>>> We’ve listed the options we have thought about in
>>>> > >>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545
>>>> (but
>>>> > that
>>>> > >>>> covers
>>>> > >>>>>>>>>>>>>> some
>>>> > >>>>>>>>>>>>>>>> more
>>>> > >>>>>>>>>>>>>>>>>> things that I don’t want to get in to in this
>>>> discussion
>>>> > >>>> such as
>>>> > >>>>>>>>>>>>>>>> possibly
>>>> > >>>>>>>>>>>>>>>>>> separating operators and executors out of a single
>>>> > >>> provider
>>>> > >>>>>>>> dist.)
>>>> > >>>>>>>>>>>>>>>>>> To give a concrete example of some code I would
>>>> like to
>>>> > >>>> share
>>>> > >>>
>>>> >
>>>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>> > >>>>>>>>>>>>>>>>>> — logging config. Another thing we will want to
>>>> share
>>>> > will
>>>> > >>>> be
>>>> > >>>>>>>> the
>>>> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class from
>>>> airflow.configuration
>>>> > (but
>>>> > >>>>>>>> notably:
>>>> > >>>>>>>>>>>>>>> only
>>>> > >>>>>>>>>>>>>>>> the
>>>> > >>>>>>>>>>>>>>>>>> parser class, _not_ the default config values,
>>>> again,
>>>> > lets
>>>> > >>>> not
>>>> > >>>>>>>>>> dwell
>>>> > >>>>>>>>>>>>>>> on
>>>> > >>>>>>>>>>>>>>>> the
>>>> > >>>>>>>>>>>>>>>>>> specifics of that)
>>>> > >>>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>>> So to bring the options listed in the issue here
>>>> for
>>>> > >>>> discussion,
>>>> > >>>>>>>>>>>>>>> broadly
>>>> > >>>>>>>>>>>>>>>>>> speaking there are two high-level approaches:
>>>> > >>>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>>> 1. A single shared distribution
>>>> > >>>>>>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
>>>> > >>>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>>> The advantage of Approach 1 is that we only have
>>>> the
>>>> > code
>>>> > >>>> in one
>>>> > >>>>>>>>>>>>>>> place.
>>>> > >>>>>>>>>>>>>>>>>> However for me, at least in this specific case of
>>>> > Logging
>>>> > >>>> config
>>>> > >>>>>>>>>> or
>>>> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class is that backwards
>>>> > compatibility
>>>> > >>> is
>>>> > >>>>>>>> much
>>>> > >>>>>>>>>>>>>> much
>>>> > >>>>>>>>>>>>>>>>>> harder.
>>>> > >>>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>>> The main advantage of Approach 2 is the the code is
>>>> > >>> released
>>>> > >>>>>>>>>>>>>>>> with/embedded
>>>> > >>>>>>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would
>>>> contain
>>>> > >>> the
>>>> > >>>>>>>> right
>>>> > >>>>>>>>>>>>>>>> version
>>>> > >>>>>>>>>>>>>>>>>> of the logging config and ConfigParser etc). The
>>>> > downside
>>>> > >>> is
>>>> > >>>>>>>> that
>>>> > >>>>>>>>>>>>>>> either
>>>> > >>>>>>>>>>>>>>>>>> the code will need to be duplicated in the repo, or
>>>> > better
>>>> > >>>> yet
>>>> > >>>>>>>> it
>>>> > >>>>>>>>>>>>>>> would
>>>> > >>>>>>>>>>>>>>>>>> live in a single place in the repo, but some
>>>> tooling
>>>> > (TBD)
>>>> > >>>> will
>>>> > >>>>>>>>>>>>>>>>>> automatically handle the duplication, either at
>>>> commit
>>>> > >>>> time, or
>>>> > >>>>>>>> my
>>>> > >>>>>>>>>>>>>>>>>> preference, at release time.
>>>> > >>>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>>> For this kind of shared “utility” code I am very
>>>> > strongly
>>>> > >>>>>>>> leaning
>>>> > >>>>>>>>>>>>>>>> towards
>>>> > >>>>>>>>>>>>>>>>>> option 2 with automation, as otherwise I think the
>>>> > >>> backwards
>>>> > >>>>>>>>>>>>>>>> compatibility
>>>> > >>>>>>>>>>>>>>>>>> requirements would make it unworkable (very
>>>> quickly over
>>>> > >>>> time
>>>> > >>>>>>>> the
>>>> > >>>>>>>>>>>>>>>>>> combinations we would have to test would just be
>>>> > >>>> unreasonable)
>>>> > >>>>>>>>>> and I
>>>> > >>>>>>>>>>>>>>>> don’t
>>>> > >>>>>>>>>>>>>>>>>> feel confident we can have things as stable as we
>>>> need
>>>> > to
>>>> > >>>> really
>>>> > >>>>>>>>>>>>>>> deliver
>>>> > >>>>>>>>>>>>>>>>>> the version separation/independency I want to
>>>> delivery
>>>> > >>> with
>>>> > >>>>>>>>>> AIP-72.
>>>> > >>>>>>>>>>>>>>>>>> So unless someone feels very strongly about this,
>>>> I will
>>>> > >>>> come up
>>>> > >>>>>>>>>>>>>> with
>>>> > >>>>>>>>>>>>>>> a
>>>> > >>>>>>>>>>>>>>>>>> draft PR for further discussion that will
>>>> implement code
>>>> > >>>> sharing
>>>> > >>>>>>>>>> via
>>>> > >>>>>>>>>>>>>>>>>> “vendoring” it at build time. I have an idea of
>>>> how I
>>>> > can
>>>> > >>>>>>>> achieve
>>>> > >>>>>>>>>>>>>> this
>>>> > >>>>>>>>>>>>>>>> so
>>>> > >>>>>>>>>>>>>>>>>> we have a single version in the repo and it’ll work
>>>> > there,
>>>> > >>>> but
>>>> > >>>>>>>> at
>>>> > >>>>>>>>>>>>>>>> runtime
>>>> > >>>>>>>>>>>>>>>>>> we vendor it in to the shipped dist so it lives at
>>>> > >>> something
>>>> > >>>>>>>> like
>>>> > >>>>>>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
>>>> > >>>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>>> In terms of repo layout, this likely means we
>>>> would end
>>>> > up
>>>> > >>>> with:
>>>> > >>>>>>>>>>>>>>>>>> airflow-core/pyproject.toml
>>>> > >>>>>>>>>>>>>>>>>> airflow-core/src/
>>>> > >>>>>>>>>>>>>>>>>> airflow-core/tests/
>>>> > >>>>>>>>>>>>>>>>>> task-sdk/pyproject.toml
>>>> > >>>>>>>>>>>>>>>>>> task-sdk/src/
>>>> > >>>>>>>>>>>>>>>>>> task-sdk/tests/
>>>> > >>>>>>>>>>>>>>>>>> airflow-common/src
>>>> > >>>>>>>>>>>>>>>>>> airflow-common/tests/
>>>> > >>>>>>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as
>>>> deps
>>>> > would
>>>> > >>>> be
>>>> > >>>>>>>>>>>>>> included
>>>> > >>>>>>>>>>>>>>>> in
>>>> > >>>>>>>>>>>>>>>>>> the downstream projects. TBD.
>>>> > >>>>>>>>>>>>>>>>>>
>>>> > >>>>>>>>>>>>>>>>>> Thoughts and feedback welcomed.
>>>> > >>>>
>>>> ---------------------------------------------------------------------
>>>> > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>> > dev-unsubscribe@airflow.apache.org
>>>> > >>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>> > >>> dev-help@airflow.apache.org
>>>> > >>>>>>>>>>>>>>>>
>>>> > >>>>>>>>
>>>> > >>>
>>>> ---------------------------------------------------------------------
>>>> > >>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>> > >>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>> > >>>>>>>>
>>>> > >>>>>>>>
>>>> > >>>>>>
>>>> > ---------------------------------------------------------------------
>>>> > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>> > >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>> > >>>>>>
>>>> > >>>>>
>>>> ---------------------------------------------------------------------
>>>> > >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>> > >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>> > >>>>>
>>>> > >>>>
>>>> ---------------------------------------------------------------------
>>>> > >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>> > >>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>> > >>>>
>>>> > >>>>
>>>> > >
>>>> > >
>>>> ---------------------------------------------------------------------
>>>> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>> > > For additional commands, e-mail: dev-help@airflow.apache.org
>>>> > >
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>> > For additional commands, e-mail: dev-help@airflow.apache.org
>>>> >
>>>> >
>>>>
>>>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Amogh Desai <am...@gmail.com>.
Yeah, I think what you are showcasing here is a step ahead of the
initial proposal from Ash.

From the original proposal, the `core_and_task_sdk` *can* have the things
relevant to just those two
distros. Logging, Config are modules that might be needed by airflow-ctl
for example, so ideally, those
would not be good candidates to be put in there, ideally speaking.

The example of Kubernetes Utils sounds to be a good example, it will be
used by KubeExecutor (lets say this is a
module called "executors") and by KPO (providers), and the
"shared/kubernetes" would probably be a good
candidate for that.

Not that I am against your idea and we can surely expand as we need but we
would not need to expand the
"core_and_task_sdk" if we put only the relevant items into it.

Thanks & Regards,
Amogh Desai


On Tue, Jul 8, 2025 at 12:28 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> > @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
> there, I am understanding the direction
> but could you elaborate a bit more please?
>
> Let me elaborate:
>
> As I understand (maybe I am wrong?),  the proposal is that we have a
> "core-and-task-sdk" folder which is a shared distribution that is
> vendored-in into both "airflow-core" and "airflow-task-sdk". This
> contains some shared code that we want to include in both distributions
> (note that we never ever release "core-and-task-sdk"
> distribution because it only contains code that is shared between the two
> distributions we release.
>
> That's fine and cool.
>
> Imagine that this distribution contains "logging" (shared code
> for logging) and "config" (shared code for configuration). - both needed in
> "airflow-core" and "airflow-task-sdk". So far so good. But what happen if
> we want to use logging in the same fashion in say "airflow-ctl" (that also
> applies for other distributions that we might come up with) ? Are we going
> to vendor in the whole "core-and-task-sdk" distribution in "airflow-ctl" ?
> It would be far better if we just vendor in "logging" and do not vendor-in
> "config".
>
> And if we are going to have a mechanism to vendor-in "a distribution" -
> there is nothing wrong with having the same way to vendor-in multiple
> distributions - so we can easily do it this way (i added "orm" ,
> "serialization". and "fast_api" as an example thing that we might want to
> share - not sure if that is really something we want to do but it will
> allow to illustrate my idea better)
>
> /
>   airflow-ctl/
>   task-sdk/...
>   airflow-core/...
>   ....
>   shared/
>     kubernetes/
>       pyproject.toml
>       src/
>         airflow_shared_kubernetes/__init__.py
>     logging/
>       pyproject.toml
>       src/
>         airflow_shared_logging/__init__.py
>     config/
>       pyproject.toml
>       src/
>         airflow_shared_config/__init__.py
>     orm/
>       pyproject.toml
>       src/
>         airflow_shared_orm/__init__.py
>      serialization/
>       pyproject.toml
>       src/
>         airflow_shared_serialization/__init__.py
>      fast_api/
>       pyproject.toml
>       src/
>         airflow_shared_fast_api/__init__.py
>     ...
>
> This has multiple benefits (and I see no real drawbacks):
>
> * the code can be really well modularised.  Those "things" we share (and
> this also connects to the __init__.py discussion) - can be independent -
> and (it follow Jens comment) it allows to keep low cyclomatic complexity
> https://en.wikipedia.org/wiki/Cyclomatic_complexity . It will be way
> easier to implement logging in the way that it does not import or use
> config. This means for example that configuration for logging will need to
> be injected when logging is initialized - and that's exactly what we want,
> we do not want logging code to use configuration code directly - they
> should be independent from each other and you should be free to vendor-in
> either logging or config independently if you also vendored-in the other.
>
> * it's much more logical. We split based on functionality we want to share
> - not about the "distributions" we want to produce. That allows us - in the
> future - to make different decisions on how we split our distributions. For
> example (i do not tell we have to do it, or that we will do it but we will
> have such a possibility) - we can add more shared utilities we find useful
> in the same way and decide that "scheduler", "api_server" or "scheduler" or
> "triggerer" or "dag processor" are split to separate distributions -
> because for example we want to keep a number of dependencies down. And for
> example "api_server" might use "fast_api", "config", "logging", "orm" and
> "fast_api" and "serialization" , where the scheduler should not need
> "fast_api". The "dag_processor" eventually might not need "orm" nor
> "fast_api" and only use the other
>
> This seems like a natural approach. If we have a mechanism to "share" the
> code, it does not add complexity, but allows us to isolate independent
> functionality into "isolated" boxes and use them
>
> Also for cyclomatic complexity that is a complex word (badly chosen as it
> scares people away) and has some math behind, but it really boils down to
> very simple "rules of thumb" (and yes I am a big proponent of having low
> cyclomatic complexity).
>
> a) when you are building the "final" product (i.e. distribution you want
> to release) - make sure that you only "use" things - that nothing else is
> "using" you as a library.
> b) when you are building a "shared" thing (a library) - make sure that
> library is only "used" by others but it does not really "use" anything
> else. For example in the case I explained above - we can achieve
> low-cyclomatic complexity when:
>
> * airflow-core uses: logging, config, orm, serialization, fast_api
> * none of the "logging, config, orm, serialization, fast_api" use each
> other - they are at the bottom of the "user -> used" tree
>
> J.
>
>
>
> On Tue, Jul 8, 2025 at 8:16 AM Amogh Desai <am...@gmail.com>
> wrote:
>
>> I like the folder structure proposed by Ash and have no objections with
>> it.
>>
>> "core_and_task_sdk" sounds good to me and justifies what it should do
>> pretty well.
>>
>> @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
>> there, I am understanding the direction
>> but could you elaborate a bit more please?
>>
>> Naming is REALLY hard!
>>
>> Thanks & Regards,
>> Amogh Desai
>>
>>
>> On Tue, Jul 8, 2025 at 2:52 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> How about splitting it even more and having each shared "thing" named?
>>> "logging", "config" and sharing them explicitly and separately with the
>>> right "user" ?
>>> That sounds way more modular and  we will be able to choose which of the
>>> shared "utils" we use where.
>>>
>>> J.
>>>
>>>
>>> On Mon, Jul 7, 2025 at 11:13 PM Jens Scheffler
>>> <j_...@gmx.de.invalid>
>>> wrote:
>>>
>>> > I like "core_and_task_sdk the same like core-and-task-sdk - I have no
>>> > problem and it is a path only.
>>> >
>>> > if we get to "dag-parser-scheduler-task-sdk-and-triggerer" which is a
>>> > bit bulky we then should name it "all-not-api-server" :-D
>>> >
>>> > On 07.07.25 22:57, Ash Berlin-Taylor wrote:
>>> > > In case I did a bad job explaining it, the “core and task sdk” is
>>> not in
>>> > the module name/import name, just in the file path.
>>> > >
>>> > > Anyone have other ideas?
>>> > >
>>> > >> On 7 Jul 2025, at 21:37, Buğra Öztürk <oz...@gmail.com>
>>> wrote:
>>> > >>
>>> > >> Thanks Ash! Looks cool! I like the structure. This will enable all
>>> the
>>> > >> combinations and structure looks easy to grasp. No strong stance on
>>> the
>>> > >> naming other than maybe it is a bit long with `and`, `core_ctl`
>>> could be
>>> > >> shorter, since no import path is defined like that, we can give any
>>> name
>>> > >> for sure.
>>> > >>
>>> > >> Best regards,
>>> > >>
>>> > >>> On Mon, 7 Jul 2025, 21:51 Jarek Potiuk, <ja...@potiuk.com> wrote:
>>> > >>>
>>> > >>> Looks good but I think we should find some better logical name for
>>> > >>> core_and_sdk :)
>>> > >>>
>>> > >>> pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
>>> > >>> <j_...@gmx.de.invalid> napisał:
>>> > >>>
>>> > >>>> Cool! Especially the "shared" folder with the ability to have
>>> > >>>> N-combinations w/o exploding project repo root!
>>> > >>>>
>>> > >>>> On 07.07.25 14:43, Ash Berlin-Taylor wrote:
>>> > >>>>> Oh, and all of this will be explain in shared/README.md
>>> > >>>>>
>>> > >>>>>> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org>
>>> wrote:
>>> > >>>>>>
>>> > >>>>>> Okay, so it seems we have agreement on the approach here, so
>>> I’ll
>>> > >>>> continue with this, and on the dev call it was mentioned that
>>> > >>>> “airflow-common” wasn’t a great name, so here is my proposal for
>>> the
>>> > file
>>> > >>>> structure;
>>> > >>>>>> ```
>>> > >>>>>> /
>>> > >>>>>>   task-sdk/...
>>> > >>>>>>   airflow-core/...
>>> > >>>>>>   shared/
>>> > >>>>>>     kuberenetes/
>>> > >>>>>>       pyproject.toml
>>> > >>>>>>       src/
>>> > >>>>>>         airflow_kube/__init__.py
>>> > >>>>>>     core-and-tasksdk/
>>> > >>>>>>       pyproject.toml
>>> > >>>>>>       src/
>>> > >>>>>>         airflow_shared/__init__.py
>>> > >>>>>> ```
>>> > >>>>>>
>>> > >>>>>> Things to note here: the “shared” folder has (the possibility)
>>> of
>>> > >>>> having multiple different shared “libraries” in it, in this
>>> example I
>>> > am
>>> > >>>> supposing a hypothetical shared kuberenetes folder a world in
>>> which we
>>> > >>>> split the KubePodOperator and the KubeExecutor in to two separate
>>> > >>>> distributions (example only, not proposing we do that right now,
>>> and
>>> > that
>>> > >>>> will be a separate discussion)
>>> > >>>>>> The other things to note here:
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>>> - the folder name in shared aims to be “self-documenting”,
>>> hence the
>>> > >>>> verbose “core-and-tasksdk” to say where the shared library is
>>> > intended to
>>> > >>>> be used.
>>> > >>>>>> - the python module itself should almost always have an
>>> `airflow_`
>>> > (or
>>> > >>>> maybe `_airflow_`?) prefix so that it does not conflict with
>>> anything
>>> > >>> else
>>> > >>>> we might use. It won’t matter “in production” as those will be
>>> > vendored
>>> > >>> in
>>> > >>>> to be imported as `airflow/_vendor/airflow_shared` etc, but
>>> avoiding
>>> > >>>> conflicts at dev time with the Finder approach is a good safety
>>> > measure.
>>> > >>>>>> I will start making a real PR for this proposal now, but I’m
>>> open to
>>> > >>>> feedback (either here, or in the PR when I open it)
>>> > >>>>>> -ash
>>> > >>>>>>
>>> > >>>>>>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>> > >>>>>>>
>>> > >>>>>>> Yeah we have to try it and test - also building packages
>>> happens
>>> > semi
>>> > >>>>>>> frequently when you run `uv sync` (they use some kind of
>>> heuristics
>>> > >>> to
>>> > >>>>>>> decide when) and you can force it with `--reinstall` or
>>> > `--refresh`.
>>> > >>>>>>> Package build also happens every time when you run "ci-image
>>> build`
>>> > >>>> now in
>>> > >>>>>>> breeze so it seems like it will nicely integrate in our
>>> workflows.
>>> > >>>>>>>
>>> > >>>>>>> Looks really cool Ash.
>>> > >>>>>>>
>>> > >>>>>>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <
>>> ash@apache.org>
>>> > >>>> wrote:
>>> > >>>>>>>> It’s not just release time, but any time we build a package
>>> which
>>> > >>>> happens
>>> > >>>>>>>> on “every” CI run. The normal unit tests will use code from
>>> > >>>>>>>> airflow-common/src/airflow_common; the kube tests which build
>>> an
>>> > >>>> image will
>>> > >>>>>>>> build the dists and vendor in the code from that commit.
>>> > >>>>>>>>
>>> > >>>>>>>> There is only a single copy of the shared code committed to
>>> the
>>> > >>> repo,
>>> > >>>> so
>>> > >>>>>>>> there is never anything to synchronise.
>>> > >>>>>>>>
>>> > >>>>>>>>> On 4 Jul 2025, at 15:53, Amogh Desai <
>>> amoghdesai.oss@gmail.com>
>>> > >>>> wrote:
>>> > >>>>>>>>> Thanks Ash.
>>> > >>>>>>>>>
>>> > >>>>>>>>> This is really cool and helpful that you were able to test
>>> both
>>> > >>>> scenarios
>>> > >>>>>>>>> -- repo checkout
>>> > >>>>>>>>> and also installing from the vendored package and the
>>> resolution
>>> > >>>> worked
>>> > >>>>>>>>> fine too.
>>> > >>>>>>>>>
>>> > >>>>>>>>> I like this idea compared the to relative import one for few
>>> > >>> reasons:
>>> > >>>>>>>>> - It feels like it will take some time to adjust to the new
>>> > coding
>>> > >>>>>>>> standard
>>> > >>>>>>>>> that we will lay
>>> > >>>>>>>>> if we impose relative imports in the shared dist
>>> > >>>>>>>>> - We can continue using repo wise absolute import standards,
>>> it
>>> > is
>>> > >>>> also
>>> > >>>>>>>>> much easier for situations
>>> > >>>>>>>>> when we do global search in IDE to find + replace, this could
>>> > mean
>>> > >>> a
>>> > >>>>>>>> change
>>> > >>>>>>>>> there
>>> > >>>>>>>>> - The vendoring work is a proven and established paradigm
>>> across
>>> > >>>> projects
>>> > >>>>>>>>> and would
>>> > >>>>>>>>> out of box give us the build tooling we need also
>>> > >>>>>>>>>
>>> > >>>>>>>>> Nothing too against the relative import but with the evidence
>>> > >>>> provided
>>> > >>>>>>>>> above, vendored approach
>>> > >>>>>>>>> seems to only do us good.
>>> > >>>>>>>>>
>>> > >>>>>>>>> Regarding synchronizing it, release time should be fine as
>>> long
>>> > as
>>> > >>> we
>>> > >>>>>>>> have
>>> > >>>>>>>>> a good CI workflow to probably
>>> > >>>>>>>>> catch such issues per PR if changes are made in shared dist?
>>> > >>>> (precommit
>>> > >>>>>>>>> would make it really slow i guess)
>>> > >>>>>>>>>
>>> > >>>>>>>>> If we can run our tests with vendored code we should be
>>> mostly
>>> > >>>> covered.
>>> > >>>>>>>>> Good effort all!
>>> > >>>>>>>>>
>>> > >>>>>>>>> Thanks & Regards,
>>> > >>>>>>>>> Amogh Desai
>>> > >>>>>>>>>
>>> > >>>>>>>>>
>>> > >>>>>>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <
>>> > ash@apache.org>
>>> > >>>>>>>> wrote:
>>> > >>>>>>>>>> Okay, I think I’ve got something that works and I’m happy
>>> with.
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>
>>> > >>>
>>> >
>>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>>> > >>>>>>>>>> This produces the following from `uv build task-sdk`
>>> > >>>>>>>>>> -
>>> > >>>>>>>>>>
>>> > >>>
>>> >
>>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>>> > >>>>>>>>>> -
>>> > >>>>>>>>>>
>>> > >>>
>>> >
>>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>>> > >>>>>>>>>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> ```
>>> > >>>>>>>>>> ❯ unzip -l
>>> > >>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
>>> > >>>>>>>> grep
>>> > >>>>>>>>>> _vendor
>>> > >>>>>>>>>>      50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
>>> > >>>>>>>>>>    2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
>>> > >>>>>>>>>>      28  02-02-2020 00:00
>>> >  airflow/sdk/_vendor/airflow_common.pyi
>>> > >>>>>>>>>>      18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
>>> > >>>>>>>>>>     785  02-02-2020 00:00
>>> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
>>> > >>>>>>>>>>   10628  02-02-2020 00:00
>>> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
>>> > >>>>>>>>>> ```
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> And similarly in the .tar.gz, so our “sdist” is complete
>>> too:
>>> > >>>>>>>>>> ```
>>> > >>>>>>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep
>>> > _vendor
>>> > >>>>>>>>>>
>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>>> > >>>>>>>>>>
>>> > apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>>> > >>>>>>>>>>
>>> > >>>>
>>> >
>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>>> > >>>>>>>>>>
>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>
>>> > >>>
>>> >
>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>>> > >>>
>>> >
>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>>> > >>>>>>>>>> ```
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> The plugin works at build time by including/copying the libs
>>> > >>>> specified
>>> > >>>>>>>> in
>>> > >>>>>>>>>> vendor.txt into place (and let `vendoring` take care of
>>> import
>>> > >>>>>>>> rewrites.)
>>> > >>>>>>>>>> For the imports to continue to work at “dev” time/from a
>>> repo
>>> > >>>> checkout,
>>> > >>>>>>>> I
>>> > >>>>>>>>>> have added a import finder to `sys.meta_path`, and since
>>> it’s at
>>> > >>> the
>>> > >>>>>>>> end of
>>> > >>>>>>>>>> the list it will only be used if the normal import can’t
>>> find
>>> > >>>> things.
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>
>>> > >>>
>>> >
>>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>>> > >>>>>>>>>> This doesn’t quite give us the same runtime effect “import
>>> > >>>> rewriting”
>>> > >>>>>>>>>> affect, as in this approach `airflow_common` is directly
>>> loaded
>>> > >>>> (i.e.
>>> > >>>>>>>>>> airflow.sdk._vendor.airflow_common and airflow_common exist
>>> in
>>> > >>>>>>>>>> sys.modules), but it does work for everything that I was
>>> able to
>>> > >>>> test..
>>> > >>>>>>>>>> I tested it with the diff at the end of this message. My
>>> test
>>> > >>>> ipython
>>> > >>>>>>>>>> shell:
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> ```
>>> > >>>>>>>>>> In [1]: from airflow.sdk._vendor.airflow_common.timezone
>>> import
>>> > >>> foo
>>> > >>>>>>>>>> In [2]: foo
>>> > >>>>>>>>>> Out[2]: 1
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> In [3]: import airflow.sdk._vendor.airflow_common
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> In [4]: import airflow.sdk._vendor.airflow_common.timezone
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>>> > >>>>>>>>>> Out[5]:
>>> > >>>>>>>>>>
>>> > >>>
>>> >
>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>>> > >>>>>>>>>> In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
>>> > >>>>>>>>>> Out[6]:
>>> > >>>>>>>>>>
>>> > >>>
>>> >
>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>>> > >>>>>>>>>> ```
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> And in an standalone environment with the SDK dist I built
>>> (it
>>> > >>>> needed
>>> > >>>>>>>> the
>>> > >>>>>>>>>> matching airflow-core right now, but that is nothing to do
>>> with
>>> > >>> this
>>> > >>>>>>>>>> discussion):
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> ```
>>> > >>>>>>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>>> > >>>>>>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>>> > >>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
>>> > >>>>>>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
>>> > >>>>>>>>>> Type 'copyright', 'credits' or 'license' for more
>>> information
>>> > >>>>>>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type '?'
>>> for
>>> > >>> help.
>>> > >>>>>>>>>> Tip: You can use `%hist` to view history, see the options
>>> with
>>> > >>>>>>>> `%history?`
>>> > >>>>>>>>>> In [1]: import airflow.sdk._vendor.airflow_common.timezone
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
>>> > >>>>>>>>>> Out[2]:
>>> > >>>>>>>>>>
>>> > >>>
>>> >
>>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>>> > >>>>>>>>>> ``
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> ```diff
>>> > >>>>>>>>>> diff --git a/airflow-common/src/airflow_common/__init__.py
>>> > >>>>>>>>>> b/airflow-common/src/airflow_common/__init__.py
>>> > >>>>>>>>>> index 13a83393a9..927b7c6b61 100644
>>> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/__init__.py
>>> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
>>> > >>>>>>>>>> @@ -14,3 +14,5 @@
>>> > >>>>>>>>>> # KIND, either express or implied.  See the License for the
>>> > >>>>>>>>>> # specific language governing permissions and limitations
>>> > >>>>>>>>>> # under the License.
>>> > >>>>>>>>>> +
>>> > >>>>>>>>>> +foo = 1
>>> > >>>>>>>>>> diff --git a/airflow-common/src/airflow_common/timezone.py
>>> > >>>>>>>>>> b/airflow-common/src/airflow_common/timezone.py
>>> > >>>>>>>>>> index 340b924c66..58384ef20f 100644
>>> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/timezone.py
>>> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
>>> > >>>>>>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>>> > >>>>>>>>>> version.parse(metadata.version("pendulum")).major == 3
>>> > >>>>>>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
>>> > >>>>>>>>>> utc = pendulum.UTC
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> +
>>> > >>>>>>>>>> +from airflow_common import foo
>>> > >>>>>>>>>> +
>>> > >>>>>>>>>> TIMEZONE: Timezone
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>
>>> > >>>>>>>>>> ```
>>> > >>>>>>>>>>
>>> > >>>>>>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com>
>>> > wrote:
>>> > >>>>>>>>>>> I think both approaches are doable:
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>> 1) -> We can very easily prevent bad imports by pre-commit
>>> when
>>> > >>>>>>>> importing
>>> > >>>>>>>>>>> from different distributions and make sure we are only
>>> doing
>>> > >>>> relative
>>> > >>>>>>>>>>> imports in the shared modules. We are doing plenty of this
>>> > >>>> already. And
>>> > >>>>>>>>>> yes
>>> > >>>>>>>>>>> it would require relative links we currently do not allow.
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>> 2) -> has one disadvantage that someone at some point in
>>> time
>>> > >>> will
>>> > >>>> have
>>> > >>>>>>>>>> to
>>> > >>>>>>>>>>> decide to synchronize this and if it happens just before
>>> > release
>>> > >>>> (I bet
>>> > >>>>>>>>>>> this is going to happen) this will lead to solving problems
>>> > that
>>> > >>>> would
>>> > >>>>>>>>>>> normally be solved during PR when you make a change (i.e.
>>> > >>> symbolic
>>> > >>>> link
>>> > >>>>>>>>>> has
>>> > >>>>>>>>>>> the advantage that whoever modifies shared code will be
>>> > >>> immediately
>>> > >>>>>>>>>>> notified in their PR - that they broke something because
>>> either
>>> > >>>> static
>>> > >>>>>>>>>>> checks or mypy or tests fail.
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>> Ash, do you have an idea of a process (who and when) does
>>> the
>>> > >>>>>>>>>>> synchronisation in case of vendoring? Maybe we could solve
>>> it
>>> > if
>>> > >>>> it is
>>> > >>>>>>>>>> done
>>> > >>>>>>>>>>> more frequently and with some regularity? We could
>>> potentially
>>> > >>>> force
>>> > >>>>>>>>>>> re-vendoring at PR time as well any time shared code
>>> changes
>>> > (and
>>> > >>>>>>>> prevent
>>> > >>>>>>>>>>> it by pre-commit. And I can't think of some place (other
>>> than
>>> > >>>> releases)
>>> > >>>>>>>>>> in
>>> > >>>>>>>>>>> our development workflow and that seems to be a bit too
>>> late as
>>> > >>>> puts an
>>> > >>>>>>>>>>> extra effort on fixing potential incompatibilities
>>> introduced
>>> > on
>>> > >>>>>>>> release
>>> > >>>>>>>>>>> manager and delays the release. WDYT?
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>> Re: relative links. I think for a shared library we could
>>> > >>>> potentially
>>> > >>>>>>>>>> relax
>>> > >>>>>>>>>>> this and allow them (and actually disallow absolute links
>>> in
>>> > the
>>> > >>>> pieces
>>> > >>>>>>>>>> of
>>> > >>>>>>>>>>> code that are shared - again, by pre-commit). As I recall,
>>> the
>>> > >>> only
>>> > >>>>>>>>>> reason
>>> > >>>>>>>>>>> we forbade the relative links is because of how we are (or
>>> > maybe
>>> > >>>> were)
>>> > >>>>>>>>>>> doing DAG parsing and failures resulting from it. So we
>>> decided
>>> > >>> to
>>> > >>>> just
>>> > >>>>>>>>>> not
>>> > >>>>>>>>>>> allow it to keep consistency. The way how Dag parsing
>>> works is
>>> > >>> that
>>> > >>>>>>>> when
>>> > >>>>>>>>>>> you are using importlib to read the Dag from a file, the
>>> > relative
>>> > >>>>>>>> imports
>>> > >>>>>>>>>>> do not work as it does not know what they should be
>>> relative
>>> > to.
>>> > >>>> But if
>>> > >>>>>>>>>>> relative import is done from an imported package, it
>>> should be
>>> > no
>>> > >>>>>>>>>> problem,
>>> > >>>>>>>>>>> I think - otherwise our Dags would not be able to import
>>> any
>>> > >>>> library
>>> > >>>>>>>> that
>>> > >>>>>>>>>>> uses relative imports.
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>> Of course consistency might be the reason why we do not
>>> want to
>>> > >>>>>>>> introduce
>>> > >>>>>>>>>>> relative imports. I don't see it as an issue if it is
>>> guarded
>>> > by
>>> > >>>>>>>>>> pre-commit
>>> > >>>>>>>>>>> though.
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>> J.
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>> J.
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
>>> > >>>> ash@apache.org>
>>> > >>>>>>>>>>> napisał:
>>> > >>>>>>>>>>>
>>> > >>>>>>>>>>>> Oh yes, symlinks will work, with one big caveat: It does
>>> mean
>>> > >>> you
>>> > >>>>>>>> can’t
>>> > >>>>>>>>>>>> use absolute imports in one common module to another.
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>> For example
>>> > >>>>>>>>>>>>
>>> > >>>
>>> >
>>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>>> > >>>>>>>>>>>> where we have
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>> ```
>>> > >>>>>>>>>>>> from airflow.utils.module_loading import import_string
>>> > >>>>>>>>>>>> ```
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>> if we want to move serve_logs into this common lib that is
>>> > then
>>> > >>>>>>>>>> symlinked
>>> > >>>>>>>>>>>> then we wouldn’t be able to have `from
>>> > >>>> airflow_common.module_loading
>>> > >>>>>>>>>> import
>>> > >>>>>>>>>>>> import_string`.
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>> I can think of two possible solutions here.
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>> 1) is to allow/require relative imports in this shared
>>> lib, so
>>> > >>>> `from
>>> > >>>>>>>>>>>> .module_loading import import_string`
>>> > >>>>>>>>>>>> 2) is to use `vendoring`[1] (from the pip maintainers)
>>> which
>>> > >>> will
>>> > >>>>>>>> handle
>>> > >>>>>>>>>>>> import-rewriting for us.
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>> I’d entirely forgot that symlinks in repos was a thing,
>>> so I
>>> > >>>> prepared
>>> > >>>>>>>> a
>>> > >>>>>>>>>>>> minimal POC/demo of what vendoring approach could look
>>> like
>>> > here
>>> > >>>>>>>>>>>>
>>> > >>>
>>> >
>>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>>> > >>>>>>>>>>>> Now personally I am more than happy with relative
>>> imports, but
>>> > >>>>>>>> generally
>>> > >>>>>>>>>>>> as a project we have avoided them, so I think that limits
>>> what
>>> > >>> we
>>> > >>>>>>>> could
>>> > >>>>>>>>>> do
>>> > >>>>>>>>>>>> with a symlink based approach.
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>> -ash
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>> [1] https://github.com/pradyunsg/vendoring
>>> > >>>>>>>>>>>>
>>> > >>>>>>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
>>> > >>>>>>>> gopidesupavan@gmail.com>
>>> > >>>>>>>>>>>> wrote:
>>> > >>>>>>>>>>>>> Thanks Ash
>>> > >>>>>>>>>>>>>
>>> > >>>>>>>>>>>>> Yes agree option 2 would be preferred for me. Making
>>> sure we
>>> > >>>> have all
>>> > >>>>>>>>>> the
>>> > >>>>>>>>>>>>> gaurdriles to protect any unwanted behaviour in code
>>> sharing
>>> > >>> and
>>> > >>>>>>>>>>>> executing
>>> > >>>>>>>>>>>>> right of tests between the packages.
>>> > >>>>>>>>>>>>>
>>> > >>>>>>>>>>>>> Agree with others, option 2 would be
>>> > >>>>>>>>>>>>>
>>> > >>>>>>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
>>> > >>>>>>>> amoghdesai.oss@gmail.com>
>>> > >>>>>>>>>>>>> wrote:
>>> > >>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>> Thanks for starting this discussion, Ash.
>>> > >>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>> I would prefer option 2 here with proper tooling to
>>> handle
>>> > the
>>> > >>>> code
>>> > >>>>>>>>>>>>>> duplication at *release* time.
>>> > >>>>>>>>>>>>>> It is best to have a dist that has all it needs in
>>> itself.
>>> > >>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>> Option 1 could very quickly get out of hand and if we
>>> decide
>>> > >>> to
>>> > >>>>>>>>>> separate
>>> > >>>>>>>>>>>>>> triggerer /
>>> > >>>>>>>>>>>>>> dag processor / config etc etc as separate packages,
>>> back
>>> > >>>> compat is
>>> > >>>>>>>>>>>> going
>>> > >>>>>>>>>>>>>> to be a nightmare
>>> > >>>>>>>>>>>>>> and will bite us harder than we anticipate.
>>> > >>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>> Thanks & Regards,
>>> > >>>>>>>>>>>>>> Amogh Desai
>>> > >>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <
>>> > >>> kaxilnaik@gmail.com>
>>> > >>>>>>>>>> wrote:
>>> > >>>>>>>>>>>>>>> I prefer Option 2 as well to avoid matrix of
>>> dependencies
>>> > >>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>>> > >>>>>>>>>> <j_scheffler@gmx.de.invalid
>>> > >>>>>>>>>>>>>>> wrote:
>>> > >>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>> I'd also rather prefer option 2 - reason here is it is
>>> > >>> rather
>>> > >>>>>>>>>>>> pragmatic
>>> > >>>>>>>>>>>>>>>> and we no not need to cut another package and have
>>> less
>>> > >>>> package
>>> > >>>>>>>>>> counts
>>> > >>>>>>>>>>>>>>>> and dependencies.
>>> > >>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>> I remember some time ago I was checking (together with
>>> > >>> Jarek,
>>> > >>>> I am
>>> > >>>>>>>>>> not
>>> > >>>>>>>>>>>>>>>> sure anymore...) if the usage of symlinks would be
>>> > possible.
>>> > >>>> To
>>> > >>>>>>>> keep
>>> > >>>>>>>>>>>>>> the
>>> > >>>>>>>>>>>>>>>> source in one package but "symlink" it into another.
>>> If
>>> > then
>>> > >>>> at
>>> > >>>>>>>>>> point
>>> > >>>>>>>>>>>>>> of
>>> > >>>>>>>>>>>>>>>> packaging/release the files are materialized we have
>>> 1 set
>>> > >>> of
>>> > >>>>>>>> code.
>>> > >>>>>>>>>>>>>>>> Otherwise if not possible still the redundancy could
>>> be
>>> > >>>> solved by
>>> > >>>>>>>> a
>>> > >>>>>>>>>>>>>>>> pre-commit hook - and in Git the files are
>>> de-duplicated
>>> > >>>> anyway
>>> > >>>>>>>>>> based
>>> > >>>>>>>>>>>>>> on
>>> > >>>>>>>>>>>>>>>> content hash, so this does not hurt.
>>> > >>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>> > >>>>>>>>>>>>>>>>> I support option 2 with proper automation & CI - the
>>> > >>>> reasonings
>>> > >>>>>>>>>>>>>> you've
>>> > >>>>>>>>>>>>>>>>> shown for that make sense to me.
>>> > >>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>> Shahar
>>> > >>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <
>>> > >>>> ash@apache.org
>>> > >>>>>>>>>>>>>>> wrote:
>>> > >>>>>>>>>>>>>>>>>> Hello everyone,
>>> > >>>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>>> As we work on finishing off the code-level
>>> separation of
>>> > >>>> Task
>>> > >>>>>>>> SDK
>>> > >>>>>>>>>>>>>> and
>>> > >>>>>>>>>>>>>>>> Core
>>> > >>>>>>>>>>>>>>>>>> (scheduler etc) we have come across some situations
>>> > where
>>> > >>> we
>>> > >>>>>>>> would
>>> > >>>>>>>>>>>>>>> like
>>> > >>>>>>>>>>>>>>>> to
>>> > >>>>>>>>>>>>>>>>>> share code between these.
>>> > >>>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>>> However it’s not as straight forward of “just put
>>> it in
>>> > a
>>> > >>>> common
>>> > >>>>>>>>>>>>>> dist
>>> > >>>>>>>>>>>>>>>> they
>>> > >>>>>>>>>>>>>>>>>> both depend upon” because one of the goals of the
>>> Task
>>> > SDK
>>> > >>>>>>>>>>>>>> separation
>>> > >>>>>>>>>>>>>>>> was
>>> > >>>>>>>>>>>>>>>>>> to have 100% complete version independence between
>>> the
>>> > >>> two,
>>> > >>>>>>>>>> ideally
>>> > >>>>>>>>>>>>>>>> even if
>>> > >>>>>>>>>>>>>>>>>> they are built into the same image and venv. Most
>>> of the
>>> > >>>> reason
>>> > >>>>>>>>>> why
>>> > >>>>>>>>>>>>>>> this
>>> > >>>>>>>>>>>>>>>>>> isn’t straight forward comes down to backwards
>>> > >>>> compatibility -
>>> > >>>>>>>> if
>>> > >>>>>>>>>> we
>>> > >>>>>>>>>>>>>>>> make
>>> > >>>>>>>>>>>>>>>>>> an change to the common/shared distribution
>>> > >>>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>>> We’ve listed the options we have thought about in
>>> > >>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545 (but
>>> > that
>>> > >>>> covers
>>> > >>>>>>>>>>>>>> some
>>> > >>>>>>>>>>>>>>>> more
>>> > >>>>>>>>>>>>>>>>>> things that I don’t want to get in to in this
>>> discussion
>>> > >>>> such as
>>> > >>>>>>>>>>>>>>>> possibly
>>> > >>>>>>>>>>>>>>>>>> separating operators and executors out of a single
>>> > >>> provider
>>> > >>>>>>>> dist.)
>>> > >>>>>>>>>>>>>>>>>> To give a concrete example of some code I would
>>> like to
>>> > >>>> share
>>> > >>>
>>> >
>>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>> > >>>>>>>>>>>>>>>>>> — logging config. Another thing we will want to
>>> share
>>> > will
>>> > >>>> be
>>> > >>>>>>>> the
>>> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class from airflow.configuration
>>> > (but
>>> > >>>>>>>> notably:
>>> > >>>>>>>>>>>>>>> only
>>> > >>>>>>>>>>>>>>>> the
>>> > >>>>>>>>>>>>>>>>>> parser class, _not_ the default config values,
>>> again,
>>> > lets
>>> > >>>> not
>>> > >>>>>>>>>> dwell
>>> > >>>>>>>>>>>>>>> on
>>> > >>>>>>>>>>>>>>>> the
>>> > >>>>>>>>>>>>>>>>>> specifics of that)
>>> > >>>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>>> So to bring the options listed in the issue here for
>>> > >>>> discussion,
>>> > >>>>>>>>>>>>>>> broadly
>>> > >>>>>>>>>>>>>>>>>> speaking there are two high-level approaches:
>>> > >>>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>>> 1. A single shared distribution
>>> > >>>>>>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
>>> > >>>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>>> The advantage of Approach 1 is that we only have the
>>> > code
>>> > >>>> in one
>>> > >>>>>>>>>>>>>>> place.
>>> > >>>>>>>>>>>>>>>>>> However for me, at least in this specific case of
>>> > Logging
>>> > >>>> config
>>> > >>>>>>>>>> or
>>> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class is that backwards
>>> > compatibility
>>> > >>> is
>>> > >>>>>>>> much
>>> > >>>>>>>>>>>>>> much
>>> > >>>>>>>>>>>>>>>>>> harder.
>>> > >>>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>>> The main advantage of Approach 2 is the the code is
>>> > >>> released
>>> > >>>>>>>>>>>>>>>> with/embedded
>>> > >>>>>>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would
>>> contain
>>> > >>> the
>>> > >>>>>>>> right
>>> > >>>>>>>>>>>>>>>> version
>>> > >>>>>>>>>>>>>>>>>> of the logging config and ConfigParser etc). The
>>> > downside
>>> > >>> is
>>> > >>>>>>>> that
>>> > >>>>>>>>>>>>>>> either
>>> > >>>>>>>>>>>>>>>>>> the code will need to be duplicated in the repo, or
>>> > better
>>> > >>>> yet
>>> > >>>>>>>> it
>>> > >>>>>>>>>>>>>>> would
>>> > >>>>>>>>>>>>>>>>>> live in a single place in the repo, but some tooling
>>> > (TBD)
>>> > >>>> will
>>> > >>>>>>>>>>>>>>>>>> automatically handle the duplication, either at
>>> commit
>>> > >>>> time, or
>>> > >>>>>>>> my
>>> > >>>>>>>>>>>>>>>>>> preference, at release time.
>>> > >>>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>>> For this kind of shared “utility” code I am very
>>> > strongly
>>> > >>>>>>>> leaning
>>> > >>>>>>>>>>>>>>>> towards
>>> > >>>>>>>>>>>>>>>>>> option 2 with automation, as otherwise I think the
>>> > >>> backwards
>>> > >>>>>>>>>>>>>>>> compatibility
>>> > >>>>>>>>>>>>>>>>>> requirements would make it unworkable (very quickly
>>> over
>>> > >>>> time
>>> > >>>>>>>> the
>>> > >>>>>>>>>>>>>>>>>> combinations we would have to test would just be
>>> > >>>> unreasonable)
>>> > >>>>>>>>>> and I
>>> > >>>>>>>>>>>>>>>> don’t
>>> > >>>>>>>>>>>>>>>>>> feel confident we can have things as stable as we
>>> need
>>> > to
>>> > >>>> really
>>> > >>>>>>>>>>>>>>> deliver
>>> > >>>>>>>>>>>>>>>>>> the version separation/independency I want to
>>> delivery
>>> > >>> with
>>> > >>>>>>>>>> AIP-72.
>>> > >>>>>>>>>>>>>>>>>> So unless someone feels very strongly about this, I
>>> will
>>> > >>>> come up
>>> > >>>>>>>>>>>>>> with
>>> > >>>>>>>>>>>>>>> a
>>> > >>>>>>>>>>>>>>>>>> draft PR for further discussion that will implement
>>> code
>>> > >>>> sharing
>>> > >>>>>>>>>> via
>>> > >>>>>>>>>>>>>>>>>> “vendoring” it at build time. I have an idea of how
>>> I
>>> > can
>>> > >>>>>>>> achieve
>>> > >>>>>>>>>>>>>> this
>>> > >>>>>>>>>>>>>>>> so
>>> > >>>>>>>>>>>>>>>>>> we have a single version in the repo and it’ll work
>>> > there,
>>> > >>>> but
>>> > >>>>>>>> at
>>> > >>>>>>>>>>>>>>>> runtime
>>> > >>>>>>>>>>>>>>>>>> we vendor it in to the shipped dist so it lives at
>>> > >>> something
>>> > >>>>>>>> like
>>> > >>>>>>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
>>> > >>>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>>> In terms of repo layout, this likely means we would
>>> end
>>> > up
>>> > >>>> with:
>>> > >>>>>>>>>>>>>>>>>> airflow-core/pyproject.toml
>>> > >>>>>>>>>>>>>>>>>> airflow-core/src/
>>> > >>>>>>>>>>>>>>>>>> airflow-core/tests/
>>> > >>>>>>>>>>>>>>>>>> task-sdk/pyproject.toml
>>> > >>>>>>>>>>>>>>>>>> task-sdk/src/
>>> > >>>>>>>>>>>>>>>>>> task-sdk/tests/
>>> > >>>>>>>>>>>>>>>>>> airflow-common/src
>>> > >>>>>>>>>>>>>>>>>> airflow-common/tests/
>>> > >>>>>>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps
>>> > would
>>> > >>>> be
>>> > >>>>>>>>>>>>>> included
>>> > >>>>>>>>>>>>>>>> in
>>> > >>>>>>>>>>>>>>>>>> the downstream projects. TBD.
>>> > >>>>>>>>>>>>>>>>>>
>>> > >>>>>>>>>>>>>>>>>> Thoughts and feedback welcomed.
>>> > >>>>
>>> ---------------------------------------------------------------------
>>> > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>> > dev-unsubscribe@airflow.apache.org
>>> > >>>>>>>>>>>>>>>> For additional commands, e-mail:
>>> > >>> dev-help@airflow.apache.org
>>> > >>>>>>>>>>>>>>>>
>>> > >>>>>>>>
>>> > >>>
>>> ---------------------------------------------------------------------
>>> > >>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> > >>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>> > >>>>>>>>
>>> > >>>>>>>>
>>> > >>>>>>
>>> > ---------------------------------------------------------------------
>>> > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> > >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>> > >>>>>>
>>> > >>>>>
>>> ---------------------------------------------------------------------
>>> > >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> > >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>> > >>>>>
>>> > >>>>
>>> ---------------------------------------------------------------------
>>> > >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> > >>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>> > >>>>
>>> > >>>>
>>> > >
>>> > > ---------------------------------------------------------------------
>>> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> > > For additional commands, e-mail: dev-help@airflow.apache.org
>>> > >
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> > For additional commands, e-mail: dev-help@airflow.apache.org
>>> >
>>> >
>>>
>>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Jarek Potiuk <ja...@potiuk.com>.
> @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
there, I am understanding the direction
but could you elaborate a bit more please?

Let me elaborate:

As I understand (maybe I am wrong?),  the proposal is that we have a
"core-and-task-sdk" folder which is a shared distribution that is
vendored-in into both "airflow-core" and "airflow-task-sdk". This
contains some shared code that we want to include in both distributions
(note that we never ever release "core-and-task-sdk"
distribution because it only contains code that is shared between the two
distributions we release.

That's fine and cool.

Imagine that this distribution contains "logging" (shared code for logging)
and "config" (shared code for configuration). - both needed in
"airflow-core" and "airflow-task-sdk". So far so good. But what happen if
we want to use logging in the same fashion in say "airflow-ctl" (that also
applies for other distributions that we might come up with) ? Are we going
to vendor in the whole "core-and-task-sdk" distribution in "airflow-ctl" ?
It would be far better if we just vendor in "logging" and do not vendor-in
"config".

And if we are going to have a mechanism to vendor-in "a distribution" -
there is nothing wrong with having the same way to vendor-in multiple
distributions - so we can easily do it this way (i added "orm" ,
"serialization". and "fast_api" as an example thing that we might want to
share - not sure if that is really something we want to do but it will
allow to illustrate my idea better)

/
  airflow-ctl/
  task-sdk/...
  airflow-core/...
  ....
  shared/
    kubernetes/
      pyproject.toml
      src/
        airflow_shared_kubernetes/__init__.py
    logging/
      pyproject.toml
      src/
        airflow_shared_logging/__init__.py
    config/
      pyproject.toml
      src/
        airflow_shared_config/__init__.py
    orm/
      pyproject.toml
      src/
        airflow_shared_orm/__init__.py
     serialization/
      pyproject.toml
      src/
        airflow_shared_serialization/__init__.py
     fast_api/
      pyproject.toml
      src/
        airflow_shared_fast_api/__init__.py
    ...

This has multiple benefits (and I see no real drawbacks):

* the code can be really well modularised.  Those "things" we share (and
this also connects to the __init__.py discussion) - can be independent -
and (it follow Jens comment) it allows to keep low cyclomatic complexity
https://en.wikipedia.org/wiki/Cyclomatic_complexity . It will be way easier
to implement logging in the way that it does not import or use config. This
means for example that configuration for logging will need to be injected
when logging is initialized - and that's exactly what we want, we do not
want logging code to use configuration code directly - they should be
independent from each other and you should be free to vendor-in either
logging or config independently if you also vendored-in the other.

* it's much more logical. We split based on functionality we want to share
- not about the "distributions" we want to produce. That allows us - in the
future - to make different decisions on how we split our distributions. For
example (i do not tell we have to do it, or that we will do it but we will
have such a possibility) - we can add more shared utilities we find useful
in the same way and decide that "scheduler", "api_server" or "scheduler" or
"triggerer" or "dag processor" are split to separate distributions -
because for example we want to keep a number of dependencies down. And for
example "api_server" might use "fast_api", "config", "logging", "orm" and
"fast_api" and "serialization" , where the scheduler should not need
"fast_api". The "dag_processor" eventually might not need "orm" nor
"fast_api" and only use the other

This seems like a natural approach. If we have a mechanism to "share" the
code, it does not add complexity, but allows us to isolate independent
functionality into "isolated" boxes and use them

Also for cyclomatic complexity that is a complex word (badly chosen as it
scares people away) and has some math behind, but it really boils down to
very simple "rules of thumb" (and yes I am a big proponent of having low
cyclomatic complexity).

a) when you are building the "final" product (i.e. distribution you want to
release) - make sure that you only "use" things - that nothing else is
"using" you as a library.
b) when you are building a "shared" thing (a library) - make sure that
library is only "used" by others but it does not really "use" anything
else. For example in the case I explained above - we can achieve
low-cyclomatic complexity when:

* airflow-core uses: logging, config, orm, serialization, fast_api
* none of the "logging, config, orm, serialization, fast_api" use each
other - they are at the bottom of the "user -> used" tree

J.



On Tue, Jul 8, 2025 at 8:16 AM Amogh Desai <am...@gmail.com> wrote:

> I like the folder structure proposed by Ash and have no objections with it.
>
> "core_and_task_sdk" sounds good to me and justifies what it should do
> pretty well.
>
> @Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean
> there, I am understanding the direction
> but could you elaborate a bit more please?
>
> Naming is REALLY hard!
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Tue, Jul 8, 2025 at 2:52 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> How about splitting it even more and having each shared "thing" named?
>> "logging", "config" and sharing them explicitly and separately with the
>> right "user" ?
>> That sounds way more modular and  we will be able to choose which of the
>> shared "utils" we use where.
>>
>> J.
>>
>>
>> On Mon, Jul 7, 2025 at 11:13 PM Jens Scheffler <j_scheffler@gmx.de.invalid
>> >
>> wrote:
>>
>> > I like "core_and_task_sdk the same like core-and-task-sdk - I have no
>> > problem and it is a path only.
>> >
>> > if we get to "dag-parser-scheduler-task-sdk-and-triggerer" which is a
>> > bit bulky we then should name it "all-not-api-server" :-D
>> >
>> > On 07.07.25 22:57, Ash Berlin-Taylor wrote:
>> > > In case I did a bad job explaining it, the “core and task sdk” is not
>> in
>> > the module name/import name, just in the file path.
>> > >
>> > > Anyone have other ideas?
>> > >
>> > >> On 7 Jul 2025, at 21:37, Buğra Öztürk <oz...@gmail.com>
>> wrote:
>> > >>
>> > >> Thanks Ash! Looks cool! I like the structure. This will enable all
>> the
>> > >> combinations and structure looks easy to grasp. No strong stance on
>> the
>> > >> naming other than maybe it is a bit long with `and`, `core_ctl`
>> could be
>> > >> shorter, since no import path is defined like that, we can give any
>> name
>> > >> for sure.
>> > >>
>> > >> Best regards,
>> > >>
>> > >>> On Mon, 7 Jul 2025, 21:51 Jarek Potiuk, <ja...@potiuk.com> wrote:
>> > >>>
>> > >>> Looks good but I think we should find some better logical name for
>> > >>> core_and_sdk :)
>> > >>>
>> > >>> pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
>> > >>> <j_...@gmx.de.invalid> napisał:
>> > >>>
>> > >>>> Cool! Especially the "shared" folder with the ability to have
>> > >>>> N-combinations w/o exploding project repo root!
>> > >>>>
>> > >>>> On 07.07.25 14:43, Ash Berlin-Taylor wrote:
>> > >>>>> Oh, and all of this will be explain in shared/README.md
>> > >>>>>
>> > >>>>>> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org>
>> wrote:
>> > >>>>>>
>> > >>>>>> Okay, so it seems we have agreement on the approach here, so I’ll
>> > >>>> continue with this, and on the dev call it was mentioned that
>> > >>>> “airflow-common” wasn’t a great name, so here is my proposal for
>> the
>> > file
>> > >>>> structure;
>> > >>>>>> ```
>> > >>>>>> /
>> > >>>>>>   task-sdk/...
>> > >>>>>>   airflow-core/...
>> > >>>>>>   shared/
>> > >>>>>>     kuberenetes/
>> > >>>>>>       pyproject.toml
>> > >>>>>>       src/
>> > >>>>>>         airflow_kube/__init__.py
>> > >>>>>>     core-and-tasksdk/
>> > >>>>>>       pyproject.toml
>> > >>>>>>       src/
>> > >>>>>>         airflow_shared/__init__.py
>> > >>>>>> ```
>> > >>>>>>
>> > >>>>>> Things to note here: the “shared” folder has (the possibility) of
>> > >>>> having multiple different shared “libraries” in it, in this
>> example I
>> > am
>> > >>>> supposing a hypothetical shared kuberenetes folder a world in
>> which we
>> > >>>> split the KubePodOperator and the KubeExecutor in to two separate
>> > >>>> distributions (example only, not proposing we do that right now,
>> and
>> > that
>> > >>>> will be a separate discussion)
>> > >>>>>> The other things to note here:
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> - the folder name in shared aims to be “self-documenting”, hence
>> the
>> > >>>> verbose “core-and-tasksdk” to say where the shared library is
>> > intended to
>> > >>>> be used.
>> > >>>>>> - the python module itself should almost always have an
>> `airflow_`
>> > (or
>> > >>>> maybe `_airflow_`?) prefix so that it does not conflict with
>> anything
>> > >>> else
>> > >>>> we might use. It won’t matter “in production” as those will be
>> > vendored
>> > >>> in
>> > >>>> to be imported as `airflow/_vendor/airflow_shared` etc, but
>> avoiding
>> > >>>> conflicts at dev time with the Finder approach is a good safety
>> > measure.
>> > >>>>>> I will start making a real PR for this proposal now, but I’m
>> open to
>> > >>>> feedback (either here, or in the PR when I open it)
>> > >>>>>> -ash
>> > >>>>>>
>> > >>>>>>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com> wrote:
>> > >>>>>>>
>> > >>>>>>> Yeah we have to try it and test - also building packages happens
>> > semi
>> > >>>>>>> frequently when you run `uv sync` (they use some kind of
>> heuristics
>> > >>> to
>> > >>>>>>> decide when) and you can force it with `--reinstall` or
>> > `--refresh`.
>> > >>>>>>> Package build also happens every time when you run "ci-image
>> build`
>> > >>>> now in
>> > >>>>>>> breeze so it seems like it will nicely integrate in our
>> workflows.
>> > >>>>>>>
>> > >>>>>>> Looks really cool Ash.
>> > >>>>>>>
>> > >>>>>>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <
>> ash@apache.org>
>> > >>>> wrote:
>> > >>>>>>>> It’s not just release time, but any time we build a package
>> which
>> > >>>> happens
>> > >>>>>>>> on “every” CI run. The normal unit tests will use code from
>> > >>>>>>>> airflow-common/src/airflow_common; the kube tests which build
>> an
>> > >>>> image will
>> > >>>>>>>> build the dists and vendor in the code from that commit.
>> > >>>>>>>>
>> > >>>>>>>> There is only a single copy of the shared code committed to the
>> > >>> repo,
>> > >>>> so
>> > >>>>>>>> there is never anything to synchronise.
>> > >>>>>>>>
>> > >>>>>>>>> On 4 Jul 2025, at 15:53, Amogh Desai <
>> amoghdesai.oss@gmail.com>
>> > >>>> wrote:
>> > >>>>>>>>> Thanks Ash.
>> > >>>>>>>>>
>> > >>>>>>>>> This is really cool and helpful that you were able to test
>> both
>> > >>>> scenarios
>> > >>>>>>>>> -- repo checkout
>> > >>>>>>>>> and also installing from the vendored package and the
>> resolution
>> > >>>> worked
>> > >>>>>>>>> fine too.
>> > >>>>>>>>>
>> > >>>>>>>>> I like this idea compared the to relative import one for few
>> > >>> reasons:
>> > >>>>>>>>> - It feels like it will take some time to adjust to the new
>> > coding
>> > >>>>>>>> standard
>> > >>>>>>>>> that we will lay
>> > >>>>>>>>> if we impose relative imports in the shared dist
>> > >>>>>>>>> - We can continue using repo wise absolute import standards,
>> it
>> > is
>> > >>>> also
>> > >>>>>>>>> much easier for situations
>> > >>>>>>>>> when we do global search in IDE to find + replace, this could
>> > mean
>> > >>> a
>> > >>>>>>>> change
>> > >>>>>>>>> there
>> > >>>>>>>>> - The vendoring work is a proven and established paradigm
>> across
>> > >>>> projects
>> > >>>>>>>>> and would
>> > >>>>>>>>> out of box give us the build tooling we need also
>> > >>>>>>>>>
>> > >>>>>>>>> Nothing too against the relative import but with the evidence
>> > >>>> provided
>> > >>>>>>>>> above, vendored approach
>> > >>>>>>>>> seems to only do us good.
>> > >>>>>>>>>
>> > >>>>>>>>> Regarding synchronizing it, release time should be fine as
>> long
>> > as
>> > >>> we
>> > >>>>>>>> have
>> > >>>>>>>>> a good CI workflow to probably
>> > >>>>>>>>> catch such issues per PR if changes are made in shared dist?
>> > >>>> (precommit
>> > >>>>>>>>> would make it really slow i guess)
>> > >>>>>>>>>
>> > >>>>>>>>> If we can run our tests with vendored code we should be mostly
>> > >>>> covered.
>> > >>>>>>>>> Good effort all!
>> > >>>>>>>>>
>> > >>>>>>>>> Thanks & Regards,
>> > >>>>>>>>> Amogh Desai
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <
>> > ash@apache.org>
>> > >>>>>>>> wrote:
>> > >>>>>>>>>> Okay, I think I’ve got something that works and I’m happy
>> with.
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>
>> >
>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>> > >>>>>>>>>> This produces the following from `uv build task-sdk`
>> > >>>>>>>>>> -
>> > >>>>>>>>>>
>> > >>>
>> >
>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>> > >>>>>>>>>> -
>> > >>>>>>>>>>
>> > >>>
>> >
>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>> > >>>>>>>>>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
>> > >>>>>>>>>>
>> > >>>>>>>>>> ```
>> > >>>>>>>>>> ❯ unzip -l
>> > >>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
>> > >>>>>>>> grep
>> > >>>>>>>>>> _vendor
>> > >>>>>>>>>>      50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
>> > >>>>>>>>>>    2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
>> > >>>>>>>>>>      28  02-02-2020 00:00
>> >  airflow/sdk/_vendor/airflow_common.pyi
>> > >>>>>>>>>>      18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
>> > >>>>>>>>>>     785  02-02-2020 00:00
>> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
>> > >>>>>>>>>>   10628  02-02-2020 00:00
>> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
>> > >>>>>>>>>> ```
>> > >>>>>>>>>>
>> > >>>>>>>>>> And similarly in the .tar.gz, so our “sdist” is complete too:
>> > >>>>>>>>>> ```
>> > >>>>>>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep
>> > _vendor
>> > >>>>>>>>>>
>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>> > >>>>>>>>>>
>> > apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>> > >>>>>>>>>>
>> > >>>>
>> > apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>> > >>>>>>>>>>
>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>
>> >
>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>> > >>>
>> >
>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>> > >>>>>>>>>> ```
>> > >>>>>>>>>>
>> > >>>>>>>>>> The plugin works at build time by including/copying the libs
>> > >>>> specified
>> > >>>>>>>> in
>> > >>>>>>>>>> vendor.txt into place (and let `vendoring` take care of
>> import
>> > >>>>>>>> rewrites.)
>> > >>>>>>>>>> For the imports to continue to work at “dev” time/from a repo
>> > >>>> checkout,
>> > >>>>>>>> I
>> > >>>>>>>>>> have added a import finder to `sys.meta_path`, and since
>> it’s at
>> > >>> the
>> > >>>>>>>> end of
>> > >>>>>>>>>> the list it will only be used if the normal import can’t find
>> > >>>> things.
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>
>> >
>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>> > >>>>>>>>>> This doesn’t quite give us the same runtime effect “import
>> > >>>> rewriting”
>> > >>>>>>>>>> affect, as in this approach `airflow_common` is directly
>> loaded
>> > >>>> (i.e.
>> > >>>>>>>>>> airflow.sdk._vendor.airflow_common and airflow_common exist
>> in
>> > >>>>>>>>>> sys.modules), but it does work for everything that I was
>> able to
>> > >>>> test..
>> > >>>>>>>>>> I tested it with the diff at the end of this message. My test
>> > >>>> ipython
>> > >>>>>>>>>> shell:
>> > >>>>>>>>>>
>> > >>>>>>>>>> ```
>> > >>>>>>>>>> In [1]: from airflow.sdk._vendor.airflow_common.timezone
>> import
>> > >>> foo
>> > >>>>>>>>>> In [2]: foo
>> > >>>>>>>>>> Out[2]: 1
>> > >>>>>>>>>>
>> > >>>>>>>>>> In [3]: import airflow.sdk._vendor.airflow_common
>> > >>>>>>>>>>
>> > >>>>>>>>>> In [4]: import airflow.sdk._vendor.airflow_common.timezone
>> > >>>>>>>>>>
>> > >>>>>>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>> > >>>>>>>>>> Out[5]:
>> > >>>>>>>>>>
>> > >>>
>> >
>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>> > >>>>>>>>>> In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
>> > >>>>>>>>>> Out[6]:
>> > >>>>>>>>>>
>> > >>>
>> >
>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>> > >>>>>>>>>> ```
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> And in an standalone environment with the SDK dist I built
>> (it
>> > >>>> needed
>> > >>>>>>>> the
>> > >>>>>>>>>> matching airflow-core right now, but that is nothing to do
>> with
>> > >>> this
>> > >>>>>>>>>> discussion):
>> > >>>>>>>>>>
>> > >>>>>>>>>> ```
>> > >>>>>>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>> > >>>>>>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>> > >>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
>> > >>>>>>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
>> > >>>>>>>>>> Type 'copyright', 'credits' or 'license' for more information
>> > >>>>>>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for
>> > >>> help.
>> > >>>>>>>>>> Tip: You can use `%hist` to view history, see the options
>> with
>> > >>>>>>>> `%history?`
>> > >>>>>>>>>> In [1]: import airflow.sdk._vendor.airflow_common.timezone
>> > >>>>>>>>>>
>> > >>>>>>>>>> In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
>> > >>>>>>>>>> Out[2]:
>> > >>>>>>>>>>
>> > >>>
>> >
>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>> > >>>>>>>>>> ``
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> ```diff
>> > >>>>>>>>>> diff --git a/airflow-common/src/airflow_common/__init__.py
>> > >>>>>>>>>> b/airflow-common/src/airflow_common/__init__.py
>> > >>>>>>>>>> index 13a83393a9..927b7c6b61 100644
>> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/__init__.py
>> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
>> > >>>>>>>>>> @@ -14,3 +14,5 @@
>> > >>>>>>>>>> # KIND, either express or implied.  See the License for the
>> > >>>>>>>>>> # specific language governing permissions and limitations
>> > >>>>>>>>>> # under the License.
>> > >>>>>>>>>> +
>> > >>>>>>>>>> +foo = 1
>> > >>>>>>>>>> diff --git a/airflow-common/src/airflow_common/timezone.py
>> > >>>>>>>>>> b/airflow-common/src/airflow_common/timezone.py
>> > >>>>>>>>>> index 340b924c66..58384ef20f 100644
>> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/timezone.py
>> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
>> > >>>>>>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>> > >>>>>>>>>> version.parse(metadata.version("pendulum")).major == 3
>> > >>>>>>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
>> > >>>>>>>>>> utc = pendulum.UTC
>> > >>>>>>>>>>
>> > >>>>>>>>>> +
>> > >>>>>>>>>> +from airflow_common import foo
>> > >>>>>>>>>> +
>> > >>>>>>>>>> TIMEZONE: Timezone
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> ```
>> > >>>>>>>>>>
>> > >>>>>>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com>
>> > wrote:
>> > >>>>>>>>>>> I think both approaches are doable:
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> 1) -> We can very easily prevent bad imports by pre-commit
>> when
>> > >>>>>>>> importing
>> > >>>>>>>>>>> from different distributions and make sure we are only doing
>> > >>>> relative
>> > >>>>>>>>>>> imports in the shared modules. We are doing plenty of this
>> > >>>> already. And
>> > >>>>>>>>>> yes
>> > >>>>>>>>>>> it would require relative links we currently do not allow.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> 2) -> has one disadvantage that someone at some point in
>> time
>> > >>> will
>> > >>>> have
>> > >>>>>>>>>> to
>> > >>>>>>>>>>> decide to synchronize this and if it happens just before
>> > release
>> > >>>> (I bet
>> > >>>>>>>>>>> this is going to happen) this will lead to solving problems
>> > that
>> > >>>> would
>> > >>>>>>>>>>> normally be solved during PR when you make a change (i.e.
>> > >>> symbolic
>> > >>>> link
>> > >>>>>>>>>> has
>> > >>>>>>>>>>> the advantage that whoever modifies shared code will be
>> > >>> immediately
>> > >>>>>>>>>>> notified in their PR - that they broke something because
>> either
>> > >>>> static
>> > >>>>>>>>>>> checks or mypy or tests fail.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> Ash, do you have an idea of a process (who and when) does
>> the
>> > >>>>>>>>>>> synchronisation in case of vendoring? Maybe we could solve
>> it
>> > if
>> > >>>> it is
>> > >>>>>>>>>> done
>> > >>>>>>>>>>> more frequently and with some regularity? We could
>> potentially
>> > >>>> force
>> > >>>>>>>>>>> re-vendoring at PR time as well any time shared code changes
>> > (and
>> > >>>>>>>> prevent
>> > >>>>>>>>>>> it by pre-commit. And I can't think of some place (other
>> than
>> > >>>> releases)
>> > >>>>>>>>>> in
>> > >>>>>>>>>>> our development workflow and that seems to be a bit too
>> late as
>> > >>>> puts an
>> > >>>>>>>>>>> extra effort on fixing potential incompatibilities
>> introduced
>> > on
>> > >>>>>>>> release
>> > >>>>>>>>>>> manager and delays the release. WDYT?
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> Re: relative links. I think for a shared library we could
>> > >>>> potentially
>> > >>>>>>>>>> relax
>> > >>>>>>>>>>> this and allow them (and actually disallow absolute links in
>> > the
>> > >>>> pieces
>> > >>>>>>>>>> of
>> > >>>>>>>>>>> code that are shared - again, by pre-commit). As I recall,
>> the
>> > >>> only
>> > >>>>>>>>>> reason
>> > >>>>>>>>>>> we forbade the relative links is because of how we are (or
>> > maybe
>> > >>>> were)
>> > >>>>>>>>>>> doing DAG parsing and failures resulting from it. So we
>> decided
>> > >>> to
>> > >>>> just
>> > >>>>>>>>>> not
>> > >>>>>>>>>>> allow it to keep consistency. The way how Dag parsing works
>> is
>> > >>> that
>> > >>>>>>>> when
>> > >>>>>>>>>>> you are using importlib to read the Dag from a file, the
>> > relative
>> > >>>>>>>> imports
>> > >>>>>>>>>>> do not work as it does not know what they should be relative
>> > to.
>> > >>>> But if
>> > >>>>>>>>>>> relative import is done from an imported package, it should
>> be
>> > no
>> > >>>>>>>>>> problem,
>> > >>>>>>>>>>> I think - otherwise our Dags would not be able to import any
>> > >>>> library
>> > >>>>>>>> that
>> > >>>>>>>>>>> uses relative imports.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> Of course consistency might be the reason why we do not
>> want to
>> > >>>>>>>> introduce
>> > >>>>>>>>>>> relative imports. I don't see it as an issue if it is
>> guarded
>> > by
>> > >>>>>>>>>> pre-commit
>> > >>>>>>>>>>> though.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> J.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> J.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
>> > >>>> ash@apache.org>
>> > >>>>>>>>>>> napisał:
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>> Oh yes, symlinks will work, with one big caveat: It does
>> mean
>> > >>> you
>> > >>>>>>>> can’t
>> > >>>>>>>>>>>> use absolute imports in one common module to another.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> For example
>> > >>>>>>>>>>>>
>> > >>>
>> >
>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>> > >>>>>>>>>>>> where we have
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> ```
>> > >>>>>>>>>>>> from airflow.utils.module_loading import import_string
>> > >>>>>>>>>>>> ```
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> if we want to move serve_logs into this common lib that is
>> > then
>> > >>>>>>>>>> symlinked
>> > >>>>>>>>>>>> then we wouldn’t be able to have `from
>> > >>>> airflow_common.module_loading
>> > >>>>>>>>>> import
>> > >>>>>>>>>>>> import_string`.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> I can think of two possible solutions here.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> 1) is to allow/require relative imports in this shared
>> lib, so
>> > >>>> `from
>> > >>>>>>>>>>>> .module_loading import import_string`
>> > >>>>>>>>>>>> 2) is to use `vendoring`[1] (from the pip maintainers)
>> which
>> > >>> will
>> > >>>>>>>> handle
>> > >>>>>>>>>>>> import-rewriting for us.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> I’d entirely forgot that symlinks in repos was a thing, so
>> I
>> > >>>> prepared
>> > >>>>>>>> a
>> > >>>>>>>>>>>> minimal POC/demo of what vendoring approach could look like
>> > here
>> > >>>>>>>>>>>>
>> > >>>
>> >
>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>> > >>>>>>>>>>>> Now personally I am more than happy with relative imports,
>> but
>> > >>>>>>>> generally
>> > >>>>>>>>>>>> as a project we have avoided them, so I think that limits
>> what
>> > >>> we
>> > >>>>>>>> could
>> > >>>>>>>>>> do
>> > >>>>>>>>>>>> with a symlink based approach.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> -ash
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> [1] https://github.com/pradyunsg/vendoring
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
>> > >>>>>>>> gopidesupavan@gmail.com>
>> > >>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>> Thanks Ash
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> Yes agree option 2 would be preferred for me. Making sure
>> we
>> > >>>> have all
>> > >>>>>>>>>> the
>> > >>>>>>>>>>>>> gaurdriles to protect any unwanted behaviour in code
>> sharing
>> > >>> and
>> > >>>>>>>>>>>> executing
>> > >>>>>>>>>>>>> right of tests between the packages.
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> Agree with others, option 2 would be
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
>> > >>>>>>>> amoghdesai.oss@gmail.com>
>> > >>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Thanks for starting this discussion, Ash.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> I would prefer option 2 here with proper tooling to
>> handle
>> > the
>> > >>>> code
>> > >>>>>>>>>>>>>> duplication at *release* time.
>> > >>>>>>>>>>>>>> It is best to have a dist that has all it needs in
>> itself.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Option 1 could very quickly get out of hand and if we
>> decide
>> > >>> to
>> > >>>>>>>>>> separate
>> > >>>>>>>>>>>>>> triggerer /
>> > >>>>>>>>>>>>>> dag processor / config etc etc as separate packages, back
>> > >>>> compat is
>> > >>>>>>>>>>>> going
>> > >>>>>>>>>>>>>> to be a nightmare
>> > >>>>>>>>>>>>>> and will bite us harder than we anticipate.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Thanks & Regards,
>> > >>>>>>>>>>>>>> Amogh Desai
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <
>> > >>> kaxilnaik@gmail.com>
>> > >>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>> I prefer Option 2 as well to avoid matrix of
>> dependencies
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>> > >>>>>>>>>> <j_scheffler@gmx.de.invalid
>> > >>>>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> I'd also rather prefer option 2 - reason here is it is
>> > >>> rather
>> > >>>>>>>>>>>> pragmatic
>> > >>>>>>>>>>>>>>>> and we no not need to cut another package and have less
>> > >>>> package
>> > >>>>>>>>>> counts
>> > >>>>>>>>>>>>>>>> and dependencies.
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> I remember some time ago I was checking (together with
>> > >>> Jarek,
>> > >>>> I am
>> > >>>>>>>>>> not
>> > >>>>>>>>>>>>>>>> sure anymore...) if the usage of symlinks would be
>> > possible.
>> > >>>> To
>> > >>>>>>>> keep
>> > >>>>>>>>>>>>>> the
>> > >>>>>>>>>>>>>>>> source in one package but "symlink" it into another. If
>> > then
>> > >>>> at
>> > >>>>>>>>>> point
>> > >>>>>>>>>>>>>> of
>> > >>>>>>>>>>>>>>>> packaging/release the files are materialized we have 1
>> set
>> > >>> of
>> > >>>>>>>> code.
>> > >>>>>>>>>>>>>>>> Otherwise if not possible still the redundancy could be
>> > >>>> solved by
>> > >>>>>>>> a
>> > >>>>>>>>>>>>>>>> pre-commit hook - and in Git the files are
>> de-duplicated
>> > >>>> anyway
>> > >>>>>>>>>> based
>> > >>>>>>>>>>>>>> on
>> > >>>>>>>>>>>>>>>> content hash, so this does not hurt.
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>> > >>>>>>>>>>>>>>>>> I support option 2 with proper automation & CI - the
>> > >>>> reasonings
>> > >>>>>>>>>>>>>> you've
>> > >>>>>>>>>>>>>>>>> shown for that make sense to me.
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>> Shahar
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <
>> > >>>> ash@apache.org
>> > >>>>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>>>> Hello everyone,
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> As we work on finishing off the code-level
>> separation of
>> > >>>> Task
>> > >>>>>>>> SDK
>> > >>>>>>>>>>>>>> and
>> > >>>>>>>>>>>>>>>> Core
>> > >>>>>>>>>>>>>>>>>> (scheduler etc) we have come across some situations
>> > where
>> > >>> we
>> > >>>>>>>> would
>> > >>>>>>>>>>>>>>> like
>> > >>>>>>>>>>>>>>>> to
>> > >>>>>>>>>>>>>>>>>> share code between these.
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> However it’s not as straight forward of “just put it
>> in
>> > a
>> > >>>> common
>> > >>>>>>>>>>>>>> dist
>> > >>>>>>>>>>>>>>>> they
>> > >>>>>>>>>>>>>>>>>> both depend upon” because one of the goals of the
>> Task
>> > SDK
>> > >>>>>>>>>>>>>> separation
>> > >>>>>>>>>>>>>>>> was
>> > >>>>>>>>>>>>>>>>>> to have 100% complete version independence between
>> the
>> > >>> two,
>> > >>>>>>>>>> ideally
>> > >>>>>>>>>>>>>>>> even if
>> > >>>>>>>>>>>>>>>>>> they are built into the same image and venv. Most of
>> the
>> > >>>> reason
>> > >>>>>>>>>> why
>> > >>>>>>>>>>>>>>> this
>> > >>>>>>>>>>>>>>>>>> isn’t straight forward comes down to backwards
>> > >>>> compatibility -
>> > >>>>>>>> if
>> > >>>>>>>>>> we
>> > >>>>>>>>>>>>>>>> make
>> > >>>>>>>>>>>>>>>>>> an change to the common/shared distribution
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> We’ve listed the options we have thought about in
>> > >>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545 (but
>> > that
>> > >>>> covers
>> > >>>>>>>>>>>>>> some
>> > >>>>>>>>>>>>>>>> more
>> > >>>>>>>>>>>>>>>>>> things that I don’t want to get in to in this
>> discussion
>> > >>>> such as
>> > >>>>>>>>>>>>>>>> possibly
>> > >>>>>>>>>>>>>>>>>> separating operators and executors out of a single
>> > >>> provider
>> > >>>>>>>> dist.)
>> > >>>>>>>>>>>>>>>>>> To give a concrete example of some code I would like
>> to
>> > >>>> share
>> > >>>
>> >
>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>> > >>>>>>>>>>>>>>>>>> — logging config. Another thing we will want to share
>> > will
>> > >>>> be
>> > >>>>>>>> the
>> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class from airflow.configuration
>> > (but
>> > >>>>>>>> notably:
>> > >>>>>>>>>>>>>>> only
>> > >>>>>>>>>>>>>>>> the
>> > >>>>>>>>>>>>>>>>>> parser class, _not_ the default config values, again,
>> > lets
>> > >>>> not
>> > >>>>>>>>>> dwell
>> > >>>>>>>>>>>>>>> on
>> > >>>>>>>>>>>>>>>> the
>> > >>>>>>>>>>>>>>>>>> specifics of that)
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> So to bring the options listed in the issue here for
>> > >>>> discussion,
>> > >>>>>>>>>>>>>>> broadly
>> > >>>>>>>>>>>>>>>>>> speaking there are two high-level approaches:
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> 1. A single shared distribution
>> > >>>>>>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> The advantage of Approach 1 is that we only have the
>> > code
>> > >>>> in one
>> > >>>>>>>>>>>>>>> place.
>> > >>>>>>>>>>>>>>>>>> However for me, at least in this specific case of
>> > Logging
>> > >>>> config
>> > >>>>>>>>>> or
>> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class is that backwards
>> > compatibility
>> > >>> is
>> > >>>>>>>> much
>> > >>>>>>>>>>>>>> much
>> > >>>>>>>>>>>>>>>>>> harder.
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> The main advantage of Approach 2 is the the code is
>> > >>> released
>> > >>>>>>>>>>>>>>>> with/embedded
>> > >>>>>>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would
>> contain
>> > >>> the
>> > >>>>>>>> right
>> > >>>>>>>>>>>>>>>> version
>> > >>>>>>>>>>>>>>>>>> of the logging config and ConfigParser etc). The
>> > downside
>> > >>> is
>> > >>>>>>>> that
>> > >>>>>>>>>>>>>>> either
>> > >>>>>>>>>>>>>>>>>> the code will need to be duplicated in the repo, or
>> > better
>> > >>>> yet
>> > >>>>>>>> it
>> > >>>>>>>>>>>>>>> would
>> > >>>>>>>>>>>>>>>>>> live in a single place in the repo, but some tooling
>> > (TBD)
>> > >>>> will
>> > >>>>>>>>>>>>>>>>>> automatically handle the duplication, either at
>> commit
>> > >>>> time, or
>> > >>>>>>>> my
>> > >>>>>>>>>>>>>>>>>> preference, at release time.
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> For this kind of shared “utility” code I am very
>> > strongly
>> > >>>>>>>> leaning
>> > >>>>>>>>>>>>>>>> towards
>> > >>>>>>>>>>>>>>>>>> option 2 with automation, as otherwise I think the
>> > >>> backwards
>> > >>>>>>>>>>>>>>>> compatibility
>> > >>>>>>>>>>>>>>>>>> requirements would make it unworkable (very quickly
>> over
>> > >>>> time
>> > >>>>>>>> the
>> > >>>>>>>>>>>>>>>>>> combinations we would have to test would just be
>> > >>>> unreasonable)
>> > >>>>>>>>>> and I
>> > >>>>>>>>>>>>>>>> don’t
>> > >>>>>>>>>>>>>>>>>> feel confident we can have things as stable as we
>> need
>> > to
>> > >>>> really
>> > >>>>>>>>>>>>>>> deliver
>> > >>>>>>>>>>>>>>>>>> the version separation/independency I want to
>> delivery
>> > >>> with
>> > >>>>>>>>>> AIP-72.
>> > >>>>>>>>>>>>>>>>>> So unless someone feels very strongly about this, I
>> will
>> > >>>> come up
>> > >>>>>>>>>>>>>> with
>> > >>>>>>>>>>>>>>> a
>> > >>>>>>>>>>>>>>>>>> draft PR for further discussion that will implement
>> code
>> > >>>> sharing
>> > >>>>>>>>>> via
>> > >>>>>>>>>>>>>>>>>> “vendoring” it at build time. I have an idea of how I
>> > can
>> > >>>>>>>> achieve
>> > >>>>>>>>>>>>>> this
>> > >>>>>>>>>>>>>>>> so
>> > >>>>>>>>>>>>>>>>>> we have a single version in the repo and it’ll work
>> > there,
>> > >>>> but
>> > >>>>>>>> at
>> > >>>>>>>>>>>>>>>> runtime
>> > >>>>>>>>>>>>>>>>>> we vendor it in to the shipped dist so it lives at
>> > >>> something
>> > >>>>>>>> like
>> > >>>>>>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> In terms of repo layout, this likely means we would
>> end
>> > up
>> > >>>> with:
>> > >>>>>>>>>>>>>>>>>> airflow-core/pyproject.toml
>> > >>>>>>>>>>>>>>>>>> airflow-core/src/
>> > >>>>>>>>>>>>>>>>>> airflow-core/tests/
>> > >>>>>>>>>>>>>>>>>> task-sdk/pyproject.toml
>> > >>>>>>>>>>>>>>>>>> task-sdk/src/
>> > >>>>>>>>>>>>>>>>>> task-sdk/tests/
>> > >>>>>>>>>>>>>>>>>> airflow-common/src
>> > >>>>>>>>>>>>>>>>>> airflow-common/tests/
>> > >>>>>>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps
>> > would
>> > >>>> be
>> > >>>>>>>>>>>>>> included
>> > >>>>>>>>>>>>>>>> in
>> > >>>>>>>>>>>>>>>>>> the downstream projects. TBD.
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> Thoughts and feedback welcomed.
>> > >>>>
>> ---------------------------------------------------------------------
>> > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>> > dev-unsubscribe@airflow.apache.org
>> > >>>>>>>>>>>>>>>> For additional commands, e-mail:
>> > >>> dev-help@airflow.apache.org
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>
>> > >>>
>> ---------------------------------------------------------------------
>> > >>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > >>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>
>> > ---------------------------------------------------------------------
>> > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>> > >>>>>>
>> > >>>>>
>> ---------------------------------------------------------------------
>> > >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>> > >>>>>
>> > >>>>
>> ---------------------------------------------------------------------
>> > >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > >>>> For additional commands, e-mail: dev-help@airflow.apache.org
>> > >>>>
>> > >>>>
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > > For additional commands, e-mail: dev-help@airflow.apache.org
>> > >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > For additional commands, e-mail: dev-help@airflow.apache.org
>> >
>> >
>>
>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Amogh Desai <am...@gmail.com>.
I like the folder structure proposed by Ash and have no objections with it.

"core_and_task_sdk" sounds good to me and justifies what it should do
pretty well.

@Jarek Potiuk <ja...@potiuk.com> a little confused on what you mean there,
I am understanding the direction
but could you elaborate a bit more please?

Naming is REALLY hard!

Thanks & Regards,
Amogh Desai


On Tue, Jul 8, 2025 at 2:52 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> How about splitting it even more and having each shared "thing" named?
> "logging", "config" and sharing them explicitly and separately with the
> right "user" ?
> That sounds way more modular and  we will be able to choose which of the
> shared "utils" we use where.
>
> J.
>
>
> On Mon, Jul 7, 2025 at 11:13 PM Jens Scheffler <j_scheffler@gmx.de.invalid
> >
> wrote:
>
> > I like "core_and_task_sdk the same like core-and-task-sdk - I have no
> > problem and it is a path only.
> >
> > if we get to "dag-parser-scheduler-task-sdk-and-triggerer" which is a
> > bit bulky we then should name it "all-not-api-server" :-D
> >
> > On 07.07.25 22:57, Ash Berlin-Taylor wrote:
> > > In case I did a bad job explaining it, the “core and task sdk” is not
> in
> > the module name/import name, just in the file path.
> > >
> > > Anyone have other ideas?
> > >
> > >> On 7 Jul 2025, at 21:37, Buğra Öztürk <oz...@gmail.com>
> wrote:
> > >>
> > >> Thanks Ash! Looks cool! I like the structure. This will enable all
> the
> > >> combinations and structure looks easy to grasp. No strong stance on
> the
> > >> naming other than maybe it is a bit long with `and`, `core_ctl` could
> be
> > >> shorter, since no import path is defined like that, we can give any
> name
> > >> for sure.
> > >>
> > >> Best regards,
> > >>
> > >>> On Mon, 7 Jul 2025, 21:51 Jarek Potiuk, <ja...@potiuk.com> wrote:
> > >>>
> > >>> Looks good but I think we should find some better logical name for
> > >>> core_and_sdk :)
> > >>>
> > >>> pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
> > >>> <j_...@gmx.de.invalid> napisał:
> > >>>
> > >>>> Cool! Especially the "shared" folder with the ability to have
> > >>>> N-combinations w/o exploding project repo root!
> > >>>>
> > >>>> On 07.07.25 14:43, Ash Berlin-Taylor wrote:
> > >>>>> Oh, and all of this will be explain in shared/README.md
> > >>>>>
> > >>>>>> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org>
> wrote:
> > >>>>>>
> > >>>>>> Okay, so it seems we have agreement on the approach here, so I’ll
> > >>>> continue with this, and on the dev call it was mentioned that
> > >>>> “airflow-common” wasn’t a great name, so here is my proposal for the
> > file
> > >>>> structure;
> > >>>>>> ```
> > >>>>>> /
> > >>>>>>   task-sdk/...
> > >>>>>>   airflow-core/...
> > >>>>>>   shared/
> > >>>>>>     kuberenetes/
> > >>>>>>       pyproject.toml
> > >>>>>>       src/
> > >>>>>>         airflow_kube/__init__.py
> > >>>>>>     core-and-tasksdk/
> > >>>>>>       pyproject.toml
> > >>>>>>       src/
> > >>>>>>         airflow_shared/__init__.py
> > >>>>>> ```
> > >>>>>>
> > >>>>>> Things to note here: the “shared” folder has (the possibility) of
> > >>>> having multiple different shared “libraries” in it, in this example
> I
> > am
> > >>>> supposing a hypothetical shared kuberenetes folder a world in which
> we
> > >>>> split the KubePodOperator and the KubeExecutor in to two separate
> > >>>> distributions (example only, not proposing we do that right now, and
> > that
> > >>>> will be a separate discussion)
> > >>>>>> The other things to note here:
> > >>>>>>
> > >>>>>>
> > >>>>>> - the folder name in shared aims to be “self-documenting”, hence
> the
> > >>>> verbose “core-and-tasksdk” to say where the shared library is
> > intended to
> > >>>> be used.
> > >>>>>> - the python module itself should almost always have an `airflow_`
> > (or
> > >>>> maybe `_airflow_`?) prefix so that it does not conflict with
> anything
> > >>> else
> > >>>> we might use. It won’t matter “in production” as those will be
> > vendored
> > >>> in
> > >>>> to be imported as `airflow/_vendor/airflow_shared` etc, but avoiding
> > >>>> conflicts at dev time with the Finder approach is a good safety
> > measure.
> > >>>>>> I will start making a real PR for this proposal now, but I’m open
> to
> > >>>> feedback (either here, or in the PR when I open it)
> > >>>>>> -ash
> > >>>>>>
> > >>>>>>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >>>>>>>
> > >>>>>>> Yeah we have to try it and test - also building packages happens
> > semi
> > >>>>>>> frequently when you run `uv sync` (they use some kind of
> heuristics
> > >>> to
> > >>>>>>> decide when) and you can force it with `--reinstall` or
> > `--refresh`.
> > >>>>>>> Package build also happens every time when you run "ci-image
> build`
> > >>>> now in
> > >>>>>>> breeze so it seems like it will nicely integrate in our
> workflows.
> > >>>>>>>
> > >>>>>>> Looks really cool Ash.
> > >>>>>>>
> > >>>>>>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <ash@apache.org
> >
> > >>>> wrote:
> > >>>>>>>> It’s not just release time, but any time we build a package
> which
> > >>>> happens
> > >>>>>>>> on “every” CI run. The normal unit tests will use code from
> > >>>>>>>> airflow-common/src/airflow_common; the kube tests which build an
> > >>>> image will
> > >>>>>>>> build the dists and vendor in the code from that commit.
> > >>>>>>>>
> > >>>>>>>> There is only a single copy of the shared code committed to the
> > >>> repo,
> > >>>> so
> > >>>>>>>> there is never anything to synchronise.
> > >>>>>>>>
> > >>>>>>>>> On 4 Jul 2025, at 15:53, Amogh Desai <amoghdesai.oss@gmail.com
> >
> > >>>> wrote:
> > >>>>>>>>> Thanks Ash.
> > >>>>>>>>>
> > >>>>>>>>> This is really cool and helpful that you were able to test both
> > >>>> scenarios
> > >>>>>>>>> -- repo checkout
> > >>>>>>>>> and also installing from the vendored package and the
> resolution
> > >>>> worked
> > >>>>>>>>> fine too.
> > >>>>>>>>>
> > >>>>>>>>> I like this idea compared the to relative import one for few
> > >>> reasons:
> > >>>>>>>>> - It feels like it will take some time to adjust to the new
> > coding
> > >>>>>>>> standard
> > >>>>>>>>> that we will lay
> > >>>>>>>>> if we impose relative imports in the shared dist
> > >>>>>>>>> - We can continue using repo wise absolute import standards, it
> > is
> > >>>> also
> > >>>>>>>>> much easier for situations
> > >>>>>>>>> when we do global search in IDE to find + replace, this could
> > mean
> > >>> a
> > >>>>>>>> change
> > >>>>>>>>> there
> > >>>>>>>>> - The vendoring work is a proven and established paradigm
> across
> > >>>> projects
> > >>>>>>>>> and would
> > >>>>>>>>> out of box give us the build tooling we need also
> > >>>>>>>>>
> > >>>>>>>>> Nothing too against the relative import but with the evidence
> > >>>> provided
> > >>>>>>>>> above, vendored approach
> > >>>>>>>>> seems to only do us good.
> > >>>>>>>>>
> > >>>>>>>>> Regarding synchronizing it, release time should be fine as long
> > as
> > >>> we
> > >>>>>>>> have
> > >>>>>>>>> a good CI workflow to probably
> > >>>>>>>>> catch such issues per PR if changes are made in shared dist?
> > >>>> (precommit
> > >>>>>>>>> would make it really slow i guess)
> > >>>>>>>>>
> > >>>>>>>>> If we can run our tests with vendored code we should be mostly
> > >>>> covered.
> > >>>>>>>>> Good effort all!
> > >>>>>>>>>
> > >>>>>>>>> Thanks & Regards,
> > >>>>>>>>> Amogh Desai
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <
> > ash@apache.org>
> > >>>>>>>> wrote:
> > >>>>>>>>>> Okay, I think I’ve got something that works and I’m happy
> with.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>
> >
> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
> > >>>>>>>>>> This produces the following from `uv build task-sdk`
> > >>>>>>>>>> -
> > >>>>>>>>>>
> > >>>
> >
> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
> > >>>>>>>>>> -
> > >>>>>>>>>>
> > >>>
> >
> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
> > >>>>>>>>>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
> > >>>>>>>>>>
> > >>>>>>>>>> ```
> > >>>>>>>>>> ❯ unzip -l
> > >>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
> > >>>>>>>> grep
> > >>>>>>>>>> _vendor
> > >>>>>>>>>>      50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
> > >>>>>>>>>>    2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
> > >>>>>>>>>>      28  02-02-2020 00:00
> >  airflow/sdk/_vendor/airflow_common.pyi
> > >>>>>>>>>>      18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
> > >>>>>>>>>>     785  02-02-2020 00:00
> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
> > >>>>>>>>>>   10628  02-02-2020 00:00
> > >>>>>>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
> > >>>>>>>>>> ```
> > >>>>>>>>>>
> > >>>>>>>>>> And similarly in the .tar.gz, so our “sdist” is complete too:
> > >>>>>>>>>> ```
> > >>>>>>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep
> > _vendor
> > >>>>>>>>>>
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
> > >>>>>>>>>>
> > apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
> > >>>>>>>>>>
> > >>>>
> > apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
> > >>>>>>>>>>
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>
> >
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
> > >>>
> >
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
> > >>>>>>>>>> ```
> > >>>>>>>>>>
> > >>>>>>>>>> The plugin works at build time by including/copying the libs
> > >>>> specified
> > >>>>>>>> in
> > >>>>>>>>>> vendor.txt into place (and let `vendoring` take care of import
> > >>>>>>>> rewrites.)
> > >>>>>>>>>> For the imports to continue to work at “dev” time/from a repo
> > >>>> checkout,
> > >>>>>>>> I
> > >>>>>>>>>> have added a import finder to `sys.meta_path`, and since it’s
> at
> > >>> the
> > >>>>>>>> end of
> > >>>>>>>>>> the list it will only be used if the normal import can’t find
> > >>>> things.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>
> >
> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
> > >>>>>>>>>> This doesn’t quite give us the same runtime effect “import
> > >>>> rewriting”
> > >>>>>>>>>> affect, as in this approach `airflow_common` is directly
> loaded
> > >>>> (i.e.
> > >>>>>>>>>> airflow.sdk._vendor.airflow_common and airflow_common exist in
> > >>>>>>>>>> sys.modules), but it does work for everything that I was able
> to
> > >>>> test..
> > >>>>>>>>>> I tested it with the diff at the end of this message. My test
> > >>>> ipython
> > >>>>>>>>>> shell:
> > >>>>>>>>>>
> > >>>>>>>>>> ```
> > >>>>>>>>>> In [1]: from airflow.sdk._vendor.airflow_common.timezone
> import
> > >>> foo
> > >>>>>>>>>> In [2]: foo
> > >>>>>>>>>> Out[2]: 1
> > >>>>>>>>>>
> > >>>>>>>>>> In [3]: import airflow.sdk._vendor.airflow_common
> > >>>>>>>>>>
> > >>>>>>>>>> In [4]: import airflow.sdk._vendor.airflow_common.timezone
> > >>>>>>>>>>
> > >>>>>>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
> > >>>>>>>>>> Out[5]:
> > >>>>>>>>>>
> > >>>
> >
> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
> > >>>>>>>>>> In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
> > >>>>>>>>>> Out[6]:
> > >>>>>>>>>>
> > >>>
> >
> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
> > >>>>>>>>>> ```
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> And in an standalone environment with the SDK dist I built (it
> > >>>> needed
> > >>>>>>>> the
> > >>>>>>>>>> matching airflow-core right now, but that is nothing to do
> with
> > >>> this
> > >>>>>>>>>> discussion):
> > >>>>>>>>>>
> > >>>>>>>>>> ```
> > >>>>>>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
> > >>>>>>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
> > >>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
> > >>>>>>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
> > >>>>>>>>>> Type 'copyright', 'credits' or 'license' for more information
> > >>>>>>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for
> > >>> help.
> > >>>>>>>>>> Tip: You can use `%hist` to view history, see the options with
> > >>>>>>>> `%history?`
> > >>>>>>>>>> In [1]: import airflow.sdk._vendor.airflow_common.timezone
> > >>>>>>>>>>
> > >>>>>>>>>> In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
> > >>>>>>>>>> Out[2]:
> > >>>>>>>>>>
> > >>>
> >
> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
> > >>>>>>>>>> ``
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> ```diff
> > >>>>>>>>>> diff --git a/airflow-common/src/airflow_common/__init__.py
> > >>>>>>>>>> b/airflow-common/src/airflow_common/__init__.py
> > >>>>>>>>>> index 13a83393a9..927b7c6b61 100644
> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/__init__.py
> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
> > >>>>>>>>>> @@ -14,3 +14,5 @@
> > >>>>>>>>>> # KIND, either express or implied.  See the License for the
> > >>>>>>>>>> # specific language governing permissions and limitations
> > >>>>>>>>>> # under the License.
> > >>>>>>>>>> +
> > >>>>>>>>>> +foo = 1
> > >>>>>>>>>> diff --git a/airflow-common/src/airflow_common/timezone.py
> > >>>>>>>>>> b/airflow-common/src/airflow_common/timezone.py
> > >>>>>>>>>> index 340b924c66..58384ef20f 100644
> > >>>>>>>>>> --- a/airflow-common/src/airflow_common/timezone.py
> > >>>>>>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
> > >>>>>>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
> > >>>>>>>>>> version.parse(metadata.version("pendulum")).major == 3
> > >>>>>>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
> > >>>>>>>>>> utc = pendulum.UTC
> > >>>>>>>>>>
> > >>>>>>>>>> +
> > >>>>>>>>>> +from airflow_common import foo
> > >>>>>>>>>> +
> > >>>>>>>>>> TIMEZONE: Timezone
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> ```
> > >>>>>>>>>>
> > >>>>>>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > >>>>>>>>>>> I think both approaches are doable:
> > >>>>>>>>>>>
> > >>>>>>>>>>> 1) -> We can very easily prevent bad imports by pre-commit
> when
> > >>>>>>>> importing
> > >>>>>>>>>>> from different distributions and make sure we are only doing
> > >>>> relative
> > >>>>>>>>>>> imports in the shared modules. We are doing plenty of this
> > >>>> already. And
> > >>>>>>>>>> yes
> > >>>>>>>>>>> it would require relative links we currently do not allow.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 2) -> has one disadvantage that someone at some point in time
> > >>> will
> > >>>> have
> > >>>>>>>>>> to
> > >>>>>>>>>>> decide to synchronize this and if it happens just before
> > release
> > >>>> (I bet
> > >>>>>>>>>>> this is going to happen) this will lead to solving problems
> > that
> > >>>> would
> > >>>>>>>>>>> normally be solved during PR when you make a change (i.e.
> > >>> symbolic
> > >>>> link
> > >>>>>>>>>> has
> > >>>>>>>>>>> the advantage that whoever modifies shared code will be
> > >>> immediately
> > >>>>>>>>>>> notified in their PR - that they broke something because
> either
> > >>>> static
> > >>>>>>>>>>> checks or mypy or tests fail.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Ash, do you have an idea of a process (who and when) does the
> > >>>>>>>>>>> synchronisation in case of vendoring? Maybe we could solve it
> > if
> > >>>> it is
> > >>>>>>>>>> done
> > >>>>>>>>>>> more frequently and with some regularity? We could
> potentially
> > >>>> force
> > >>>>>>>>>>> re-vendoring at PR time as well any time shared code changes
> > (and
> > >>>>>>>> prevent
> > >>>>>>>>>>> it by pre-commit. And I can't think of some place (other than
> > >>>> releases)
> > >>>>>>>>>> in
> > >>>>>>>>>>> our development workflow and that seems to be a bit too late
> as
> > >>>> puts an
> > >>>>>>>>>>> extra effort on fixing potential incompatibilities introduced
> > on
> > >>>>>>>> release
> > >>>>>>>>>>> manager and delays the release. WDYT?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Re: relative links. I think for a shared library we could
> > >>>> potentially
> > >>>>>>>>>> relax
> > >>>>>>>>>>> this and allow them (and actually disallow absolute links in
> > the
> > >>>> pieces
> > >>>>>>>>>> of
> > >>>>>>>>>>> code that are shared - again, by pre-commit). As I recall,
> the
> > >>> only
> > >>>>>>>>>> reason
> > >>>>>>>>>>> we forbade the relative links is because of how we are (or
> > maybe
> > >>>> were)
> > >>>>>>>>>>> doing DAG parsing and failures resulting from it. So we
> decided
> > >>> to
> > >>>> just
> > >>>>>>>>>> not
> > >>>>>>>>>>> allow it to keep consistency. The way how Dag parsing works
> is
> > >>> that
> > >>>>>>>> when
> > >>>>>>>>>>> you are using importlib to read the Dag from a file, the
> > relative
> > >>>>>>>> imports
> > >>>>>>>>>>> do not work as it does not know what they should be relative
> > to.
> > >>>> But if
> > >>>>>>>>>>> relative import is done from an imported package, it should
> be
> > no
> > >>>>>>>>>> problem,
> > >>>>>>>>>>> I think - otherwise our Dags would not be able to import any
> > >>>> library
> > >>>>>>>> that
> > >>>>>>>>>>> uses relative imports.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Of course consistency might be the reason why we do not want
> to
> > >>>>>>>> introduce
> > >>>>>>>>>>> relative imports. I don't see it as an issue if it is guarded
> > by
> > >>>>>>>>>> pre-commit
> > >>>>>>>>>>> though.
> > >>>>>>>>>>>
> > >>>>>>>>>>> J.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> J.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
> > >>>> ash@apache.org>
> > >>>>>>>>>>> napisał:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Oh yes, symlinks will work, with one big caveat: It does
> mean
> > >>> you
> > >>>>>>>> can’t
> > >>>>>>>>>>>> use absolute imports in one common module to another.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> For example
> > >>>>>>>>>>>>
> > >>>
> >
> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
> > >>>>>>>>>>>> where we have
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ```
> > >>>>>>>>>>>> from airflow.utils.module_loading import import_string
> > >>>>>>>>>>>> ```
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> if we want to move serve_logs into this common lib that is
> > then
> > >>>>>>>>>> symlinked
> > >>>>>>>>>>>> then we wouldn’t be able to have `from
> > >>>> airflow_common.module_loading
> > >>>>>>>>>> import
> > >>>>>>>>>>>> import_string`.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I can think of two possible solutions here.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1) is to allow/require relative imports in this shared lib,
> so
> > >>>> `from
> > >>>>>>>>>>>> .module_loading import import_string`
> > >>>>>>>>>>>> 2) is to use `vendoring`[1] (from the pip maintainers) which
> > >>> will
> > >>>>>>>> handle
> > >>>>>>>>>>>> import-rewriting for us.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I’d entirely forgot that symlinks in repos was a thing, so I
> > >>>> prepared
> > >>>>>>>> a
> > >>>>>>>>>>>> minimal POC/demo of what vendoring approach could look like
> > here
> > >>>>>>>>>>>>
> > >>>
> >
> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
> > >>>>>>>>>>>> Now personally I am more than happy with relative imports,
> but
> > >>>>>>>> generally
> > >>>>>>>>>>>> as a project we have avoided them, so I think that limits
> what
> > >>> we
> > >>>>>>>> could
> > >>>>>>>>>> do
> > >>>>>>>>>>>> with a symlink based approach.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> -ash
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> [1] https://github.com/pradyunsg/vendoring
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
> > >>>>>>>> gopidesupavan@gmail.com>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>> Thanks Ash
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Yes agree option 2 would be preferred for me. Making sure
> we
> > >>>> have all
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> gaurdriles to protect any unwanted behaviour in code
> sharing
> > >>> and
> > >>>>>>>>>>>> executing
> > >>>>>>>>>>>>> right of tests between the packages.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Agree with others, option 2 would be
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
> > >>>>>>>> amoghdesai.oss@gmail.com>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for starting this discussion, Ash.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I would prefer option 2 here with proper tooling to handle
> > the
> > >>>> code
> > >>>>>>>>>>>>>> duplication at *release* time.
> > >>>>>>>>>>>>>> It is best to have a dist that has all it needs in itself.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Option 1 could very quickly get out of hand and if we
> decide
> > >>> to
> > >>>>>>>>>> separate
> > >>>>>>>>>>>>>> triggerer /
> > >>>>>>>>>>>>>> dag processor / config etc etc as separate packages, back
> > >>>> compat is
> > >>>>>>>>>>>> going
> > >>>>>>>>>>>>>> to be a nightmare
> > >>>>>>>>>>>>>> and will bite us harder than we anticipate.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks & Regards,
> > >>>>>>>>>>>>>> Amogh Desai
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <
> > >>> kaxilnaik@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>> I prefer Option 2 as well to avoid matrix of dependencies
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
> > >>>>>>>>>> <j_scheffler@gmx.de.invalid
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I'd also rather prefer option 2 - reason here is it is
> > >>> rather
> > >>>>>>>>>>>> pragmatic
> > >>>>>>>>>>>>>>>> and we no not need to cut another package and have less
> > >>>> package
> > >>>>>>>>>> counts
> > >>>>>>>>>>>>>>>> and dependencies.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I remember some time ago I was checking (together with
> > >>> Jarek,
> > >>>> I am
> > >>>>>>>>>> not
> > >>>>>>>>>>>>>>>> sure anymore...) if the usage of symlinks would be
> > possible.
> > >>>> To
> > >>>>>>>> keep
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> source in one package but "symlink" it into another. If
> > then
> > >>>> at
> > >>>>>>>>>> point
> > >>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>> packaging/release the files are materialized we have 1
> set
> > >>> of
> > >>>>>>>> code.
> > >>>>>>>>>>>>>>>> Otherwise if not possible still the redundancy could be
> > >>>> solved by
> > >>>>>>>> a
> > >>>>>>>>>>>>>>>> pre-commit hook - and in Git the files are de-duplicated
> > >>>> anyway
> > >>>>>>>>>> based
> > >>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>> content hash, so this does not hurt.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
> > >>>>>>>>>>>>>>>>> I support option 2 with proper automation & CI - the
> > >>>> reasonings
> > >>>>>>>>>>>>>> you've
> > >>>>>>>>>>>>>>>>> shown for that make sense to me.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Shahar
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <
> > >>>> ash@apache.org
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> As we work on finishing off the code-level separation
> of
> > >>>> Task
> > >>>>>>>> SDK
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>> Core
> > >>>>>>>>>>>>>>>>>> (scheduler etc) we have come across some situations
> > where
> > >>> we
> > >>>>>>>> would
> > >>>>>>>>>>>>>>> like
> > >>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>> share code between these.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> However it’s not as straight forward of “just put it
> in
> > a
> > >>>> common
> > >>>>>>>>>>>>>> dist
> > >>>>>>>>>>>>>>>> they
> > >>>>>>>>>>>>>>>>>> both depend upon” because one of the goals of the Task
> > SDK
> > >>>>>>>>>>>>>> separation
> > >>>>>>>>>>>>>>>> was
> > >>>>>>>>>>>>>>>>>> to have 100% complete version independence between the
> > >>> two,
> > >>>>>>>>>> ideally
> > >>>>>>>>>>>>>>>> even if
> > >>>>>>>>>>>>>>>>>> they are built into the same image and venv. Most of
> the
> > >>>> reason
> > >>>>>>>>>> why
> > >>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>> isn’t straight forward comes down to backwards
> > >>>> compatibility -
> > >>>>>>>> if
> > >>>>>>>>>> we
> > >>>>>>>>>>>>>>>> make
> > >>>>>>>>>>>>>>>>>> an change to the common/shared distribution
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> We’ve listed the options we have thought about in
> > >>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545 (but
> > that
> > >>>> covers
> > >>>>>>>>>>>>>> some
> > >>>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>> things that I don’t want to get in to in this
> discussion
> > >>>> such as
> > >>>>>>>>>>>>>>>> possibly
> > >>>>>>>>>>>>>>>>>> separating operators and executors out of a single
> > >>> provider
> > >>>>>>>> dist.)
> > >>>>>>>>>>>>>>>>>> To give a concrete example of some code I would like
> to
> > >>>> share
> > >>>
> >
> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
> > >>>>>>>>>>>>>>>>>> — logging config. Another thing we will want to share
> > will
> > >>>> be
> > >>>>>>>> the
> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class from airflow.configuration
> > (but
> > >>>>>>>> notably:
> > >>>>>>>>>>>>>>> only
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> parser class, _not_ the default config values, again,
> > lets
> > >>>> not
> > >>>>>>>>>> dwell
> > >>>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> specifics of that)
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> So to bring the options listed in the issue here for
> > >>>> discussion,
> > >>>>>>>>>>>>>>> broadly
> > >>>>>>>>>>>>>>>>>> speaking there are two high-level approaches:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> 1. A single shared distribution
> > >>>>>>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> The advantage of Approach 1 is that we only have the
> > code
> > >>>> in one
> > >>>>>>>>>>>>>>> place.
> > >>>>>>>>>>>>>>>>>> However for me, at least in this specific case of
> > Logging
> > >>>> config
> > >>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>> AirflowConfigParser class is that backwards
> > compatibility
> > >>> is
> > >>>>>>>> much
> > >>>>>>>>>>>>>> much
> > >>>>>>>>>>>>>>>>>> harder.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> The main advantage of Approach 2 is the the code is
> > >>> released
> > >>>>>>>>>>>>>>>> with/embedded
> > >>>>>>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would
> contain
> > >>> the
> > >>>>>>>> right
> > >>>>>>>>>>>>>>>> version
> > >>>>>>>>>>>>>>>>>> of the logging config and ConfigParser etc). The
> > downside
> > >>> is
> > >>>>>>>> that
> > >>>>>>>>>>>>>>> either
> > >>>>>>>>>>>>>>>>>> the code will need to be duplicated in the repo, or
> > better
> > >>>> yet
> > >>>>>>>> it
> > >>>>>>>>>>>>>>> would
> > >>>>>>>>>>>>>>>>>> live in a single place in the repo, but some tooling
> > (TBD)
> > >>>> will
> > >>>>>>>>>>>>>>>>>> automatically handle the duplication, either at commit
> > >>>> time, or
> > >>>>>>>> my
> > >>>>>>>>>>>>>>>>>> preference, at release time.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> For this kind of shared “utility” code I am very
> > strongly
> > >>>>>>>> leaning
> > >>>>>>>>>>>>>>>> towards
> > >>>>>>>>>>>>>>>>>> option 2 with automation, as otherwise I think the
> > >>> backwards
> > >>>>>>>>>>>>>>>> compatibility
> > >>>>>>>>>>>>>>>>>> requirements would make it unworkable (very quickly
> over
> > >>>> time
> > >>>>>>>> the
> > >>>>>>>>>>>>>>>>>> combinations we would have to test would just be
> > >>>> unreasonable)
> > >>>>>>>>>> and I
> > >>>>>>>>>>>>>>>> don’t
> > >>>>>>>>>>>>>>>>>> feel confident we can have things as stable as we need
> > to
> > >>>> really
> > >>>>>>>>>>>>>>> deliver
> > >>>>>>>>>>>>>>>>>> the version separation/independency I want to delivery
> > >>> with
> > >>>>>>>>>> AIP-72.
> > >>>>>>>>>>>>>>>>>> So unless someone feels very strongly about this, I
> will
> > >>>> come up
> > >>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>> draft PR for further discussion that will implement
> code
> > >>>> sharing
> > >>>>>>>>>> via
> > >>>>>>>>>>>>>>>>>> “vendoring” it at build time. I have an idea of how I
> > can
> > >>>>>>>> achieve
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>> so
> > >>>>>>>>>>>>>>>>>> we have a single version in the repo and it’ll work
> > there,
> > >>>> but
> > >>>>>>>> at
> > >>>>>>>>>>>>>>>> runtime
> > >>>>>>>>>>>>>>>>>> we vendor it in to the shipped dist so it lives at
> > >>> something
> > >>>>>>>> like
> > >>>>>>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> In terms of repo layout, this likely means we would
> end
> > up
> > >>>> with:
> > >>>>>>>>>>>>>>>>>> airflow-core/pyproject.toml
> > >>>>>>>>>>>>>>>>>> airflow-core/src/
> > >>>>>>>>>>>>>>>>>> airflow-core/tests/
> > >>>>>>>>>>>>>>>>>> task-sdk/pyproject.toml
> > >>>>>>>>>>>>>>>>>> task-sdk/src/
> > >>>>>>>>>>>>>>>>>> task-sdk/tests/
> > >>>>>>>>>>>>>>>>>> airflow-common/src
> > >>>>>>>>>>>>>>>>>> airflow-common/tests/
> > >>>>>>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps
> > would
> > >>>> be
> > >>>>>>>>>>>>>> included
> > >>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>> the downstream projects. TBD.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thoughts and feedback welcomed.
> > >>>>
> ---------------------------------------------------------------------
> > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> > dev-unsubscribe@airflow.apache.org
> > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > >>> dev-help@airflow.apache.org
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>
> > >>> ---------------------------------------------------------------------
> > >>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > ---------------------------------------------------------------------
> > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>>>>>
> > >>>>>
> ---------------------------------------------------------------------
> > >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>>>
> > >>>>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Jarek Potiuk <ja...@potiuk.com>.
How about splitting it even more and having each shared "thing" named?
"logging", "config" and sharing them explicitly and separately with the
right "user" ?
That sounds way more modular and  we will be able to choose which of the
shared "utils" we use where.

J.


On Mon, Jul 7, 2025 at 11:13 PM Jens Scheffler <j_...@gmx.de.invalid>
wrote:

> I like "core_and_task_sdk the same like core-and-task-sdk - I have no
> problem and it is a path only.
>
> if we get to "dag-parser-scheduler-task-sdk-and-triggerer" which is a
> bit bulky we then should name it "all-not-api-server" :-D
>
> On 07.07.25 22:57, Ash Berlin-Taylor wrote:
> > In case I did a bad job explaining it, the “core and task sdk” is not in
> the module name/import name, just in the file path.
> >
> > Anyone have other ideas?
> >
> >> On 7 Jul 2025, at 21:37, Buğra Öztürk <oz...@gmail.com> wrote:
> >>
> >> Thanks Ash! Looks cool! I like the structure. This will enable all the
> >> combinations and structure looks easy to grasp. No strong stance on the
> >> naming other than maybe it is a bit long with `and`, `core_ctl` could be
> >> shorter, since no import path is defined like that, we can give any name
> >> for sure.
> >>
> >> Best regards,
> >>
> >>> On Mon, 7 Jul 2025, 21:51 Jarek Potiuk, <ja...@potiuk.com> wrote:
> >>>
> >>> Looks good but I think we should find some better logical name for
> >>> core_and_sdk :)
> >>>
> >>> pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
> >>> <j_...@gmx.de.invalid> napisał:
> >>>
> >>>> Cool! Especially the "shared" folder with the ability to have
> >>>> N-combinations w/o exploding project repo root!
> >>>>
> >>>> On 07.07.25 14:43, Ash Berlin-Taylor wrote:
> >>>>> Oh, and all of this will be explain in shared/README.md
> >>>>>
> >>>>>> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org> wrote:
> >>>>>>
> >>>>>> Okay, so it seems we have agreement on the approach here, so I’ll
> >>>> continue with this, and on the dev call it was mentioned that
> >>>> “airflow-common” wasn’t a great name, so here is my proposal for the
> file
> >>>> structure;
> >>>>>> ```
> >>>>>> /
> >>>>>>   task-sdk/...
> >>>>>>   airflow-core/...
> >>>>>>   shared/
> >>>>>>     kuberenetes/
> >>>>>>       pyproject.toml
> >>>>>>       src/
> >>>>>>         airflow_kube/__init__.py
> >>>>>>     core-and-tasksdk/
> >>>>>>       pyproject.toml
> >>>>>>       src/
> >>>>>>         airflow_shared/__init__.py
> >>>>>> ```
> >>>>>>
> >>>>>> Things to note here: the “shared” folder has (the possibility) of
> >>>> having multiple different shared “libraries” in it, in this example I
> am
> >>>> supposing a hypothetical shared kuberenetes folder a world in which we
> >>>> split the KubePodOperator and the KubeExecutor in to two separate
> >>>> distributions (example only, not proposing we do that right now, and
> that
> >>>> will be a separate discussion)
> >>>>>> The other things to note here:
> >>>>>>
> >>>>>>
> >>>>>> - the folder name in shared aims to be “self-documenting”, hence the
> >>>> verbose “core-and-tasksdk” to say where the shared library is
> intended to
> >>>> be used.
> >>>>>> - the python module itself should almost always have an `airflow_`
> (or
> >>>> maybe `_airflow_`?) prefix so that it does not conflict with anything
> >>> else
> >>>> we might use. It won’t matter “in production” as those will be
> vendored
> >>> in
> >>>> to be imported as `airflow/_vendor/airflow_shared` etc, but avoiding
> >>>> conflicts at dev time with the Finder approach is a good safety
> measure.
> >>>>>> I will start making a real PR for this proposal now, but I’m open to
> >>>> feedback (either here, or in the PR when I open it)
> >>>>>> -ash
> >>>>>>
> >>>>>>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>>>>>
> >>>>>>> Yeah we have to try it and test - also building packages happens
> semi
> >>>>>>> frequently when you run `uv sync` (they use some kind of heuristics
> >>> to
> >>>>>>> decide when) and you can force it with `--reinstall` or
> `--refresh`.
> >>>>>>> Package build also happens every time when you run "ci-image build`
> >>>> now in
> >>>>>>> breeze so it seems like it will nicely integrate in our workflows.
> >>>>>>>
> >>>>>>> Looks really cool Ash.
> >>>>>>>
> >>>>>>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <as...@apache.org>
> >>>> wrote:
> >>>>>>>> It’s not just release time, but any time we build a package which
> >>>> happens
> >>>>>>>> on “every” CI run. The normal unit tests will use code from
> >>>>>>>> airflow-common/src/airflow_common; the kube tests which build an
> >>>> image will
> >>>>>>>> build the dists and vendor in the code from that commit.
> >>>>>>>>
> >>>>>>>> There is only a single copy of the shared code committed to the
> >>> repo,
> >>>> so
> >>>>>>>> there is never anything to synchronise.
> >>>>>>>>
> >>>>>>>>> On 4 Jul 2025, at 15:53, Amogh Desai <am...@gmail.com>
> >>>> wrote:
> >>>>>>>>> Thanks Ash.
> >>>>>>>>>
> >>>>>>>>> This is really cool and helpful that you were able to test both
> >>>> scenarios
> >>>>>>>>> -- repo checkout
> >>>>>>>>> and also installing from the vendored package and the resolution
> >>>> worked
> >>>>>>>>> fine too.
> >>>>>>>>>
> >>>>>>>>> I like this idea compared the to relative import one for few
> >>> reasons:
> >>>>>>>>> - It feels like it will take some time to adjust to the new
> coding
> >>>>>>>> standard
> >>>>>>>>> that we will lay
> >>>>>>>>> if we impose relative imports in the shared dist
> >>>>>>>>> - We can continue using repo wise absolute import standards, it
> is
> >>>> also
> >>>>>>>>> much easier for situations
> >>>>>>>>> when we do global search in IDE to find + replace, this could
> mean
> >>> a
> >>>>>>>> change
> >>>>>>>>> there
> >>>>>>>>> - The vendoring work is a proven and established paradigm across
> >>>> projects
> >>>>>>>>> and would
> >>>>>>>>> out of box give us the build tooling we need also
> >>>>>>>>>
> >>>>>>>>> Nothing too against the relative import but with the evidence
> >>>> provided
> >>>>>>>>> above, vendored approach
> >>>>>>>>> seems to only do us good.
> >>>>>>>>>
> >>>>>>>>> Regarding synchronizing it, release time should be fine as long
> as
> >>> we
> >>>>>>>> have
> >>>>>>>>> a good CI workflow to probably
> >>>>>>>>> catch such issues per PR if changes are made in shared dist?
> >>>> (precommit
> >>>>>>>>> would make it really slow i guess)
> >>>>>>>>>
> >>>>>>>>> If we can run our tests with vendored code we should be mostly
> >>>> covered.
> >>>>>>>>> Good effort all!
> >>>>>>>>>
> >>>>>>>>> Thanks & Regards,
> >>>>>>>>> Amogh Desai
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <
> ash@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>> Okay, I think I’ve got something that works and I’m happy with.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>
> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
> >>>>>>>>>> This produces the following from `uv build task-sdk`
> >>>>>>>>>> -
> >>>>>>>>>>
> >>>
> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
> >>>>>>>>>> -
> >>>>>>>>>>
> >>>
> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
> >>>>>>>>>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
> >>>>>>>>>>
> >>>>>>>>>> ```
> >>>>>>>>>> ❯ unzip -l
> >>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
> >>>>>>>> grep
> >>>>>>>>>> _vendor
> >>>>>>>>>>      50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
> >>>>>>>>>>    2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
> >>>>>>>>>>      28  02-02-2020 00:00
>  airflow/sdk/_vendor/airflow_common.pyi
> >>>>>>>>>>      18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
> >>>>>>>>>>     785  02-02-2020 00:00
> >>>>>>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
> >>>>>>>>>>   10628  02-02-2020 00:00
> >>>>>>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
> >>>>>>>>>> ```
> >>>>>>>>>>
> >>>>>>>>>> And similarly in the .tar.gz, so our “sdist” is complete too:
> >>>>>>>>>> ```
> >>>>>>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep
> _vendor
> >>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
> >>>>>>>>>>
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
> >>>>>>>>>>
> >>>>
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
> >>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
> >>>>>>>>>>
> >>>>>>>>>>
> >>>
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
> >>>
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
> >>>>>>>>>> ```
> >>>>>>>>>>
> >>>>>>>>>> The plugin works at build time by including/copying the libs
> >>>> specified
> >>>>>>>> in
> >>>>>>>>>> vendor.txt into place (and let `vendoring` take care of import
> >>>>>>>> rewrites.)
> >>>>>>>>>> For the imports to continue to work at “dev” time/from a repo
> >>>> checkout,
> >>>>>>>> I
> >>>>>>>>>> have added a import finder to `sys.meta_path`, and since it’s at
> >>> the
> >>>>>>>> end of
> >>>>>>>>>> the list it will only be used if the normal import can’t find
> >>>> things.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>
> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
> >>>>>>>>>> This doesn’t quite give us the same runtime effect “import
> >>>> rewriting”
> >>>>>>>>>> affect, as in this approach `airflow_common` is directly loaded
> >>>> (i.e.
> >>>>>>>>>> airflow.sdk._vendor.airflow_common and airflow_common exist in
> >>>>>>>>>> sys.modules), but it does work for everything that I was able to
> >>>> test..
> >>>>>>>>>> I tested it with the diff at the end of this message. My test
> >>>> ipython
> >>>>>>>>>> shell:
> >>>>>>>>>>
> >>>>>>>>>> ```
> >>>>>>>>>> In [1]: from airflow.sdk._vendor.airflow_common.timezone import
> >>> foo
> >>>>>>>>>> In [2]: foo
> >>>>>>>>>> Out[2]: 1
> >>>>>>>>>>
> >>>>>>>>>> In [3]: import airflow.sdk._vendor.airflow_common
> >>>>>>>>>>
> >>>>>>>>>> In [4]: import airflow.sdk._vendor.airflow_common.timezone
> >>>>>>>>>>
> >>>>>>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
> >>>>>>>>>> Out[5]:
> >>>>>>>>>>
> >>>
> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
> >>>>>>>>>> In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
> >>>>>>>>>> Out[6]:
> >>>>>>>>>>
> >>>
> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
> >>>>>>>>>> ```
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> And in an standalone environment with the SDK dist I built (it
> >>>> needed
> >>>>>>>> the
> >>>>>>>>>> matching airflow-core right now, but that is nothing to do with
> >>> this
> >>>>>>>>>> discussion):
> >>>>>>>>>>
> >>>>>>>>>> ```
> >>>>>>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
> >>>>>>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
> >>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
> >>>>>>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
> >>>>>>>>>> Type 'copyright', 'credits' or 'license' for more information
> >>>>>>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for
> >>> help.
> >>>>>>>>>> Tip: You can use `%hist` to view history, see the options with
> >>>>>>>> `%history?`
> >>>>>>>>>> In [1]: import airflow.sdk._vendor.airflow_common.timezone
> >>>>>>>>>>
> >>>>>>>>>> In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
> >>>>>>>>>> Out[2]:
> >>>>>>>>>>
> >>>
> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
> >>>>>>>>>> ``
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ```diff
> >>>>>>>>>> diff --git a/airflow-common/src/airflow_common/__init__.py
> >>>>>>>>>> b/airflow-common/src/airflow_common/__init__.py
> >>>>>>>>>> index 13a83393a9..927b7c6b61 100644
> >>>>>>>>>> --- a/airflow-common/src/airflow_common/__init__.py
> >>>>>>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
> >>>>>>>>>> @@ -14,3 +14,5 @@
> >>>>>>>>>> # KIND, either express or implied.  See the License for the
> >>>>>>>>>> # specific language governing permissions and limitations
> >>>>>>>>>> # under the License.
> >>>>>>>>>> +
> >>>>>>>>>> +foo = 1
> >>>>>>>>>> diff --git a/airflow-common/src/airflow_common/timezone.py
> >>>>>>>>>> b/airflow-common/src/airflow_common/timezone.py
> >>>>>>>>>> index 340b924c66..58384ef20f 100644
> >>>>>>>>>> --- a/airflow-common/src/airflow_common/timezone.py
> >>>>>>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
> >>>>>>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
> >>>>>>>>>> version.parse(metadata.version("pendulum")).major == 3
> >>>>>>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
> >>>>>>>>>> utc = pendulum.UTC
> >>>>>>>>>>
> >>>>>>>>>> +
> >>>>>>>>>> +from airflow_common import foo
> >>>>>>>>>> +
> >>>>>>>>>> TIMEZONE: Timezone
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ```
> >>>>>>>>>>
> >>>>>>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >>>>>>>>>>> I think both approaches are doable:
> >>>>>>>>>>>
> >>>>>>>>>>> 1) -> We can very easily prevent bad imports by pre-commit when
> >>>>>>>> importing
> >>>>>>>>>>> from different distributions and make sure we are only doing
> >>>> relative
> >>>>>>>>>>> imports in the shared modules. We are doing plenty of this
> >>>> already. And
> >>>>>>>>>> yes
> >>>>>>>>>>> it would require relative links we currently do not allow.
> >>>>>>>>>>>
> >>>>>>>>>>> 2) -> has one disadvantage that someone at some point in time
> >>> will
> >>>> have
> >>>>>>>>>> to
> >>>>>>>>>>> decide to synchronize this and if it happens just before
> release
> >>>> (I bet
> >>>>>>>>>>> this is going to happen) this will lead to solving problems
> that
> >>>> would
> >>>>>>>>>>> normally be solved during PR when you make a change (i.e.
> >>> symbolic
> >>>> link
> >>>>>>>>>> has
> >>>>>>>>>>> the advantage that whoever modifies shared code will be
> >>> immediately
> >>>>>>>>>>> notified in their PR - that they broke something because either
> >>>> static
> >>>>>>>>>>> checks or mypy or tests fail.
> >>>>>>>>>>>
> >>>>>>>>>>> Ash, do you have an idea of a process (who and when) does the
> >>>>>>>>>>> synchronisation in case of vendoring? Maybe we could solve it
> if
> >>>> it is
> >>>>>>>>>> done
> >>>>>>>>>>> more frequently and with some regularity? We could potentially
> >>>> force
> >>>>>>>>>>> re-vendoring at PR time as well any time shared code changes
> (and
> >>>>>>>> prevent
> >>>>>>>>>>> it by pre-commit. And I can't think of some place (other than
> >>>> releases)
> >>>>>>>>>> in
> >>>>>>>>>>> our development workflow and that seems to be a bit too late as
> >>>> puts an
> >>>>>>>>>>> extra effort on fixing potential incompatibilities introduced
> on
> >>>>>>>> release
> >>>>>>>>>>> manager and delays the release. WDYT?
> >>>>>>>>>>>
> >>>>>>>>>>> Re: relative links. I think for a shared library we could
> >>>> potentially
> >>>>>>>>>> relax
> >>>>>>>>>>> this and allow them (and actually disallow absolute links in
> the
> >>>> pieces
> >>>>>>>>>> of
> >>>>>>>>>>> code that are shared - again, by pre-commit). As I recall, the
> >>> only
> >>>>>>>>>> reason
> >>>>>>>>>>> we forbade the relative links is because of how we are (or
> maybe
> >>>> were)
> >>>>>>>>>>> doing DAG parsing and failures resulting from it. So we decided
> >>> to
> >>>> just
> >>>>>>>>>> not
> >>>>>>>>>>> allow it to keep consistency. The way how Dag parsing works is
> >>> that
> >>>>>>>> when
> >>>>>>>>>>> you are using importlib to read the Dag from a file, the
> relative
> >>>>>>>> imports
> >>>>>>>>>>> do not work as it does not know what they should be relative
> to.
> >>>> But if
> >>>>>>>>>>> relative import is done from an imported package, it should be
> no
> >>>>>>>>>> problem,
> >>>>>>>>>>> I think - otherwise our Dags would not be able to import any
> >>>> library
> >>>>>>>> that
> >>>>>>>>>>> uses relative imports.
> >>>>>>>>>>>
> >>>>>>>>>>> Of course consistency might be the reason why we do not want to
> >>>>>>>> introduce
> >>>>>>>>>>> relative imports. I don't see it as an issue if it is guarded
> by
> >>>>>>>>>> pre-commit
> >>>>>>>>>>> though.
> >>>>>>>>>>>
> >>>>>>>>>>> J.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> J.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
> >>>> ash@apache.org>
> >>>>>>>>>>> napisał:
> >>>>>>>>>>>
> >>>>>>>>>>>> Oh yes, symlinks will work, with one big caveat: It does mean
> >>> you
> >>>>>>>> can’t
> >>>>>>>>>>>> use absolute imports in one common module to another.
> >>>>>>>>>>>>
> >>>>>>>>>>>> For example
> >>>>>>>>>>>>
> >>>
> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
> >>>>>>>>>>>> where we have
> >>>>>>>>>>>>
> >>>>>>>>>>>> ```
> >>>>>>>>>>>> from airflow.utils.module_loading import import_string
> >>>>>>>>>>>> ```
> >>>>>>>>>>>>
> >>>>>>>>>>>> if we want to move serve_logs into this common lib that is
> then
> >>>>>>>>>> symlinked
> >>>>>>>>>>>> then we wouldn’t be able to have `from
> >>>> airflow_common.module_loading
> >>>>>>>>>> import
> >>>>>>>>>>>> import_string`.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I can think of two possible solutions here.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1) is to allow/require relative imports in this shared lib, so
> >>>> `from
> >>>>>>>>>>>> .module_loading import import_string`
> >>>>>>>>>>>> 2) is to use `vendoring`[1] (from the pip maintainers) which
> >>> will
> >>>>>>>> handle
> >>>>>>>>>>>> import-rewriting for us.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I’d entirely forgot that symlinks in repos was a thing, so I
> >>>> prepared
> >>>>>>>> a
> >>>>>>>>>>>> minimal POC/demo of what vendoring approach could look like
> here
> >>>>>>>>>>>>
> >>>
> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
> >>>>>>>>>>>> Now personally I am more than happy with relative imports, but
> >>>>>>>> generally
> >>>>>>>>>>>> as a project we have avoided them, so I think that limits what
> >>> we
> >>>>>>>> could
> >>>>>>>>>> do
> >>>>>>>>>>>> with a symlink based approach.
> >>>>>>>>>>>>
> >>>>>>>>>>>> -ash
> >>>>>>>>>>>>
> >>>>>>>>>>>> [1] https://github.com/pradyunsg/vendoring
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
> >>>>>>>> gopidesupavan@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>> Thanks Ash
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yes agree option 2 would be preferred for me. Making sure we
> >>>> have all
> >>>>>>>>>> the
> >>>>>>>>>>>>> gaurdriles to protect any unwanted behaviour in code sharing
> >>> and
> >>>>>>>>>>>> executing
> >>>>>>>>>>>>> right of tests between the packages.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Agree with others, option 2 would be
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
> >>>>>>>> amoghdesai.oss@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for starting this discussion, Ash.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I would prefer option 2 here with proper tooling to handle
> the
> >>>> code
> >>>>>>>>>>>>>> duplication at *release* time.
> >>>>>>>>>>>>>> It is best to have a dist that has all it needs in itself.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Option 1 could very quickly get out of hand and if we decide
> >>> to
> >>>>>>>>>> separate
> >>>>>>>>>>>>>> triggerer /
> >>>>>>>>>>>>>> dag processor / config etc etc as separate packages, back
> >>>> compat is
> >>>>>>>>>>>> going
> >>>>>>>>>>>>>> to be a nightmare
> >>>>>>>>>>>>>> and will bite us harder than we anticipate.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks & Regards,
> >>>>>>>>>>>>>> Amogh Desai
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <
> >>> kaxilnaik@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>> I prefer Option 2 as well to avoid matrix of dependencies
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
> >>>>>>>>>> <j_scheffler@gmx.de.invalid
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I'd also rather prefer option 2 - reason here is it is
> >>> rather
> >>>>>>>>>>>> pragmatic
> >>>>>>>>>>>>>>>> and we no not need to cut another package and have less
> >>>> package
> >>>>>>>>>> counts
> >>>>>>>>>>>>>>>> and dependencies.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I remember some time ago I was checking (together with
> >>> Jarek,
> >>>> I am
> >>>>>>>>>> not
> >>>>>>>>>>>>>>>> sure anymore...) if the usage of symlinks would be
> possible.
> >>>> To
> >>>>>>>> keep
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> source in one package but "symlink" it into another. If
> then
> >>>> at
> >>>>>>>>>> point
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>> packaging/release the files are materialized we have 1 set
> >>> of
> >>>>>>>> code.
> >>>>>>>>>>>>>>>> Otherwise if not possible still the redundancy could be
> >>>> solved by
> >>>>>>>> a
> >>>>>>>>>>>>>>>> pre-commit hook - and in Git the files are de-duplicated
> >>>> anyway
> >>>>>>>>>> based
> >>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>> content hash, so this does not hurt.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
> >>>>>>>>>>>>>>>>> I support option 2 with proper automation & CI - the
> >>>> reasonings
> >>>>>>>>>>>>>> you've
> >>>>>>>>>>>>>>>>> shown for that make sense to me.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Shahar
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <
> >>>> ash@apache.org
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> As we work on finishing off the code-level separation of
> >>>> Task
> >>>>>>>> SDK
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> Core
> >>>>>>>>>>>>>>>>>> (scheduler etc) we have come across some situations
> where
> >>> we
> >>>>>>>> would
> >>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> share code between these.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> However it’s not as straight forward of “just put it in
> a
> >>>> common
> >>>>>>>>>>>>>> dist
> >>>>>>>>>>>>>>>> they
> >>>>>>>>>>>>>>>>>> both depend upon” because one of the goals of the Task
> SDK
> >>>>>>>>>>>>>> separation
> >>>>>>>>>>>>>>>> was
> >>>>>>>>>>>>>>>>>> to have 100% complete version independence between the
> >>> two,
> >>>>>>>>>> ideally
> >>>>>>>>>>>>>>>> even if
> >>>>>>>>>>>>>>>>>> they are built into the same image and venv. Most of the
> >>>> reason
> >>>>>>>>>> why
> >>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> isn’t straight forward comes down to backwards
> >>>> compatibility -
> >>>>>>>> if
> >>>>>>>>>> we
> >>>>>>>>>>>>>>>> make
> >>>>>>>>>>>>>>>>>> an change to the common/shared distribution
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We’ve listed the options we have thought about in
> >>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545 (but
> that
> >>>> covers
> >>>>>>>>>>>>>> some
> >>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>> things that I don’t want to get in to in this discussion
> >>>> such as
> >>>>>>>>>>>>>>>> possibly
> >>>>>>>>>>>>>>>>>> separating operators and executors out of a single
> >>> provider
> >>>>>>>> dist.)
> >>>>>>>>>>>>>>>>>> To give a concrete example of some code I would like to
> >>>> share
> >>>
> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
> >>>>>>>>>>>>>>>>>> — logging config. Another thing we will want to share
> will
> >>>> be
> >>>>>>>> the
> >>>>>>>>>>>>>>>>>> AirflowConfigParser class from airflow.configuration
> (but
> >>>>>>>> notably:
> >>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> parser class, _not_ the default config values, again,
> lets
> >>>> not
> >>>>>>>>>> dwell
> >>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> specifics of that)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> So to bring the options listed in the issue here for
> >>>> discussion,
> >>>>>>>>>>>>>>> broadly
> >>>>>>>>>>>>>>>>>> speaking there are two high-level approaches:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 1. A single shared distribution
> >>>>>>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The advantage of Approach 1 is that we only have the
> code
> >>>> in one
> >>>>>>>>>>>>>>> place.
> >>>>>>>>>>>>>>>>>> However for me, at least in this specific case of
> Logging
> >>>> config
> >>>>>>>>>> or
> >>>>>>>>>>>>>>>>>> AirflowConfigParser class is that backwards
> compatibility
> >>> is
> >>>>>>>> much
> >>>>>>>>>>>>>> much
> >>>>>>>>>>>>>>>>>> harder.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The main advantage of Approach 2 is the the code is
> >>> released
> >>>>>>>>>>>>>>>> with/embedded
> >>>>>>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would contain
> >>> the
> >>>>>>>> right
> >>>>>>>>>>>>>>>> version
> >>>>>>>>>>>>>>>>>> of the logging config and ConfigParser etc). The
> downside
> >>> is
> >>>>>>>> that
> >>>>>>>>>>>>>>> either
> >>>>>>>>>>>>>>>>>> the code will need to be duplicated in the repo, or
> better
> >>>> yet
> >>>>>>>> it
> >>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>> live in a single place in the repo, but some tooling
> (TBD)
> >>>> will
> >>>>>>>>>>>>>>>>>> automatically handle the duplication, either at commit
> >>>> time, or
> >>>>>>>> my
> >>>>>>>>>>>>>>>>>> preference, at release time.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> For this kind of shared “utility” code I am very
> strongly
> >>>>>>>> leaning
> >>>>>>>>>>>>>>>> towards
> >>>>>>>>>>>>>>>>>> option 2 with automation, as otherwise I think the
> >>> backwards
> >>>>>>>>>>>>>>>> compatibility
> >>>>>>>>>>>>>>>>>> requirements would make it unworkable (very quickly over
> >>>> time
> >>>>>>>> the
> >>>>>>>>>>>>>>>>>> combinations we would have to test would just be
> >>>> unreasonable)
> >>>>>>>>>> and I
> >>>>>>>>>>>>>>>> don’t
> >>>>>>>>>>>>>>>>>> feel confident we can have things as stable as we need
> to
> >>>> really
> >>>>>>>>>>>>>>> deliver
> >>>>>>>>>>>>>>>>>> the version separation/independency I want to delivery
> >>> with
> >>>>>>>>>> AIP-72.
> >>>>>>>>>>>>>>>>>> So unless someone feels very strongly about this, I will
> >>>> come up
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>> draft PR for further discussion that will implement code
> >>>> sharing
> >>>>>>>>>> via
> >>>>>>>>>>>>>>>>>> “vendoring” it at build time. I have an idea of how I
> can
> >>>>>>>> achieve
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>>> we have a single version in the repo and it’ll work
> there,
> >>>> but
> >>>>>>>> at
> >>>>>>>>>>>>>>>> runtime
> >>>>>>>>>>>>>>>>>> we vendor it in to the shipped dist so it lives at
> >>> something
> >>>>>>>> like
> >>>>>>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> In terms of repo layout, this likely means we would end
> up
> >>>> with:
> >>>>>>>>>>>>>>>>>> airflow-core/pyproject.toml
> >>>>>>>>>>>>>>>>>> airflow-core/src/
> >>>>>>>>>>>>>>>>>> airflow-core/tests/
> >>>>>>>>>>>>>>>>>> task-sdk/pyproject.toml
> >>>>>>>>>>>>>>>>>> task-sdk/src/
> >>>>>>>>>>>>>>>>>> task-sdk/tests/
> >>>>>>>>>>>>>>>>>> airflow-common/src
> >>>>>>>>>>>>>>>>>> airflow-common/tests/
> >>>>>>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps
> would
> >>>> be
> >>>>>>>>>>>>>> included
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> the downstream projects. TBD.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thoughts and feedback welcomed.
> >>>> ---------------------------------------------------------------------
> >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> dev-unsubscribe@airflow.apache.org
> >>>>>>>>>>>>>>>> For additional commands, e-mail:
> >>> dev-help@airflow.apache.org
> >>>>>>>>>>>>>>>>
> >>>>>>>>
> >>> ---------------------------------------------------------------------
> >>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>
> >>>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Jens Scheffler <j_...@gmx.de.INVALID>.
I like "core_and_task_sdk the same like core-and-task-sdk - I have no 
problem and it is a path only.

if we get to "dag-parser-scheduler-task-sdk-and-triggerer" which is a 
bit bulky we then should name it "all-not-api-server" :-D

On 07.07.25 22:57, Ash Berlin-Taylor wrote:
> In case I did a bad job explaining it, the “core and task sdk” is not in the module name/import name, just in the file path.
>
> Anyone have other ideas?
>
>> On 7 Jul 2025, at 21:37, Buğra Öztürk <oz...@gmail.com> wrote:
>>
>> Thanks Ash! Looks cool! I like the structure. This will enable all the
>> combinations and structure looks easy to grasp. No strong stance on the
>> naming other than maybe it is a bit long with `and`, `core_ctl` could be
>> shorter, since no import path is defined like that, we can give any name
>> for sure.
>>
>> Best regards,
>>
>>> On Mon, 7 Jul 2025, 21:51 Jarek Potiuk, <ja...@potiuk.com> wrote:
>>>
>>> Looks good but I think we should find some better logical name for
>>> core_and_sdk :)
>>>
>>> pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
>>> <j_...@gmx.de.invalid> napisał:
>>>
>>>> Cool! Especially the "shared" folder with the ability to have
>>>> N-combinations w/o exploding project repo root!
>>>>
>>>> On 07.07.25 14:43, Ash Berlin-Taylor wrote:
>>>>> Oh, and all of this will be explain in shared/README.md
>>>>>
>>>>>> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>>>>
>>>>>> Okay, so it seems we have agreement on the approach here, so I’ll
>>>> continue with this, and on the dev call it was mentioned that
>>>> “airflow-common” wasn’t a great name, so here is my proposal for the file
>>>> structure;
>>>>>> ```
>>>>>> /
>>>>>>   task-sdk/...
>>>>>>   airflow-core/...
>>>>>>   shared/
>>>>>>     kuberenetes/
>>>>>>       pyproject.toml
>>>>>>       src/
>>>>>>         airflow_kube/__init__.py
>>>>>>     core-and-tasksdk/
>>>>>>       pyproject.toml
>>>>>>       src/
>>>>>>         airflow_shared/__init__.py
>>>>>> ```
>>>>>>
>>>>>> Things to note here: the “shared” folder has (the possibility) of
>>>> having multiple different shared “libraries” in it, in this example I am
>>>> supposing a hypothetical shared kuberenetes folder a world in which we
>>>> split the KubePodOperator and the KubeExecutor in to two separate
>>>> distributions (example only, not proposing we do that right now, and that
>>>> will be a separate discussion)
>>>>>> The other things to note here:
>>>>>>
>>>>>>
>>>>>> - the folder name in shared aims to be “self-documenting”, hence the
>>>> verbose “core-and-tasksdk” to say where the shared library is intended to
>>>> be used.
>>>>>> - the python module itself should almost always have an `airflow_` (or
>>>> maybe `_airflow_`?) prefix so that it does not conflict with anything
>>> else
>>>> we might use. It won’t matter “in production” as those will be vendored
>>> in
>>>> to be imported as `airflow/_vendor/airflow_shared` etc, but avoiding
>>>> conflicts at dev time with the Finder approach is a good safety measure.
>>>>>> I will start making a real PR for this proposal now, but I’m open to
>>>> feedback (either here, or in the PR when I open it)
>>>>>> -ash
>>>>>>
>>>>>>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>>
>>>>>>> Yeah we have to try it and test - also building packages happens semi
>>>>>>> frequently when you run `uv sync` (they use some kind of heuristics
>>> to
>>>>>>> decide when) and you can force it with `--reinstall` or `--refresh`.
>>>>>>> Package build also happens every time when you run "ci-image build`
>>>> now in
>>>>>>> breeze so it seems like it will nicely integrate in our workflows.
>>>>>>>
>>>>>>> Looks really cool Ash.
>>>>>>>
>>>>>>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <as...@apache.org>
>>>> wrote:
>>>>>>>> It’s not just release time, but any time we build a package which
>>>> happens
>>>>>>>> on “every” CI run. The normal unit tests will use code from
>>>>>>>> airflow-common/src/airflow_common; the kube tests which build an
>>>> image will
>>>>>>>> build the dists and vendor in the code from that commit.
>>>>>>>>
>>>>>>>> There is only a single copy of the shared code committed to the
>>> repo,
>>>> so
>>>>>>>> there is never anything to synchronise.
>>>>>>>>
>>>>>>>>> On 4 Jul 2025, at 15:53, Amogh Desai <am...@gmail.com>
>>>> wrote:
>>>>>>>>> Thanks Ash.
>>>>>>>>>
>>>>>>>>> This is really cool and helpful that you were able to test both
>>>> scenarios
>>>>>>>>> -- repo checkout
>>>>>>>>> and also installing from the vendored package and the resolution
>>>> worked
>>>>>>>>> fine too.
>>>>>>>>>
>>>>>>>>> I like this idea compared the to relative import one for few
>>> reasons:
>>>>>>>>> - It feels like it will take some time to adjust to the new coding
>>>>>>>> standard
>>>>>>>>> that we will lay
>>>>>>>>> if we impose relative imports in the shared dist
>>>>>>>>> - We can continue using repo wise absolute import standards, it is
>>>> also
>>>>>>>>> much easier for situations
>>>>>>>>> when we do global search in IDE to find + replace, this could mean
>>> a
>>>>>>>> change
>>>>>>>>> there
>>>>>>>>> - The vendoring work is a proven and established paradigm across
>>>> projects
>>>>>>>>> and would
>>>>>>>>> out of box give us the build tooling we need also
>>>>>>>>>
>>>>>>>>> Nothing too against the relative import but with the evidence
>>>> provided
>>>>>>>>> above, vendored approach
>>>>>>>>> seems to only do us good.
>>>>>>>>>
>>>>>>>>> Regarding synchronizing it, release time should be fine as long as
>>> we
>>>>>>>> have
>>>>>>>>> a good CI workflow to probably
>>>>>>>>> catch such issues per PR if changes are made in shared dist?
>>>> (precommit
>>>>>>>>> would make it really slow i guess)
>>>>>>>>>
>>>>>>>>> If we can run our tests with vendored code we should be mostly
>>>> covered.
>>>>>>>>> Good effort all!
>>>>>>>>>
>>>>>>>>> Thanks & Regards,
>>>>>>>>> Amogh Desai
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <as...@apache.org>
>>>>>>>> wrote:
>>>>>>>>>> Okay, I think I’ve got something that works and I’m happy with.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>>>>>>>>>> This produces the following from `uv build task-sdk`
>>>>>>>>>> -
>>>>>>>>>>
>>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>>>>>>> -
>>>>>>>>>>
>>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>>>>>>>>>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
>>>>>>>>>>
>>>>>>>>>> ```
>>>>>>>>>> ❯ unzip -l
>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
>>>>>>>> grep
>>>>>>>>>> _vendor
>>>>>>>>>>      50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
>>>>>>>>>>    2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
>>>>>>>>>>      28  02-02-2020 00:00   airflow/sdk/_vendor/airflow_common.pyi
>>>>>>>>>>      18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
>>>>>>>>>>     785  02-02-2020 00:00
>>>>>>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>>>>>>   10628  02-02-2020 00:00
>>>>>>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>>>>>> ```
>>>>>>>>>>
>>>>>>>>>> And similarly in the .tar.gz, so our “sdist” is complete too:
>>>>>>>>>> ```
>>>>>>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep _vendor
>>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>>>>>>>>>>
>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>>>>>>>>>>
>>>>>>>>>>
>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>>>>>> ```
>>>>>>>>>>
>>>>>>>>>> The plugin works at build time by including/copying the libs
>>>> specified
>>>>>>>> in
>>>>>>>>>> vendor.txt into place (and let `vendoring` take care of import
>>>>>>>> rewrites.)
>>>>>>>>>> For the imports to continue to work at “dev” time/from a repo
>>>> checkout,
>>>>>>>> I
>>>>>>>>>> have added a import finder to `sys.meta_path`, and since it’s at
>>> the
>>>>>>>> end of
>>>>>>>>>> the list it will only be used if the normal import can’t find
>>>> things.
>>>>>>>>>>
>>>>>>>>>>
>>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>>>>>>>>>> This doesn’t quite give us the same runtime effect “import
>>>> rewriting”
>>>>>>>>>> affect, as in this approach `airflow_common` is directly loaded
>>>> (i.e.
>>>>>>>>>> airflow.sdk._vendor.airflow_common and airflow_common exist in
>>>>>>>>>> sys.modules), but it does work for everything that I was able to
>>>> test..
>>>>>>>>>> I tested it with the diff at the end of this message. My test
>>>> ipython
>>>>>>>>>> shell:
>>>>>>>>>>
>>>>>>>>>> ```
>>>>>>>>>> In [1]: from airflow.sdk._vendor.airflow_common.timezone import
>>> foo
>>>>>>>>>> In [2]: foo
>>>>>>>>>> Out[2]: 1
>>>>>>>>>>
>>>>>>>>>> In [3]: import airflow.sdk._vendor.airflow_common
>>>>>>>>>>
>>>>>>>>>> In [4]: import airflow.sdk._vendor.airflow_common.timezone
>>>>>>>>>>
>>>>>>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>>>>>>>>>> Out[5]:
>>>>>>>>>>
>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>>>>>>>>>> In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>>>>>> Out[6]:
>>>>>>>>>>
>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>>>>>>>>>> ```
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> And in an standalone environment with the SDK dist I built (it
>>>> needed
>>>>>>>> the
>>>>>>>>>> matching airflow-core right now, but that is nothing to do with
>>> this
>>>>>>>>>> discussion):
>>>>>>>>>>
>>>>>>>>>> ```
>>>>>>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>>>>>>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
>>>>>>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
>>>>>>>>>> Type 'copyright', 'credits' or 'license' for more information
>>>>>>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for
>>> help.
>>>>>>>>>> Tip: You can use `%hist` to view history, see the options with
>>>>>>>> `%history?`
>>>>>>>>>> In [1]: import airflow.sdk._vendor.airflow_common.timezone
>>>>>>>>>>
>>>>>>>>>> In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>>>>>> Out[2]:
>>>>>>>>>>
>>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>>>>>>>>>> ``
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ```diff
>>>>>>>>>> diff --git a/airflow-common/src/airflow_common/__init__.py
>>>>>>>>>> b/airflow-common/src/airflow_common/__init__.py
>>>>>>>>>> index 13a83393a9..927b7c6b61 100644
>>>>>>>>>> --- a/airflow-common/src/airflow_common/__init__.py
>>>>>>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
>>>>>>>>>> @@ -14,3 +14,5 @@
>>>>>>>>>> # KIND, either express or implied.  See the License for the
>>>>>>>>>> # specific language governing permissions and limitations
>>>>>>>>>> # under the License.
>>>>>>>>>> +
>>>>>>>>>> +foo = 1
>>>>>>>>>> diff --git a/airflow-common/src/airflow_common/timezone.py
>>>>>>>>>> b/airflow-common/src/airflow_common/timezone.py
>>>>>>>>>> index 340b924c66..58384ef20f 100644
>>>>>>>>>> --- a/airflow-common/src/airflow_common/timezone.py
>>>>>>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
>>>>>>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>>>>>>>>>> version.parse(metadata.version("pendulum")).major == 3
>>>>>>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
>>>>>>>>>> utc = pendulum.UTC
>>>>>>>>>>
>>>>>>>>>> +
>>>>>>>>>> +from airflow_common import foo
>>>>>>>>>> +
>>>>>>>>>> TIMEZONE: Timezone
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ```
>>>>>>>>>>
>>>>>>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>>>>>> I think both approaches are doable:
>>>>>>>>>>>
>>>>>>>>>>> 1) -> We can very easily prevent bad imports by pre-commit when
>>>>>>>> importing
>>>>>>>>>>> from different distributions and make sure we are only doing
>>>> relative
>>>>>>>>>>> imports in the shared modules. We are doing plenty of this
>>>> already. And
>>>>>>>>>> yes
>>>>>>>>>>> it would require relative links we currently do not allow.
>>>>>>>>>>>
>>>>>>>>>>> 2) -> has one disadvantage that someone at some point in time
>>> will
>>>> have
>>>>>>>>>> to
>>>>>>>>>>> decide to synchronize this and if it happens just before release
>>>> (I bet
>>>>>>>>>>> this is going to happen) this will lead to solving problems that
>>>> would
>>>>>>>>>>> normally be solved during PR when you make a change (i.e.
>>> symbolic
>>>> link
>>>>>>>>>> has
>>>>>>>>>>> the advantage that whoever modifies shared code will be
>>> immediately
>>>>>>>>>>> notified in their PR - that they broke something because either
>>>> static
>>>>>>>>>>> checks or mypy or tests fail.
>>>>>>>>>>>
>>>>>>>>>>> Ash, do you have an idea of a process (who and when) does the
>>>>>>>>>>> synchronisation in case of vendoring? Maybe we could solve it if
>>>> it is
>>>>>>>>>> done
>>>>>>>>>>> more frequently and with some regularity? We could potentially
>>>> force
>>>>>>>>>>> re-vendoring at PR time as well any time shared code changes (and
>>>>>>>> prevent
>>>>>>>>>>> it by pre-commit. And I can't think of some place (other than
>>>> releases)
>>>>>>>>>> in
>>>>>>>>>>> our development workflow and that seems to be a bit too late as
>>>> puts an
>>>>>>>>>>> extra effort on fixing potential incompatibilities introduced on
>>>>>>>> release
>>>>>>>>>>> manager and delays the release. WDYT?
>>>>>>>>>>>
>>>>>>>>>>> Re: relative links. I think for a shared library we could
>>>> potentially
>>>>>>>>>> relax
>>>>>>>>>>> this and allow them (and actually disallow absolute links in the
>>>> pieces
>>>>>>>>>> of
>>>>>>>>>>> code that are shared - again, by pre-commit). As I recall, the
>>> only
>>>>>>>>>> reason
>>>>>>>>>>> we forbade the relative links is because of how we are (or maybe
>>>> were)
>>>>>>>>>>> doing DAG parsing and failures resulting from it. So we decided
>>> to
>>>> just
>>>>>>>>>> not
>>>>>>>>>>> allow it to keep consistency. The way how Dag parsing works is
>>> that
>>>>>>>> when
>>>>>>>>>>> you are using importlib to read the Dag from a file, the relative
>>>>>>>> imports
>>>>>>>>>>> do not work as it does not know what they should be relative to.
>>>> But if
>>>>>>>>>>> relative import is done from an imported package, it should be no
>>>>>>>>>> problem,
>>>>>>>>>>> I think - otherwise our Dags would not be able to import any
>>>> library
>>>>>>>> that
>>>>>>>>>>> uses relative imports.
>>>>>>>>>>>
>>>>>>>>>>> Of course consistency might be the reason why we do not want to
>>>>>>>> introduce
>>>>>>>>>>> relative imports. I don't see it as an issue if it is guarded by
>>>>>>>>>> pre-commit
>>>>>>>>>>> though.
>>>>>>>>>>>
>>>>>>>>>>> J.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> J.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
>>>> ash@apache.org>
>>>>>>>>>>> napisał:
>>>>>>>>>>>
>>>>>>>>>>>> Oh yes, symlinks will work, with one big caveat: It does mean
>>> you
>>>>>>>> can’t
>>>>>>>>>>>> use absolute imports in one common module to another.
>>>>>>>>>>>>
>>>>>>>>>>>> For example
>>>>>>>>>>>>
>>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>>>>>>>>>>>> where we have
>>>>>>>>>>>>
>>>>>>>>>>>> ```
>>>>>>>>>>>> from airflow.utils.module_loading import import_string
>>>>>>>>>>>> ```
>>>>>>>>>>>>
>>>>>>>>>>>> if we want to move serve_logs into this common lib that is then
>>>>>>>>>> symlinked
>>>>>>>>>>>> then we wouldn’t be able to have `from
>>>> airflow_common.module_loading
>>>>>>>>>> import
>>>>>>>>>>>> import_string`.
>>>>>>>>>>>>
>>>>>>>>>>>> I can think of two possible solutions here.
>>>>>>>>>>>>
>>>>>>>>>>>> 1) is to allow/require relative imports in this shared lib, so
>>>> `from
>>>>>>>>>>>> .module_loading import import_string`
>>>>>>>>>>>> 2) is to use `vendoring`[1] (from the pip maintainers) which
>>> will
>>>>>>>> handle
>>>>>>>>>>>> import-rewriting for us.
>>>>>>>>>>>>
>>>>>>>>>>>> I’d entirely forgot that symlinks in repos was a thing, so I
>>>> prepared
>>>>>>>> a
>>>>>>>>>>>> minimal POC/demo of what vendoring approach could look like here
>>>>>>>>>>>>
>>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>>>>>>>>>>>> Now personally I am more than happy with relative imports, but
>>>>>>>> generally
>>>>>>>>>>>> as a project we have avoided them, so I think that limits what
>>> we
>>>>>>>> could
>>>>>>>>>> do
>>>>>>>>>>>> with a symlink based approach.
>>>>>>>>>>>>
>>>>>>>>>>>> -ash
>>>>>>>>>>>>
>>>>>>>>>>>> [1] https://github.com/pradyunsg/vendoring
>>>>>>>>>>>>
>>>>>>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
>>>>>>>> gopidesupavan@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> Thanks Ash
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes agree option 2 would be preferred for me. Making sure we
>>>> have all
>>>>>>>>>> the
>>>>>>>>>>>>> gaurdriles to protect any unwanted behaviour in code sharing
>>> and
>>>>>>>>>>>> executing
>>>>>>>>>>>>> right of tests between the packages.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Agree with others, option 2 would be
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
>>>>>>>> amoghdesai.oss@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for starting this discussion, Ash.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I would prefer option 2 here with proper tooling to handle the
>>>> code
>>>>>>>>>>>>>> duplication at *release* time.
>>>>>>>>>>>>>> It is best to have a dist that has all it needs in itself.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Option 1 could very quickly get out of hand and if we decide
>>> to
>>>>>>>>>> separate
>>>>>>>>>>>>>> triggerer /
>>>>>>>>>>>>>> dag processor / config etc etc as separate packages, back
>>>> compat is
>>>>>>>>>>>> going
>>>>>>>>>>>>>> to be a nightmare
>>>>>>>>>>>>>> and will bite us harder than we anticipate.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>>> Amogh Desai
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <
>>> kaxilnaik@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> I prefer Option 2 as well to avoid matrix of dependencies
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>>>>>>>>>> <j_scheffler@gmx.de.invalid
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd also rather prefer option 2 - reason here is it is
>>> rather
>>>>>>>>>>>> pragmatic
>>>>>>>>>>>>>>>> and we no not need to cut another package and have less
>>>> package
>>>>>>>>>> counts
>>>>>>>>>>>>>>>> and dependencies.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I remember some time ago I was checking (together with
>>> Jarek,
>>>> I am
>>>>>>>>>> not
>>>>>>>>>>>>>>>> sure anymore...) if the usage of symlinks would be possible.
>>>> To
>>>>>>>> keep
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> source in one package but "symlink" it into another. If then
>>>> at
>>>>>>>>>> point
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> packaging/release the files are materialized we have 1 set
>>> of
>>>>>>>> code.
>>>>>>>>>>>>>>>> Otherwise if not possible still the redundancy could be
>>>> solved by
>>>>>>>> a
>>>>>>>>>>>>>>>> pre-commit hook - and in Git the files are de-duplicated
>>>> anyway
>>>>>>>>>> based
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> content hash, so this does not hurt.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>>>>>>>>>>>>>>> I support option 2 with proper automation & CI - the
>>>> reasonings
>>>>>>>>>>>>>> you've
>>>>>>>>>>>>>>>>> shown for that make sense to me.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Shahar
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <
>>>> ash@apache.org
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> As we work on finishing off the code-level separation of
>>>> Task
>>>>>>>> SDK
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> Core
>>>>>>>>>>>>>>>>>> (scheduler etc) we have come across some situations where
>>> we
>>>>>>>> would
>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> share code between these.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> However it’s not as straight forward of “just put it in a
>>>> common
>>>>>>>>>>>>>> dist
>>>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>>>> both depend upon” because one of the goals of the Task SDK
>>>>>>>>>>>>>> separation
>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>> to have 100% complete version independence between the
>>> two,
>>>>>>>>>> ideally
>>>>>>>>>>>>>>>> even if
>>>>>>>>>>>>>>>>>> they are built into the same image and venv. Most of the
>>>> reason
>>>>>>>>>> why
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> isn’t straight forward comes down to backwards
>>>> compatibility -
>>>>>>>> if
>>>>>>>>>> we
>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>>> an change to the common/shared distribution
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We’ve listed the options we have thought about in
>>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545 (but that
>>>> covers
>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>> things that I don’t want to get in to in this discussion
>>>> such as
>>>>>>>>>>>>>>>> possibly
>>>>>>>>>>>>>>>>>> separating operators and executors out of a single
>>> provider
>>>>>>>> dist.)
>>>>>>>>>>>>>>>>>> To give a concrete example of some code I would like to
>>>> share
>>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>>>>>>>>>>>>>>>> — logging config. Another thing we will want to share will
>>>> be
>>>>>>>> the
>>>>>>>>>>>>>>>>>> AirflowConfigParser class from airflow.configuration (but
>>>>>>>> notably:
>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> parser class, _not_ the default config values, again, lets
>>>> not
>>>>>>>>>> dwell
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> specifics of that)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> So to bring the options listed in the issue here for
>>>> discussion,
>>>>>>>>>>>>>>> broadly
>>>>>>>>>>>>>>>>>> speaking there are two high-level approaches:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1. A single shared distribution
>>>>>>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The advantage of Approach 1 is that we only have the code
>>>> in one
>>>>>>>>>>>>>>> place.
>>>>>>>>>>>>>>>>>> However for me, at least in this specific case of Logging
>>>> config
>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> AirflowConfigParser class is that backwards compatibility
>>> is
>>>>>>>> much
>>>>>>>>>>>>>> much
>>>>>>>>>>>>>>>>>> harder.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The main advantage of Approach 2 is the the code is
>>> released
>>>>>>>>>>>>>>>> with/embedded
>>>>>>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would contain
>>> the
>>>>>>>> right
>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>> of the logging config and ConfigParser etc). The downside
>>> is
>>>>>>>> that
>>>>>>>>>>>>>>> either
>>>>>>>>>>>>>>>>>> the code will need to be duplicated in the repo, or better
>>>> yet
>>>>>>>> it
>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>> live in a single place in the repo, but some tooling (TBD)
>>>> will
>>>>>>>>>>>>>>>>>> automatically handle the duplication, either at commit
>>>> time, or
>>>>>>>> my
>>>>>>>>>>>>>>>>>> preference, at release time.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For this kind of shared “utility” code I am very strongly
>>>>>>>> leaning
>>>>>>>>>>>>>>>> towards
>>>>>>>>>>>>>>>>>> option 2 with automation, as otherwise I think the
>>> backwards
>>>>>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>>>>>> requirements would make it unworkable (very quickly over
>>>> time
>>>>>>>> the
>>>>>>>>>>>>>>>>>> combinations we would have to test would just be
>>>> unreasonable)
>>>>>>>>>> and I
>>>>>>>>>>>>>>>> don’t
>>>>>>>>>>>>>>>>>> feel confident we can have things as stable as we need to
>>>> really
>>>>>>>>>>>>>>> deliver
>>>>>>>>>>>>>>>>>> the version separation/independency I want to delivery
>>> with
>>>>>>>>>> AIP-72.
>>>>>>>>>>>>>>>>>> So unless someone feels very strongly about this, I will
>>>> come up
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> draft PR for further discussion that will implement code
>>>> sharing
>>>>>>>>>> via
>>>>>>>>>>>>>>>>>> “vendoring” it at build time. I have an idea of how I can
>>>>>>>> achieve
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>> we have a single version in the repo and it’ll work there,
>>>> but
>>>>>>>> at
>>>>>>>>>>>>>>>> runtime
>>>>>>>>>>>>>>>>>> we vendor it in to the shipped dist so it lives at
>>> something
>>>>>>>> like
>>>>>>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> In terms of repo layout, this likely means we would end up
>>>> with:
>>>>>>>>>>>>>>>>>> airflow-core/pyproject.toml
>>>>>>>>>>>>>>>>>> airflow-core/src/
>>>>>>>>>>>>>>>>>> airflow-core/tests/
>>>>>>>>>>>>>>>>>> task-sdk/pyproject.toml
>>>>>>>>>>>>>>>>>> task-sdk/src/
>>>>>>>>>>>>>>>>>> task-sdk/tests/
>>>>>>>>>>>>>>>>>> airflow-common/src
>>>>>>>>>>>>>>>>>> airflow-common/tests/
>>>>>>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps would
>>>> be
>>>>>>>>>>>>>> included
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> the downstream projects. TBD.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thoughts and feedback welcomed.
>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>> dev-help@airflow.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>
>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>>
>>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>
>>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org


Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Ash Berlin-Taylor <as...@apache.org>.
In case I did a bad job explaining it, the “core and task sdk” is not in the module name/import name, just in the file path. 

Anyone have other ideas?

> On 7 Jul 2025, at 21:37, Buğra Öztürk <oz...@gmail.com> wrote:
> 
> Thanks Ash! Looks cool! I like the structure. This will enable all the
> combinations and structure looks easy to grasp. No strong stance on the
> naming other than maybe it is a bit long with `and`, `core_ctl` could be
> shorter, since no import path is defined like that, we can give any name
> for sure.
> 
> Best regards,
> 
>> On Mon, 7 Jul 2025, 21:51 Jarek Potiuk, <ja...@potiuk.com> wrote:
>> 
>> Looks good but I think we should find some better logical name for
>> core_and_sdk :)
>> 
>> pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
>> <j_...@gmx.de.invalid> napisał:
>> 
>>> Cool! Especially the "shared" folder with the ability to have
>>> N-combinations w/o exploding project repo root!
>>> 
>>> On 07.07.25 14:43, Ash Berlin-Taylor wrote:
>>>> Oh, and all of this will be explain in shared/README.md
>>>> 
>>>>> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>>> 
>>>>> Okay, so it seems we have agreement on the approach here, so I’ll
>>> continue with this, and on the dev call it was mentioned that
>>> “airflow-common” wasn’t a great name, so here is my proposal for the file
>>> structure;
>>>>> 
>>>>> ```
>>>>> /
>>>>>  task-sdk/...
>>>>>  airflow-core/...
>>>>>  shared/
>>>>>    kuberenetes/
>>>>>      pyproject.toml
>>>>>      src/
>>>>>        airflow_kube/__init__.py
>>>>>    core-and-tasksdk/
>>>>>      pyproject.toml
>>>>>      src/
>>>>>        airflow_shared/__init__.py
>>>>> ```
>>>>> 
>>>>> Things to note here: the “shared” folder has (the possibility) of
>>> having multiple different shared “libraries” in it, in this example I am
>>> supposing a hypothetical shared kuberenetes folder a world in which we
>>> split the KubePodOperator and the KubeExecutor in to two separate
>>> distributions (example only, not proposing we do that right now, and that
>>> will be a separate discussion)
>>>>> 
>>>>> The other things to note here:
>>>>> 
>>>>> 
>>>>> - the folder name in shared aims to be “self-documenting”, hence the
>>> verbose “core-and-tasksdk” to say where the shared library is intended to
>>> be used.
>>>>> - the python module itself should almost always have an `airflow_` (or
>>> maybe `_airflow_`?) prefix so that it does not conflict with anything
>> else
>>> we might use. It won’t matter “in production” as those will be vendored
>> in
>>> to be imported as `airflow/_vendor/airflow_shared` etc, but avoiding
>>> conflicts at dev time with the Finder approach is a good safety measure.
>>>>> 
>>>>> I will start making a real PR for this proposal now, but I’m open to
>>> feedback (either here, or in the PR when I open it)
>>>>> 
>>>>> -ash
>>>>> 
>>>>>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>> 
>>>>>> Yeah we have to try it and test - also building packages happens semi
>>>>>> frequently when you run `uv sync` (they use some kind of heuristics
>> to
>>>>>> decide when) and you can force it with `--reinstall` or `--refresh`.
>>>>>> Package build also happens every time when you run "ci-image build`
>>> now in
>>>>>> breeze so it seems like it will nicely integrate in our workflows.
>>>>>> 
>>>>>> Looks really cool Ash.
>>>>>> 
>>>>>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <as...@apache.org>
>>> wrote:
>>>>>> 
>>>>>>> It’s not just release time, but any time we build a package which
>>> happens
>>>>>>> on “every” CI run. The normal unit tests will use code from
>>>>>>> airflow-common/src/airflow_common; the kube tests which build an
>>> image will
>>>>>>> build the dists and vendor in the code from that commit.
>>>>>>> 
>>>>>>> There is only a single copy of the shared code committed to the
>> repo,
>>> so
>>>>>>> there is never anything to synchronise.
>>>>>>> 
>>>>>>>> On 4 Jul 2025, at 15:53, Amogh Desai <am...@gmail.com>
>>> wrote:
>>>>>>>> 
>>>>>>>> Thanks Ash.
>>>>>>>> 
>>>>>>>> This is really cool and helpful that you were able to test both
>>> scenarios
>>>>>>>> -- repo checkout
>>>>>>>> and also installing from the vendored package and the resolution
>>> worked
>>>>>>>> fine too.
>>>>>>>> 
>>>>>>>> I like this idea compared the to relative import one for few
>> reasons:
>>>>>>>> - It feels like it will take some time to adjust to the new coding
>>>>>>> standard
>>>>>>>> that we will lay
>>>>>>>> if we impose relative imports in the shared dist
>>>>>>>> - We can continue using repo wise absolute import standards, it is
>>> also
>>>>>>>> much easier for situations
>>>>>>>> when we do global search in IDE to find + replace, this could mean
>> a
>>>>>>> change
>>>>>>>> there
>>>>>>>> - The vendoring work is a proven and established paradigm across
>>> projects
>>>>>>>> and would
>>>>>>>> out of box give us the build tooling we need also
>>>>>>>> 
>>>>>>>> Nothing too against the relative import but with the evidence
>>> provided
>>>>>>>> above, vendored approach
>>>>>>>> seems to only do us good.
>>>>>>>> 
>>>>>>>> Regarding synchronizing it, release time should be fine as long as
>> we
>>>>>>> have
>>>>>>>> a good CI workflow to probably
>>>>>>>> catch such issues per PR if changes are made in shared dist?
>>> (precommit
>>>>>>>> would make it really slow i guess)
>>>>>>>> 
>>>>>>>> If we can run our tests with vendored code we should be mostly
>>> covered.
>>>>>>>> 
>>>>>>>> Good effort all!
>>>>>>>> 
>>>>>>>> Thanks & Regards,
>>>>>>>> Amogh Desai
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <as...@apache.org>
>>>>>>> wrote:
>>>>>>>>> Okay, I think I’ve got something that works and I’m happy with.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>> 
>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>>>>>>>>> This produces the following from `uv build task-sdk`
>>>>>>>>> -
>>>>>>>>> 
>>>>>>> 
>>> 
>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>>>>>> -
>>>>>>>>> 
>>>>>>> 
>>> 
>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>>>>>>>>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
>>>>>>>>> 
>>>>>>>>> ```
>>>>>>>>> ❯ unzip -l
>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
>>>>>>> grep
>>>>>>>>> _vendor
>>>>>>>>>     50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
>>>>>>>>>   2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
>>>>>>>>>     28  02-02-2020 00:00   airflow/sdk/_vendor/airflow_common.pyi
>>>>>>>>>     18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
>>>>>>>>>    785  02-02-2020 00:00
>>>>>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>>>>>  10628  02-02-2020 00:00
>>>>>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>>>>> ```
>>>>>>>>> 
>>>>>>>>> And similarly in the .tar.gz, so our “sdist” is complete too:
>>>>>>>>> ```
>>>>>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep _vendor
>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>>>>>>>>> 
>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>>>>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>> 
>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>>>>> 
>>>>>>> 
>>> 
>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>>>>> ```
>>>>>>>>> 
>>>>>>>>> The plugin works at build time by including/copying the libs
>>> specified
>>>>>>> in
>>>>>>>>> vendor.txt into place (and let `vendoring` take care of import
>>>>>>> rewrites.)
>>>>>>>>> For the imports to continue to work at “dev” time/from a repo
>>> checkout,
>>>>>>> I
>>>>>>>>> have added a import finder to `sys.meta_path`, and since it’s at
>> the
>>>>>>> end of
>>>>>>>>> the list it will only be used if the normal import can’t find
>>> things.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>> 
>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>>>>>>>>> This doesn’t quite give us the same runtime effect “import
>>> rewriting”
>>>>>>>>> affect, as in this approach `airflow_common` is directly loaded
>>> (i.e.
>>>>>>>>> airflow.sdk._vendor.airflow_common and airflow_common exist in
>>>>>>>>> sys.modules), but it does work for everything that I was able to
>>> test..
>>>>>>>>> 
>>>>>>>>> I tested it with the diff at the end of this message. My test
>>> ipython
>>>>>>>>> shell:
>>>>>>>>> 
>>>>>>>>> ```
>>>>>>>>> In [1]: from airflow.sdk._vendor.airflow_common.timezone import
>> foo
>>>>>>>>> 
>>>>>>>>> In [2]: foo
>>>>>>>>> Out[2]: 1
>>>>>>>>> 
>>>>>>>>> In [3]: import airflow.sdk._vendor.airflow_common
>>>>>>>>> 
>>>>>>>>> In [4]: import airflow.sdk._vendor.airflow_common.timezone
>>>>>>>>> 
>>>>>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>>>>>>>>> Out[5]:
>>>>>>>>> 
>>>>>>> 
>>> 
>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>>>>>>>>> In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>>>>> Out[6]:
>>>>>>>>> 
>>>>>>> 
>>> 
>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>>>>>>>>> ```
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> And in an standalone environment with the SDK dist I built (it
>>> needed
>>>>>>> the
>>>>>>>>> matching airflow-core right now, but that is nothing to do with
>> this
>>>>>>>>> discussion):
>>>>>>>>> 
>>>>>>>>> ```
>>>>>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>>>>>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>>>>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
>>>>>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
>>>>>>>>> Type 'copyright', 'credits' or 'license' for more information
>>>>>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for
>> help.
>>>>>>>>> Tip: You can use `%hist` to view history, see the options with
>>>>>>> `%history?`
>>>>>>>>> In [1]: import airflow.sdk._vendor.airflow_common.timezone
>>>>>>>>> 
>>>>>>>>> In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>>>>> Out[2]:
>>>>>>>>> 
>>>>>>> 
>>> 
>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>>>>>>>>> ``
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ```diff
>>>>>>>>> diff --git a/airflow-common/src/airflow_common/__init__.py
>>>>>>>>> b/airflow-common/src/airflow_common/__init__.py
>>>>>>>>> index 13a83393a9..927b7c6b61 100644
>>>>>>>>> --- a/airflow-common/src/airflow_common/__init__.py
>>>>>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
>>>>>>>>> @@ -14,3 +14,5 @@
>>>>>>>>> # KIND, either express or implied.  See the License for the
>>>>>>>>> # specific language governing permissions and limitations
>>>>>>>>> # under the License.
>>>>>>>>> +
>>>>>>>>> +foo = 1
>>>>>>>>> diff --git a/airflow-common/src/airflow_common/timezone.py
>>>>>>>>> b/airflow-common/src/airflow_common/timezone.py
>>>>>>>>> index 340b924c66..58384ef20f 100644
>>>>>>>>> --- a/airflow-common/src/airflow_common/timezone.py
>>>>>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
>>>>>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>>>>>>>>> version.parse(metadata.version("pendulum")).major == 3
>>>>>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
>>>>>>>>> utc = pendulum.UTC
>>>>>>>>> 
>>>>>>>>> +
>>>>>>>>> +from airflow_common import foo
>>>>>>>>> +
>>>>>>>>> TIMEZONE: Timezone
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ```
>>>>>>>>> 
>>>>>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>>>>> I think both approaches are doable:
>>>>>>>>>> 
>>>>>>>>>> 1) -> We can very easily prevent bad imports by pre-commit when
>>>>>>> importing
>>>>>>>>>> from different distributions and make sure we are only doing
>>> relative
>>>>>>>>>> imports in the shared modules. We are doing plenty of this
>>> already. And
>>>>>>>>> yes
>>>>>>>>>> it would require relative links we currently do not allow.
>>>>>>>>>> 
>>>>>>>>>> 2) -> has one disadvantage that someone at some point in time
>> will
>>> have
>>>>>>>>> to
>>>>>>>>>> decide to synchronize this and if it happens just before release
>>> (I bet
>>>>>>>>>> this is going to happen) this will lead to solving problems that
>>> would
>>>>>>>>>> normally be solved during PR when you make a change (i.e.
>> symbolic
>>> link
>>>>>>>>> has
>>>>>>>>>> the advantage that whoever modifies shared code will be
>> immediately
>>>>>>>>>> notified in their PR - that they broke something because either
>>> static
>>>>>>>>>> checks or mypy or tests fail.
>>>>>>>>>> 
>>>>>>>>>> Ash, do you have an idea of a process (who and when) does the
>>>>>>>>>> synchronisation in case of vendoring? Maybe we could solve it if
>>> it is
>>>>>>>>> done
>>>>>>>>>> more frequently and with some regularity? We could potentially
>>> force
>>>>>>>>>> re-vendoring at PR time as well any time shared code changes (and
>>>>>>> prevent
>>>>>>>>>> it by pre-commit. And I can't think of some place (other than
>>> releases)
>>>>>>>>> in
>>>>>>>>>> our development workflow and that seems to be a bit too late as
>>> puts an
>>>>>>>>>> extra effort on fixing potential incompatibilities introduced on
>>>>>>> release
>>>>>>>>>> manager and delays the release. WDYT?
>>>>>>>>>> 
>>>>>>>>>> Re: relative links. I think for a shared library we could
>>> potentially
>>>>>>>>> relax
>>>>>>>>>> this and allow them (and actually disallow absolute links in the
>>> pieces
>>>>>>>>> of
>>>>>>>>>> code that are shared - again, by pre-commit). As I recall, the
>> only
>>>>>>>>> reason
>>>>>>>>>> we forbade the relative links is because of how we are (or maybe
>>> were)
>>>>>>>>>> doing DAG parsing and failures resulting from it. So we decided
>> to
>>> just
>>>>>>>>> not
>>>>>>>>>> allow it to keep consistency. The way how Dag parsing works is
>> that
>>>>>>> when
>>>>>>>>>> you are using importlib to read the Dag from a file, the relative
>>>>>>> imports
>>>>>>>>>> do not work as it does not know what they should be relative to.
>>> But if
>>>>>>>>>> relative import is done from an imported package, it should be no
>>>>>>>>> problem,
>>>>>>>>>> I think - otherwise our Dags would not be able to import any
>>> library
>>>>>>> that
>>>>>>>>>> uses relative imports.
>>>>>>>>>> 
>>>>>>>>>> Of course consistency might be the reason why we do not want to
>>>>>>> introduce
>>>>>>>>>> relative imports. I don't see it as an issue if it is guarded by
>>>>>>>>> pre-commit
>>>>>>>>>> though.
>>>>>>>>>> 
>>>>>>>>>> J.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> J.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
>>> ash@apache.org>
>>>>>>>>>> napisał:
>>>>>>>>>> 
>>>>>>>>>>> Oh yes, symlinks will work, with one big caveat: It does mean
>> you
>>>>>>> can’t
>>>>>>>>>>> use absolute imports in one common module to another.
>>>>>>>>>>> 
>>>>>>>>>>> For example
>>>>>>>>>>> 
>>>>>>> 
>>> 
>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>>>>>>>>>>> where we have
>>>>>>>>>>> 
>>>>>>>>>>> ```
>>>>>>>>>>> from airflow.utils.module_loading import import_string
>>>>>>>>>>> ```
>>>>>>>>>>> 
>>>>>>>>>>> if we want to move serve_logs into this common lib that is then
>>>>>>>>> symlinked
>>>>>>>>>>> then we wouldn’t be able to have `from
>>> airflow_common.module_loading
>>>>>>>>> import
>>>>>>>>>>> import_string`.
>>>>>>>>>>> 
>>>>>>>>>>> I can think of two possible solutions here.
>>>>>>>>>>> 
>>>>>>>>>>> 1) is to allow/require relative imports in this shared lib, so
>>> `from
>>>>>>>>>>> .module_loading import import_string`
>>>>>>>>>>> 2) is to use `vendoring`[1] (from the pip maintainers) which
>> will
>>>>>>> handle
>>>>>>>>>>> import-rewriting for us.
>>>>>>>>>>> 
>>>>>>>>>>> I’d entirely forgot that symlinks in repos was a thing, so I
>>> prepared
>>>>>>> a
>>>>>>>>>>> minimal POC/demo of what vendoring approach could look like here
>>>>>>>>>>> 
>>>>>>> 
>>> 
>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>>>>>>>>>>> Now personally I am more than happy with relative imports, but
>>>>>>> generally
>>>>>>>>>>> as a project we have avoided them, so I think that limits what
>> we
>>>>>>> could
>>>>>>>>> do
>>>>>>>>>>> with a symlink based approach.
>>>>>>>>>>> 
>>>>>>>>>>> -ash
>>>>>>>>>>> 
>>>>>>>>>>> [1] https://github.com/pradyunsg/vendoring
>>>>>>>>>>> 
>>>>>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
>>>>>>> gopidesupavan@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Thanks Ash
>>>>>>>>>>>> 
>>>>>>>>>>>> Yes agree option 2 would be preferred for me. Making sure we
>>> have all
>>>>>>>>> the
>>>>>>>>>>>> gaurdriles to protect any unwanted behaviour in code sharing
>> and
>>>>>>>>>>> executing
>>>>>>>>>>>> right of tests between the packages.
>>>>>>>>>>>> 
>>>>>>>>>>>> Agree with others, option 2 would be
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
>>>>>>> amoghdesai.oss@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for starting this discussion, Ash.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I would prefer option 2 here with proper tooling to handle the
>>> code
>>>>>>>>>>>>> duplication at *release* time.
>>>>>>>>>>>>> It is best to have a dist that has all it needs in itself.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Option 1 could very quickly get out of hand and if we decide
>> to
>>>>>>>>> separate
>>>>>>>>>>>>> triggerer /
>>>>>>>>>>>>> dag processor / config etc etc as separate packages, back
>>> compat is
>>>>>>>>>>> going
>>>>>>>>>>>>> to be a nightmare
>>>>>>>>>>>>> and will bite us harder than we anticipate.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks & Regards,
>>>>>>>>>>>>> Amogh Desai
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <
>> kaxilnaik@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>> I prefer Option 2 as well to avoid matrix of dependencies
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>>>>>>>>> <j_scheffler@gmx.de.invalid
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I'd also rather prefer option 2 - reason here is it is
>> rather
>>>>>>>>>>> pragmatic
>>>>>>>>>>>>>>> and we no not need to cut another package and have less
>>> package
>>>>>>>>> counts
>>>>>>>>>>>>>>> and dependencies.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I remember some time ago I was checking (together with
>> Jarek,
>>> I am
>>>>>>>>> not
>>>>>>>>>>>>>>> sure anymore...) if the usage of symlinks would be possible.
>>> To
>>>>>>> keep
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> source in one package but "symlink" it into another. If then
>>> at
>>>>>>>>> point
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> packaging/release the files are materialized we have 1 set
>> of
>>>>>>> code.
>>>>>>>>>>>>>>> Otherwise if not possible still the redundancy could be
>>> solved by
>>>>>>> a
>>>>>>>>>>>>>>> pre-commit hook - and in Git the files are de-duplicated
>>> anyway
>>>>>>>>> based
>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> content hash, so this does not hurt.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>>>>>>>>>>>>>> I support option 2 with proper automation & CI - the
>>> reasonings
>>>>>>>>>>>>> you've
>>>>>>>>>>>>>>>> shown for that make sense to me.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Shahar
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <
>>> ash@apache.org
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> As we work on finishing off the code-level separation of
>>> Task
>>>>>>> SDK
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> Core
>>>>>>>>>>>>>>>>> (scheduler etc) we have come across some situations where
>> we
>>>>>>> would
>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> share code between these.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> However it’s not as straight forward of “just put it in a
>>> common
>>>>>>>>>>>>> dist
>>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>>> both depend upon” because one of the goals of the Task SDK
>>>>>>>>>>>>> separation
>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>> to have 100% complete version independence between the
>> two,
>>>>>>>>> ideally
>>>>>>>>>>>>>>> even if
>>>>>>>>>>>>>>>>> they are built into the same image and venv. Most of the
>>> reason
>>>>>>>>> why
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> isn’t straight forward comes down to backwards
>>> compatibility -
>>>>>>> if
>>>>>>>>> we
>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>> an change to the common/shared distribution
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> We’ve listed the options we have thought about in
>>>>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545 (but that
>>> covers
>>>>>>>>>>>>> some
>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>> things that I don’t want to get in to in this discussion
>>> such as
>>>>>>>>>>>>>>> possibly
>>>>>>>>>>>>>>>>> separating operators and executors out of a single
>> provider
>>>>>>> dist.)
>>>>>>>>>>>>>>>>> To give a concrete example of some code I would like to
>>> share
>>>>>>>>>>>>>>>>> 
>>>>>>> 
>>> 
>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>>>>>>>>>>>>>>> — logging config. Another thing we will want to share will
>>> be
>>>>>>> the
>>>>>>>>>>>>>>>>> AirflowConfigParser class from airflow.configuration (but
>>>>>>> notably:
>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> parser class, _not_ the default config values, again, lets
>>> not
>>>>>>>>> dwell
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> specifics of that)
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> So to bring the options listed in the issue here for
>>> discussion,
>>>>>>>>>>>>>> broadly
>>>>>>>>>>>>>>>>> speaking there are two high-level approaches:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 1. A single shared distribution
>>>>>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The advantage of Approach 1 is that we only have the code
>>> in one
>>>>>>>>>>>>>> place.
>>>>>>>>>>>>>>>>> However for me, at least in this specific case of Logging
>>> config
>>>>>>>>> or
>>>>>>>>>>>>>>>>> AirflowConfigParser class is that backwards compatibility
>> is
>>>>>>> much
>>>>>>>>>>>>> much
>>>>>>>>>>>>>>>>> harder.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The main advantage of Approach 2 is the the code is
>> released
>>>>>>>>>>>>>>> with/embedded
>>>>>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would contain
>> the
>>>>>>> right
>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>> of the logging config and ConfigParser etc). The downside
>> is
>>>>>>> that
>>>>>>>>>>>>>> either
>>>>>>>>>>>>>>>>> the code will need to be duplicated in the repo, or better
>>> yet
>>>>>>> it
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>> live in a single place in the repo, but some tooling (TBD)
>>> will
>>>>>>>>>>>>>>>>> automatically handle the duplication, either at commit
>>> time, or
>>>>>>> my
>>>>>>>>>>>>>>>>> preference, at release time.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> For this kind of shared “utility” code I am very strongly
>>>>>>> leaning
>>>>>>>>>>>>>>> towards
>>>>>>>>>>>>>>>>> option 2 with automation, as otherwise I think the
>> backwards
>>>>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>>>>> requirements would make it unworkable (very quickly over
>>> time
>>>>>>> the
>>>>>>>>>>>>>>>>> combinations we would have to test would just be
>>> unreasonable)
>>>>>>>>> and I
>>>>>>>>>>>>>>> don’t
>>>>>>>>>>>>>>>>> feel confident we can have things as stable as we need to
>>> really
>>>>>>>>>>>>>> deliver
>>>>>>>>>>>>>>>>> the version separation/independency I want to delivery
>> with
>>>>>>>>> AIP-72.
>>>>>>>>>>>>>>>>> So unless someone feels very strongly about this, I will
>>> come up
>>>>>>>>>>>>> with
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> draft PR for further discussion that will implement code
>>> sharing
>>>>>>>>> via
>>>>>>>>>>>>>>>>> “vendoring” it at build time. I have an idea of how I can
>>>>>>> achieve
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>> we have a single version in the repo and it’ll work there,
>>> but
>>>>>>> at
>>>>>>>>>>>>>>> runtime
>>>>>>>>>>>>>>>>> we vendor it in to the shipped dist so it lives at
>> something
>>>>>>> like
>>>>>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> In terms of repo layout, this likely means we would end up
>>> with:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> airflow-core/pyproject.toml
>>>>>>>>>>>>>>>>> airflow-core/src/
>>>>>>>>>>>>>>>>> airflow-core/tests/
>>>>>>>>>>>>>>>>> task-sdk/pyproject.toml
>>>>>>>>>>>>>>>>> task-sdk/src/
>>>>>>>>>>>>>>>>> task-sdk/tests/
>>>>>>>>>>>>>>>>> airflow-common/src
>>>>>>>>>>>>>>>>> airflow-common/tests/
>>>>>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps would
>>> be
>>>>>>>>>>>>> included
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the downstream projects. TBD.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thoughts and feedback welcomed.
>>>>>>>>>>>>>>> 
>>>>>>>>> 
>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>>>>>>> For additional commands, e-mail:
>> dev-help@airflow.apache.org
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>> 
>>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org


Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Buğra Öztürk <oz...@gmail.com>.
Thanks Ash! Looks cool! I like the structure. This will enable all the
combinations and structure looks easy to grasp. No strong stance on the
naming other than maybe it is a bit long with `and`, `core_ctl` could be
shorter, since no import path is defined like that, we can give any name
for sure.

Best regards,

On Mon, 7 Jul 2025, 21:51 Jarek Potiuk, <ja...@potiuk.com> wrote:

> Looks good but I think we should find some better logical name for
> core_and_sdk :)
>
> pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
> <j_...@gmx.de.invalid> napisał:
>
> > Cool! Especially the "shared" folder with the ability to have
> > N-combinations w/o exploding project repo root!
> >
> > On 07.07.25 14:43, Ash Berlin-Taylor wrote:
> > > Oh, and all of this will be explain in shared/README.md
> > >
> > >> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org> wrote:
> > >>
> > >> Okay, so it seems we have agreement on the approach here, so I’ll
> > continue with this, and on the dev call it was mentioned that
> > “airflow-common” wasn’t a great name, so here is my proposal for the file
> > structure;
> > >>
> > >> ```
> > >> /
> > >>   task-sdk/...
> > >>   airflow-core/...
> > >>   shared/
> > >>     kuberenetes/
> > >>       pyproject.toml
> > >>       src/
> > >>         airflow_kube/__init__.py
> > >>     core-and-tasksdk/
> > >>       pyproject.toml
> > >>       src/
> > >>         airflow_shared/__init__.py
> > >> ```
> > >>
> > >> Things to note here: the “shared” folder has (the possibility) of
> > having multiple different shared “libraries” in it, in this example I am
> > supposing a hypothetical shared kuberenetes folder a world in which we
> > split the KubePodOperator and the KubeExecutor in to two separate
> > distributions (example only, not proposing we do that right now, and that
> > will be a separate discussion)
> > >>
> > >> The other things to note here:
> > >>
> > >>
> > >> - the folder name in shared aims to be “self-documenting”, hence the
> > verbose “core-and-tasksdk” to say where the shared library is intended to
> > be used.
> > >> - the python module itself should almost always have an `airflow_` (or
> > maybe `_airflow_`?) prefix so that it does not conflict with anything
> else
> > we might use. It won’t matter “in production” as those will be vendored
> in
> > to be imported as `airflow/_vendor/airflow_shared` etc, but avoiding
> > conflicts at dev time with the Finder approach is a good safety measure.
> > >>
> > >> I will start making a real PR for this proposal now, but I’m open to
> > feedback (either here, or in the PR when I open it)
> > >>
> > >> -ash
> > >>
> > >>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >>>
> > >>> Yeah we have to try it and test - also building packages happens semi
> > >>> frequently when you run `uv sync` (they use some kind of heuristics
> to
> > >>> decide when) and you can force it with `--reinstall` or `--refresh`.
> > >>> Package build also happens every time when you run "ci-image build`
> > now in
> > >>> breeze so it seems like it will nicely integrate in our workflows.
> > >>>
> > >>> Looks really cool Ash.
> > >>>
> > >>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <as...@apache.org>
> > wrote:
> > >>>
> > >>>> It’s not just release time, but any time we build a package which
> > happens
> > >>>> on “every” CI run. The normal unit tests will use code from
> > >>>> airflow-common/src/airflow_common; the kube tests which build an
> > image will
> > >>>> build the dists and vendor in the code from that commit.
> > >>>>
> > >>>> There is only a single copy of the shared code committed to the
> repo,
> > so
> > >>>> there is never anything to synchronise.
> > >>>>
> > >>>>> On 4 Jul 2025, at 15:53, Amogh Desai <am...@gmail.com>
> > wrote:
> > >>>>>
> > >>>>> Thanks Ash.
> > >>>>>
> > >>>>> This is really cool and helpful that you were able to test both
> > scenarios
> > >>>>> -- repo checkout
> > >>>>> and also installing from the vendored package and the resolution
> > worked
> > >>>>> fine too.
> > >>>>>
> > >>>>> I like this idea compared the to relative import one for few
> reasons:
> > >>>>> - It feels like it will take some time to adjust to the new coding
> > >>>> standard
> > >>>>> that we will lay
> > >>>>> if we impose relative imports in the shared dist
> > >>>>> - We can continue using repo wise absolute import standards, it is
> > also
> > >>>>> much easier for situations
> > >>>>> when we do global search in IDE to find + replace, this could mean
> a
> > >>>> change
> > >>>>> there
> > >>>>> - The vendoring work is a proven and established paradigm across
> > projects
> > >>>>> and would
> > >>>>> out of box give us the build tooling we need also
> > >>>>>
> > >>>>> Nothing too against the relative import but with the evidence
> > provided
> > >>>>> above, vendored approach
> > >>>>> seems to only do us good.
> > >>>>>
> > >>>>> Regarding synchronizing it, release time should be fine as long as
> we
> > >>>> have
> > >>>>> a good CI workflow to probably
> > >>>>> catch such issues per PR if changes are made in shared dist?
> > (precommit
> > >>>>> would make it really slow i guess)
> > >>>>>
> > >>>>> If we can run our tests with vendored code we should be mostly
> > covered.
> > >>>>>
> > >>>>> Good effort all!
> > >>>>>
> > >>>>> Thanks & Regards,
> > >>>>> Amogh Desai
> > >>>>>
> > >>>>>
> > >>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <as...@apache.org>
> > >>>> wrote:
> > >>>>>> Okay, I think I’ve got something that works and I’m happy with.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>
> >
> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
> > >>>>>> This produces the following from `uv build task-sdk`
> > >>>>>> -
> > >>>>>>
> > >>>>
> >
> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
> > >>>>>> -
> > >>>>>>
> > >>>>
> >
> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
> > >>>>>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
> > >>>>>>
> > >>>>>> ```
> > >>>>>> ❯ unzip -l
> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
> > >>>> grep
> > >>>>>> _vendor
> > >>>>>>      50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
> > >>>>>>    2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
> > >>>>>>      28  02-02-2020 00:00   airflow/sdk/_vendor/airflow_common.pyi
> > >>>>>>      18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
> > >>>>>>     785  02-02-2020 00:00
> > >>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
> > >>>>>>   10628  02-02-2020 00:00
> > >>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
> > >>>>>> ```
> > >>>>>>
> > >>>>>> And similarly in the .tar.gz, so our “sdist” is complete too:
> > >>>>>> ```
> > >>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep _vendor
> > >>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
> > >>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
> > >>>>>>
> > apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
> > >>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
> > >>>>>>
> > >>>>>>
> > >>>>
> >
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
> > >>>>>>
> > >>>>
> >
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
> > >>>>>> ```
> > >>>>>>
> > >>>>>> The plugin works at build time by including/copying the libs
> > specified
> > >>>> in
> > >>>>>> vendor.txt into place (and let `vendoring` take care of import
> > >>>> rewrites.)
> > >>>>>> For the imports to continue to work at “dev” time/from a repo
> > checkout,
> > >>>> I
> > >>>>>> have added a import finder to `sys.meta_path`, and since it’s at
> the
> > >>>> end of
> > >>>>>> the list it will only be used if the normal import can’t find
> > things.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>
> >
> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
> > >>>>>> This doesn’t quite give us the same runtime effect “import
> > rewriting”
> > >>>>>> affect, as in this approach `airflow_common` is directly loaded
> > (i.e.
> > >>>>>> airflow.sdk._vendor.airflow_common and airflow_common exist in
> > >>>>>> sys.modules), but it does work for everything that I was able to
> > test..
> > >>>>>>
> > >>>>>> I tested it with the diff at the end of this message. My test
> > ipython
> > >>>>>> shell:
> > >>>>>>
> > >>>>>> ```
> > >>>>>> In [1]: from airflow.sdk._vendor.airflow_common.timezone import
> foo
> > >>>>>>
> > >>>>>> In [2]: foo
> > >>>>>> Out[2]: 1
> > >>>>>>
> > >>>>>> In [3]: import airflow.sdk._vendor.airflow_common
> > >>>>>>
> > >>>>>> In [4]: import airflow.sdk._vendor.airflow_common.timezone
> > >>>>>>
> > >>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
> > >>>>>> Out[5]:
> > >>>>>>
> > >>>>
> >
> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
> > >>>>>> In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
> > >>>>>> Out[6]:
> > >>>>>>
> > >>>>
> >
> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
> > >>>>>> ```
> > >>>>>>
> > >>>>>>
> > >>>>>> And in an standalone environment with the SDK dist I built (it
> > needed
> > >>>> the
> > >>>>>> matching airflow-core right now, but that is nothing to do with
> this
> > >>>>>> discussion):
> > >>>>>>
> > >>>>>> ```
> > >>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
> > >>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
> > >>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
> > >>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
> > >>>>>> Type 'copyright', 'credits' or 'license' for more information
> > >>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for
> help.
> > >>>>>> Tip: You can use `%hist` to view history, see the options with
> > >>>> `%history?`
> > >>>>>> In [1]: import airflow.sdk._vendor.airflow_common.timezone
> > >>>>>>
> > >>>>>> In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
> > >>>>>> Out[2]:
> > >>>>>>
> > >>>>
> >
> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
> > >>>>>> ``
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> ```diff
> > >>>>>> diff --git a/airflow-common/src/airflow_common/__init__.py
> > >>>>>> b/airflow-common/src/airflow_common/__init__.py
> > >>>>>> index 13a83393a9..927b7c6b61 100644
> > >>>>>> --- a/airflow-common/src/airflow_common/__init__.py
> > >>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
> > >>>>>> @@ -14,3 +14,5 @@
> > >>>>>> # KIND, either express or implied.  See the License for the
> > >>>>>> # specific language governing permissions and limitations
> > >>>>>> # under the License.
> > >>>>>> +
> > >>>>>> +foo = 1
> > >>>>>> diff --git a/airflow-common/src/airflow_common/timezone.py
> > >>>>>> b/airflow-common/src/airflow_common/timezone.py
> > >>>>>> index 340b924c66..58384ef20f 100644
> > >>>>>> --- a/airflow-common/src/airflow_common/timezone.py
> > >>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
> > >>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
> > >>>>>> version.parse(metadata.version("pendulum")).major == 3
> > >>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
> > >>>>>> utc = pendulum.UTC
> > >>>>>>
> > >>>>>> +
> > >>>>>> +from airflow_common import foo
> > >>>>>> +
> > >>>>>> TIMEZONE: Timezone
> > >>>>>>
> > >>>>>>
> > >>>>>> ```
> > >>>>>>
> > >>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >>>>>>> I think both approaches are doable:
> > >>>>>>>
> > >>>>>>> 1) -> We can very easily prevent bad imports by pre-commit when
> > >>>> importing
> > >>>>>>> from different distributions and make sure we are only doing
> > relative
> > >>>>>>> imports in the shared modules. We are doing plenty of this
> > already. And
> > >>>>>> yes
> > >>>>>>> it would require relative links we currently do not allow.
> > >>>>>>>
> > >>>>>>> 2) -> has one disadvantage that someone at some point in time
> will
> > have
> > >>>>>> to
> > >>>>>>> decide to synchronize this and if it happens just before release
> > (I bet
> > >>>>>>> this is going to happen) this will lead to solving problems that
> > would
> > >>>>>>> normally be solved during PR when you make a change (i.e.
> symbolic
> > link
> > >>>>>> has
> > >>>>>>> the advantage that whoever modifies shared code will be
> immediately
> > >>>>>>> notified in their PR - that they broke something because either
> > static
> > >>>>>>> checks or mypy or tests fail.
> > >>>>>>>
> > >>>>>>> Ash, do you have an idea of a process (who and when) does the
> > >>>>>>> synchronisation in case of vendoring? Maybe we could solve it if
> > it is
> > >>>>>> done
> > >>>>>>> more frequently and with some regularity? We could potentially
> > force
> > >>>>>>> re-vendoring at PR time as well any time shared code changes (and
> > >>>> prevent
> > >>>>>>> it by pre-commit. And I can't think of some place (other than
> > releases)
> > >>>>>> in
> > >>>>>>> our development workflow and that seems to be a bit too late as
> > puts an
> > >>>>>>> extra effort on fixing potential incompatibilities introduced on
> > >>>> release
> > >>>>>>> manager and delays the release. WDYT?
> > >>>>>>>
> > >>>>>>> Re: relative links. I think for a shared library we could
> > potentially
> > >>>>>> relax
> > >>>>>>> this and allow them (and actually disallow absolute links in the
> > pieces
> > >>>>>> of
> > >>>>>>> code that are shared - again, by pre-commit). As I recall, the
> only
> > >>>>>> reason
> > >>>>>>> we forbade the relative links is because of how we are (or maybe
> > were)
> > >>>>>>> doing DAG parsing and failures resulting from it. So we decided
> to
> > just
> > >>>>>> not
> > >>>>>>> allow it to keep consistency. The way how Dag parsing works is
> that
> > >>>> when
> > >>>>>>> you are using importlib to read the Dag from a file, the relative
> > >>>> imports
> > >>>>>>> do not work as it does not know what they should be relative to.
> > But if
> > >>>>>>> relative import is done from an imported package, it should be no
> > >>>>>> problem,
> > >>>>>>> I think - otherwise our Dags would not be able to import any
> > library
> > >>>> that
> > >>>>>>> uses relative imports.
> > >>>>>>>
> > >>>>>>> Of course consistency might be the reason why we do not want to
> > >>>> introduce
> > >>>>>>> relative imports. I don't see it as an issue if it is guarded by
> > >>>>>> pre-commit
> > >>>>>>> though.
> > >>>>>>>
> > >>>>>>> J.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> J.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
> > ash@apache.org>
> > >>>>>>> napisał:
> > >>>>>>>
> > >>>>>>>> Oh yes, symlinks will work, with one big caveat: It does mean
> you
> > >>>> can’t
> > >>>>>>>> use absolute imports in one common module to another.
> > >>>>>>>>
> > >>>>>>>> For example
> > >>>>>>>>
> > >>>>
> >
> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
> > >>>>>>>> where we have
> > >>>>>>>>
> > >>>>>>>> ```
> > >>>>>>>> from airflow.utils.module_loading import import_string
> > >>>>>>>> ```
> > >>>>>>>>
> > >>>>>>>> if we want to move serve_logs into this common lib that is then
> > >>>>>> symlinked
> > >>>>>>>> then we wouldn’t be able to have `from
> > airflow_common.module_loading
> > >>>>>> import
> > >>>>>>>> import_string`.
> > >>>>>>>>
> > >>>>>>>> I can think of two possible solutions here.
> > >>>>>>>>
> > >>>>>>>> 1) is to allow/require relative imports in this shared lib, so
> > `from
> > >>>>>>>> .module_loading import import_string`
> > >>>>>>>> 2) is to use `vendoring`[1] (from the pip maintainers) which
> will
> > >>>> handle
> > >>>>>>>> import-rewriting for us.
> > >>>>>>>>
> > >>>>>>>> I’d entirely forgot that symlinks in repos was a thing, so I
> > prepared
> > >>>> a
> > >>>>>>>> minimal POC/demo of what vendoring approach could look like here
> > >>>>>>>>
> > >>>>
> >
> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
> > >>>>>>>> Now personally I am more than happy with relative imports, but
> > >>>> generally
> > >>>>>>>> as a project we have avoided them, so I think that limits what
> we
> > >>>> could
> > >>>>>> do
> > >>>>>>>> with a symlink based approach.
> > >>>>>>>>
> > >>>>>>>> -ash
> > >>>>>>>>
> > >>>>>>>> [1] https://github.com/pradyunsg/vendoring
> > >>>>>>>>
> > >>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
> > >>>> gopidesupavan@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>> Thanks Ash
> > >>>>>>>>>
> > >>>>>>>>> Yes agree option 2 would be preferred for me. Making sure we
> > have all
> > >>>>>> the
> > >>>>>>>>> gaurdriles to protect any unwanted behaviour in code sharing
> and
> > >>>>>>>> executing
> > >>>>>>>>> right of tests between the packages.
> > >>>>>>>>>
> > >>>>>>>>> Agree with others, option 2 would be
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
> > >>>> amoghdesai.oss@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Thanks for starting this discussion, Ash.
> > >>>>>>>>>>
> > >>>>>>>>>> I would prefer option 2 here with proper tooling to handle the
> > code
> > >>>>>>>>>> duplication at *release* time.
> > >>>>>>>>>> It is best to have a dist that has all it needs in itself.
> > >>>>>>>>>>
> > >>>>>>>>>> Option 1 could very quickly get out of hand and if we decide
> to
> > >>>>>> separate
> > >>>>>>>>>> triggerer /
> > >>>>>>>>>> dag processor / config etc etc as separate packages, back
> > compat is
> > >>>>>>>> going
> > >>>>>>>>>> to be a nightmare
> > >>>>>>>>>> and will bite us harder than we anticipate.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks & Regards,
> > >>>>>>>>>> Amogh Desai
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <
> kaxilnaik@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>>>>> I prefer Option 2 as well to avoid matrix of dependencies
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
> > >>>>>> <j_scheffler@gmx.de.invalid
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> I'd also rather prefer option 2 - reason here is it is
> rather
> > >>>>>>>> pragmatic
> > >>>>>>>>>>>> and we no not need to cut another package and have less
> > package
> > >>>>>> counts
> > >>>>>>>>>>>> and dependencies.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I remember some time ago I was checking (together with
> Jarek,
> > I am
> > >>>>>> not
> > >>>>>>>>>>>> sure anymore...) if the usage of symlinks would be possible.
> > To
> > >>>> keep
> > >>>>>>>>>> the
> > >>>>>>>>>>>> source in one package but "symlink" it into another. If then
> > at
> > >>>>>> point
> > >>>>>>>>>> of
> > >>>>>>>>>>>> packaging/release the files are materialized we have 1 set
> of
> > >>>> code.
> > >>>>>>>>>>>> Otherwise if not possible still the redundancy could be
> > solved by
> > >>>> a
> > >>>>>>>>>>>> pre-commit hook - and in Git the files are de-duplicated
> > anyway
> > >>>>>> based
> > >>>>>>>>>> on
> > >>>>>>>>>>>> content hash, so this does not hurt.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
> > >>>>>>>>>>>>> I support option 2 with proper automation & CI - the
> > reasonings
> > >>>>>>>>>> you've
> > >>>>>>>>>>>>> shown for that make sense to me.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Shahar
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <
> > ash@apache.org
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> As we work on finishing off the code-level separation of
> > Task
> > >>>> SDK
> > >>>>>>>>>> and
> > >>>>>>>>>>>> Core
> > >>>>>>>>>>>>>> (scheduler etc) we have come across some situations where
> we
> > >>>> would
> > >>>>>>>>>>> like
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>> share code between these.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> However it’s not as straight forward of “just put it in a
> > common
> > >>>>>>>>>> dist
> > >>>>>>>>>>>> they
> > >>>>>>>>>>>>>> both depend upon” because one of the goals of the Task SDK
> > >>>>>>>>>> separation
> > >>>>>>>>>>>> was
> > >>>>>>>>>>>>>> to have 100% complete version independence between the
> two,
> > >>>>>> ideally
> > >>>>>>>>>>>> even if
> > >>>>>>>>>>>>>> they are built into the same image and venv. Most of the
> > reason
> > >>>>>> why
> > >>>>>>>>>>> this
> > >>>>>>>>>>>>>> isn’t straight forward comes down to backwards
> > compatibility -
> > >>>> if
> > >>>>>> we
> > >>>>>>>>>>>> make
> > >>>>>>>>>>>>>> an change to the common/shared distribution
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> We’ve listed the options we have thought about in
> > >>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545 (but that
> > covers
> > >>>>>>>>>> some
> > >>>>>>>>>>>> more
> > >>>>>>>>>>>>>> things that I don’t want to get in to in this discussion
> > such as
> > >>>>>>>>>>>> possibly
> > >>>>>>>>>>>>>> separating operators and executors out of a single
> provider
> > >>>> dist.)
> > >>>>>>>>>>>>>> To give a concrete example of some code I would like to
> > share
> > >>>>>>>>>>>>>>
> > >>>>
> >
> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
> > >>>>>>>>>>>>>> — logging config. Another thing we will want to share will
> > be
> > >>>> the
> > >>>>>>>>>>>>>> AirflowConfigParser class from airflow.configuration (but
> > >>>> notably:
> > >>>>>>>>>>> only
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>> parser class, _not_ the default config values, again, lets
> > not
> > >>>>>> dwell
> > >>>>>>>>>>> on
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>> specifics of that)
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> So to bring the options listed in the issue here for
> > discussion,
> > >>>>>>>>>>> broadly
> > >>>>>>>>>>>>>> speaking there are two high-level approaches:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 1. A single shared distribution
> > >>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> The advantage of Approach 1 is that we only have the code
> > in one
> > >>>>>>>>>>> place.
> > >>>>>>>>>>>>>> However for me, at least in this specific case of Logging
> > config
> > >>>>>> or
> > >>>>>>>>>>>>>> AirflowConfigParser class is that backwards compatibility
> is
> > >>>> much
> > >>>>>>>>>> much
> > >>>>>>>>>>>>>> harder.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> The main advantage of Approach 2 is the the code is
> released
> > >>>>>>>>>>>> with/embedded
> > >>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would contain
> the
> > >>>> right
> > >>>>>>>>>>>> version
> > >>>>>>>>>>>>>> of the logging config and ConfigParser etc). The downside
> is
> > >>>> that
> > >>>>>>>>>>> either
> > >>>>>>>>>>>>>> the code will need to be duplicated in the repo, or better
> > yet
> > >>>> it
> > >>>>>>>>>>> would
> > >>>>>>>>>>>>>> live in a single place in the repo, but some tooling (TBD)
> > will
> > >>>>>>>>>>>>>> automatically handle the duplication, either at commit
> > time, or
> > >>>> my
> > >>>>>>>>>>>>>> preference, at release time.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> For this kind of shared “utility” code I am very strongly
> > >>>> leaning
> > >>>>>>>>>>>> towards
> > >>>>>>>>>>>>>> option 2 with automation, as otherwise I think the
> backwards
> > >>>>>>>>>>>> compatibility
> > >>>>>>>>>>>>>> requirements would make it unworkable (very quickly over
> > time
> > >>>> the
> > >>>>>>>>>>>>>> combinations we would have to test would just be
> > unreasonable)
> > >>>>>> and I
> > >>>>>>>>>>>> don’t
> > >>>>>>>>>>>>>> feel confident we can have things as stable as we need to
> > really
> > >>>>>>>>>>> deliver
> > >>>>>>>>>>>>>> the version separation/independency I want to delivery
> with
> > >>>>>> AIP-72.
> > >>>>>>>>>>>>>> So unless someone feels very strongly about this, I will
> > come up
> > >>>>>>>>>> with
> > >>>>>>>>>>> a
> > >>>>>>>>>>>>>> draft PR for further discussion that will implement code
> > sharing
> > >>>>>> via
> > >>>>>>>>>>>>>> “vendoring” it at build time. I have an idea of how I can
> > >>>> achieve
> > >>>>>>>>>> this
> > >>>>>>>>>>>> so
> > >>>>>>>>>>>>>> we have a single version in the repo and it’ll work there,
> > but
> > >>>> at
> > >>>>>>>>>>>> runtime
> > >>>>>>>>>>>>>> we vendor it in to the shipped dist so it lives at
> something
> > >>>> like
> > >>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> In terms of repo layout, this likely means we would end up
> > with:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> airflow-core/pyproject.toml
> > >>>>>>>>>>>>>> airflow-core/src/
> > >>>>>>>>>>>>>> airflow-core/tests/
> > >>>>>>>>>>>>>> task-sdk/pyproject.toml
> > >>>>>>>>>>>>>> task-sdk/src/
> > >>>>>>>>>>>>>> task-sdk/tests/
> > >>>>>>>>>>>>>> airflow-common/src
> > >>>>>>>>>>>>>> airflow-common/tests/
> > >>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps would
> > be
> > >>>>>>>>>> included
> > >>>>>>>>>>>> in
> > >>>>>>>>>>>>>> the downstream projects. TBD.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thoughts and feedback welcomed.
> > >>>>>>>>>>>>
> > >>>>>>
> > ---------------------------------------------------------------------
> > >>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >>>>>>>>>>>> For additional commands, e-mail:
> dev-help@airflow.apache.org
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>>>
> > >>>>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Jarek Potiuk <ja...@potiuk.com>.
Looks good but I think we should find some better logical name for
core_and_sdk :)

pon., 7 lip 2025, 21:44 użytkownik Jens Scheffler
<j_...@gmx.de.invalid> napisał:

> Cool! Especially the "shared" folder with the ability to have
> N-combinations w/o exploding project repo root!
>
> On 07.07.25 14:43, Ash Berlin-Taylor wrote:
> > Oh, and all of this will be explain in shared/README.md
> >
> >> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org> wrote:
> >>
> >> Okay, so it seems we have agreement on the approach here, so I’ll
> continue with this, and on the dev call it was mentioned that
> “airflow-common” wasn’t a great name, so here is my proposal for the file
> structure;
> >>
> >> ```
> >> /
> >>   task-sdk/...
> >>   airflow-core/...
> >>   shared/
> >>     kuberenetes/
> >>       pyproject.toml
> >>       src/
> >>         airflow_kube/__init__.py
> >>     core-and-tasksdk/
> >>       pyproject.toml
> >>       src/
> >>         airflow_shared/__init__.py
> >> ```
> >>
> >> Things to note here: the “shared” folder has (the possibility) of
> having multiple different shared “libraries” in it, in this example I am
> supposing a hypothetical shared kuberenetes folder a world in which we
> split the KubePodOperator and the KubeExecutor in to two separate
> distributions (example only, not proposing we do that right now, and that
> will be a separate discussion)
> >>
> >> The other things to note here:
> >>
> >>
> >> - the folder name in shared aims to be “self-documenting”, hence the
> verbose “core-and-tasksdk” to say where the shared library is intended to
> be used.
> >> - the python module itself should almost always have an `airflow_` (or
> maybe `_airflow_`?) prefix so that it does not conflict with anything else
> we might use. It won’t matter “in production” as those will be vendored in
> to be imported as `airflow/_vendor/airflow_shared` etc, but avoiding
> conflicts at dev time with the Finder approach is a good safety measure.
> >>
> >> I will start making a real PR for this proposal now, but I’m open to
> feedback (either here, or in the PR when I open it)
> >>
> >> -ash
> >>
> >>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>> Yeah we have to try it and test - also building packages happens semi
> >>> frequently when you run `uv sync` (they use some kind of heuristics to
> >>> decide when) and you can force it with `--reinstall` or `--refresh`.
> >>> Package build also happens every time when you run "ci-image build`
> now in
> >>> breeze so it seems like it will nicely integrate in our workflows.
> >>>
> >>> Looks really cool Ash.
> >>>
> >>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >>>
> >>>> It’s not just release time, but any time we build a package which
> happens
> >>>> on “every” CI run. The normal unit tests will use code from
> >>>> airflow-common/src/airflow_common; the kube tests which build an
> image will
> >>>> build the dists and vendor in the code from that commit.
> >>>>
> >>>> There is only a single copy of the shared code committed to the repo,
> so
> >>>> there is never anything to synchronise.
> >>>>
> >>>>> On 4 Jul 2025, at 15:53, Amogh Desai <am...@gmail.com>
> wrote:
> >>>>>
> >>>>> Thanks Ash.
> >>>>>
> >>>>> This is really cool and helpful that you were able to test both
> scenarios
> >>>>> -- repo checkout
> >>>>> and also installing from the vendored package and the resolution
> worked
> >>>>> fine too.
> >>>>>
> >>>>> I like this idea compared the to relative import one for few reasons:
> >>>>> - It feels like it will take some time to adjust to the new coding
> >>>> standard
> >>>>> that we will lay
> >>>>> if we impose relative imports in the shared dist
> >>>>> - We can continue using repo wise absolute import standards, it is
> also
> >>>>> much easier for situations
> >>>>> when we do global search in IDE to find + replace, this could mean a
> >>>> change
> >>>>> there
> >>>>> - The vendoring work is a proven and established paradigm across
> projects
> >>>>> and would
> >>>>> out of box give us the build tooling we need also
> >>>>>
> >>>>> Nothing too against the relative import but with the evidence
> provided
> >>>>> above, vendored approach
> >>>>> seems to only do us good.
> >>>>>
> >>>>> Regarding synchronizing it, release time should be fine as long as we
> >>>> have
> >>>>> a good CI workflow to probably
> >>>>> catch such issues per PR if changes are made in shared dist?
> (precommit
> >>>>> would make it really slow i guess)
> >>>>>
> >>>>> If we can run our tests with vendored code we should be mostly
> covered.
> >>>>>
> >>>>> Good effort all!
> >>>>>
> >>>>> Thanks & Regards,
> >>>>> Amogh Desai
> >>>>>
> >>>>>
> >>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <as...@apache.org>
> >>>> wrote:
> >>>>>> Okay, I think I’ve got something that works and I’m happy with.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
> >>>>>> This produces the following from `uv build task-sdk`
> >>>>>> -
> >>>>>>
> >>>>
> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
> >>>>>> -
> >>>>>>
> >>>>
> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
> >>>>>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
> >>>>>>
> >>>>>> ```
> >>>>>> ❯ unzip -l dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
> >>>> grep
> >>>>>> _vendor
> >>>>>>      50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
> >>>>>>    2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
> >>>>>>      28  02-02-2020 00:00   airflow/sdk/_vendor/airflow_common.pyi
> >>>>>>      18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
> >>>>>>     785  02-02-2020 00:00
> >>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
> >>>>>>   10628  02-02-2020 00:00
> >>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
> >>>>>> ```
> >>>>>>
> >>>>>> And similarly in the .tar.gz, so our “sdist” is complete too:
> >>>>>> ```
> >>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep _vendor
> >>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
> >>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
> >>>>>>
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
> >>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
> >>>>>>
> >>>>>>
> >>>>
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
> >>>>>>
> >>>>
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
> >>>>>> ```
> >>>>>>
> >>>>>> The plugin works at build time by including/copying the libs
> specified
> >>>> in
> >>>>>> vendor.txt into place (and let `vendoring` take care of import
> >>>> rewrites.)
> >>>>>> For the imports to continue to work at “dev” time/from a repo
> checkout,
> >>>> I
> >>>>>> have added a import finder to `sys.meta_path`, and since it’s at the
> >>>> end of
> >>>>>> the list it will only be used if the normal import can’t find
> things.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
> >>>>>> This doesn’t quite give us the same runtime effect “import
> rewriting”
> >>>>>> affect, as in this approach `airflow_common` is directly loaded
> (i.e.
> >>>>>> airflow.sdk._vendor.airflow_common and airflow_common exist in
> >>>>>> sys.modules), but it does work for everything that I was able to
> test..
> >>>>>>
> >>>>>> I tested it with the diff at the end of this message. My test
> ipython
> >>>>>> shell:
> >>>>>>
> >>>>>> ```
> >>>>>> In [1]: from airflow.sdk._vendor.airflow_common.timezone import foo
> >>>>>>
> >>>>>> In [2]: foo
> >>>>>> Out[2]: 1
> >>>>>>
> >>>>>> In [3]: import airflow.sdk._vendor.airflow_common
> >>>>>>
> >>>>>> In [4]: import airflow.sdk._vendor.airflow_common.timezone
> >>>>>>
> >>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
> >>>>>> Out[5]:
> >>>>>>
> >>>>
> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
> >>>>>> In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
> >>>>>> Out[6]:
> >>>>>>
> >>>>
> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
> >>>>>> ```
> >>>>>>
> >>>>>>
> >>>>>> And in an standalone environment with the SDK dist I built (it
> needed
> >>>> the
> >>>>>> matching airflow-core right now, but that is nothing to do with this
> >>>>>> discussion):
> >>>>>>
> >>>>>> ```
> >>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
> >>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
> >>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
> >>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
> >>>>>> Type 'copyright', 'credits' or 'license' for more information
> >>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for help.
> >>>>>> Tip: You can use `%hist` to view history, see the options with
> >>>> `%history?`
> >>>>>> In [1]: import airflow.sdk._vendor.airflow_common.timezone
> >>>>>>
> >>>>>> In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
> >>>>>> Out[2]:
> >>>>>>
> >>>>
> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
> >>>>>> ``
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ```diff
> >>>>>> diff --git a/airflow-common/src/airflow_common/__init__.py
> >>>>>> b/airflow-common/src/airflow_common/__init__.py
> >>>>>> index 13a83393a9..927b7c6b61 100644
> >>>>>> --- a/airflow-common/src/airflow_common/__init__.py
> >>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
> >>>>>> @@ -14,3 +14,5 @@
> >>>>>> # KIND, either express or implied.  See the License for the
> >>>>>> # specific language governing permissions and limitations
> >>>>>> # under the License.
> >>>>>> +
> >>>>>> +foo = 1
> >>>>>> diff --git a/airflow-common/src/airflow_common/timezone.py
> >>>>>> b/airflow-common/src/airflow_common/timezone.py
> >>>>>> index 340b924c66..58384ef20f 100644
> >>>>>> --- a/airflow-common/src/airflow_common/timezone.py
> >>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
> >>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
> >>>>>> version.parse(metadata.version("pendulum")).major == 3
> >>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
> >>>>>> utc = pendulum.UTC
> >>>>>>
> >>>>>> +
> >>>>>> +from airflow_common import foo
> >>>>>> +
> >>>>>> TIMEZONE: Timezone
> >>>>>>
> >>>>>>
> >>>>>> ```
> >>>>>>
> >>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>>>>> I think both approaches are doable:
> >>>>>>>
> >>>>>>> 1) -> We can very easily prevent bad imports by pre-commit when
> >>>> importing
> >>>>>>> from different distributions and make sure we are only doing
> relative
> >>>>>>> imports in the shared modules. We are doing plenty of this
> already. And
> >>>>>> yes
> >>>>>>> it would require relative links we currently do not allow.
> >>>>>>>
> >>>>>>> 2) -> has one disadvantage that someone at some point in time will
> have
> >>>>>> to
> >>>>>>> decide to synchronize this and if it happens just before release
> (I bet
> >>>>>>> this is going to happen) this will lead to solving problems that
> would
> >>>>>>> normally be solved during PR when you make a change (i.e. symbolic
> link
> >>>>>> has
> >>>>>>> the advantage that whoever modifies shared code will be immediately
> >>>>>>> notified in their PR - that they broke something because either
> static
> >>>>>>> checks or mypy or tests fail.
> >>>>>>>
> >>>>>>> Ash, do you have an idea of a process (who and when) does the
> >>>>>>> synchronisation in case of vendoring? Maybe we could solve it if
> it is
> >>>>>> done
> >>>>>>> more frequently and with some regularity? We could potentially
> force
> >>>>>>> re-vendoring at PR time as well any time shared code changes (and
> >>>> prevent
> >>>>>>> it by pre-commit. And I can't think of some place (other than
> releases)
> >>>>>> in
> >>>>>>> our development workflow and that seems to be a bit too late as
> puts an
> >>>>>>> extra effort on fixing potential incompatibilities introduced on
> >>>> release
> >>>>>>> manager and delays the release. WDYT?
> >>>>>>>
> >>>>>>> Re: relative links. I think for a shared library we could
> potentially
> >>>>>> relax
> >>>>>>> this and allow them (and actually disallow absolute links in the
> pieces
> >>>>>> of
> >>>>>>> code that are shared - again, by pre-commit). As I recall, the only
> >>>>>> reason
> >>>>>>> we forbade the relative links is because of how we are (or maybe
> were)
> >>>>>>> doing DAG parsing and failures resulting from it. So we decided to
> just
> >>>>>> not
> >>>>>>> allow it to keep consistency. The way how Dag parsing works is that
> >>>> when
> >>>>>>> you are using importlib to read the Dag from a file, the relative
> >>>> imports
> >>>>>>> do not work as it does not know what they should be relative to.
> But if
> >>>>>>> relative import is done from an imported package, it should be no
> >>>>>> problem,
> >>>>>>> I think - otherwise our Dags would not be able to import any
> library
> >>>> that
> >>>>>>> uses relative imports.
> >>>>>>>
> >>>>>>> Of course consistency might be the reason why we do not want to
> >>>> introduce
> >>>>>>> relative imports. I don't see it as an issue if it is guarded by
> >>>>>> pre-commit
> >>>>>>> though.
> >>>>>>>
> >>>>>>> J.
> >>>>>>>
> >>>>>>>
> >>>>>>> J.
> >>>>>>>
> >>>>>>>
> >>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <
> ash@apache.org>
> >>>>>>> napisał:
> >>>>>>>
> >>>>>>>> Oh yes, symlinks will work, with one big caveat: It does mean you
> >>>> can’t
> >>>>>>>> use absolute imports in one common module to another.
> >>>>>>>>
> >>>>>>>> For example
> >>>>>>>>
> >>>>
> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
> >>>>>>>> where we have
> >>>>>>>>
> >>>>>>>> ```
> >>>>>>>> from airflow.utils.module_loading import import_string
> >>>>>>>> ```
> >>>>>>>>
> >>>>>>>> if we want to move serve_logs into this common lib that is then
> >>>>>> symlinked
> >>>>>>>> then we wouldn’t be able to have `from
> airflow_common.module_loading
> >>>>>> import
> >>>>>>>> import_string`.
> >>>>>>>>
> >>>>>>>> I can think of two possible solutions here.
> >>>>>>>>
> >>>>>>>> 1) is to allow/require relative imports in this shared lib, so
> `from
> >>>>>>>> .module_loading import import_string`
> >>>>>>>> 2) is to use `vendoring`[1] (from the pip maintainers) which will
> >>>> handle
> >>>>>>>> import-rewriting for us.
> >>>>>>>>
> >>>>>>>> I’d entirely forgot that symlinks in repos was a thing, so I
> prepared
> >>>> a
> >>>>>>>> minimal POC/demo of what vendoring approach could look like here
> >>>>>>>>
> >>>>
> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
> >>>>>>>> Now personally I am more than happy with relative imports, but
> >>>> generally
> >>>>>>>> as a project we have avoided them, so I think that limits what we
> >>>> could
> >>>>>> do
> >>>>>>>> with a symlink based approach.
> >>>>>>>>
> >>>>>>>> -ash
> >>>>>>>>
> >>>>>>>> [1] https://github.com/pradyunsg/vendoring
> >>>>>>>>
> >>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
> >>>> gopidesupavan@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>> Thanks Ash
> >>>>>>>>>
> >>>>>>>>> Yes agree option 2 would be preferred for me. Making sure we
> have all
> >>>>>> the
> >>>>>>>>> gaurdriles to protect any unwanted behaviour in code sharing and
> >>>>>>>> executing
> >>>>>>>>> right of tests between the packages.
> >>>>>>>>>
> >>>>>>>>> Agree with others, option 2 would be
> >>>>>>>>>
> >>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
> >>>> amoghdesai.oss@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Thanks for starting this discussion, Ash.
> >>>>>>>>>>
> >>>>>>>>>> I would prefer option 2 here with proper tooling to handle the
> code
> >>>>>>>>>> duplication at *release* time.
> >>>>>>>>>> It is best to have a dist that has all it needs in itself.
> >>>>>>>>>>
> >>>>>>>>>> Option 1 could very quickly get out of hand and if we decide to
> >>>>>> separate
> >>>>>>>>>> triggerer /
> >>>>>>>>>> dag processor / config etc etc as separate packages, back
> compat is
> >>>>>>>> going
> >>>>>>>>>> to be a nightmare
> >>>>>>>>>> and will bite us harder than we anticipate.
> >>>>>>>>>>
> >>>>>>>>>> Thanks & Regards,
> >>>>>>>>>> Amogh Desai
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <ka...@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>> I prefer Option 2 as well to avoid matrix of dependencies
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
> >>>>>> <j_scheffler@gmx.de.invalid
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> I'd also rather prefer option 2 - reason here is it is rather
> >>>>>>>> pragmatic
> >>>>>>>>>>>> and we no not need to cut another package and have less
> package
> >>>>>> counts
> >>>>>>>>>>>> and dependencies.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I remember some time ago I was checking (together with Jarek,
> I am
> >>>>>> not
> >>>>>>>>>>>> sure anymore...) if the usage of symlinks would be possible.
> To
> >>>> keep
> >>>>>>>>>> the
> >>>>>>>>>>>> source in one package but "symlink" it into another. If then
> at
> >>>>>> point
> >>>>>>>>>> of
> >>>>>>>>>>>> packaging/release the files are materialized we have 1 set of
> >>>> code.
> >>>>>>>>>>>> Otherwise if not possible still the redundancy could be
> solved by
> >>>> a
> >>>>>>>>>>>> pre-commit hook - and in Git the files are de-duplicated
> anyway
> >>>>>> based
> >>>>>>>>>> on
> >>>>>>>>>>>> content hash, so this does not hurt.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
> >>>>>>>>>>>>> I support option 2 with proper automation & CI - the
> reasonings
> >>>>>>>>>> you've
> >>>>>>>>>>>>> shown for that make sense to me.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Shahar
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <
> ash@apache.org
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> As we work on finishing off the code-level separation of
> Task
> >>>> SDK
> >>>>>>>>>> and
> >>>>>>>>>>>> Core
> >>>>>>>>>>>>>> (scheduler etc) we have come across some situations where we
> >>>> would
> >>>>>>>>>>> like
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> share code between these.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> However it’s not as straight forward of “just put it in a
> common
> >>>>>>>>>> dist
> >>>>>>>>>>>> they
> >>>>>>>>>>>>>> both depend upon” because one of the goals of the Task SDK
> >>>>>>>>>> separation
> >>>>>>>>>>>> was
> >>>>>>>>>>>>>> to have 100% complete version independence between the two,
> >>>>>> ideally
> >>>>>>>>>>>> even if
> >>>>>>>>>>>>>> they are built into the same image and venv. Most of the
> reason
> >>>>>> why
> >>>>>>>>>>> this
> >>>>>>>>>>>>>> isn’t straight forward comes down to backwards
> compatibility -
> >>>> if
> >>>>>> we
> >>>>>>>>>>>> make
> >>>>>>>>>>>>>> an change to the common/shared distribution
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We’ve listed the options we have thought about in
> >>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545 (but that
> covers
> >>>>>>>>>> some
> >>>>>>>>>>>> more
> >>>>>>>>>>>>>> things that I don’t want to get in to in this discussion
> such as
> >>>>>>>>>>>> possibly
> >>>>>>>>>>>>>> separating operators and executors out of a single provider
> >>>> dist.)
> >>>>>>>>>>>>>> To give a concrete example of some code I would like to
> share
> >>>>>>>>>>>>>>
> >>>>
> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
> >>>>>>>>>>>>>> — logging config. Another thing we will want to share will
> be
> >>>> the
> >>>>>>>>>>>>>> AirflowConfigParser class from airflow.configuration (but
> >>>> notably:
> >>>>>>>>>>> only
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> parser class, _not_ the default config values, again, lets
> not
> >>>>>> dwell
> >>>>>>>>>>> on
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>> specifics of that)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> So to bring the options listed in the issue here for
> discussion,
> >>>>>>>>>>> broadly
> >>>>>>>>>>>>>> speaking there are two high-level approaches:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 1. A single shared distribution
> >>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The advantage of Approach 1 is that we only have the code
> in one
> >>>>>>>>>>> place.
> >>>>>>>>>>>>>> However for me, at least in this specific case of Logging
> config
> >>>>>> or
> >>>>>>>>>>>>>> AirflowConfigParser class is that backwards compatibility is
> >>>> much
> >>>>>>>>>> much
> >>>>>>>>>>>>>> harder.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The main advantage of Approach 2 is the the code is released
> >>>>>>>>>>>> with/embedded
> >>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would contain the
> >>>> right
> >>>>>>>>>>>> version
> >>>>>>>>>>>>>> of the logging config and ConfigParser etc). The downside is
> >>>> that
> >>>>>>>>>>> either
> >>>>>>>>>>>>>> the code will need to be duplicated in the repo, or better
> yet
> >>>> it
> >>>>>>>>>>> would
> >>>>>>>>>>>>>> live in a single place in the repo, but some tooling (TBD)
> will
> >>>>>>>>>>>>>> automatically handle the duplication, either at commit
> time, or
> >>>> my
> >>>>>>>>>>>>>> preference, at release time.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> For this kind of shared “utility” code I am very strongly
> >>>> leaning
> >>>>>>>>>>>> towards
> >>>>>>>>>>>>>> option 2 with automation, as otherwise I think the backwards
> >>>>>>>>>>>> compatibility
> >>>>>>>>>>>>>> requirements would make it unworkable (very quickly over
> time
> >>>> the
> >>>>>>>>>>>>>> combinations we would have to test would just be
> unreasonable)
> >>>>>> and I
> >>>>>>>>>>>> don’t
> >>>>>>>>>>>>>> feel confident we can have things as stable as we need to
> really
> >>>>>>>>>>> deliver
> >>>>>>>>>>>>>> the version separation/independency I want to delivery with
> >>>>>> AIP-72.
> >>>>>>>>>>>>>> So unless someone feels very strongly about this, I will
> come up
> >>>>>>>>>> with
> >>>>>>>>>>> a
> >>>>>>>>>>>>>> draft PR for further discussion that will implement code
> sharing
> >>>>>> via
> >>>>>>>>>>>>>> “vendoring” it at build time. I have an idea of how I can
> >>>> achieve
> >>>>>>>>>> this
> >>>>>>>>>>>> so
> >>>>>>>>>>>>>> we have a single version in the repo and it’ll work there,
> but
> >>>> at
> >>>>>>>>>>>> runtime
> >>>>>>>>>>>>>> we vendor it in to the shipped dist so it lives at something
> >>>> like
> >>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> In terms of repo layout, this likely means we would end up
> with:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> airflow-core/pyproject.toml
> >>>>>>>>>>>>>> airflow-core/src/
> >>>>>>>>>>>>>> airflow-core/tests/
> >>>>>>>>>>>>>> task-sdk/pyproject.toml
> >>>>>>>>>>>>>> task-sdk/src/
> >>>>>>>>>>>>>> task-sdk/tests/
> >>>>>>>>>>>>>> airflow-common/src
> >>>>>>>>>>>>>> airflow-common/tests/
> >>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps would
> be
> >>>>>>>>>> included
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>> the downstream projects. TBD.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thoughts and feedback welcomed.
> >>>>>>>>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>>>>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>
> >>>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> For additional commands, e-mail: dev-help@airflow.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Jens Scheffler <j_...@gmx.de.INVALID>.
Cool! Especially the "shared" folder with the ability to have 
N-combinations w/o exploding project repo root!

On 07.07.25 14:43, Ash Berlin-Taylor wrote:
> Oh, and all of this will be explain in shared/README.md
>
>> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>> Okay, so it seems we have agreement on the approach here, so I’ll continue with this, and on the dev call it was mentioned that “airflow-common” wasn’t a great name, so here is my proposal for the file structure;
>>
>> ```
>> /
>>   task-sdk/...
>>   airflow-core/...
>>   shared/
>>     kuberenetes/
>>       pyproject.toml
>>       src/
>>         airflow_kube/__init__.py
>>     core-and-tasksdk/
>>       pyproject.toml
>>       src/
>>         airflow_shared/__init__.py
>> ```
>>
>> Things to note here: the “shared” folder has (the possibility) of having multiple different shared “libraries” in it, in this example I am supposing a hypothetical shared kuberenetes folder a world in which we split the KubePodOperator and the KubeExecutor in to two separate distributions (example only, not proposing we do that right now, and that will be a separate discussion)
>>
>> The other things to note here:
>>
>>
>> - the folder name in shared aims to be “self-documenting”, hence the verbose “core-and-tasksdk” to say where the shared library is intended to be used.
>> - the python module itself should almost always have an `airflow_` (or maybe `_airflow_`?) prefix so that it does not conflict with anything else we might use. It won’t matter “in production” as those will be vendored in to be imported as `airflow/_vendor/airflow_shared` etc, but avoiding conflicts at dev time with the Finder approach is a good safety measure.
>>
>> I will start making a real PR for this proposal now, but I’m open to feedback (either here, or in the PR when I open it)
>>
>> -ash
>>
>>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>> Yeah we have to try it and test - also building packages happens semi
>>> frequently when you run `uv sync` (they use some kind of heuristics to
>>> decide when) and you can force it with `--reinstall` or `--refresh`.
>>> Package build also happens every time when you run "ci-image build` now in
>>> breeze so it seems like it will nicely integrate in our workflows.
>>>
>>> Looks really cool Ash.
>>>
>>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>>>
>>>> It’s not just release time, but any time we build a package which happens
>>>> on “every” CI run. The normal unit tests will use code from
>>>> airflow-common/src/airflow_common; the kube tests which build an image will
>>>> build the dists and vendor in the code from that commit.
>>>>
>>>> There is only a single copy of the shared code committed to the repo, so
>>>> there is never anything to synchronise.
>>>>
>>>>> On 4 Jul 2025, at 15:53, Amogh Desai <am...@gmail.com> wrote:
>>>>>
>>>>> Thanks Ash.
>>>>>
>>>>> This is really cool and helpful that you were able to test both scenarios
>>>>> -- repo checkout
>>>>> and also installing from the vendored package and the resolution worked
>>>>> fine too.
>>>>>
>>>>> I like this idea compared the to relative import one for few reasons:
>>>>> - It feels like it will take some time to adjust to the new coding
>>>> standard
>>>>> that we will lay
>>>>> if we impose relative imports in the shared dist
>>>>> - We can continue using repo wise absolute import standards, it is also
>>>>> much easier for situations
>>>>> when we do global search in IDE to find + replace, this could mean a
>>>> change
>>>>> there
>>>>> - The vendoring work is a proven and established paradigm across projects
>>>>> and would
>>>>> out of box give us the build tooling we need also
>>>>>
>>>>> Nothing too against the relative import but with the evidence provided
>>>>> above, vendored approach
>>>>> seems to only do us good.
>>>>>
>>>>> Regarding synchronizing it, release time should be fine as long as we
>>>> have
>>>>> a good CI workflow to probably
>>>>> catch such issues per PR if changes are made in shared dist? (precommit
>>>>> would make it really slow i guess)
>>>>>
>>>>> If we can run our tests with vendored code we should be mostly covered.
>>>>>
>>>>> Good effort all!
>>>>>
>>>>> Thanks & Regards,
>>>>> Amogh Desai
>>>>>
>>>>>
>>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <as...@apache.org>
>>>> wrote:
>>>>>> Okay, I think I’ve got something that works and I’m happy with.
>>>>>>
>>>>>>
>>>>>>
>>>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>>>>>> This produces the following from `uv build task-sdk`
>>>>>> -
>>>>>>
>>>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>>> -
>>>>>>
>>>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>>>>>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
>>>>>>
>>>>>> ```
>>>>>> ❯ unzip -l dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
>>>> grep
>>>>>> _vendor
>>>>>>      50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
>>>>>>    2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
>>>>>>      28  02-02-2020 00:00   airflow/sdk/_vendor/airflow_common.pyi
>>>>>>      18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
>>>>>>     785  02-02-2020 00:00
>>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>>   10628  02-02-2020 00:00
>>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>> ```
>>>>>>
>>>>>> And similarly in the .tar.gz, so our “sdist” is complete too:
>>>>>> ```
>>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep _vendor
>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>>>>>>
>>>>>>
>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>>
>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>>>>>> ```
>>>>>>
>>>>>> The plugin works at build time by including/copying the libs specified
>>>> in
>>>>>> vendor.txt into place (and let `vendoring` take care of import
>>>> rewrites.)
>>>>>> For the imports to continue to work at “dev” time/from a repo checkout,
>>>> I
>>>>>> have added a import finder to `sys.meta_path`, and since it’s at the
>>>> end of
>>>>>> the list it will only be used if the normal import can’t find things.
>>>>>>
>>>>>>
>>>>>>
>>>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>>>>>> This doesn’t quite give us the same runtime effect “import rewriting”
>>>>>> affect, as in this approach `airflow_common` is directly loaded (i.e.
>>>>>> airflow.sdk._vendor.airflow_common and airflow_common exist in
>>>>>> sys.modules), but it does work for everything that I was able to test..
>>>>>>
>>>>>> I tested it with the diff at the end of this message. My test ipython
>>>>>> shell:
>>>>>>
>>>>>> ```
>>>>>> In [1]: from airflow.sdk._vendor.airflow_common.timezone import foo
>>>>>>
>>>>>> In [2]: foo
>>>>>> Out[2]: 1
>>>>>>
>>>>>> In [3]: import airflow.sdk._vendor.airflow_common
>>>>>>
>>>>>> In [4]: import airflow.sdk._vendor.airflow_common.timezone
>>>>>>
>>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>>>>>> Out[5]:
>>>>>>
>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>>>>>> In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>> Out[6]:
>>>>>>
>>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>>>>>> ```
>>>>>>
>>>>>>
>>>>>> And in an standalone environment with the SDK dist I built (it needed
>>>> the
>>>>>> matching airflow-core right now, but that is nothing to do with this
>>>>>> discussion):
>>>>>>
>>>>>> ```
>>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
>>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
>>>>>> Type 'copyright', 'credits' or 'license' for more information
>>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for help.
>>>>>> Tip: You can use `%hist` to view history, see the options with
>>>> `%history?`
>>>>>> In [1]: import airflow.sdk._vendor.airflow_common.timezone
>>>>>>
>>>>>> In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>>> Out[2]:
>>>>>>
>>>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>>>>>> ``
>>>>>>
>>>>>>
>>>>>>
>>>>>> ```diff
>>>>>> diff --git a/airflow-common/src/airflow_common/__init__.py
>>>>>> b/airflow-common/src/airflow_common/__init__.py
>>>>>> index 13a83393a9..927b7c6b61 100644
>>>>>> --- a/airflow-common/src/airflow_common/__init__.py
>>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
>>>>>> @@ -14,3 +14,5 @@
>>>>>> # KIND, either express or implied.  See the License for the
>>>>>> # specific language governing permissions and limitations
>>>>>> # under the License.
>>>>>> +
>>>>>> +foo = 1
>>>>>> diff --git a/airflow-common/src/airflow_common/timezone.py
>>>>>> b/airflow-common/src/airflow_common/timezone.py
>>>>>> index 340b924c66..58384ef20f 100644
>>>>>> --- a/airflow-common/src/airflow_common/timezone.py
>>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
>>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>>>>>> version.parse(metadata.version("pendulum")).major == 3
>>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
>>>>>> utc = pendulum.UTC
>>>>>>
>>>>>> +
>>>>>> +from airflow_common import foo
>>>>>> +
>>>>>> TIMEZONE: Timezone
>>>>>>
>>>>>>
>>>>>> ```
>>>>>>
>>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>>> I think both approaches are doable:
>>>>>>>
>>>>>>> 1) -> We can very easily prevent bad imports by pre-commit when
>>>> importing
>>>>>>> from different distributions and make sure we are only doing relative
>>>>>>> imports in the shared modules. We are doing plenty of this already. And
>>>>>> yes
>>>>>>> it would require relative links we currently do not allow.
>>>>>>>
>>>>>>> 2) -> has one disadvantage that someone at some point in time will have
>>>>>> to
>>>>>>> decide to synchronize this and if it happens just before release (I bet
>>>>>>> this is going to happen) this will lead to solving problems that would
>>>>>>> normally be solved during PR when you make a change (i.e. symbolic link
>>>>>> has
>>>>>>> the advantage that whoever modifies shared code will be immediately
>>>>>>> notified in their PR - that they broke something because either static
>>>>>>> checks or mypy or tests fail.
>>>>>>>
>>>>>>> Ash, do you have an idea of a process (who and when) does the
>>>>>>> synchronisation in case of vendoring? Maybe we could solve it if it is
>>>>>> done
>>>>>>> more frequently and with some regularity? We could potentially force
>>>>>>> re-vendoring at PR time as well any time shared code changes (and
>>>> prevent
>>>>>>> it by pre-commit. And I can't think of some place (other than releases)
>>>>>> in
>>>>>>> our development workflow and that seems to be a bit too late as puts an
>>>>>>> extra effort on fixing potential incompatibilities introduced on
>>>> release
>>>>>>> manager and delays the release. WDYT?
>>>>>>>
>>>>>>> Re: relative links. I think for a shared library we could potentially
>>>>>> relax
>>>>>>> this and allow them (and actually disallow absolute links in the pieces
>>>>>> of
>>>>>>> code that are shared - again, by pre-commit). As I recall, the only
>>>>>> reason
>>>>>>> we forbade the relative links is because of how we are (or maybe were)
>>>>>>> doing DAG parsing and failures resulting from it. So we decided to just
>>>>>> not
>>>>>>> allow it to keep consistency. The way how Dag parsing works is that
>>>> when
>>>>>>> you are using importlib to read the Dag from a file, the relative
>>>> imports
>>>>>>> do not work as it does not know what they should be relative to. But if
>>>>>>> relative import is done from an imported package, it should be no
>>>>>> problem,
>>>>>>> I think - otherwise our Dags would not be able to import any library
>>>> that
>>>>>>> uses relative imports.
>>>>>>>
>>>>>>> Of course consistency might be the reason why we do not want to
>>>> introduce
>>>>>>> relative imports. I don't see it as an issue if it is guarded by
>>>>>> pre-commit
>>>>>>> though.
>>>>>>>
>>>>>>> J.
>>>>>>>
>>>>>>>
>>>>>>> J.
>>>>>>>
>>>>>>>
>>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <as...@apache.org>
>>>>>>> napisał:
>>>>>>>
>>>>>>>> Oh yes, symlinks will work, with one big caveat: It does mean you
>>>> can’t
>>>>>>>> use absolute imports in one common module to another.
>>>>>>>>
>>>>>>>> For example
>>>>>>>>
>>>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>>>>>>>> where we have
>>>>>>>>
>>>>>>>> ```
>>>>>>>> from airflow.utils.module_loading import import_string
>>>>>>>> ```
>>>>>>>>
>>>>>>>> if we want to move serve_logs into this common lib that is then
>>>>>> symlinked
>>>>>>>> then we wouldn’t be able to have `from airflow_common.module_loading
>>>>>> import
>>>>>>>> import_string`.
>>>>>>>>
>>>>>>>> I can think of two possible solutions here.
>>>>>>>>
>>>>>>>> 1) is to allow/require relative imports in this shared lib, so `from
>>>>>>>> .module_loading import import_string`
>>>>>>>> 2) is to use `vendoring`[1] (from the pip maintainers) which will
>>>> handle
>>>>>>>> import-rewriting for us.
>>>>>>>>
>>>>>>>> I’d entirely forgot that symlinks in repos was a thing, so I prepared
>>>> a
>>>>>>>> minimal POC/demo of what vendoring approach could look like here
>>>>>>>>
>>>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>>>>>>>> Now personally I am more than happy with relative imports, but
>>>> generally
>>>>>>>> as a project we have avoided them, so I think that limits what we
>>>> could
>>>>>> do
>>>>>>>> with a symlink based approach.
>>>>>>>>
>>>>>>>> -ash
>>>>>>>>
>>>>>>>> [1] https://github.com/pradyunsg/vendoring
>>>>>>>>
>>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
>>>> gopidesupavan@gmail.com>
>>>>>>>> wrote:
>>>>>>>>> Thanks Ash
>>>>>>>>>
>>>>>>>>> Yes agree option 2 would be preferred for me. Making sure we have all
>>>>>> the
>>>>>>>>> gaurdriles to protect any unwanted behaviour in code sharing and
>>>>>>>> executing
>>>>>>>>> right of tests between the packages.
>>>>>>>>>
>>>>>>>>> Agree with others, option 2 would be
>>>>>>>>>
>>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
>>>> amoghdesai.oss@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for starting this discussion, Ash.
>>>>>>>>>>
>>>>>>>>>> I would prefer option 2 here with proper tooling to handle the code
>>>>>>>>>> duplication at *release* time.
>>>>>>>>>> It is best to have a dist that has all it needs in itself.
>>>>>>>>>>
>>>>>>>>>> Option 1 could very quickly get out of hand and if we decide to
>>>>>> separate
>>>>>>>>>> triggerer /
>>>>>>>>>> dag processor / config etc etc as separate packages, back compat is
>>>>>>>> going
>>>>>>>>>> to be a nightmare
>>>>>>>>>> and will bite us harder than we anticipate.
>>>>>>>>>>
>>>>>>>>>> Thanks & Regards,
>>>>>>>>>> Amogh Desai
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <ka...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>> I prefer Option 2 as well to avoid matrix of dependencies
>>>>>>>>>>>
>>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>>>>>> <j_scheffler@gmx.de.invalid
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I'd also rather prefer option 2 - reason here is it is rather
>>>>>>>> pragmatic
>>>>>>>>>>>> and we no not need to cut another package and have less package
>>>>>> counts
>>>>>>>>>>>> and dependencies.
>>>>>>>>>>>>
>>>>>>>>>>>> I remember some time ago I was checking (together with Jarek, I am
>>>>>> not
>>>>>>>>>>>> sure anymore...) if the usage of symlinks would be possible. To
>>>> keep
>>>>>>>>>> the
>>>>>>>>>>>> source in one package but "symlink" it into another. If then at
>>>>>> point
>>>>>>>>>> of
>>>>>>>>>>>> packaging/release the files are materialized we have 1 set of
>>>> code.
>>>>>>>>>>>> Otherwise if not possible still the redundancy could be solved by
>>>> a
>>>>>>>>>>>> pre-commit hook - and in Git the files are de-duplicated anyway
>>>>>> based
>>>>>>>>>> on
>>>>>>>>>>>> content hash, so this does not hurt.
>>>>>>>>>>>>
>>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>>>>>>>>>>> I support option 2 with proper automation & CI - the reasonings
>>>>>>>>>> you've
>>>>>>>>>>>>> shown for that make sense to me.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Shahar
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <ash@apache.org
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As we work on finishing off the code-level separation of Task
>>>> SDK
>>>>>>>>>> and
>>>>>>>>>>>> Core
>>>>>>>>>>>>>> (scheduler etc) we have come across some situations where we
>>>> would
>>>>>>>>>>> like
>>>>>>>>>>>> to
>>>>>>>>>>>>>> share code between these.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> However it’s not as straight forward of “just put it in a common
>>>>>>>>>> dist
>>>>>>>>>>>> they
>>>>>>>>>>>>>> both depend upon” because one of the goals of the Task SDK
>>>>>>>>>> separation
>>>>>>>>>>>> was
>>>>>>>>>>>>>> to have 100% complete version independence between the two,
>>>>>> ideally
>>>>>>>>>>>> even if
>>>>>>>>>>>>>> they are built into the same image and venv. Most of the reason
>>>>>> why
>>>>>>>>>>> this
>>>>>>>>>>>>>> isn’t straight forward comes down to backwards compatibility -
>>>> if
>>>>>> we
>>>>>>>>>>>> make
>>>>>>>>>>>>>> an change to the common/shared distribution
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We’ve listed the options we have thought about in
>>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545 (but that covers
>>>>>>>>>> some
>>>>>>>>>>>> more
>>>>>>>>>>>>>> things that I don’t want to get in to in this discussion such as
>>>>>>>>>>>> possibly
>>>>>>>>>>>>>> separating operators and executors out of a single provider
>>>> dist.)
>>>>>>>>>>>>>> To give a concrete example of some code I would like to share
>>>>>>>>>>>>>>
>>>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>>>>>>>>>>>> — logging config. Another thing we will want to share will be
>>>> the
>>>>>>>>>>>>>> AirflowConfigParser class from airflow.configuration (but
>>>> notably:
>>>>>>>>>>> only
>>>>>>>>>>>> the
>>>>>>>>>>>>>> parser class, _not_ the default config values, again, lets not
>>>>>> dwell
>>>>>>>>>>> on
>>>>>>>>>>>> the
>>>>>>>>>>>>>> specifics of that)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So to bring the options listed in the issue here for discussion,
>>>>>>>>>>> broadly
>>>>>>>>>>>>>> speaking there are two high-level approaches:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. A single shared distribution
>>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The advantage of Approach 1 is that we only have the code in one
>>>>>>>>>>> place.
>>>>>>>>>>>>>> However for me, at least in this specific case of Logging config
>>>>>> or
>>>>>>>>>>>>>> AirflowConfigParser class is that backwards compatibility is
>>>> much
>>>>>>>>>> much
>>>>>>>>>>>>>> harder.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The main advantage of Approach 2 is the the code is released
>>>>>>>>>>>> with/embedded
>>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would contain the
>>>> right
>>>>>>>>>>>> version
>>>>>>>>>>>>>> of the logging config and ConfigParser etc). The downside is
>>>> that
>>>>>>>>>>> either
>>>>>>>>>>>>>> the code will need to be duplicated in the repo, or better yet
>>>> it
>>>>>>>>>>> would
>>>>>>>>>>>>>> live in a single place in the repo, but some tooling (TBD) will
>>>>>>>>>>>>>> automatically handle the duplication, either at commit time, or
>>>> my
>>>>>>>>>>>>>> preference, at release time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For this kind of shared “utility” code I am very strongly
>>>> leaning
>>>>>>>>>>>> towards
>>>>>>>>>>>>>> option 2 with automation, as otherwise I think the backwards
>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>> requirements would make it unworkable (very quickly over time
>>>> the
>>>>>>>>>>>>>> combinations we would have to test would just be unreasonable)
>>>>>> and I
>>>>>>>>>>>> don’t
>>>>>>>>>>>>>> feel confident we can have things as stable as we need to really
>>>>>>>>>>> deliver
>>>>>>>>>>>>>> the version separation/independency I want to delivery with
>>>>>> AIP-72.
>>>>>>>>>>>>>> So unless someone feels very strongly about this, I will come up
>>>>>>>>>> with
>>>>>>>>>>> a
>>>>>>>>>>>>>> draft PR for further discussion that will implement code sharing
>>>>>> via
>>>>>>>>>>>>>> “vendoring” it at build time. I have an idea of how I can
>>>> achieve
>>>>>>>>>> this
>>>>>>>>>>>> so
>>>>>>>>>>>>>> we have a single version in the repo and it’ll work there, but
>>>> at
>>>>>>>>>>>> runtime
>>>>>>>>>>>>>> we vendor it in to the shipped dist so it lives at something
>>>> like
>>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In terms of repo layout, this likely means we would end up with:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> airflow-core/pyproject.toml
>>>>>>>>>>>>>> airflow-core/src/
>>>>>>>>>>>>>> airflow-core/tests/
>>>>>>>>>>>>>> task-sdk/pyproject.toml
>>>>>>>>>>>>>> task-sdk/src/
>>>>>>>>>>>>>> task-sdk/tests/
>>>>>>>>>>>>>> airflow-common/src
>>>>>>>>>>>>>> airflow-common/tests/
>>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps would be
>>>>>>>>>> included
>>>>>>>>>>>> in
>>>>>>>>>>>>>> the downstream projects. TBD.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thoughts and feedback welcomed.
>>>>>>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> For additional commands, e-mail: dev-help@airflow.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org


Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Ash Berlin-Taylor <as...@apache.org>.
Oh, and all of this will be explain in shared/README.md

> On 7 Jul 2025, at 13:41, Ash Berlin-Taylor <as...@apache.org> wrote:
> 
> Okay, so it seems we have agreement on the approach here, so I’ll continue with this, and on the dev call it was mentioned that “airflow-common” wasn’t a great name, so here is my proposal for the file structure;
> 
> ```
> /
>  task-sdk/...
>  airflow-core/...
>  shared/
>    kuberenetes/
>      pyproject.toml
>      src/
>        airflow_kube/__init__.py
>    core-and-tasksdk/
>      pyproject.toml
>      src/
>        airflow_shared/__init__.py
> ```
> 
> Things to note here: the “shared” folder has (the possibility) of having multiple different shared “libraries” in it, in this example I am supposing a hypothetical shared kuberenetes folder a world in which we split the KubePodOperator and the KubeExecutor in to two separate distributions (example only, not proposing we do that right now, and that will be a separate discussion)
> 
> The other things to note here:
> 
> 
> - the folder name in shared aims to be “self-documenting”, hence the verbose “core-and-tasksdk” to say where the shared library is intended to be used. 
> - the python module itself should almost always have an `airflow_` (or maybe `_airflow_`?) prefix so that it does not conflict with anything else we might use. It won’t matter “in production” as those will be vendored in to be imported as `airflow/_vendor/airflow_shared` etc, but avoiding conflicts at dev time with the Finder approach is a good safety measure.
> 
> I will start making a real PR for this proposal now, but I’m open to feedback (either here, or in the PR when I open it)
> 
> -ash
> 
>> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com> wrote:
>> 
>> Yeah we have to try it and test - also building packages happens semi
>> frequently when you run `uv sync` (they use some kind of heuristics to
>> decide when) and you can force it with `--reinstall` or `--refresh`.
>> Package build also happens every time when you run "ci-image build` now in
>> breeze so it seems like it will nicely integrate in our workflows.
>> 
>> Looks really cool Ash.
>> 
>> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>> 
>>> It’s not just release time, but any time we build a package which happens
>>> on “every” CI run. The normal unit tests will use code from
>>> airflow-common/src/airflow_common; the kube tests which build an image will
>>> build the dists and vendor in the code from that commit.
>>> 
>>> There is only a single copy of the shared code committed to the repo, so
>>> there is never anything to synchronise.
>>> 
>>>> On 4 Jul 2025, at 15:53, Amogh Desai <am...@gmail.com> wrote:
>>>> 
>>>> Thanks Ash.
>>>> 
>>>> This is really cool and helpful that you were able to test both scenarios
>>>> -- repo checkout
>>>> and also installing from the vendored package and the resolution worked
>>>> fine too.
>>>> 
>>>> I like this idea compared the to relative import one for few reasons:
>>>> - It feels like it will take some time to adjust to the new coding
>>> standard
>>>> that we will lay
>>>> if we impose relative imports in the shared dist
>>>> - We can continue using repo wise absolute import standards, it is also
>>>> much easier for situations
>>>> when we do global search in IDE to find + replace, this could mean a
>>> change
>>>> there
>>>> - The vendoring work is a proven and established paradigm across projects
>>>> and would
>>>> out of box give us the build tooling we need also
>>>> 
>>>> Nothing too against the relative import but with the evidence provided
>>>> above, vendored approach
>>>> seems to only do us good.
>>>> 
>>>> Regarding synchronizing it, release time should be fine as long as we
>>> have
>>>> a good CI workflow to probably
>>>> catch such issues per PR if changes are made in shared dist? (precommit
>>>> would make it really slow i guess)
>>>> 
>>>> If we can run our tests with vendored code we should be mostly covered.
>>>> 
>>>> Good effort all!
>>>> 
>>>> Thanks & Regards,
>>>> Amogh Desai
>>>> 
>>>> 
>>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <as...@apache.org>
>>> wrote:
>>>>> 
>>>>> Okay, I think I’ve got something that works and I’m happy with.
>>>>> 
>>>>> 
>>>>> 
>>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>>>>> 
>>>>> This produces the following from `uv build task-sdk`
>>>>> -
>>>>> 
>>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>>>>> -
>>>>> 
>>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>>>>> 
>>>>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
>>>>> 
>>>>> ```
>>>>> ❯ unzip -l dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
>>> grep
>>>>> _vendor
>>>>>     50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
>>>>>   2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
>>>>>     28  02-02-2020 00:00   airflow/sdk/_vendor/airflow_common.pyi
>>>>>     18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
>>>>>    785  02-02-2020 00:00
>>>>> airflow/sdk/_vendor/airflow_common/__init__.py
>>>>>  10628  02-02-2020 00:00
>>>>> airflow/sdk/_vendor/airflow_common/timezone.py
>>>>> ```
>>>>> 
>>>>> And similarly in the .tar.gz, so our “sdist” is complete too:
>>>>> ```
>>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep _vendor
>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>>>>> 
>>>>> 
>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>>>>> 
>>>>> 
>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>>>>> ```
>>>>> 
>>>>> The plugin works at build time by including/copying the libs specified
>>> in
>>>>> vendor.txt into place (and let `vendoring` take care of import
>>> rewrites.)
>>>>> 
>>>>> For the imports to continue to work at “dev” time/from a repo checkout,
>>> I
>>>>> have added a import finder to `sys.meta_path`, and since it’s at the
>>> end of
>>>>> the list it will only be used if the normal import can’t find things.
>>>>> 
>>>>> 
>>>>> 
>>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>>>>> 
>>>>> This doesn’t quite give us the same runtime effect “import rewriting”
>>>>> affect, as in this approach `airflow_common` is directly loaded (i.e.
>>>>> airflow.sdk._vendor.airflow_common and airflow_common exist in
>>>>> sys.modules), but it does work for everything that I was able to test..
>>>>> 
>>>>> I tested it with the diff at the end of this message. My test ipython
>>>>> shell:
>>>>> 
>>>>> ```
>>>>> In [1]: from airflow.sdk._vendor.airflow_common.timezone import foo
>>>>> 
>>>>> In [2]: foo
>>>>> Out[2]: 1
>>>>> 
>>>>> In [3]: import airflow.sdk._vendor.airflow_common
>>>>> 
>>>>> In [4]: import airflow.sdk._vendor.airflow_common.timezone
>>>>> 
>>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>>>>> Out[5]:
>>>>> 
>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>>>>> 
>>>>> In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>> Out[6]:
>>>>> 
>>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>>>>> 
>>>>> ```
>>>>> 
>>>>> 
>>>>> And in an standalone environment with the SDK dist I built (it needed
>>> the
>>>>> matching airflow-core right now, but that is nothing to do with this
>>>>> discussion):
>>>>> 
>>>>> ```
>>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
>>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
>>>>> Type 'copyright', 'credits' or 'license' for more information
>>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for help.
>>>>> Tip: You can use `%hist` to view history, see the options with
>>> `%history?`
>>>>> 
>>>>> In [1]: import airflow.sdk._vendor.airflow_common.timezone
>>>>> 
>>>>> In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
>>>>> Out[2]:
>>>>> 
>>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>>>>> ``
>>>>> 
>>>>> 
>>>>> 
>>>>> ```diff
>>>>> diff --git a/airflow-common/src/airflow_common/__init__.py
>>>>> b/airflow-common/src/airflow_common/__init__.py
>>>>> index 13a83393a9..927b7c6b61 100644
>>>>> --- a/airflow-common/src/airflow_common/__init__.py
>>>>> +++ b/airflow-common/src/airflow_common/__init__.py
>>>>> @@ -14,3 +14,5 @@
>>>>> # KIND, either express or implied.  See the License for the
>>>>> # specific language governing permissions and limitations
>>>>> # under the License.
>>>>> +
>>>>> +foo = 1
>>>>> diff --git a/airflow-common/src/airflow_common/timezone.py
>>>>> b/airflow-common/src/airflow_common/timezone.py
>>>>> index 340b924c66..58384ef20f 100644
>>>>> --- a/airflow-common/src/airflow_common/timezone.py
>>>>> +++ b/airflow-common/src/airflow_common/timezone.py
>>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>>>>> version.parse(metadata.version("pendulum")).major == 3
>>>>> # - FixedTimezone(0, "UTC") in pendulum 2
>>>>> utc = pendulum.UTC
>>>>> 
>>>>> +
>>>>> +from airflow_common import foo
>>>>> +
>>>>> TIMEZONE: Timezone
>>>>> 
>>>>> 
>>>>> ```
>>>>> 
>>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>>> 
>>>>>> I think both approaches are doable:
>>>>>> 
>>>>>> 1) -> We can very easily prevent bad imports by pre-commit when
>>> importing
>>>>>> from different distributions and make sure we are only doing relative
>>>>>> imports in the shared modules. We are doing plenty of this already. And
>>>>> yes
>>>>>> it would require relative links we currently do not allow.
>>>>>> 
>>>>>> 2) -> has one disadvantage that someone at some point in time will have
>>>>> to
>>>>>> decide to synchronize this and if it happens just before release (I bet
>>>>>> this is going to happen) this will lead to solving problems that would
>>>>>> normally be solved during PR when you make a change (i.e. symbolic link
>>>>> has
>>>>>> the advantage that whoever modifies shared code will be immediately
>>>>>> notified in their PR - that they broke something because either static
>>>>>> checks or mypy or tests fail.
>>>>>> 
>>>>>> Ash, do you have an idea of a process (who and when) does the
>>>>>> synchronisation in case of vendoring? Maybe we could solve it if it is
>>>>> done
>>>>>> more frequently and with some regularity? We could potentially force
>>>>>> re-vendoring at PR time as well any time shared code changes (and
>>> prevent
>>>>>> it by pre-commit. And I can't think of some place (other than releases)
>>>>> in
>>>>>> our development workflow and that seems to be a bit too late as puts an
>>>>>> extra effort on fixing potential incompatibilities introduced on
>>> release
>>>>>> manager and delays the release. WDYT?
>>>>>> 
>>>>>> Re: relative links. I think for a shared library we could potentially
>>>>> relax
>>>>>> this and allow them (and actually disallow absolute links in the pieces
>>>>> of
>>>>>> code that are shared - again, by pre-commit). As I recall, the only
>>>>> reason
>>>>>> we forbade the relative links is because of how we are (or maybe were)
>>>>>> doing DAG parsing and failures resulting from it. So we decided to just
>>>>> not
>>>>>> allow it to keep consistency. The way how Dag parsing works is that
>>> when
>>>>>> you are using importlib to read the Dag from a file, the relative
>>> imports
>>>>>> do not work as it does not know what they should be relative to. But if
>>>>>> relative import is done from an imported package, it should be no
>>>>> problem,
>>>>>> I think - otherwise our Dags would not be able to import any library
>>> that
>>>>>> uses relative imports.
>>>>>> 
>>>>>> Of course consistency might be the reason why we do not want to
>>> introduce
>>>>>> relative imports. I don't see it as an issue if it is guarded by
>>>>> pre-commit
>>>>>> though.
>>>>>> 
>>>>>> J.
>>>>>> 
>>>>>> 
>>>>>> J.
>>>>>> 
>>>>>> 
>>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <as...@apache.org>
>>>>>> napisał:
>>>>>> 
>>>>>>> Oh yes, symlinks will work, with one big caveat: It does mean you
>>> can’t
>>>>>>> use absolute imports in one common module to another.
>>>>>>> 
>>>>>>> For example
>>>>>>> 
>>>>> 
>>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>>>>>>> where we have
>>>>>>> 
>>>>>>> ```
>>>>>>> from airflow.utils.module_loading import import_string
>>>>>>> ```
>>>>>>> 
>>>>>>> if we want to move serve_logs into this common lib that is then
>>>>> symlinked
>>>>>>> then we wouldn’t be able to have `from airflow_common.module_loading
>>>>> import
>>>>>>> import_string`.
>>>>>>> 
>>>>>>> I can think of two possible solutions here.
>>>>>>> 
>>>>>>> 1) is to allow/require relative imports in this shared lib, so `from
>>>>>>> .module_loading import import_string`
>>>>>>> 2) is to use `vendoring`[1] (from the pip maintainers) which will
>>> handle
>>>>>>> import-rewriting for us.
>>>>>>> 
>>>>>>> I’d entirely forgot that symlinks in repos was a thing, so I prepared
>>> a
>>>>>>> minimal POC/demo of what vendoring approach could look like here
>>>>>>> 
>>>>> 
>>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>>>>>>> 
>>>>>>> Now personally I am more than happy with relative imports, but
>>> generally
>>>>>>> as a project we have avoided them, so I think that limits what we
>>> could
>>>>> do
>>>>>>> with a symlink based approach.
>>>>>>> 
>>>>>>> -ash
>>>>>>> 
>>>>>>> [1] https://github.com/pradyunsg/vendoring
>>>>>>> 
>>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
>>> gopidesupavan@gmail.com>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Thanks Ash
>>>>>>>> 
>>>>>>>> Yes agree option 2 would be preferred for me. Making sure we have all
>>>>> the
>>>>>>>> gaurdriles to protect any unwanted behaviour in code sharing and
>>>>>>> executing
>>>>>>>> right of tests between the packages.
>>>>>>>> 
>>>>>>>> Agree with others, option 2 would be
>>>>>>>> 
>>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
>>> amoghdesai.oss@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Thanks for starting this discussion, Ash.
>>>>>>>>> 
>>>>>>>>> I would prefer option 2 here with proper tooling to handle the code
>>>>>>>>> duplication at *release* time.
>>>>>>>>> It is best to have a dist that has all it needs in itself.
>>>>>>>>> 
>>>>>>>>> Option 1 could very quickly get out of hand and if we decide to
>>>>> separate
>>>>>>>>> triggerer /
>>>>>>>>> dag processor / config etc etc as separate packages, back compat is
>>>>>>> going
>>>>>>>>> to be a nightmare
>>>>>>>>> and will bite us harder than we anticipate.
>>>>>>>>> 
>>>>>>>>> Thanks & Regards,
>>>>>>>>> Amogh Desai
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <ka...@gmail.com>
>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> I prefer Option 2 as well to avoid matrix of dependencies
>>>>>>>>>> 
>>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>>>>> <j_scheffler@gmx.de.invalid
>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> I'd also rather prefer option 2 - reason here is it is rather
>>>>>>> pragmatic
>>>>>>>>>>> and we no not need to cut another package and have less package
>>>>> counts
>>>>>>>>>>> and dependencies.
>>>>>>>>>>> 
>>>>>>>>>>> I remember some time ago I was checking (together with Jarek, I am
>>>>> not
>>>>>>>>>>> sure anymore...) if the usage of symlinks would be possible. To
>>> keep
>>>>>>>>> the
>>>>>>>>>>> source in one package but "symlink" it into another. If then at
>>>>> point
>>>>>>>>> of
>>>>>>>>>>> packaging/release the files are materialized we have 1 set of
>>> code.
>>>>>>>>>>> 
>>>>>>>>>>> Otherwise if not possible still the redundancy could be solved by
>>> a
>>>>>>>>>>> pre-commit hook - and in Git the files are de-duplicated anyway
>>>>> based
>>>>>>>>> on
>>>>>>>>>>> content hash, so this does not hurt.
>>>>>>>>>>> 
>>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>>>>>>>>>> I support option 2 with proper automation & CI - the reasonings
>>>>>>>>> you've
>>>>>>>>>>>> shown for that make sense to me.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Shahar
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <ash@apache.org
>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> As we work on finishing off the code-level separation of Task
>>> SDK
>>>>>>>>> and
>>>>>>>>>>> Core
>>>>>>>>>>>>> (scheduler etc) we have come across some situations where we
>>> would
>>>>>>>>>> like
>>>>>>>>>>> to
>>>>>>>>>>>>> share code between these.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> However it’s not as straight forward of “just put it in a common
>>>>>>>>> dist
>>>>>>>>>>> they
>>>>>>>>>>>>> both depend upon” because one of the goals of the Task SDK
>>>>>>>>> separation
>>>>>>>>>>> was
>>>>>>>>>>>>> to have 100% complete version independence between the two,
>>>>> ideally
>>>>>>>>>>> even if
>>>>>>>>>>>>> they are built into the same image and venv. Most of the reason
>>>>> why
>>>>>>>>>> this
>>>>>>>>>>>>> isn’t straight forward comes down to backwards compatibility -
>>> if
>>>>> we
>>>>>>>>>>> make
>>>>>>>>>>>>> an change to the common/shared distribution
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> We’ve listed the options we have thought about in
>>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545 (but that covers
>>>>>>>>> some
>>>>>>>>>>> more
>>>>>>>>>>>>> things that I don’t want to get in to in this discussion such as
>>>>>>>>>>> possibly
>>>>>>>>>>>>> separating operators and executors out of a single provider
>>> dist.)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> To give a concrete example of some code I would like to share
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>>>>>>>>>>> — logging config. Another thing we will want to share will be
>>> the
>>>>>>>>>>>>> AirflowConfigParser class from airflow.configuration (but
>>> notably:
>>>>>>>>>> only
>>>>>>>>>>> the
>>>>>>>>>>>>> parser class, _not_ the default config values, again, lets not
>>>>> dwell
>>>>>>>>>> on
>>>>>>>>>>> the
>>>>>>>>>>>>> specifics of that)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> So to bring the options listed in the issue here for discussion,
>>>>>>>>>> broadly
>>>>>>>>>>>>> speaking there are two high-level approaches:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 1. A single shared distribution
>>>>>>>>>>>>> 2. No shared package and copy/duplicate code
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The advantage of Approach 1 is that we only have the code in one
>>>>>>>>>> place.
>>>>>>>>>>>>> However for me, at least in this specific case of Logging config
>>>>> or
>>>>>>>>>>>>> AirflowConfigParser class is that backwards compatibility is
>>> much
>>>>>>>>> much
>>>>>>>>>>>>> harder.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The main advantage of Approach 2 is the the code is released
>>>>>>>>>>> with/embedded
>>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would contain the
>>> right
>>>>>>>>>>> version
>>>>>>>>>>>>> of the logging config and ConfigParser etc). The downside is
>>> that
>>>>>>>>>> either
>>>>>>>>>>>>> the code will need to be duplicated in the repo, or better yet
>>> it
>>>>>>>>>> would
>>>>>>>>>>>>> live in a single place in the repo, but some tooling (TBD) will
>>>>>>>>>>>>> automatically handle the duplication, either at commit time, or
>>> my
>>>>>>>>>>>>> preference, at release time.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> For this kind of shared “utility” code I am very strongly
>>> leaning
>>>>>>>>>>> towards
>>>>>>>>>>>>> option 2 with automation, as otherwise I think the backwards
>>>>>>>>>>> compatibility
>>>>>>>>>>>>> requirements would make it unworkable (very quickly over time
>>> the
>>>>>>>>>>>>> combinations we would have to test would just be unreasonable)
>>>>> and I
>>>>>>>>>>> don’t
>>>>>>>>>>>>> feel confident we can have things as stable as we need to really
>>>>>>>>>> deliver
>>>>>>>>>>>>> the version separation/independency I want to delivery with
>>>>> AIP-72.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> So unless someone feels very strongly about this, I will come up
>>>>>>>>> with
>>>>>>>>>> a
>>>>>>>>>>>>> draft PR for further discussion that will implement code sharing
>>>>> via
>>>>>>>>>>>>> “vendoring” it at build time. I have an idea of how I can
>>> achieve
>>>>>>>>> this
>>>>>>>>>>> so
>>>>>>>>>>>>> we have a single version in the repo and it’ll work there, but
>>> at
>>>>>>>>>>> runtime
>>>>>>>>>>>>> we vendor it in to the shipped dist so it lives at something
>>> like
>>>>>>>>>>>>> `airflow.sdk._vendor` etc.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> In terms of repo layout, this likely means we would end up with:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> airflow-core/pyproject.toml
>>>>>>>>>>>>> airflow-core/src/
>>>>>>>>>>>>> airflow-core/tests/
>>>>>>>>>>>>> task-sdk/pyproject.toml
>>>>>>>>>>>>> task-sdk/src/
>>>>>>>>>>>>> task-sdk/tests/
>>>>>>>>>>>>> airflow-common/src
>>>>>>>>>>>>> airflow-common/tests/
>>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps would be
>>>>>>>>> included
>>>>>>>>>>> in
>>>>>>>>>>>>> the downstream projects. TBD.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thoughts and feedback welcomed.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>> 
>>> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org


Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Ash Berlin-Taylor <as...@apache.org>.
Okay, so it seems we have agreement on the approach here, so I’ll continue with this, and on the dev call it was mentioned that “airflow-common” wasn’t a great name, so here is my proposal for the file structure;

```
/
  task-sdk/...
  airflow-core/...
  shared/
    kuberenetes/
      pyproject.toml
      src/
        airflow_kube/__init__.py
    core-and-tasksdk/
      pyproject.toml
      src/
        airflow_shared/__init__.py
```

Things to note here: the “shared” folder has (the possibility) of having multiple different shared “libraries” in it, in this example I am supposing a hypothetical shared kuberenetes folder a world in which we split the KubePodOperator and the KubeExecutor in to two separate distributions (example only, not proposing we do that right now, and that will be a separate discussion)

The other things to note here:


- the folder name in shared aims to be “self-documenting”, hence the verbose “core-and-tasksdk” to say where the shared library is intended to be used. 
- the python module itself should almost always have an `airflow_` (or maybe `_airflow_`?) prefix so that it does not conflict with anything else we might use. It won’t matter “in production” as those will be vendored in to be imported as `airflow/_vendor/airflow_shared` etc, but avoiding conflicts at dev time with the Finder approach is a good safety measure.

I will start making a real PR for this proposal now, but I’m open to feedback (either here, or in the PR when I open it)

-ash

> On 4 Jul 2025, at 16:55, Jarek Potiuk <ja...@potiuk.com> wrote:
> 
> Yeah we have to try it and test - also building packages happens semi
> frequently when you run `uv sync` (they use some kind of heuristics to
> decide when) and you can force it with `--reinstall` or `--refresh`.
> Package build also happens every time when you run "ci-image build` now in
> breeze so it seems like it will nicely integrate in our workflows.
> 
> Looks really cool Ash.
> 
> On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> 
>> It’s not just release time, but any time we build a package which happens
>> on “every” CI run. The normal unit tests will use code from
>> airflow-common/src/airflow_common; the kube tests which build an image will
>> build the dists and vendor in the code from that commit.
>> 
>> There is only a single copy of the shared code committed to the repo, so
>> there is never anything to synchronise.
>> 
>>> On 4 Jul 2025, at 15:53, Amogh Desai <am...@gmail.com> wrote:
>>> 
>>> Thanks Ash.
>>> 
>>> This is really cool and helpful that you were able to test both scenarios
>>> -- repo checkout
>>> and also installing from the vendored package and the resolution worked
>>> fine too.
>>> 
>>> I like this idea compared the to relative import one for few reasons:
>>> - It feels like it will take some time to adjust to the new coding
>> standard
>>> that we will lay
>>> if we impose relative imports in the shared dist
>>> - We can continue using repo wise absolute import standards, it is also
>>> much easier for situations
>>> when we do global search in IDE to find + replace, this could mean a
>> change
>>> there
>>> - The vendoring work is a proven and established paradigm across projects
>>> and would
>>> out of box give us the build tooling we need also
>>> 
>>> Nothing too against the relative import but with the evidence provided
>>> above, vendored approach
>>> seems to only do us good.
>>> 
>>> Regarding synchronizing it, release time should be fine as long as we
>> have
>>> a good CI workflow to probably
>>> catch such issues per PR if changes are made in shared dist? (precommit
>>> would make it really slow i guess)
>>> 
>>> If we can run our tests with vendored code we should be mostly covered.
>>> 
>>> Good effort all!
>>> 
>>> Thanks & Regards,
>>> Amogh Desai
>>> 
>>> 
>>>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <as...@apache.org>
>> wrote:
>>>> 
>>>> Okay, I think I’ve got something that works and I’m happy with.
>>>> 
>>>> 
>>>> 
>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>>>> 
>>>> This produces the following from `uv build task-sdk`
>>>> -
>>>> 
>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>>>> -
>>>> 
>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>>>> 
>>>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
>>>> 
>>>> ```
>>>> ❯ unzip -l dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
>> grep
>>>> _vendor
>>>>      50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
>>>>    2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
>>>>      28  02-02-2020 00:00   airflow/sdk/_vendor/airflow_common.pyi
>>>>      18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
>>>>     785  02-02-2020 00:00
>>>> airflow/sdk/_vendor/airflow_common/__init__.py
>>>>   10628  02-02-2020 00:00
>>>> airflow/sdk/_vendor/airflow_common/timezone.py
>>>> ```
>>>> 
>>>> And similarly in the .tar.gz, so our “sdist” is complete too:
>>>> ```
>>>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep _vendor
>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>>>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>>>> 
>>>> 
>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>>>> 
>>>> 
>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>>>> ```
>>>> 
>>>> The plugin works at build time by including/copying the libs specified
>> in
>>>> vendor.txt into place (and let `vendoring` take care of import
>> rewrites.)
>>>> 
>>>> For the imports to continue to work at “dev” time/from a repo checkout,
>> I
>>>> have added a import finder to `sys.meta_path`, and since it’s at the
>> end of
>>>> the list it will only be used if the normal import can’t find things.
>>>> 
>>>> 
>>>> 
>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>>>> 
>>>> This doesn’t quite give us the same runtime effect “import rewriting”
>>>> affect, as in this approach `airflow_common` is directly loaded (i.e.
>>>> airflow.sdk._vendor.airflow_common and airflow_common exist in
>>>> sys.modules), but it does work for everything that I was able to test..
>>>> 
>>>> I tested it with the diff at the end of this message. My test ipython
>>>> shell:
>>>> 
>>>> ```
>>>> In [1]: from airflow.sdk._vendor.airflow_common.timezone import foo
>>>> 
>>>> In [2]: foo
>>>> Out[2]: 1
>>>> 
>>>> In [3]: import airflow.sdk._vendor.airflow_common
>>>> 
>>>> In [4]: import airflow.sdk._vendor.airflow_common.timezone
>>>> 
>>>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>>>> Out[5]:
>>>> 
>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>>>> 
>>>> In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
>>>> Out[6]:
>>>> 
>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>>>> 
>>>> ```
>>>> 
>>>> 
>>>> And in an standalone environment with the SDK dist I built (it needed
>> the
>>>> matching airflow-core right now, but that is nothing to do with this
>>>> discussion):
>>>> 
>>>> ```
>>>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>>>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>>>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
>>>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
>>>> Type 'copyright', 'credits' or 'license' for more information
>>>> IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for help.
>>>> Tip: You can use `%hist` to view history, see the options with
>> `%history?`
>>>> 
>>>> In [1]: import airflow.sdk._vendor.airflow_common.timezone
>>>> 
>>>> In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
>>>> Out[2]:
>>>> 
>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>>>> ``
>>>> 
>>>> 
>>>> 
>>>> ```diff
>>>> diff --git a/airflow-common/src/airflow_common/__init__.py
>>>> b/airflow-common/src/airflow_common/__init__.py
>>>> index 13a83393a9..927b7c6b61 100644
>>>> --- a/airflow-common/src/airflow_common/__init__.py
>>>> +++ b/airflow-common/src/airflow_common/__init__.py
>>>> @@ -14,3 +14,5 @@
>>>> # KIND, either express or implied.  See the License for the
>>>> # specific language governing permissions and limitations
>>>> # under the License.
>>>> +
>>>> +foo = 1
>>>> diff --git a/airflow-common/src/airflow_common/timezone.py
>>>> b/airflow-common/src/airflow_common/timezone.py
>>>> index 340b924c66..58384ef20f 100644
>>>> --- a/airflow-common/src/airflow_common/timezone.py
>>>> +++ b/airflow-common/src/airflow_common/timezone.py
>>>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>>>> version.parse(metadata.version("pendulum")).major == 3
>>>> # - FixedTimezone(0, "UTC") in pendulum 2
>>>> utc = pendulum.UTC
>>>> 
>>>> +
>>>> +from airflow_common import foo
>>>> +
>>>> TIMEZONE: Timezone
>>>> 
>>>> 
>>>> ```
>>>> 
>>>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>> 
>>>>> I think both approaches are doable:
>>>>> 
>>>>> 1) -> We can very easily prevent bad imports by pre-commit when
>> importing
>>>>> from different distributions and make sure we are only doing relative
>>>>> imports in the shared modules. We are doing plenty of this already. And
>>>> yes
>>>>> it would require relative links we currently do not allow.
>>>>> 
>>>>> 2) -> has one disadvantage that someone at some point in time will have
>>>> to
>>>>> decide to synchronize this and if it happens just before release (I bet
>>>>> this is going to happen) this will lead to solving problems that would
>>>>> normally be solved during PR when you make a change (i.e. symbolic link
>>>> has
>>>>> the advantage that whoever modifies shared code will be immediately
>>>>> notified in their PR - that they broke something because either static
>>>>> checks or mypy or tests fail.
>>>>> 
>>>>> Ash, do you have an idea of a process (who and when) does the
>>>>> synchronisation in case of vendoring? Maybe we could solve it if it is
>>>> done
>>>>> more frequently and with some regularity? We could potentially force
>>>>> re-vendoring at PR time as well any time shared code changes (and
>> prevent
>>>>> it by pre-commit. And I can't think of some place (other than releases)
>>>> in
>>>>> our development workflow and that seems to be a bit too late as puts an
>>>>> extra effort on fixing potential incompatibilities introduced on
>> release
>>>>> manager and delays the release. WDYT?
>>>>> 
>>>>> Re: relative links. I think for a shared library we could potentially
>>>> relax
>>>>> this and allow them (and actually disallow absolute links in the pieces
>>>> of
>>>>> code that are shared - again, by pre-commit). As I recall, the only
>>>> reason
>>>>> we forbade the relative links is because of how we are (or maybe were)
>>>>> doing DAG parsing and failures resulting from it. So we decided to just
>>>> not
>>>>> allow it to keep consistency. The way how Dag parsing works is that
>> when
>>>>> you are using importlib to read the Dag from a file, the relative
>> imports
>>>>> do not work as it does not know what they should be relative to. But if
>>>>> relative import is done from an imported package, it should be no
>>>> problem,
>>>>> I think - otherwise our Dags would not be able to import any library
>> that
>>>>> uses relative imports.
>>>>> 
>>>>> Of course consistency might be the reason why we do not want to
>> introduce
>>>>> relative imports. I don't see it as an issue if it is guarded by
>>>> pre-commit
>>>>> though.
>>>>> 
>>>>> J.
>>>>> 
>>>>> 
>>>>> J.
>>>>> 
>>>>> 
>>>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <as...@apache.org>
>>>>> napisał:
>>>>> 
>>>>>> Oh yes, symlinks will work, with one big caveat: It does mean you
>> can’t
>>>>>> use absolute imports in one common module to another.
>>>>>> 
>>>>>> For example
>>>>>> 
>>>> 
>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>>>>>> where we have
>>>>>> 
>>>>>> ```
>>>>>> from airflow.utils.module_loading import import_string
>>>>>> ```
>>>>>> 
>>>>>> if we want to move serve_logs into this common lib that is then
>>>> symlinked
>>>>>> then we wouldn’t be able to have `from airflow_common.module_loading
>>>> import
>>>>>> import_string`.
>>>>>> 
>>>>>> I can think of two possible solutions here.
>>>>>> 
>>>>>> 1) is to allow/require relative imports in this shared lib, so `from
>>>>>> .module_loading import import_string`
>>>>>> 2) is to use `vendoring`[1] (from the pip maintainers) which will
>> handle
>>>>>> import-rewriting for us.
>>>>>> 
>>>>>> I’d entirely forgot that symlinks in repos was a thing, so I prepared
>> a
>>>>>> minimal POC/demo of what vendoring approach could look like here
>>>>>> 
>>>> 
>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>>>>>> 
>>>>>> Now personally I am more than happy with relative imports, but
>> generally
>>>>>> as a project we have avoided them, so I think that limits what we
>> could
>>>> do
>>>>>> with a symlink based approach.
>>>>>> 
>>>>>> -ash
>>>>>> 
>>>>>> [1] https://github.com/pradyunsg/vendoring
>>>>>> 
>>>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
>> gopidesupavan@gmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> Thanks Ash
>>>>>>> 
>>>>>>> Yes agree option 2 would be preferred for me. Making sure we have all
>>>> the
>>>>>>> gaurdriles to protect any unwanted behaviour in code sharing and
>>>>>> executing
>>>>>>> right of tests between the packages.
>>>>>>> 
>>>>>>> Agree with others, option 2 would be
>>>>>>> 
>>>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
>> amoghdesai.oss@gmail.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Thanks for starting this discussion, Ash.
>>>>>>>> 
>>>>>>>> I would prefer option 2 here with proper tooling to handle the code
>>>>>>>> duplication at *release* time.
>>>>>>>> It is best to have a dist that has all it needs in itself.
>>>>>>>> 
>>>>>>>> Option 1 could very quickly get out of hand and if we decide to
>>>> separate
>>>>>>>> triggerer /
>>>>>>>> dag processor / config etc etc as separate packages, back compat is
>>>>>> going
>>>>>>>> to be a nightmare
>>>>>>>> and will bite us harder than we anticipate.
>>>>>>>> 
>>>>>>>> Thanks & Regards,
>>>>>>>> Amogh Desai
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <ka...@gmail.com>
>>>> wrote:
>>>>>>>> 
>>>>>>>>> I prefer Option 2 as well to avoid matrix of dependencies
>>>>>>>>> 
>>>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>>>> <j_scheffler@gmx.de.invalid
>>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> I'd also rather prefer option 2 - reason here is it is rather
>>>>>> pragmatic
>>>>>>>>>> and we no not need to cut another package and have less package
>>>> counts
>>>>>>>>>> and dependencies.
>>>>>>>>>> 
>>>>>>>>>> I remember some time ago I was checking (together with Jarek, I am
>>>> not
>>>>>>>>>> sure anymore...) if the usage of symlinks would be possible. To
>> keep
>>>>>>>> the
>>>>>>>>>> source in one package but "symlink" it into another. If then at
>>>> point
>>>>>>>> of
>>>>>>>>>> packaging/release the files are materialized we have 1 set of
>> code.
>>>>>>>>>> 
>>>>>>>>>> Otherwise if not possible still the redundancy could be solved by
>> a
>>>>>>>>>> pre-commit hook - and in Git the files are de-duplicated anyway
>>>> based
>>>>>>>> on
>>>>>>>>>> content hash, so this does not hurt.
>>>>>>>>>> 
>>>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>>>>>>>>> I support option 2 with proper automation & CI - the reasonings
>>>>>>>> you've
>>>>>>>>>>> shown for that make sense to me.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Shahar
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <ash@apache.org
>>> 
>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>> 
>>>>>>>>>>>> As we work on finishing off the code-level separation of Task
>> SDK
>>>>>>>> and
>>>>>>>>>> Core
>>>>>>>>>>>> (scheduler etc) we have come across some situations where we
>> would
>>>>>>>>> like
>>>>>>>>>> to
>>>>>>>>>>>> share code between these.
>>>>>>>>>>>> 
>>>>>>>>>>>> However it’s not as straight forward of “just put it in a common
>>>>>>>> dist
>>>>>>>>>> they
>>>>>>>>>>>> both depend upon” because one of the goals of the Task SDK
>>>>>>>> separation
>>>>>>>>>> was
>>>>>>>>>>>> to have 100% complete version independence between the two,
>>>> ideally
>>>>>>>>>> even if
>>>>>>>>>>>> they are built into the same image and venv. Most of the reason
>>>> why
>>>>>>>>> this
>>>>>>>>>>>> isn’t straight forward comes down to backwards compatibility -
>> if
>>>> we
>>>>>>>>>> make
>>>>>>>>>>>> an change to the common/shared distribution
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> We’ve listed the options we have thought about in
>>>>>>>>>>>> https://github.com/apache/airflow/issues/51545 (but that covers
>>>>>>>> some
>>>>>>>>>> more
>>>>>>>>>>>> things that I don’t want to get in to in this discussion such as
>>>>>>>>>> possibly
>>>>>>>>>>>> separating operators and executors out of a single provider
>> dist.)
>>>>>>>>>>>> 
>>>>>>>>>>>> To give a concrete example of some code I would like to share
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>>>>>>>>>> — logging config. Another thing we will want to share will be
>> the
>>>>>>>>>>>> AirflowConfigParser class from airflow.configuration (but
>> notably:
>>>>>>>>> only
>>>>>>>>>> the
>>>>>>>>>>>> parser class, _not_ the default config values, again, lets not
>>>> dwell
>>>>>>>>> on
>>>>>>>>>> the
>>>>>>>>>>>> specifics of that)
>>>>>>>>>>>> 
>>>>>>>>>>>> So to bring the options listed in the issue here for discussion,
>>>>>>>>> broadly
>>>>>>>>>>>> speaking there are two high-level approaches:
>>>>>>>>>>>> 
>>>>>>>>>>>> 1. A single shared distribution
>>>>>>>>>>>> 2. No shared package and copy/duplicate code
>>>>>>>>>>>> 
>>>>>>>>>>>> The advantage of Approach 1 is that we only have the code in one
>>>>>>>>> place.
>>>>>>>>>>>> However for me, at least in this specific case of Logging config
>>>> or
>>>>>>>>>>>> AirflowConfigParser class is that backwards compatibility is
>> much
>>>>>>>> much
>>>>>>>>>>>> harder.
>>>>>>>>>>>> 
>>>>>>>>>>>> The main advantage of Approach 2 is the the code is released
>>>>>>>>>> with/embedded
>>>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would contain the
>> right
>>>>>>>>>> version
>>>>>>>>>>>> of the logging config and ConfigParser etc). The downside is
>> that
>>>>>>>>> either
>>>>>>>>>>>> the code will need to be duplicated in the repo, or better yet
>> it
>>>>>>>>> would
>>>>>>>>>>>> live in a single place in the repo, but some tooling (TBD) will
>>>>>>>>>>>> automatically handle the duplication, either at commit time, or
>> my
>>>>>>>>>>>> preference, at release time.
>>>>>>>>>>>> 
>>>>>>>>>>>> For this kind of shared “utility” code I am very strongly
>> leaning
>>>>>>>>>> towards
>>>>>>>>>>>> option 2 with automation, as otherwise I think the backwards
>>>>>>>>>> compatibility
>>>>>>>>>>>> requirements would make it unworkable (very quickly over time
>> the
>>>>>>>>>>>> combinations we would have to test would just be unreasonable)
>>>> and I
>>>>>>>>>> don’t
>>>>>>>>>>>> feel confident we can have things as stable as we need to really
>>>>>>>>> deliver
>>>>>>>>>>>> the version separation/independency I want to delivery with
>>>> AIP-72.
>>>>>>>>>>>> 
>>>>>>>>>>>> So unless someone feels very strongly about this, I will come up
>>>>>>>> with
>>>>>>>>> a
>>>>>>>>>>>> draft PR for further discussion that will implement code sharing
>>>> via
>>>>>>>>>>>> “vendoring” it at build time. I have an idea of how I can
>> achieve
>>>>>>>> this
>>>>>>>>>> so
>>>>>>>>>>>> we have a single version in the repo and it’ll work there, but
>> at
>>>>>>>>>> runtime
>>>>>>>>>>>> we vendor it in to the shipped dist so it lives at something
>> like
>>>>>>>>>>>> `airflow.sdk._vendor` etc.
>>>>>>>>>>>> 
>>>>>>>>>>>> In terms of repo layout, this likely means we would end up with:
>>>>>>>>>>>> 
>>>>>>>>>>>> airflow-core/pyproject.toml
>>>>>>>>>>>> airflow-core/src/
>>>>>>>>>>>> airflow-core/tests/
>>>>>>>>>>>> task-sdk/pyproject.toml
>>>>>>>>>>>> task-sdk/src/
>>>>>>>>>>>> task-sdk/tests/
>>>>>>>>>>>> airflow-common/src
>>>>>>>>>>>> airflow-common/tests/
>>>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps would be
>>>>>>>> included
>>>>>>>>>> in
>>>>>>>>>>>> the downstream projects. TBD.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thoughts and feedback welcomed.
>>>>>>>>>> 
>>>>>>>>>> 
>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> For additional commands, e-mail: dev-help@airflow.apache.org
>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org


Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Jarek Potiuk <ja...@potiuk.com>.
Yeah we have to try it and test - also building packages happens semi
frequently when you run `uv sync` (they use some kind of heuristics to
decide when) and you can force it with `--reinstall` or `--refresh`.
Package build also happens every time when you run "ci-image build` now in
breeze so it seems like it will nicely integrate in our workflows.

Looks really cool Ash.

On Fri, Jul 4, 2025 at 5:14 PM Ash Berlin-Taylor <as...@apache.org> wrote:

> It’s not just release time, but any time we build a package which happens
> on “every” CI run. The normal unit tests will use code from
> airflow-common/src/airflow_common; the kube tests which build an image will
> build the dists and vendor in the code from that commit.
>
> There is only a single copy of the shared code committed to the repo, so
> there is never anything to synchronise.
>
> > On 4 Jul 2025, at 15:53, Amogh Desai <am...@gmail.com> wrote:
> >
> > Thanks Ash.
> >
> > This is really cool and helpful that you were able to test both scenarios
> > -- repo checkout
> > and also installing from the vendored package and the resolution worked
> > fine too.
> >
> > I like this idea compared the to relative import one for few reasons:
> > - It feels like it will take some time to adjust to the new coding
> standard
> > that we will lay
> > if we impose relative imports in the shared dist
> > - We can continue using repo wise absolute import standards, it is also
> > much easier for situations
> > when we do global search in IDE to find + replace, this could mean a
> change
> > there
> > - The vendoring work is a proven and established paradigm across projects
> > and would
> > out of box give us the build tooling we need also
> >
> > Nothing too against the relative import but with the evidence provided
> > above, vendored approach
> > seems to only do us good.
> >
> > Regarding synchronizing it, release time should be fine as long as we
> have
> > a good CI workflow to probably
> > catch such issues per PR if changes are made in shared dist? (precommit
> > would make it really slow i guess)
> >
> > If we can run our tests with vendored code we should be mostly covered.
> >
> > Good effort all!
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> >> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >>
> >> Okay, I think I’ve got something that works and I’m happy with.
> >>
> >>
> >>
> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
> >>
> >> This produces the following from `uv build task-sdk`
> >> -
> >>
> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
> >> -
> >>
> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
> >>
> >> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
> >>
> >> ```
> >> ❯ unzip -l dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip |
> grep
> >> _vendor
> >>       50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
> >>     2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
> >>       28  02-02-2020 00:00   airflow/sdk/_vendor/airflow_common.pyi
> >>       18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
> >>      785  02-02-2020 00:00
> >> airflow/sdk/_vendor/airflow_common/__init__.py
> >>    10628  02-02-2020 00:00
> >> airflow/sdk/_vendor/airflow_common/timezone.py
> >> ```
> >>
> >> And similarly in the .tar.gz, so our “sdist” is complete too:
> >> ```
> >> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep _vendor
> >> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
> >> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
> >> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
> >> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
> >>
> >>
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
> >>
> >>
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
> >> ```
> >>
> >> The plugin works at build time by including/copying the libs specified
> in
> >> vendor.txt into place (and let `vendoring` take care of import
> rewrites.)
> >>
> >> For the imports to continue to work at “dev” time/from a repo checkout,
> I
> >> have added a import finder to `sys.meta_path`, and since it’s at the
> end of
> >> the list it will only be used if the normal import can’t find things.
> >>
> >>
> >>
> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
> >>
> >> This doesn’t quite give us the same runtime effect “import rewriting”
> >> affect, as in this approach `airflow_common` is directly loaded (i.e.
> >> airflow.sdk._vendor.airflow_common and airflow_common exist in
> >> sys.modules), but it does work for everything that I was able to test..
> >>
> >> I tested it with the diff at the end of this message. My test ipython
> >> shell:
> >>
> >> ```
> >> In [1]: from airflow.sdk._vendor.airflow_common.timezone import foo
> >>
> >> In [2]: foo
> >> Out[2]: 1
> >>
> >> In [3]: import airflow.sdk._vendor.airflow_common
> >>
> >> In [4]: import airflow.sdk._vendor.airflow_common.timezone
> >>
> >> In [5]: airflow.sdk._vendor.airflow_common.__file__
> >> Out[5]:
> >>
> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
> >>
> >> In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
> >> Out[6]:
> >>
> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
> >>
> >> ```
> >>
> >>
> >> And in an standalone environment with the SDK dist I built (it needed
> the
> >> matching airflow-core right now, but that is nothing to do with this
> >> discussion):
> >>
> >> ```
> >> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
> >> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
> >> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
> >> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
> >> Type 'copyright', 'credits' or 'license' for more information
> >> IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for help.
> >> Tip: You can use `%hist` to view history, see the options with
> `%history?`
> >>
> >> In [1]: import airflow.sdk._vendor.airflow_common.timezone
> >>
> >> In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
> >> Out[2]:
> >>
> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
> >> ``
> >>
> >>
> >>
> >> ```diff
> >> diff --git a/airflow-common/src/airflow_common/__init__.py
> >> b/airflow-common/src/airflow_common/__init__.py
> >> index 13a83393a9..927b7c6b61 100644
> >> --- a/airflow-common/src/airflow_common/__init__.py
> >> +++ b/airflow-common/src/airflow_common/__init__.py
> >> @@ -14,3 +14,5 @@
> >> # KIND, either express or implied.  See the License for the
> >> # specific language governing permissions and limitations
> >> # under the License.
> >> +
> >> +foo = 1
> >> diff --git a/airflow-common/src/airflow_common/timezone.py
> >> b/airflow-common/src/airflow_common/timezone.py
> >> index 340b924c66..58384ef20f 100644
> >> --- a/airflow-common/src/airflow_common/timezone.py
> >> +++ b/airflow-common/src/airflow_common/timezone.py
> >> @@ -36,6 +36,9 @@ _PENDULUM3 =
> >> version.parse(metadata.version("pendulum")).major == 3
> >> # - FixedTimezone(0, "UTC") in pendulum 2
> >> utc = pendulum.UTC
> >>
> >> +
> >> +from airflow_common import foo
> >> +
> >> TIMEZONE: Timezone
> >>
> >>
> >> ```
> >>
> >>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>> I think both approaches are doable:
> >>>
> >>> 1) -> We can very easily prevent bad imports by pre-commit when
> importing
> >>> from different distributions and make sure we are only doing relative
> >>> imports in the shared modules. We are doing plenty of this already. And
> >> yes
> >>> it would require relative links we currently do not allow.
> >>>
> >>> 2) -> has one disadvantage that someone at some point in time will have
> >> to
> >>> decide to synchronize this and if it happens just before release (I bet
> >>> this is going to happen) this will lead to solving problems that would
> >>> normally be solved during PR when you make a change (i.e. symbolic link
> >> has
> >>> the advantage that whoever modifies shared code will be immediately
> >>> notified in their PR - that they broke something because either static
> >>> checks or mypy or tests fail.
> >>>
> >>> Ash, do you have an idea of a process (who and when) does the
> >>> synchronisation in case of vendoring? Maybe we could solve it if it is
> >> done
> >>> more frequently and with some regularity? We could potentially force
> >>> re-vendoring at PR time as well any time shared code changes (and
> prevent
> >>> it by pre-commit. And I can't think of some place (other than releases)
> >> in
> >>> our development workflow and that seems to be a bit too late as puts an
> >>> extra effort on fixing potential incompatibilities introduced on
> release
> >>> manager and delays the release. WDYT?
> >>>
> >>> Re: relative links. I think for a shared library we could potentially
> >> relax
> >>> this and allow them (and actually disallow absolute links in the pieces
> >> of
> >>> code that are shared - again, by pre-commit). As I recall, the only
> >> reason
> >>> we forbade the relative links is because of how we are (or maybe were)
> >>> doing DAG parsing and failures resulting from it. So we decided to just
> >> not
> >>> allow it to keep consistency. The way how Dag parsing works is that
> when
> >>> you are using importlib to read the Dag from a file, the relative
> imports
> >>> do not work as it does not know what they should be relative to. But if
> >>> relative import is done from an imported package, it should be no
> >> problem,
> >>> I think - otherwise our Dags would not be able to import any library
> that
> >>> uses relative imports.
> >>>
> >>> Of course consistency might be the reason why we do not want to
> introduce
> >>> relative imports. I don't see it as an issue if it is guarded by
> >> pre-commit
> >>> though.
> >>>
> >>> J.
> >>>
> >>>
> >>> J.
> >>>
> >>>
> >>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <as...@apache.org>
> >>> napisał:
> >>>
> >>>> Oh yes, symlinks will work, with one big caveat: It does mean you
> can’t
> >>>> use absolute imports in one common module to another.
> >>>>
> >>>> For example
> >>>>
> >>
> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
> >>>> where we have
> >>>>
> >>>> ```
> >>>> from airflow.utils.module_loading import import_string
> >>>> ```
> >>>>
> >>>> if we want to move serve_logs into this common lib that is then
> >> symlinked
> >>>> then we wouldn’t be able to have `from airflow_common.module_loading
> >> import
> >>>> import_string`.
> >>>>
> >>>> I can think of two possible solutions here.
> >>>>
> >>>> 1) is to allow/require relative imports in this shared lib, so `from
> >>>> .module_loading import import_string`
> >>>> 2) is to use `vendoring`[1] (from the pip maintainers) which will
> handle
> >>>> import-rewriting for us.
> >>>>
> >>>> I’d entirely forgot that symlinks in repos was a thing, so I prepared
> a
> >>>> minimal POC/demo of what vendoring approach could look like here
> >>>>
> >>
> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
> >>>>
> >>>> Now personally I am more than happy with relative imports, but
> generally
> >>>> as a project we have avoided them, so I think that limits what we
> could
> >> do
> >>>> with a symlink based approach.
> >>>>
> >>>> -ash
> >>>>
> >>>> [1] https://github.com/pradyunsg/vendoring
> >>>>
> >>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <
> gopidesupavan@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Thanks Ash
> >>>>>
> >>>>> Yes agree option 2 would be preferred for me. Making sure we have all
> >> the
> >>>>> gaurdriles to protect any unwanted behaviour in code sharing and
> >>>> executing
> >>>>> right of tests between the packages.
> >>>>>
> >>>>> Agree with others, option 2 would be
> >>>>>
> >>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <
> amoghdesai.oss@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Thanks for starting this discussion, Ash.
> >>>>>>
> >>>>>> I would prefer option 2 here with proper tooling to handle the code
> >>>>>> duplication at *release* time.
> >>>>>> It is best to have a dist that has all it needs in itself.
> >>>>>>
> >>>>>> Option 1 could very quickly get out of hand and if we decide to
> >> separate
> >>>>>> triggerer /
> >>>>>> dag processor / config etc etc as separate packages, back compat is
> >>>> going
> >>>>>> to be a nightmare
> >>>>>> and will bite us harder than we anticipate.
> >>>>>>
> >>>>>> Thanks & Regards,
> >>>>>> Amogh Desai
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <ka...@gmail.com>
> >> wrote:
> >>>>>>
> >>>>>>> I prefer Option 2 as well to avoid matrix of dependencies
> >>>>>>>
> >>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
> >> <j_scheffler@gmx.de.invalid
> >>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> I'd also rather prefer option 2 - reason here is it is rather
> >>>> pragmatic
> >>>>>>>> and we no not need to cut another package and have less package
> >> counts
> >>>>>>>> and dependencies.
> >>>>>>>>
> >>>>>>>> I remember some time ago I was checking (together with Jarek, I am
> >> not
> >>>>>>>> sure anymore...) if the usage of symlinks would be possible. To
> keep
> >>>>>> the
> >>>>>>>> source in one package but "symlink" it into another. If then at
> >> point
> >>>>>> of
> >>>>>>>> packaging/release the files are materialized we have 1 set of
> code.
> >>>>>>>>
> >>>>>>>> Otherwise if not possible still the redundancy could be solved by
> a
> >>>>>>>> pre-commit hook - and in Git the files are de-duplicated anyway
> >> based
> >>>>>> on
> >>>>>>>> content hash, so this does not hurt.
> >>>>>>>>
> >>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
> >>>>>>>>> I support option 2 with proper automation & CI - the reasonings
> >>>>>> you've
> >>>>>>>>> shown for that make sense to me.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Shahar
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <ash@apache.org
> >
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hello everyone,
> >>>>>>>>>>
> >>>>>>>>>> As we work on finishing off the code-level separation of Task
> SDK
> >>>>>> and
> >>>>>>>> Core
> >>>>>>>>>> (scheduler etc) we have come across some situations where we
> would
> >>>>>>> like
> >>>>>>>> to
> >>>>>>>>>> share code between these.
> >>>>>>>>>>
> >>>>>>>>>> However it’s not as straight forward of “just put it in a common
> >>>>>> dist
> >>>>>>>> they
> >>>>>>>>>> both depend upon” because one of the goals of the Task SDK
> >>>>>> separation
> >>>>>>>> was
> >>>>>>>>>> to have 100% complete version independence between the two,
> >> ideally
> >>>>>>>> even if
> >>>>>>>>>> they are built into the same image and venv. Most of the reason
> >> why
> >>>>>>> this
> >>>>>>>>>> isn’t straight forward comes down to backwards compatibility -
> if
> >> we
> >>>>>>>> make
> >>>>>>>>>> an change to the common/shared distribution
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> We’ve listed the options we have thought about in
> >>>>>>>>>> https://github.com/apache/airflow/issues/51545 (but that covers
> >>>>>> some
> >>>>>>>> more
> >>>>>>>>>> things that I don’t want to get in to in this discussion such as
> >>>>>>>> possibly
> >>>>>>>>>> separating operators and executors out of a single provider
> dist.)
> >>>>>>>>>>
> >>>>>>>>>> To give a concrete example of some code I would like to share
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
> >>>>>>>>>> — logging config. Another thing we will want to share will be
> the
> >>>>>>>>>> AirflowConfigParser class from airflow.configuration (but
> notably:
> >>>>>>> only
> >>>>>>>> the
> >>>>>>>>>> parser class, _not_ the default config values, again, lets not
> >> dwell
> >>>>>>> on
> >>>>>>>> the
> >>>>>>>>>> specifics of that)
> >>>>>>>>>>
> >>>>>>>>>> So to bring the options listed in the issue here for discussion,
> >>>>>>> broadly
> >>>>>>>>>> speaking there are two high-level approaches:
> >>>>>>>>>>
> >>>>>>>>>> 1. A single shared distribution
> >>>>>>>>>> 2. No shared package and copy/duplicate code
> >>>>>>>>>>
> >>>>>>>>>> The advantage of Approach 1 is that we only have the code in one
> >>>>>>> place.
> >>>>>>>>>> However for me, at least in this specific case of Logging config
> >> or
> >>>>>>>>>> AirflowConfigParser class is that backwards compatibility is
> much
> >>>>>> much
> >>>>>>>>>> harder.
> >>>>>>>>>>
> >>>>>>>>>> The main advantage of Approach 2 is the the code is released
> >>>>>>>> with/embedded
> >>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would contain the
> right
> >>>>>>>> version
> >>>>>>>>>> of the logging config and ConfigParser etc). The downside is
> that
> >>>>>>> either
> >>>>>>>>>> the code will need to be duplicated in the repo, or better yet
> it
> >>>>>>> would
> >>>>>>>>>> live in a single place in the repo, but some tooling (TBD) will
> >>>>>>>>>> automatically handle the duplication, either at commit time, or
> my
> >>>>>>>>>> preference, at release time.
> >>>>>>>>>>
> >>>>>>>>>> For this kind of shared “utility” code I am very strongly
> leaning
> >>>>>>>> towards
> >>>>>>>>>> option 2 with automation, as otherwise I think the backwards
> >>>>>>>> compatibility
> >>>>>>>>>> requirements would make it unworkable (very quickly over time
> the
> >>>>>>>>>> combinations we would have to test would just be unreasonable)
> >> and I
> >>>>>>>> don’t
> >>>>>>>>>> feel confident we can have things as stable as we need to really
> >>>>>>> deliver
> >>>>>>>>>> the version separation/independency I want to delivery with
> >> AIP-72.
> >>>>>>>>>>
> >>>>>>>>>> So unless someone feels very strongly about this, I will come up
> >>>>>> with
> >>>>>>> a
> >>>>>>>>>> draft PR for further discussion that will implement code sharing
> >> via
> >>>>>>>>>> “vendoring” it at build time. I have an idea of how I can
> achieve
> >>>>>> this
> >>>>>>>> so
> >>>>>>>>>> we have a single version in the repo and it’ll work there, but
> at
> >>>>>>>> runtime
> >>>>>>>>>> we vendor it in to the shipped dist so it lives at something
> like
> >>>>>>>>>> `airflow.sdk._vendor` etc.
> >>>>>>>>>>
> >>>>>>>>>> In terms of repo layout, this likely means we would end up with:
> >>>>>>>>>>
> >>>>>>>>>> airflow-core/pyproject.toml
> >>>>>>>>>> airflow-core/src/
> >>>>>>>>>> airflow-core/tests/
> >>>>>>>>>> task-sdk/pyproject.toml
> >>>>>>>>>> task-sdk/src/
> >>>>>>>>>> task-sdk/tests/
> >>>>>>>>>> airflow-common/src
> >>>>>>>>>> airflow-common/tests/
> >>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps would be
> >>>>>> included
> >>>>>>>> in
> >>>>>>>>>> the downstream projects. TBD.
> >>>>>>>>>>
> >>>>>>>>>> Thoughts and feedback welcomed.
> >>>>>>>>
> >>>>>>>>
> >> ---------------------------------------------------------------------
> >>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Ash Berlin-Taylor <as...@apache.org>.
It’s not just release time, but any time we build a package which happens on “every” CI run. The normal unit tests will use code from airflow-common/src/airflow_common; the kube tests which build an image will build the dists and vendor in the code from that commit. 

There is only a single copy of the shared code committed to the repo, so there is never anything to synchronise. 

> On 4 Jul 2025, at 15:53, Amogh Desai <am...@gmail.com> wrote:
> 
> Thanks Ash.
> 
> This is really cool and helpful that you were able to test both scenarios
> -- repo checkout
> and also installing from the vendored package and the resolution worked
> fine too.
> 
> I like this idea compared the to relative import one for few reasons:
> - It feels like it will take some time to adjust to the new coding standard
> that we will lay
> if we impose relative imports in the shared dist
> - We can continue using repo wise absolute import standards, it is also
> much easier for situations
> when we do global search in IDE to find + replace, this could mean a change
> there
> - The vendoring work is a proven and established paradigm across projects
> and would
> out of box give us the build tooling we need also
> 
> Nothing too against the relative import but with the evidence provided
> above, vendored approach
> seems to only do us good.
> 
> Regarding synchronizing it, release time should be fine as long as we have
> a good CI workflow to probably
> catch such issues per PR if changes are made in shared dist? (precommit
> would make it really slow i guess)
> 
> If we can run our tests with vendored code we should be mostly covered.
> 
> Good effort all!
> 
> Thanks & Regards,
> Amogh Desai
> 
> 
>> On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>> 
>> Okay, I think I’ve got something that works and I’m happy with.
>> 
>> 
>> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>> 
>> This produces the following from `uv build task-sdk`
>> -
>> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
>> -
>> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>> 
>> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
>> 
>> ```
>> ❯ unzip -l dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip | grep
>> _vendor
>>       50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
>>     2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
>>       28  02-02-2020 00:00   airflow/sdk/_vendor/airflow_common.pyi
>>       18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
>>      785  02-02-2020 00:00
>> airflow/sdk/_vendor/airflow_common/__init__.py
>>    10628  02-02-2020 00:00
>> airflow/sdk/_vendor/airflow_common/timezone.py
>> ```
>> 
>> And similarly in the .tar.gz, so our “sdist” is complete too:
>> ```
>> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep _vendor
>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>> 
>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>> 
>> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
>> ```
>> 
>> The plugin works at build time by including/copying the libs specified in
>> vendor.txt into place (and let `vendoring` take care of import rewrites.)
>> 
>> For the imports to continue to work at “dev” time/from a repo checkout, I
>> have added a import finder to `sys.meta_path`, and since it’s at the end of
>> the list it will only be used if the normal import can’t find things.
>> 
>> 
>> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>> 
>> This doesn’t quite give us the same runtime effect “import rewriting”
>> affect, as in this approach `airflow_common` is directly loaded (i.e.
>> airflow.sdk._vendor.airflow_common and airflow_common exist in
>> sys.modules), but it does work for everything that I was able to test..
>> 
>> I tested it with the diff at the end of this message. My test ipython
>> shell:
>> 
>> ```
>> In [1]: from airflow.sdk._vendor.airflow_common.timezone import foo
>> 
>> In [2]: foo
>> Out[2]: 1
>> 
>> In [3]: import airflow.sdk._vendor.airflow_common
>> 
>> In [4]: import airflow.sdk._vendor.airflow_common.timezone
>> 
>> In [5]: airflow.sdk._vendor.airflow_common.__file__
>> Out[5]:
>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>> 
>> In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
>> Out[6]:
>> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>> 
>> ```
>> 
>> 
>> And in an standalone environment with the SDK dist I built (it needed the
>> matching airflow-core right now, but that is nothing to do with this
>> discussion):
>> 
>> ```
>> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
>> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
>> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
>> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
>> Type 'copyright', 'credits' or 'license' for more information
>> IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for help.
>> Tip: You can use `%hist` to view history, see the options with `%history?`
>> 
>> In [1]: import airflow.sdk._vendor.airflow_common.timezone
>> 
>> In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
>> Out[2]:
>> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
>> ``
>> 
>> 
>> 
>> ```diff
>> diff --git a/airflow-common/src/airflow_common/__init__.py
>> b/airflow-common/src/airflow_common/__init__.py
>> index 13a83393a9..927b7c6b61 100644
>> --- a/airflow-common/src/airflow_common/__init__.py
>> +++ b/airflow-common/src/airflow_common/__init__.py
>> @@ -14,3 +14,5 @@
>> # KIND, either express or implied.  See the License for the
>> # specific language governing permissions and limitations
>> # under the License.
>> +
>> +foo = 1
>> diff --git a/airflow-common/src/airflow_common/timezone.py
>> b/airflow-common/src/airflow_common/timezone.py
>> index 340b924c66..58384ef20f 100644
>> --- a/airflow-common/src/airflow_common/timezone.py
>> +++ b/airflow-common/src/airflow_common/timezone.py
>> @@ -36,6 +36,9 @@ _PENDULUM3 =
>> version.parse(metadata.version("pendulum")).major == 3
>> # - FixedTimezone(0, "UTC") in pendulum 2
>> utc = pendulum.UTC
>> 
>> +
>> +from airflow_common import foo
>> +
>> TIMEZONE: Timezone
>> 
>> 
>> ```
>> 
>>>> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com> wrote:
>>> 
>>> I think both approaches are doable:
>>> 
>>> 1) -> We can very easily prevent bad imports by pre-commit when importing
>>> from different distributions and make sure we are only doing relative
>>> imports in the shared modules. We are doing plenty of this already. And
>> yes
>>> it would require relative links we currently do not allow.
>>> 
>>> 2) -> has one disadvantage that someone at some point in time will have
>> to
>>> decide to synchronize this and if it happens just before release (I bet
>>> this is going to happen) this will lead to solving problems that would
>>> normally be solved during PR when you make a change (i.e. symbolic link
>> has
>>> the advantage that whoever modifies shared code will be immediately
>>> notified in their PR - that they broke something because either static
>>> checks or mypy or tests fail.
>>> 
>>> Ash, do you have an idea of a process (who and when) does the
>>> synchronisation in case of vendoring? Maybe we could solve it if it is
>> done
>>> more frequently and with some regularity? We could potentially force
>>> re-vendoring at PR time as well any time shared code changes (and prevent
>>> it by pre-commit. And I can't think of some place (other than releases)
>> in
>>> our development workflow and that seems to be a bit too late as puts an
>>> extra effort on fixing potential incompatibilities introduced on release
>>> manager and delays the release. WDYT?
>>> 
>>> Re: relative links. I think for a shared library we could potentially
>> relax
>>> this and allow them (and actually disallow absolute links in the pieces
>> of
>>> code that are shared - again, by pre-commit). As I recall, the only
>> reason
>>> we forbade the relative links is because of how we are (or maybe were)
>>> doing DAG parsing and failures resulting from it. So we decided to just
>> not
>>> allow it to keep consistency. The way how Dag parsing works is that when
>>> you are using importlib to read the Dag from a file, the relative imports
>>> do not work as it does not know what they should be relative to. But if
>>> relative import is done from an imported package, it should be no
>> problem,
>>> I think - otherwise our Dags would not be able to import any library that
>>> uses relative imports.
>>> 
>>> Of course consistency might be the reason why we do not want to introduce
>>> relative imports. I don't see it as an issue if it is guarded by
>> pre-commit
>>> though.
>>> 
>>> J.
>>> 
>>> 
>>> J.
>>> 
>>> 
>>> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <as...@apache.org>
>>> napisał:
>>> 
>>>> Oh yes, symlinks will work, with one big caveat: It does mean you can’t
>>>> use absolute imports in one common module to another.
>>>> 
>>>> For example
>>>> 
>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>>>> where we have
>>>> 
>>>> ```
>>>> from airflow.utils.module_loading import import_string
>>>> ```
>>>> 
>>>> if we want to move serve_logs into this common lib that is then
>> symlinked
>>>> then we wouldn’t be able to have `from airflow_common.module_loading
>> import
>>>> import_string`.
>>>> 
>>>> I can think of two possible solutions here.
>>>> 
>>>> 1) is to allow/require relative imports in this shared lib, so `from
>>>> .module_loading import import_string`
>>>> 2) is to use `vendoring`[1] (from the pip maintainers) which will handle
>>>> import-rewriting for us.
>>>> 
>>>> I’d entirely forgot that symlinks in repos was a thing, so I prepared a
>>>> minimal POC/demo of what vendoring approach could look like here
>>>> 
>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>>>> 
>>>> Now personally I am more than happy with relative imports, but generally
>>>> as a project we have avoided them, so I think that limits what we could
>> do
>>>> with a symlink based approach.
>>>> 
>>>> -ash
>>>> 
>>>> [1] https://github.com/pradyunsg/vendoring
>>>> 
>>>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <go...@gmail.com>
>>>> wrote:
>>>>> 
>>>>> Thanks Ash
>>>>> 
>>>>> Yes agree option 2 would be preferred for me. Making sure we have all
>> the
>>>>> gaurdriles to protect any unwanted behaviour in code sharing and
>>>> executing
>>>>> right of tests between the packages.
>>>>> 
>>>>> Agree with others, option 2 would be
>>>>> 
>>>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <am...@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> Thanks for starting this discussion, Ash.
>>>>>> 
>>>>>> I would prefer option 2 here with proper tooling to handle the code
>>>>>> duplication at *release* time.
>>>>>> It is best to have a dist that has all it needs in itself.
>>>>>> 
>>>>>> Option 1 could very quickly get out of hand and if we decide to
>> separate
>>>>>> triggerer /
>>>>>> dag processor / config etc etc as separate packages, back compat is
>>>> going
>>>>>> to be a nightmare
>>>>>> and will bite us harder than we anticipate.
>>>>>> 
>>>>>> Thanks & Regards,
>>>>>> Amogh Desai
>>>>>> 
>>>>>> 
>>>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <ka...@gmail.com>
>> wrote:
>>>>>> 
>>>>>>> I prefer Option 2 as well to avoid matrix of dependencies
>>>>>>> 
>>>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
>> <j_scheffler@gmx.de.invalid
>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> I'd also rather prefer option 2 - reason here is it is rather
>>>> pragmatic
>>>>>>>> and we no not need to cut another package and have less package
>> counts
>>>>>>>> and dependencies.
>>>>>>>> 
>>>>>>>> I remember some time ago I was checking (together with Jarek, I am
>> not
>>>>>>>> sure anymore...) if the usage of symlinks would be possible. To keep
>>>>>> the
>>>>>>>> source in one package but "symlink" it into another. If then at
>> point
>>>>>> of
>>>>>>>> packaging/release the files are materialized we have 1 set of code.
>>>>>>>> 
>>>>>>>> Otherwise if not possible still the redundancy could be solved by a
>>>>>>>> pre-commit hook - and in Git the files are de-duplicated anyway
>> based
>>>>>> on
>>>>>>>> content hash, so this does not hurt.
>>>>>>>> 
>>>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>>>>>>> I support option 2 with proper automation & CI - the reasonings
>>>>>> you've
>>>>>>>>> shown for that make sense to me.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Shahar
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <as...@apache.org>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hello everyone,
>>>>>>>>>> 
>>>>>>>>>> As we work on finishing off the code-level separation of Task SDK
>>>>>> and
>>>>>>>> Core
>>>>>>>>>> (scheduler etc) we have come across some situations where we would
>>>>>>> like
>>>>>>>> to
>>>>>>>>>> share code between these.
>>>>>>>>>> 
>>>>>>>>>> However it’s not as straight forward of “just put it in a common
>>>>>> dist
>>>>>>>> they
>>>>>>>>>> both depend upon” because one of the goals of the Task SDK
>>>>>> separation
>>>>>>>> was
>>>>>>>>>> to have 100% complete version independence between the two,
>> ideally
>>>>>>>> even if
>>>>>>>>>> they are built into the same image and venv. Most of the reason
>> why
>>>>>>> this
>>>>>>>>>> isn’t straight forward comes down to backwards compatibility - if
>> we
>>>>>>>> make
>>>>>>>>>> an change to the common/shared distribution
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> We’ve listed the options we have thought about in
>>>>>>>>>> https://github.com/apache/airflow/issues/51545 (but that covers
>>>>>> some
>>>>>>>> more
>>>>>>>>>> things that I don’t want to get in to in this discussion such as
>>>>>>>> possibly
>>>>>>>>>> separating operators and executors out of a single provider dist.)
>>>>>>>>>> 
>>>>>>>>>> To give a concrete example of some code I would like to share
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>>>>>>>> — logging config. Another thing we will want to share will be the
>>>>>>>>>> AirflowConfigParser class from airflow.configuration (but notably:
>>>>>>> only
>>>>>>>> the
>>>>>>>>>> parser class, _not_ the default config values, again, lets not
>> dwell
>>>>>>> on
>>>>>>>> the
>>>>>>>>>> specifics of that)
>>>>>>>>>> 
>>>>>>>>>> So to bring the options listed in the issue here for discussion,
>>>>>>> broadly
>>>>>>>>>> speaking there are two high-level approaches:
>>>>>>>>>> 
>>>>>>>>>> 1. A single shared distribution
>>>>>>>>>> 2. No shared package and copy/duplicate code
>>>>>>>>>> 
>>>>>>>>>> The advantage of Approach 1 is that we only have the code in one
>>>>>>> place.
>>>>>>>>>> However for me, at least in this specific case of Logging config
>> or
>>>>>>>>>> AirflowConfigParser class is that backwards compatibility is much
>>>>>> much
>>>>>>>>>> harder.
>>>>>>>>>> 
>>>>>>>>>> The main advantage of Approach 2 is the the code is released
>>>>>>>> with/embedded
>>>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would contain the right
>>>>>>>> version
>>>>>>>>>> of the logging config and ConfigParser etc). The downside is that
>>>>>>> either
>>>>>>>>>> the code will need to be duplicated in the repo, or better yet it
>>>>>>> would
>>>>>>>>>> live in a single place in the repo, but some tooling (TBD) will
>>>>>>>>>> automatically handle the duplication, either at commit time, or my
>>>>>>>>>> preference, at release time.
>>>>>>>>>> 
>>>>>>>>>> For this kind of shared “utility” code I am very strongly leaning
>>>>>>>> towards
>>>>>>>>>> option 2 with automation, as otherwise I think the backwards
>>>>>>>> compatibility
>>>>>>>>>> requirements would make it unworkable (very quickly over time the
>>>>>>>>>> combinations we would have to test would just be unreasonable)
>> and I
>>>>>>>> don’t
>>>>>>>>>> feel confident we can have things as stable as we need to really
>>>>>>> deliver
>>>>>>>>>> the version separation/independency I want to delivery with
>> AIP-72.
>>>>>>>>>> 
>>>>>>>>>> So unless someone feels very strongly about this, I will come up
>>>>>> with
>>>>>>> a
>>>>>>>>>> draft PR for further discussion that will implement code sharing
>> via
>>>>>>>>>> “vendoring” it at build time. I have an idea of how I can achieve
>>>>>> this
>>>>>>>> so
>>>>>>>>>> we have a single version in the repo and it’ll work there, but at
>>>>>>>> runtime
>>>>>>>>>> we vendor it in to the shipped dist so it lives at something like
>>>>>>>>>> `airflow.sdk._vendor` etc.
>>>>>>>>>> 
>>>>>>>>>> In terms of repo layout, this likely means we would end up with:
>>>>>>>>>> 
>>>>>>>>>> airflow-core/pyproject.toml
>>>>>>>>>> airflow-core/src/
>>>>>>>>>> airflow-core/tests/
>>>>>>>>>> task-sdk/pyproject.toml
>>>>>>>>>> task-sdk/src/
>>>>>>>>>> task-sdk/tests/
>>>>>>>>>> airflow-common/src
>>>>>>>>>> airflow-common/tests/
>>>>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps would be
>>>>>> included
>>>>>>>> in
>>>>>>>>>> the downstream projects. TBD.
>>>>>>>>>> 
>>>>>>>>>> Thoughts and feedback welcomed.
>>>>>>>> 
>>>>>>>> 
>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org


Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Amogh Desai <am...@gmail.com>.
Thanks Ash.

This is really cool and helpful that you were able to test both scenarios
-- repo checkout
and also installing from the vendored package and the resolution worked
fine too.

I like this idea compared the to relative import one for few reasons:
- It feels like it will take some time to adjust to the new coding standard
that we will lay
if we impose relative imports in the shared dist
- We can continue using repo wise absolute import standards, it is also
much easier for situations
when we do global search in IDE to find + replace, this could mean a change
there
- The vendoring work is a proven and established paradigm across projects
and would
out of box give us the build tooling we need also

Nothing too against the relative import but with the evidence provided
above, vendored approach
seems to only do us good.

Regarding synchronizing it, release time should be fine as long as we have
a good CI workflow to probably
catch such issues per PR if changes are made in shared dist? (precommit
would make it really slow i guess)

If we can run our tests with vendored code we should be mostly covered.

Good effort all!

Thanks & Regards,
Amogh Desai


On Fri, Jul 4, 2025 at 7:23 PM Ash Berlin-Taylor <as...@apache.org> wrote:

> Okay, I think I’ve got something that works and I’m happy with.
>
>
> https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core
>
> This produces the following from `uv build task-sdk`
> -
> https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
> -
> https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip
>
> (`.whl.zip` as GH won't allow .whl upload, but will .zip)
>
> ```
> ❯ unzip -l dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip | grep
> _vendor
>        50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
>      2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
>        28  02-02-2020 00:00   airflow/sdk/_vendor/airflow_common.pyi
>        18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
>       785  02-02-2020 00:00
>  airflow/sdk/_vendor/airflow_common/__init__.py
>     10628  02-02-2020 00:00
>  airflow/sdk/_vendor/airflow_common/timezone.py
> ```
>
> And similarly in the .tar.gz, so our “sdist” is complete too:
> ```
> ❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep _vendor
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
>
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
>
> apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
> ```
>
> The plugin works at build time by including/copying the libs specified in
> vendor.txt into place (and let `vendoring` take care of import rewrites.)
>
> For the imports to continue to work at “dev” time/from a repo checkout, I
> have added a import finder to `sys.meta_path`, and since it’s at the end of
> the list it will only be used if the normal import can’t find things.
>
>
> https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py
>
> This doesn’t quite give us the same runtime effect “import rewriting”
> affect, as in this approach `airflow_common` is directly loaded (i.e.
> airflow.sdk._vendor.airflow_common and airflow_common exist in
> sys.modules), but it does work for everything that I was able to test..
>
> I tested it with the diff at the end of this message. My test ipython
> shell:
>
> ```
> In [1]: from airflow.sdk._vendor.airflow_common.timezone import foo
>
> In [2]: foo
> Out[2]: 1
>
> In [3]: import airflow.sdk._vendor.airflow_common
>
> In [4]: import airflow.sdk._vendor.airflow_common.timezone
>
> In [5]: airflow.sdk._vendor.airflow_common.__file__
> Out[5]:
> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'
>
> In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
> Out[6]:
> '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'
>
> ```
>
>
> And in an standalone environment with the SDK dist I built (it needed the
> matching airflow-core right now, but that is nothing to do with this
> discussion):
>
> ```
> ❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with
> dist/apache_airflow_core-3.1.0-py3-none-any.whl --with
> dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
> Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
> Type 'copyright', 'credits' or 'license' for more information
> IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for help.
> Tip: You can use `%hist` to view history, see the options with `%history?`
>
> In [1]: import airflow.sdk._vendor.airflow_common.timezone
>
> In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
> Out[2]:
> '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
> ``
>
>
>
> ```diff
> diff --git a/airflow-common/src/airflow_common/__init__.py
> b/airflow-common/src/airflow_common/__init__.py
> index 13a83393a9..927b7c6b61 100644
> --- a/airflow-common/src/airflow_common/__init__.py
> +++ b/airflow-common/src/airflow_common/__init__.py
> @@ -14,3 +14,5 @@
>  # KIND, either express or implied.  See the License for the
>  # specific language governing permissions and limitations
>  # under the License.
> +
> +foo = 1
> diff --git a/airflow-common/src/airflow_common/timezone.py
> b/airflow-common/src/airflow_common/timezone.py
> index 340b924c66..58384ef20f 100644
> --- a/airflow-common/src/airflow_common/timezone.py
> +++ b/airflow-common/src/airflow_common/timezone.py
> @@ -36,6 +36,9 @@ _PENDULUM3 =
> version.parse(metadata.version("pendulum")).major == 3
>  # - FixedTimezone(0, "UTC") in pendulum 2
>  utc = pendulum.UTC
>
> +
> +from airflow_common import foo
> +
>  TIMEZONE: Timezone
>
>
> ```
>
> > On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > I think both approaches are doable:
> >
> > 1) -> We can very easily prevent bad imports by pre-commit when importing
> > from different distributions and make sure we are only doing relative
> > imports in the shared modules. We are doing plenty of this already. And
> yes
> > it would require relative links we currently do not allow.
> >
> > 2) -> has one disadvantage that someone at some point in time will have
> to
> > decide to synchronize this and if it happens just before release (I bet
> > this is going to happen) this will lead to solving problems that would
> > normally be solved during PR when you make a change (i.e. symbolic link
> has
> > the advantage that whoever modifies shared code will be immediately
> > notified in their PR - that they broke something because either static
> > checks or mypy or tests fail.
> >
> > Ash, do you have an idea of a process (who and when) does the
> > synchronisation in case of vendoring? Maybe we could solve it if it is
> done
> > more frequently and with some regularity? We could potentially force
> > re-vendoring at PR time as well any time shared code changes (and prevent
> > it by pre-commit. And I can't think of some place (other than releases)
> in
> > our development workflow and that seems to be a bit too late as puts an
> > extra effort on fixing potential incompatibilities introduced on release
> > manager and delays the release. WDYT?
> >
> > Re: relative links. I think for a shared library we could potentially
> relax
> > this and allow them (and actually disallow absolute links in the pieces
> of
> > code that are shared - again, by pre-commit). As I recall, the only
> reason
> > we forbade the relative links is because of how we are (or maybe were)
> > doing DAG parsing and failures resulting from it. So we decided to just
> not
> > allow it to keep consistency. The way how Dag parsing works is that when
> > you are using importlib to read the Dag from a file, the relative imports
> > do not work as it does not know what they should be relative to. But if
> > relative import is done from an imported package, it should be no
> problem,
> > I think - otherwise our Dags would not be able to import any library that
> > uses relative imports.
> >
> > Of course consistency might be the reason why we do not want to introduce
> > relative imports. I don't see it as an issue if it is guarded by
> pre-commit
> > though.
> >
> > J.
> >
> >
> > J.
> >
> >
> > czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <as...@apache.org>
> > napisał:
> >
> >> Oh yes, symlinks will work, with one big caveat: It does mean you can’t
> >> use absolute imports in one common module to another.
> >>
> >> For example
> >>
> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
> >> where we have
> >>
> >> ```
> >> from airflow.utils.module_loading import import_string
> >> ```
> >>
> >> if we want to move serve_logs into this common lib that is then
> symlinked
> >> then we wouldn’t be able to have `from airflow_common.module_loading
> import
> >> import_string`.
> >>
> >> I can think of two possible solutions here.
> >>
> >> 1) is to allow/require relative imports in this shared lib, so `from
> >> .module_loading import import_string`
> >> 2) is to use `vendoring`[1] (from the pip maintainers) which will handle
> >> import-rewriting for us.
> >>
> >> I’d entirely forgot that symlinks in repos was a thing, so I prepared a
> >> minimal POC/demo of what vendoring approach could look like here
> >>
> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
> >>
> >> Now personally I am more than happy with relative imports, but generally
> >> as a project we have avoided them, so I think that limits what we could
> do
> >> with a symlink based approach.
> >>
> >> -ash
> >>
> >> [1] https://github.com/pradyunsg/vendoring
> >>
> >>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <go...@gmail.com>
> >> wrote:
> >>>
> >>> Thanks Ash
> >>>
> >>> Yes agree option 2 would be preferred for me. Making sure we have all
> the
> >>> gaurdriles to protect any unwanted behaviour in code sharing and
> >> executing
> >>> right of tests between the packages.
> >>>
> >>> Agree with others, option 2 would be
> >>>
> >>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <am...@gmail.com>
> >>> wrote:
> >>>
> >>>> Thanks for starting this discussion, Ash.
> >>>>
> >>>> I would prefer option 2 here with proper tooling to handle the code
> >>>> duplication at *release* time.
> >>>> It is best to have a dist that has all it needs in itself.
> >>>>
> >>>> Option 1 could very quickly get out of hand and if we decide to
> separate
> >>>> triggerer /
> >>>> dag processor / config etc etc as separate packages, back compat is
> >> going
> >>>> to be a nightmare
> >>>> and will bite us harder than we anticipate.
> >>>>
> >>>> Thanks & Regards,
> >>>> Amogh Desai
> >>>>
> >>>>
> >>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <ka...@gmail.com>
> wrote:
> >>>>
> >>>>> I prefer Option 2 as well to avoid matrix of dependencies
> >>>>>
> >>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler
> <j_scheffler@gmx.de.invalid
> >>>
> >>>>> wrote:
> >>>>>
> >>>>>> I'd also rather prefer option 2 - reason here is it is rather
> >> pragmatic
> >>>>>> and we no not need to cut another package and have less package
> counts
> >>>>>> and dependencies.
> >>>>>>
> >>>>>> I remember some time ago I was checking (together with Jarek, I am
> not
> >>>>>> sure anymore...) if the usage of symlinks would be possible. To keep
> >>>> the
> >>>>>> source in one package but "symlink" it into another. If then at
> point
> >>>> of
> >>>>>> packaging/release the files are materialized we have 1 set of code.
> >>>>>>
> >>>>>> Otherwise if not possible still the redundancy could be solved by a
> >>>>>> pre-commit hook - and in Git the files are de-duplicated anyway
> based
> >>>> on
> >>>>>> content hash, so this does not hurt.
> >>>>>>
> >>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
> >>>>>>> I support option 2 with proper automation & CI - the reasonings
> >>>> you've
> >>>>>>> shown for that make sense to me.
> >>>>>>>
> >>>>>>>
> >>>>>>> Shahar
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <as...@apache.org>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> Hello everyone,
> >>>>>>>>
> >>>>>>>> As we work on finishing off the code-level separation of Task SDK
> >>>> and
> >>>>>> Core
> >>>>>>>> (scheduler etc) we have come across some situations where we would
> >>>>> like
> >>>>>> to
> >>>>>>>> share code between these.
> >>>>>>>>
> >>>>>>>> However it’s not as straight forward of “just put it in a common
> >>>> dist
> >>>>>> they
> >>>>>>>> both depend upon” because one of the goals of the Task SDK
> >>>> separation
> >>>>>> was
> >>>>>>>> to have 100% complete version independence between the two,
> ideally
> >>>>>> even if
> >>>>>>>> they are built into the same image and venv. Most of the reason
> why
> >>>>> this
> >>>>>>>> isn’t straight forward comes down to backwards compatibility - if
> we
> >>>>>> make
> >>>>>>>> an change to the common/shared distribution
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> We’ve listed the options we have thought about in
> >>>>>>>> https://github.com/apache/airflow/issues/51545 (but that covers
> >>>> some
> >>>>>> more
> >>>>>>>> things that I don’t want to get in to in this discussion such as
> >>>>>> possibly
> >>>>>>>> separating operators and executors out of a single provider dist.)
> >>>>>>>>
> >>>>>>>> To give a concrete example of some code I would like to share
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
> >>>>>>>> — logging config. Another thing we will want to share will be the
> >>>>>>>> AirflowConfigParser class from airflow.configuration (but notably:
> >>>>> only
> >>>>>> the
> >>>>>>>> parser class, _not_ the default config values, again, lets not
> dwell
> >>>>> on
> >>>>>> the
> >>>>>>>> specifics of that)
> >>>>>>>>
> >>>>>>>> So to bring the options listed in the issue here for discussion,
> >>>>> broadly
> >>>>>>>> speaking there are two high-level approaches:
> >>>>>>>>
> >>>>>>>> 1. A single shared distribution
> >>>>>>>> 2. No shared package and copy/duplicate code
> >>>>>>>>
> >>>>>>>> The advantage of Approach 1 is that we only have the code in one
> >>>>> place.
> >>>>>>>> However for me, at least in this specific case of Logging config
> or
> >>>>>>>> AirflowConfigParser class is that backwards compatibility is much
> >>>> much
> >>>>>>>> harder.
> >>>>>>>>
> >>>>>>>> The main advantage of Approach 2 is the the code is released
> >>>>>> with/embedded
> >>>>>>>> in the dist (i.e. apache-airflow-task-sdk would contain the right
> >>>>>> version
> >>>>>>>> of the logging config and ConfigParser etc). The downside is that
> >>>>> either
> >>>>>>>> the code will need to be duplicated in the repo, or better yet it
> >>>>> would
> >>>>>>>> live in a single place in the repo, but some tooling (TBD) will
> >>>>>>>> automatically handle the duplication, either at commit time, or my
> >>>>>>>> preference, at release time.
> >>>>>>>>
> >>>>>>>> For this kind of shared “utility” code I am very strongly leaning
> >>>>>> towards
> >>>>>>>> option 2 with automation, as otherwise I think the backwards
> >>>>>> compatibility
> >>>>>>>> requirements would make it unworkable (very quickly over time the
> >>>>>>>> combinations we would have to test would just be unreasonable)
> and I
> >>>>>> don’t
> >>>>>>>> feel confident we can have things as stable as we need to really
> >>>>> deliver
> >>>>>>>> the version separation/independency I want to delivery with
> AIP-72.
> >>>>>>>>
> >>>>>>>> So unless someone feels very strongly about this, I will come up
> >>>> with
> >>>>> a
> >>>>>>>> draft PR for further discussion that will implement code sharing
> via
> >>>>>>>> “vendoring” it at build time. I have an idea of how I can achieve
> >>>> this
> >>>>>> so
> >>>>>>>> we have a single version in the repo and it’ll work there, but at
> >>>>>> runtime
> >>>>>>>> we vendor it in to the shipped dist so it lives at something like
> >>>>>>>> `airflow.sdk._vendor` etc.
> >>>>>>>>
> >>>>>>>> In terms of repo layout, this likely means we would end up with:
> >>>>>>>>
> >>>>>>>> airflow-core/pyproject.toml
> >>>>>>>> airflow-core/src/
> >>>>>>>> airflow-core/tests/
> >>>>>>>> task-sdk/pyproject.toml
> >>>>>>>> task-sdk/src/
> >>>>>>>> task-sdk/tests/
> >>>>>>>> airflow-common/src
> >>>>>>>> airflow-common/tests/
> >>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps would be
> >>>> included
> >>>>>> in
> >>>>>>>> the downstream projects. TBD.
> >>>>>>>>
> >>>>>>>> Thoughts and feedback welcomed.
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Ash Berlin-Taylor <as...@apache.org>.
Okay, I think I’ve got something that works and I’m happy with. 

https://github.com/astronomer/airflow/tree/shared-vendored-lib-tasksdk-and-core

This produces the following from `uv build task-sdk`
- https://github.com/user-attachments/files/21058976/apache_airflow_task_sdk-1.1.0.tar.gz
- https://github.com/user-attachments/files/21058996/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip

(`.whl.zip` as GH won't allow .whl upload, but will .zip)

```
❯ unzip -l dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl.zip | grep _vendor
       50  02-02-2020 00:00   airflow/sdk/_vendor/.gitignore
     2082  02-02-2020 00:00   airflow/sdk/_vendor/__init__.py
       28  02-02-2020 00:00   airflow/sdk/_vendor/airflow_common.pyi
       18  02-02-2020 00:00   airflow/sdk/_vendor/vendor.txt
      785  02-02-2020 00:00   airflow/sdk/_vendor/airflow_common/__init__.py
    10628  02-02-2020 00:00   airflow/sdk/_vendor/airflow_common/timezone.py
```

And similarly in the .tar.gz, so our “sdist” is complete too:
```
❯ tar -tzf dist/apache_airflow_task_sdk-1.1.0.tar.gz |grep _vendor
apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/.gitignore
apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/__init__.py
apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common.pyi
apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/vendor.txt
apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/__init__.py
apache_airflow_task_sdk-1.1.0/src/airflow/sdk/_vendor/airflow_common/timezone.py
```

The plugin works at build time by including/copying the libs specified in vendor.txt into place (and let `vendoring` take care of import rewrites.)

For the imports to continue to work at “dev” time/from a repo checkout, I have added a import finder to `sys.meta_path`, and since it’s at the end of the list it will only be used if the normal import can’t find things.

https://github.com/astronomer/airflow/blob/996817782be6071b306a87af9f36fe1cf2d3aaa3/task-sdk/src/airflow/sdk/_vendor/__init__.py 

This doesn’t quite give us the same runtime effect “import rewriting” affect, as in this approach `airflow_common` is directly loaded (i.e. airflow.sdk._vendor.airflow_common and airflow_common exist in sys.modules), but it does work for everything that I was able to test..

I tested it with the diff at the end of this message. My test ipython shell:

```
In [1]: from airflow.sdk._vendor.airflow_common.timezone import foo

In [2]: foo
Out[2]: 1

In [3]: import airflow.sdk._vendor.airflow_common

In [4]: import airflow.sdk._vendor.airflow_common.timezone

In [5]: airflow.sdk._vendor.airflow_common.__file__
Out[5]: '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/__init__.py'

In [6]: airflow.sdk._vendor.airflow_common.timezone.__file__
Out[6]: '/Users/ash/code/airflow/airflow/airflow-common/src/airflow_common/timezone.py'

```


And in an standalone environment with the SDK dist I built (it needed the matching airflow-core right now, but that is nothing to do with this discussion):

```
❯ _AIRFLOW__AS_LIBRARY=1 uvx --python 3.12 --with dist/apache_airflow_core-3.1.0-py3-none-any.whl --with dist/apache_airflow_task_sdk-1.1.0-py3-none-any.whl ipython
Python 3.12.7 (main, Oct 16 2024, 07:12:08) [Clang 18.1.8 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for help.
Tip: You can use `%hist` to view history, see the options with `%history?`

In [1]: import airflow.sdk._vendor.airflow_common.timezone

In [2]: airflow.sdk._vendor.airflow_common.timezone.__file__
Out[2]: '/Users/ash/.cache/uv/archive-v0/WWq6r65aPto2eJOyPObEH/lib/python3.12/site-packages/airflow/sdk/_vendor/airflow_common/timezone.py’
``



```diff
diff --git a/airflow-common/src/airflow_common/__init__.py b/airflow-common/src/airflow_common/__init__.py
index 13a83393a9..927b7c6b61 100644
--- a/airflow-common/src/airflow_common/__init__.py
+++ b/airflow-common/src/airflow_common/__init__.py
@@ -14,3 +14,5 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+
+foo = 1
diff --git a/airflow-common/src/airflow_common/timezone.py b/airflow-common/src/airflow_common/timezone.py
index 340b924c66..58384ef20f 100644
--- a/airflow-common/src/airflow_common/timezone.py
+++ b/airflow-common/src/airflow_common/timezone.py
@@ -36,6 +36,9 @@ _PENDULUM3 = version.parse(metadata.version("pendulum")).major == 3
 # - FixedTimezone(0, "UTC") in pendulum 2
 utc = pendulum.UTC
 
+
+from airflow_common import foo
+
 TIMEZONE: Timezone
 
 
```

> On 3 Jul 2025, at 12:43, Jarek Potiuk <ja...@potiuk.com> wrote:
> 
> I think both approaches are doable:
> 
> 1) -> We can very easily prevent bad imports by pre-commit when importing
> from different distributions and make sure we are only doing relative
> imports in the shared modules. We are doing plenty of this already. And yes
> it would require relative links we currently do not allow.
> 
> 2) -> has one disadvantage that someone at some point in time will have to
> decide to synchronize this and if it happens just before release (I bet
> this is going to happen) this will lead to solving problems that would
> normally be solved during PR when you make a change (i.e. symbolic link has
> the advantage that whoever modifies shared code will be immediately
> notified in their PR - that they broke something because either static
> checks or mypy or tests fail.
> 
> Ash, do you have an idea of a process (who and when) does the
> synchronisation in case of vendoring? Maybe we could solve it if it is done
> more frequently and with some regularity? We could potentially force
> re-vendoring at PR time as well any time shared code changes (and prevent
> it by pre-commit. And I can't think of some place (other than releases) in
> our development workflow and that seems to be a bit too late as puts an
> extra effort on fixing potential incompatibilities introduced on release
> manager and delays the release. WDYT?
> 
> Re: relative links. I think for a shared library we could potentially relax
> this and allow them (and actually disallow absolute links in the pieces of
> code that are shared - again, by pre-commit). As I recall, the only reason
> we forbade the relative links is because of how we are (or maybe were)
> doing DAG parsing and failures resulting from it. So we decided to just not
> allow it to keep consistency. The way how Dag parsing works is that when
> you are using importlib to read the Dag from a file, the relative imports
> do not work as it does not know what they should be relative to. But if
> relative import is done from an imported package, it should be no problem,
> I think - otherwise our Dags would not be able to import any library that
> uses relative imports.
> 
> Of course consistency might be the reason why we do not want to introduce
> relative imports. I don't see it as an issue if it is guarded by pre-commit
> though.
> 
> J.
> 
> 
> J.
> 
> 
> czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <as...@apache.org>
> napisał:
> 
>> Oh yes, symlinks will work, with one big caveat: It does mean you can’t
>> use absolute imports in one common module to another.
>> 
>> For example
>> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
>> where we have
>> 
>> ```
>> from airflow.utils.module_loading import import_string
>> ```
>> 
>> if we want to move serve_logs into this common lib that is then symlinked
>> then we wouldn’t be able to have `from airflow_common.module_loading import
>> import_string`.
>> 
>> I can think of two possible solutions here.
>> 
>> 1) is to allow/require relative imports in this shared lib, so `from
>> .module_loading import import_string`
>> 2) is to use `vendoring`[1] (from the pip maintainers) which will handle
>> import-rewriting for us.
>> 
>> I’d entirely forgot that symlinks in repos was a thing, so I prepared a
>> minimal POC/demo of what vendoring approach could look like here
>> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>> 
>> Now personally I am more than happy with relative imports, but generally
>> as a project we have avoided them, so I think that limits what we could do
>> with a symlink based approach.
>> 
>> -ash
>> 
>> [1] https://github.com/pradyunsg/vendoring
>> 
>>> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <go...@gmail.com>
>> wrote:
>>> 
>>> Thanks Ash
>>> 
>>> Yes agree option 2 would be preferred for me. Making sure we have all the
>>> gaurdriles to protect any unwanted behaviour in code sharing and
>> executing
>>> right of tests between the packages.
>>> 
>>> Agree with others, option 2 would be
>>> 
>>> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <am...@gmail.com>
>>> wrote:
>>> 
>>>> Thanks for starting this discussion, Ash.
>>>> 
>>>> I would prefer option 2 here with proper tooling to handle the code
>>>> duplication at *release* time.
>>>> It is best to have a dist that has all it needs in itself.
>>>> 
>>>> Option 1 could very quickly get out of hand and if we decide to separate
>>>> triggerer /
>>>> dag processor / config etc etc as separate packages, back compat is
>> going
>>>> to be a nightmare
>>>> and will bite us harder than we anticipate.
>>>> 
>>>> Thanks & Regards,
>>>> Amogh Desai
>>>> 
>>>> 
>>>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <ka...@gmail.com> wrote:
>>>> 
>>>>> I prefer Option 2 as well to avoid matrix of dependencies
>>>>> 
>>>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler <j_scheffler@gmx.de.invalid
>>> 
>>>>> wrote:
>>>>> 
>>>>>> I'd also rather prefer option 2 - reason here is it is rather
>> pragmatic
>>>>>> and we no not need to cut another package and have less package counts
>>>>>> and dependencies.
>>>>>> 
>>>>>> I remember some time ago I was checking (together with Jarek, I am not
>>>>>> sure anymore...) if the usage of symlinks would be possible. To keep
>>>> the
>>>>>> source in one package but "symlink" it into another. If then at point
>>>> of
>>>>>> packaging/release the files are materialized we have 1 set of code.
>>>>>> 
>>>>>> Otherwise if not possible still the redundancy could be solved by a
>>>>>> pre-commit hook - and in Git the files are de-duplicated anyway based
>>>> on
>>>>>> content hash, so this does not hurt.
>>>>>> 
>>>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>>>>> I support option 2 with proper automation & CI - the reasonings
>>>> you've
>>>>>>> shown for that make sense to me.
>>>>>>> 
>>>>>>> 
>>>>>>> Shahar
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <as...@apache.org>
>>>>> wrote:
>>>>>>> 
>>>>>>>> Hello everyone,
>>>>>>>> 
>>>>>>>> As we work on finishing off the code-level separation of Task SDK
>>>> and
>>>>>> Core
>>>>>>>> (scheduler etc) we have come across some situations where we would
>>>>> like
>>>>>> to
>>>>>>>> share code between these.
>>>>>>>> 
>>>>>>>> However it’s not as straight forward of “just put it in a common
>>>> dist
>>>>>> they
>>>>>>>> both depend upon” because one of the goals of the Task SDK
>>>> separation
>>>>>> was
>>>>>>>> to have 100% complete version independence between the two, ideally
>>>>>> even if
>>>>>>>> they are built into the same image and venv. Most of the reason why
>>>>> this
>>>>>>>> isn’t straight forward comes down to backwards compatibility - if we
>>>>>> make
>>>>>>>> an change to the common/shared distribution
>>>>>>>> 
>>>>>>>> 
>>>>>>>> We’ve listed the options we have thought about in
>>>>>>>> https://github.com/apache/airflow/issues/51545 (but that covers
>>>> some
>>>>>> more
>>>>>>>> things that I don’t want to get in to in this discussion such as
>>>>>> possibly
>>>>>>>> separating operators and executors out of a single provider dist.)
>>>>>>>> 
>>>>>>>> To give a concrete example of some code I would like to share
>>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>>>>>> — logging config. Another thing we will want to share will be the
>>>>>>>> AirflowConfigParser class from airflow.configuration (but notably:
>>>>> only
>>>>>> the
>>>>>>>> parser class, _not_ the default config values, again, lets not dwell
>>>>> on
>>>>>> the
>>>>>>>> specifics of that)
>>>>>>>> 
>>>>>>>> So to bring the options listed in the issue here for discussion,
>>>>> broadly
>>>>>>>> speaking there are two high-level approaches:
>>>>>>>> 
>>>>>>>> 1. A single shared distribution
>>>>>>>> 2. No shared package and copy/duplicate code
>>>>>>>> 
>>>>>>>> The advantage of Approach 1 is that we only have the code in one
>>>>> place.
>>>>>>>> However for me, at least in this specific case of Logging config or
>>>>>>>> AirflowConfigParser class is that backwards compatibility is much
>>>> much
>>>>>>>> harder.
>>>>>>>> 
>>>>>>>> The main advantage of Approach 2 is the the code is released
>>>>>> with/embedded
>>>>>>>> in the dist (i.e. apache-airflow-task-sdk would contain the right
>>>>>> version
>>>>>>>> of the logging config and ConfigParser etc). The downside is that
>>>>> either
>>>>>>>> the code will need to be duplicated in the repo, or better yet it
>>>>> would
>>>>>>>> live in a single place in the repo, but some tooling (TBD) will
>>>>>>>> automatically handle the duplication, either at commit time, or my
>>>>>>>> preference, at release time.
>>>>>>>> 
>>>>>>>> For this kind of shared “utility” code I am very strongly leaning
>>>>>> towards
>>>>>>>> option 2 with automation, as otherwise I think the backwards
>>>>>> compatibility
>>>>>>>> requirements would make it unworkable (very quickly over time the
>>>>>>>> combinations we would have to test would just be unreasonable) and I
>>>>>> don’t
>>>>>>>> feel confident we can have things as stable as we need to really
>>>>> deliver
>>>>>>>> the version separation/independency I want to delivery with AIP-72.
>>>>>>>> 
>>>>>>>> So unless someone feels very strongly about this, I will come up
>>>> with
>>>>> a
>>>>>>>> draft PR for further discussion that will implement code sharing via
>>>>>>>> “vendoring” it at build time. I have an idea of how I can achieve
>>>> this
>>>>>> so
>>>>>>>> we have a single version in the repo and it’ll work there, but at
>>>>>> runtime
>>>>>>>> we vendor it in to the shipped dist so it lives at something like
>>>>>>>> `airflow.sdk._vendor` etc.
>>>>>>>> 
>>>>>>>> In terms of repo layout, this likely means we would end up with:
>>>>>>>> 
>>>>>>>> airflow-core/pyproject.toml
>>>>>>>> airflow-core/src/
>>>>>>>> airflow-core/tests/
>>>>>>>> task-sdk/pyproject.toml
>>>>>>>> task-sdk/src/
>>>>>>>> task-sdk/tests/
>>>>>>>> airflow-common/src
>>>>>>>> airflow-common/tests/
>>>>>>>> # Possibly no airflow-common/pyproject.toml, as deps would be
>>>> included
>>>>>> in
>>>>>>>> the downstream projects. TBD.
>>>>>>>> 
>>>>>>>> Thoughts and feedback welcomed.
>>>>>> 
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
>> 


Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Jarek Potiuk <ja...@potiuk.com>.
I think both approaches are doable:

1) -> We can very easily prevent bad imports by pre-commit when importing
from different distributions and make sure we are only doing relative
imports in the shared modules. We are doing plenty of this already. And yes
it would require relative links we currently do not allow.

2) -> has one disadvantage that someone at some point in time will have to
decide to synchronize this and if it happens just before release (I bet
this is going to happen) this will lead to solving problems that would
normally be solved during PR when you make a change (i.e. symbolic link has
the advantage that whoever modifies shared code will be immediately
notified in their PR - that they broke something because either static
checks or mypy or tests fail.

Ash, do you have an idea of a process (who and when) does the
synchronisation in case of vendoring? Maybe we could solve it if it is done
more frequently and with some regularity? We could potentially force
re-vendoring at PR time as well any time shared code changes (and prevent
it by pre-commit. And I can't think of some place (other than releases) in
our development workflow and that seems to be a bit too late as puts an
extra effort on fixing potential incompatibilities introduced on release
manager and delays the release. WDYT?

Re: relative links. I think for a shared library we could potentially relax
this and allow them (and actually disallow absolute links in the pieces of
code that are shared - again, by pre-commit). As I recall, the only reason
we forbade the relative links is because of how we are (or maybe were)
doing DAG parsing and failures resulting from it. So we decided to just not
allow it to keep consistency. The way how Dag parsing works is that when
you are using importlib to read the Dag from a file, the relative imports
do not work as it does not know what they should be relative to. But if
relative import is done from an imported package, it should be no problem,
I think - otherwise our Dags would not be able to import any library that
uses relative imports.

Of course consistency might be the reason why we do not want to introduce
relative imports. I don't see it as an issue if it is guarded by pre-commit
though.

J.


J.


czw., 3 lip 2025, 12:11 użytkownik Ash Berlin-Taylor <as...@apache.org>
napisał:

> Oh yes, symlinks will work, with one big caveat: It does mean you can’t
> use absolute imports in one common module to another.
>
> For example
> https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41
> where we have
>
> ```
> from airflow.utils.module_loading import import_string
> ```
>
>  if we want to move serve_logs into this common lib that is then symlinked
> then we wouldn’t be able to have `from airflow_common.module_loading import
> import_string`.
>
> I can think of two possible solutions here.
>
> 1) is to allow/require relative imports in this shared lib, so `from
> .module_loading import import_string`
> 2) is to use `vendoring`[1] (from the pip maintainers) which will handle
> import-rewriting for us.
>
> I’d entirely forgot that symlinks in repos was a thing, so I prepared a
> minimal POC/demo of what vendoring approach could look like here
> https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3
>
> Now personally I am more than happy with relative imports, but generally
> as a project we have avoided them, so I think that limits what we could do
> with a symlink based approach.
>
> -ash
>
> [1] https://github.com/pradyunsg/vendoring
>
> > On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <go...@gmail.com>
> wrote:
> >
> > Thanks Ash
> >
> > Yes agree option 2 would be preferred for me. Making sure we have all the
> > gaurdriles to protect any unwanted behaviour in code sharing and
> executing
> > right of tests between the packages.
> >
> > Agree with others, option 2 would be
> >
> > On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <am...@gmail.com>
> > wrote:
> >
> >> Thanks for starting this discussion, Ash.
> >>
> >> I would prefer option 2 here with proper tooling to handle the code
> >> duplication at *release* time.
> >> It is best to have a dist that has all it needs in itself.
> >>
> >> Option 1 could very quickly get out of hand and if we decide to separate
> >> triggerer /
> >> dag processor / config etc etc as separate packages, back compat is
> going
> >> to be a nightmare
> >> and will bite us harder than we anticipate.
> >>
> >> Thanks & Regards,
> >> Amogh Desai
> >>
> >>
> >> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <ka...@gmail.com> wrote:
> >>
> >>> I prefer Option 2 as well to avoid matrix of dependencies
> >>>
> >>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler <j_scheffler@gmx.de.invalid
> >
> >>> wrote:
> >>>
> >>>> I'd also rather prefer option 2 - reason here is it is rather
> pragmatic
> >>>> and we no not need to cut another package and have less package counts
> >>>> and dependencies.
> >>>>
> >>>> I remember some time ago I was checking (together with Jarek, I am not
> >>>> sure anymore...) if the usage of symlinks would be possible. To keep
> >> the
> >>>> source in one package but "symlink" it into another. If then at point
> >> of
> >>>> packaging/release the files are materialized we have 1 set of code.
> >>>>
> >>>> Otherwise if not possible still the redundancy could be solved by a
> >>>> pre-commit hook - and in Git the files are de-duplicated anyway based
> >> on
> >>>> content hash, so this does not hurt.
> >>>>
> >>>> On 02.07.25 18:49, Shahar Epstein wrote:
> >>>>> I support option 2 with proper automation & CI - the reasonings
> >> you've
> >>>>> shown for that make sense to me.
> >>>>>
> >>>>>
> >>>>> Shahar
> >>>>>
> >>>>>
> >>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <as...@apache.org>
> >>> wrote:
> >>>>>
> >>>>>> Hello everyone,
> >>>>>>
> >>>>>> As we work on finishing off the code-level separation of Task SDK
> >> and
> >>>> Core
> >>>>>> (scheduler etc) we have come across some situations where we would
> >>> like
> >>>> to
> >>>>>> share code between these.
> >>>>>>
> >>>>>> However it’s not as straight forward of “just put it in a common
> >> dist
> >>>> they
> >>>>>> both depend upon” because one of the goals of the Task SDK
> >> separation
> >>>> was
> >>>>>> to have 100% complete version independence between the two, ideally
> >>>> even if
> >>>>>> they are built into the same image and venv. Most of the reason why
> >>> this
> >>>>>> isn’t straight forward comes down to backwards compatibility - if we
> >>>> make
> >>>>>> an change to the common/shared distribution
> >>>>>>
> >>>>>>
> >>>>>> We’ve listed the options we have thought about in
> >>>>>> https://github.com/apache/airflow/issues/51545 (but that covers
> >> some
> >>>> more
> >>>>>> things that I don’t want to get in to in this discussion such as
> >>>> possibly
> >>>>>> separating operators and executors out of a single provider dist.)
> >>>>>>
> >>>>>> To give a concrete example of some code I would like to share
> >>>>>>
> >>>>
> >>>
> >>
> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
> >>>>>> — logging config. Another thing we will want to share will be the
> >>>>>> AirflowConfigParser class from airflow.configuration (but notably:
> >>> only
> >>>> the
> >>>>>> parser class, _not_ the default config values, again, lets not dwell
> >>> on
> >>>> the
> >>>>>> specifics of that)
> >>>>>>
> >>>>>> So to bring the options listed in the issue here for discussion,
> >>> broadly
> >>>>>> speaking there are two high-level approaches:
> >>>>>>
> >>>>>> 1. A single shared distribution
> >>>>>> 2. No shared package and copy/duplicate code
> >>>>>>
> >>>>>> The advantage of Approach 1 is that we only have the code in one
> >>> place.
> >>>>>> However for me, at least in this specific case of Logging config or
> >>>>>> AirflowConfigParser class is that backwards compatibility is much
> >> much
> >>>>>> harder.
> >>>>>>
> >>>>>> The main advantage of Approach 2 is the the code is released
> >>>> with/embedded
> >>>>>> in the dist (i.e. apache-airflow-task-sdk would contain the right
> >>>> version
> >>>>>> of the logging config and ConfigParser etc). The downside is that
> >>> either
> >>>>>> the code will need to be duplicated in the repo, or better yet it
> >>> would
> >>>>>> live in a single place in the repo, but some tooling (TBD) will
> >>>>>> automatically handle the duplication, either at commit time, or my
> >>>>>> preference, at release time.
> >>>>>>
> >>>>>> For this kind of shared “utility” code I am very strongly leaning
> >>>> towards
> >>>>>> option 2 with automation, as otherwise I think the backwards
> >>>> compatibility
> >>>>>> requirements would make it unworkable (very quickly over time the
> >>>>>> combinations we would have to test would just be unreasonable) and I
> >>>> don’t
> >>>>>> feel confident we can have things as stable as we need to really
> >>> deliver
> >>>>>> the version separation/independency I want to delivery with AIP-72.
> >>>>>>
> >>>>>> So unless someone feels very strongly about this, I will come up
> >> with
> >>> a
> >>>>>> draft PR for further discussion that will implement code sharing via
> >>>>>> “vendoring” it at build time. I have an idea of how I can achieve
> >> this
> >>>> so
> >>>>>> we have a single version in the repo and it’ll work there, but at
> >>>> runtime
> >>>>>> we vendor it in to the shipped dist so it lives at something like
> >>>>>> `airflow.sdk._vendor` etc.
> >>>>>>
> >>>>>> In terms of repo layout, this likely means we would end up with:
> >>>>>>
> >>>>>> airflow-core/pyproject.toml
> >>>>>> airflow-core/src/
> >>>>>> airflow-core/tests/
> >>>>>> task-sdk/pyproject.toml
> >>>>>> task-sdk/src/
> >>>>>> task-sdk/tests/
> >>>>>> airflow-common/src
> >>>>>> airflow-common/tests/
> >>>>>> # Possibly no airflow-common/pyproject.toml, as deps would be
> >> included
> >>>> in
> >>>>>> the downstream projects. TBD.
> >>>>>>
> >>>>>> Thoughts and feedback welcomed.
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>
> >>>>
> >>>
> >>
>
>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Ash Berlin-Taylor <as...@apache.org>.
Oh yes, symlinks will work, with one big caveat: It does mean you can’t use absolute imports in one common module to another.

For example https://github.com/apache/airflow/blob/4c66ebd06/airflow-core/src/airflow/utils/serve_logs.py#L41 where we have

```
from airflow.utils.module_loading import import_string
```

 if we want to move serve_logs into this common lib that is then symlinked then we wouldn’t be able to have `from airflow_common.module_loading import import_string`.

I can think of two possible solutions here.

1) is to allow/require relative imports in this shared lib, so `from .module_loading import import_string`
2) is to use `vendoring`[1] (from the pip maintainers) which will handle import-rewriting for us.

I’d entirely forgot that symlinks in repos was a thing, so I prepared a minimal POC/demo of what vendoring approach could look like here https://github.com/apache/airflow/commit/996817782be6071b306a87af9f36fe1cf2d3aaa3

Now personally I am more than happy with relative imports, but generally as a project we have avoided them, so I think that limits what we could do with a symlink based approach.

-ash

[1] https://github.com/pradyunsg/vendoring 

> On 3 Jul 2025, at 10:30, Pavankumar Gopidesu <go...@gmail.com> wrote:
> 
> Thanks Ash
> 
> Yes agree option 2 would be preferred for me. Making sure we have all the
> gaurdriles to protect any unwanted behaviour in code sharing and executing
> right of tests between the packages.
> 
> Agree with others, option 2 would be
> 
> On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <am...@gmail.com>
> wrote:
> 
>> Thanks for starting this discussion, Ash.
>> 
>> I would prefer option 2 here with proper tooling to handle the code
>> duplication at *release* time.
>> It is best to have a dist that has all it needs in itself.
>> 
>> Option 1 could very quickly get out of hand and if we decide to separate
>> triggerer /
>> dag processor / config etc etc as separate packages, back compat is going
>> to be a nightmare
>> and will bite us harder than we anticipate.
>> 
>> Thanks & Regards,
>> Amogh Desai
>> 
>> 
>> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <ka...@gmail.com> wrote:
>> 
>>> I prefer Option 2 as well to avoid matrix of dependencies
>>> 
>>> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler <j_...@gmx.de.invalid>
>>> wrote:
>>> 
>>>> I'd also rather prefer option 2 - reason here is it is rather pragmatic
>>>> and we no not need to cut another package and have less package counts
>>>> and dependencies.
>>>> 
>>>> I remember some time ago I was checking (together with Jarek, I am not
>>>> sure anymore...) if the usage of symlinks would be possible. To keep
>> the
>>>> source in one package but "symlink" it into another. If then at point
>> of
>>>> packaging/release the files are materialized we have 1 set of code.
>>>> 
>>>> Otherwise if not possible still the redundancy could be solved by a
>>>> pre-commit hook - and in Git the files are de-duplicated anyway based
>> on
>>>> content hash, so this does not hurt.
>>>> 
>>>> On 02.07.25 18:49, Shahar Epstein wrote:
>>>>> I support option 2 with proper automation & CI - the reasonings
>> you've
>>>>> shown for that make sense to me.
>>>>> 
>>>>> 
>>>>> Shahar
>>>>> 
>>>>> 
>>>>> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <as...@apache.org>
>>> wrote:
>>>>> 
>>>>>> Hello everyone,
>>>>>> 
>>>>>> As we work on finishing off the code-level separation of Task SDK
>> and
>>>> Core
>>>>>> (scheduler etc) we have come across some situations where we would
>>> like
>>>> to
>>>>>> share code between these.
>>>>>> 
>>>>>> However it’s not as straight forward of “just put it in a common
>> dist
>>>> they
>>>>>> both depend upon” because one of the goals of the Task SDK
>> separation
>>>> was
>>>>>> to have 100% complete version independence between the two, ideally
>>>> even if
>>>>>> they are built into the same image and venv. Most of the reason why
>>> this
>>>>>> isn’t straight forward comes down to backwards compatibility - if we
>>>> make
>>>>>> an change to the common/shared distribution
>>>>>> 
>>>>>> 
>>>>>> We’ve listed the options we have thought about in
>>>>>> https://github.com/apache/airflow/issues/51545 (but that covers
>> some
>>>> more
>>>>>> things that I don’t want to get in to in this discussion such as
>>>> possibly
>>>>>> separating operators and executors out of a single provider dist.)
>>>>>> 
>>>>>> To give a concrete example of some code I would like to share
>>>>>> 
>>>> 
>>> 
>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>>>>>> — logging config. Another thing we will want to share will be the
>>>>>> AirflowConfigParser class from airflow.configuration (but notably:
>>> only
>>>> the
>>>>>> parser class, _not_ the default config values, again, lets not dwell
>>> on
>>>> the
>>>>>> specifics of that)
>>>>>> 
>>>>>> So to bring the options listed in the issue here for discussion,
>>> broadly
>>>>>> speaking there are two high-level approaches:
>>>>>> 
>>>>>> 1. A single shared distribution
>>>>>> 2. No shared package and copy/duplicate code
>>>>>> 
>>>>>> The advantage of Approach 1 is that we only have the code in one
>>> place.
>>>>>> However for me, at least in this specific case of Logging config or
>>>>>> AirflowConfigParser class is that backwards compatibility is much
>> much
>>>>>> harder.
>>>>>> 
>>>>>> The main advantage of Approach 2 is the the code is released
>>>> with/embedded
>>>>>> in the dist (i.e. apache-airflow-task-sdk would contain the right
>>>> version
>>>>>> of the logging config and ConfigParser etc). The downside is that
>>> either
>>>>>> the code will need to be duplicated in the repo, or better yet it
>>> would
>>>>>> live in a single place in the repo, but some tooling (TBD) will
>>>>>> automatically handle the duplication, either at commit time, or my
>>>>>> preference, at release time.
>>>>>> 
>>>>>> For this kind of shared “utility” code I am very strongly leaning
>>>> towards
>>>>>> option 2 with automation, as otherwise I think the backwards
>>>> compatibility
>>>>>> requirements would make it unworkable (very quickly over time the
>>>>>> combinations we would have to test would just be unreasonable) and I
>>>> don’t
>>>>>> feel confident we can have things as stable as we need to really
>>> deliver
>>>>>> the version separation/independency I want to delivery with AIP-72.
>>>>>> 
>>>>>> So unless someone feels very strongly about this, I will come up
>> with
>>> a
>>>>>> draft PR for further discussion that will implement code sharing via
>>>>>> “vendoring” it at build time. I have an idea of how I can achieve
>> this
>>>> so
>>>>>> we have a single version in the repo and it’ll work there, but at
>>>> runtime
>>>>>> we vendor it in to the shipped dist so it lives at something like
>>>>>> `airflow.sdk._vendor` etc.
>>>>>> 
>>>>>> In terms of repo layout, this likely means we would end up with:
>>>>>> 
>>>>>> airflow-core/pyproject.toml
>>>>>> airflow-core/src/
>>>>>> airflow-core/tests/
>>>>>> task-sdk/pyproject.toml
>>>>>> task-sdk/src/
>>>>>> task-sdk/tests/
>>>>>> airflow-common/src
>>>>>> airflow-common/tests/
>>>>>> # Possibly no airflow-common/pyproject.toml, as deps would be
>> included
>>>> in
>>>>>> the downstream projects. TBD.
>>>>>> 
>>>>>> Thoughts and feedback welcomed.
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>> 
>>>> 
>>> 
>> 


Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Pavankumar Gopidesu <go...@gmail.com>.
Thanks Ash

Yes agree option 2 would be preferred for me. Making sure we have all the
gaurdriles to protect any unwanted behaviour in code sharing and executing
right of tests between the packages.

Agree with others, option 2 would be

On Thu, Jul 3, 2025 at 10:02 AM Amogh Desai <am...@gmail.com>
wrote:

> Thanks for starting this discussion, Ash.
>
> I would prefer option 2 here with proper tooling to handle the code
> duplication at *release* time.
> It is best to have a dist that has all it needs in itself.
>
> Option 1 could very quickly get out of hand and if we decide to separate
> triggerer /
> dag processor / config etc etc as separate packages, back compat is going
> to be a nightmare
> and will bite us harder than we anticipate.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > I prefer Option 2 as well to avoid matrix of dependencies
> >
> > On Thu, 3 Jul 2025 at 01:03, Jens Scheffler <j_...@gmx.de.invalid>
> > wrote:
> >
> > > I'd also rather prefer option 2 - reason here is it is rather pragmatic
> > > and we no not need to cut another package and have less package counts
> > > and dependencies.
> > >
> > > I remember some time ago I was checking (together with Jarek, I am not
> > > sure anymore...) if the usage of symlinks would be possible. To keep
> the
> > > source in one package but "symlink" it into another. If then at point
> of
> > > packaging/release the files are materialized we have 1 set of code.
> > >
> > > Otherwise if not possible still the redundancy could be solved by a
> > > pre-commit hook - and in Git the files are de-duplicated anyway based
> on
> > > content hash, so this does not hurt.
> > >
> > > On 02.07.25 18:49, Shahar Epstein wrote:
> > > > I support option 2 with proper automation & CI - the reasonings
> you've
> > > > shown for that make sense to me.
> > > >
> > > >
> > > > Shahar
> > > >
> > > >
> > > > On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <as...@apache.org>
> > wrote:
> > > >
> > > >> Hello everyone,
> > > >>
> > > >> As we work on finishing off the code-level separation of Task SDK
> and
> > > Core
> > > >> (scheduler etc) we have come across some situations where we would
> > like
> > > to
> > > >> share code between these.
> > > >>
> > > >> However it’s not as straight forward of “just put it in a common
> dist
> > > they
> > > >> both depend upon” because one of the goals of the Task SDK
> separation
> > > was
> > > >> to have 100% complete version independence between the two, ideally
> > > even if
> > > >> they are built into the same image and venv. Most of the reason why
> > this
> > > >> isn’t straight forward comes down to backwards compatibility - if we
> > > make
> > > >> an change to the common/shared distribution
> > > >>
> > > >>
> > > >> We’ve listed the options we have thought about in
> > > >> https://github.com/apache/airflow/issues/51545 (but that covers
> some
> > > more
> > > >> things that I don’t want to get in to in this discussion such as
> > > possibly
> > > >> separating operators and executors out of a single provider dist.)
> > > >>
> > > >> To give a concrete example of some code I would like to share
> > > >>
> > >
> >
> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
> > > >> — logging config. Another thing we will want to share will be the
> > > >> AirflowConfigParser class from airflow.configuration (but notably:
> > only
> > > the
> > > >> parser class, _not_ the default config values, again, lets not dwell
> > on
> > > the
> > > >> specifics of that)
> > > >>
> > > >> So to bring the options listed in the issue here for discussion,
> > broadly
> > > >> speaking there are two high-level approaches:
> > > >>
> > > >> 1. A single shared distribution
> > > >> 2. No shared package and copy/duplicate code
> > > >>
> > > >> The advantage of Approach 1 is that we only have the code in one
> > place.
> > > >> However for me, at least in this specific case of Logging config or
> > > >> AirflowConfigParser class is that backwards compatibility is much
> much
> > > >> harder.
> > > >>
> > > >> The main advantage of Approach 2 is the the code is released
> > > with/embedded
> > > >> in the dist (i.e. apache-airflow-task-sdk would contain the right
> > > version
> > > >> of the logging config and ConfigParser etc). The downside is that
> > either
> > > >> the code will need to be duplicated in the repo, or better yet it
> > would
> > > >> live in a single place in the repo, but some tooling (TBD) will
> > > >> automatically handle the duplication, either at commit time, or my
> > > >> preference, at release time.
> > > >>
> > > >> For this kind of shared “utility” code I am very strongly leaning
> > > towards
> > > >> option 2 with automation, as otherwise I think the backwards
> > > compatibility
> > > >> requirements would make it unworkable (very quickly over time the
> > > >> combinations we would have to test would just be unreasonable) and I
> > > don’t
> > > >> feel confident we can have things as stable as we need to really
> > deliver
> > > >> the version separation/independency I want to delivery with AIP-72.
> > > >>
> > > >> So unless someone feels very strongly about this, I will come up
> with
> > a
> > > >> draft PR for further discussion that will implement code sharing via
> > > >> “vendoring” it at build time. I have an idea of how I can achieve
> this
> > > so
> > > >> we have a single version in the repo and it’ll work there, but at
> > > runtime
> > > >> we vendor it in to the shipped dist so it lives at something like
> > > >> `airflow.sdk._vendor` etc.
> > > >>
> > > >> In terms of repo layout, this likely means we would end up with:
> > > >>
> > > >> airflow-core/pyproject.toml
> > > >> airflow-core/src/
> > > >> airflow-core/tests/
> > > >> task-sdk/pyproject.toml
> > > >> task-sdk/src/
> > > >> task-sdk/tests/
> > > >> airflow-common/src
> > > >> airflow-common/tests/
> > > >> # Possibly no airflow-common/pyproject.toml, as deps would be
> included
> > > in
> > > >> the downstream projects. TBD.
> > > >>
> > > >> Thoughts and feedback welcomed.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Amogh Desai <am...@gmail.com>.
Thanks for starting this discussion, Ash.

I would prefer option 2 here with proper tooling to handle the code
duplication at *release* time.
It is best to have a dist that has all it needs in itself.

Option 1 could very quickly get out of hand and if we decide to separate
triggerer /
dag processor / config etc etc as separate packages, back compat is going
to be a nightmare
and will bite us harder than we anticipate.

Thanks & Regards,
Amogh Desai


On Thu, Jul 3, 2025 at 1:12 AM Kaxil Naik <ka...@gmail.com> wrote:

> I prefer Option 2 as well to avoid matrix of dependencies
>
> On Thu, 3 Jul 2025 at 01:03, Jens Scheffler <j_...@gmx.de.invalid>
> wrote:
>
> > I'd also rather prefer option 2 - reason here is it is rather pragmatic
> > and we no not need to cut another package and have less package counts
> > and dependencies.
> >
> > I remember some time ago I was checking (together with Jarek, I am not
> > sure anymore...) if the usage of symlinks would be possible. To keep the
> > source in one package but "symlink" it into another. If then at point of
> > packaging/release the files are materialized we have 1 set of code.
> >
> > Otherwise if not possible still the redundancy could be solved by a
> > pre-commit hook - and in Git the files are de-duplicated anyway based on
> > content hash, so this does not hurt.
> >
> > On 02.07.25 18:49, Shahar Epstein wrote:
> > > I support option 2 with proper automation & CI - the reasonings you've
> > > shown for that make sense to me.
> > >
> > >
> > > Shahar
> > >
> > >
> > > On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> > >
> > >> Hello everyone,
> > >>
> > >> As we work on finishing off the code-level separation of Task SDK and
> > Core
> > >> (scheduler etc) we have come across some situations where we would
> like
> > to
> > >> share code between these.
> > >>
> > >> However it’s not as straight forward of “just put it in a common dist
> > they
> > >> both depend upon” because one of the goals of the Task SDK separation
> > was
> > >> to have 100% complete version independence between the two, ideally
> > even if
> > >> they are built into the same image and venv. Most of the reason why
> this
> > >> isn’t straight forward comes down to backwards compatibility - if we
> > make
> > >> an change to the common/shared distribution
> > >>
> > >>
> > >> We’ve listed the options we have thought about in
> > >> https://github.com/apache/airflow/issues/51545 (but that covers some
> > more
> > >> things that I don’t want to get in to in this discussion such as
> > possibly
> > >> separating operators and executors out of a single provider dist.)
> > >>
> > >> To give a concrete example of some code I would like to share
> > >>
> >
> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
> > >> — logging config. Another thing we will want to share will be the
> > >> AirflowConfigParser class from airflow.configuration (but notably:
> only
> > the
> > >> parser class, _not_ the default config values, again, lets not dwell
> on
> > the
> > >> specifics of that)
> > >>
> > >> So to bring the options listed in the issue here for discussion,
> broadly
> > >> speaking there are two high-level approaches:
> > >>
> > >> 1. A single shared distribution
> > >> 2. No shared package and copy/duplicate code
> > >>
> > >> The advantage of Approach 1 is that we only have the code in one
> place.
> > >> However for me, at least in this specific case of Logging config or
> > >> AirflowConfigParser class is that backwards compatibility is much much
> > >> harder.
> > >>
> > >> The main advantage of Approach 2 is the the code is released
> > with/embedded
> > >> in the dist (i.e. apache-airflow-task-sdk would contain the right
> > version
> > >> of the logging config and ConfigParser etc). The downside is that
> either
> > >> the code will need to be duplicated in the repo, or better yet it
> would
> > >> live in a single place in the repo, but some tooling (TBD) will
> > >> automatically handle the duplication, either at commit time, or my
> > >> preference, at release time.
> > >>
> > >> For this kind of shared “utility” code I am very strongly leaning
> > towards
> > >> option 2 with automation, as otherwise I think the backwards
> > compatibility
> > >> requirements would make it unworkable (very quickly over time the
> > >> combinations we would have to test would just be unreasonable) and I
> > don’t
> > >> feel confident we can have things as stable as we need to really
> deliver
> > >> the version separation/independency I want to delivery with AIP-72.
> > >>
> > >> So unless someone feels very strongly about this, I will come up with
> a
> > >> draft PR for further discussion that will implement code sharing via
> > >> “vendoring” it at build time. I have an idea of how I can achieve this
> > so
> > >> we have a single version in the repo and it’ll work there, but at
> > runtime
> > >> we vendor it in to the shipped dist so it lives at something like
> > >> `airflow.sdk._vendor` etc.
> > >>
> > >> In terms of repo layout, this likely means we would end up with:
> > >>
> > >> airflow-core/pyproject.toml
> > >> airflow-core/src/
> > >> airflow-core/tests/
> > >> task-sdk/pyproject.toml
> > >> task-sdk/src/
> > >> task-sdk/tests/
> > >> airflow-common/src
> > >> airflow-common/tests/
> > >> # Possibly no airflow-common/pyproject.toml, as deps would be included
> > in
> > >> the downstream projects. TBD.
> > >>
> > >> Thoughts and feedback welcomed.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Kaxil Naik <ka...@gmail.com>.
I prefer Option 2 as well to avoid matrix of dependencies

On Thu, 3 Jul 2025 at 01:03, Jens Scheffler <j_...@gmx.de.invalid>
wrote:

> I'd also rather prefer option 2 - reason here is it is rather pragmatic
> and we no not need to cut another package and have less package counts
> and dependencies.
>
> I remember some time ago I was checking (together with Jarek, I am not
> sure anymore...) if the usage of symlinks would be possible. To keep the
> source in one package but "symlink" it into another. If then at point of
> packaging/release the files are materialized we have 1 set of code.
>
> Otherwise if not possible still the redundancy could be solved by a
> pre-commit hook - and in Git the files are de-duplicated anyway based on
> content hash, so this does not hurt.
>
> On 02.07.25 18:49, Shahar Epstein wrote:
> > I support option 2 with proper automation & CI - the reasonings you've
> > shown for that make sense to me.
> >
> >
> > Shahar
> >
> >
> > On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> >> Hello everyone,
> >>
> >> As we work on finishing off the code-level separation of Task SDK and
> Core
> >> (scheduler etc) we have come across some situations where we would like
> to
> >> share code between these.
> >>
> >> However it’s not as straight forward of “just put it in a common dist
> they
> >> both depend upon” because one of the goals of the Task SDK separation
> was
> >> to have 100% complete version independence between the two, ideally
> even if
> >> they are built into the same image and venv. Most of the reason why this
> >> isn’t straight forward comes down to backwards compatibility - if we
> make
> >> an change to the common/shared distribution
> >>
> >>
> >> We’ve listed the options we have thought about in
> >> https://github.com/apache/airflow/issues/51545 (but that covers some
> more
> >> things that I don’t want to get in to in this discussion such as
> possibly
> >> separating operators and executors out of a single provider dist.)
> >>
> >> To give a concrete example of some code I would like to share
> >>
> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
> >> — logging config. Another thing we will want to share will be the
> >> AirflowConfigParser class from airflow.configuration (but notably: only
> the
> >> parser class, _not_ the default config values, again, lets not dwell on
> the
> >> specifics of that)
> >>
> >> So to bring the options listed in the issue here for discussion, broadly
> >> speaking there are two high-level approaches:
> >>
> >> 1. A single shared distribution
> >> 2. No shared package and copy/duplicate code
> >>
> >> The advantage of Approach 1 is that we only have the code in one place.
> >> However for me, at least in this specific case of Logging config or
> >> AirflowConfigParser class is that backwards compatibility is much much
> >> harder.
> >>
> >> The main advantage of Approach 2 is the the code is released
> with/embedded
> >> in the dist (i.e. apache-airflow-task-sdk would contain the right
> version
> >> of the logging config and ConfigParser etc). The downside is that either
> >> the code will need to be duplicated in the repo, or better yet it would
> >> live in a single place in the repo, but some tooling (TBD) will
> >> automatically handle the duplication, either at commit time, or my
> >> preference, at release time.
> >>
> >> For this kind of shared “utility” code I am very strongly leaning
> towards
> >> option 2 with automation, as otherwise I think the backwards
> compatibility
> >> requirements would make it unworkable (very quickly over time the
> >> combinations we would have to test would just be unreasonable) and I
> don’t
> >> feel confident we can have things as stable as we need to really deliver
> >> the version separation/independency I want to delivery with AIP-72.
> >>
> >> So unless someone feels very strongly about this, I will come up with a
> >> draft PR for further discussion that will implement code sharing via
> >> “vendoring” it at build time. I have an idea of how I can achieve this
> so
> >> we have a single version in the repo and it’ll work there, but at
> runtime
> >> we vendor it in to the shipped dist so it lives at something like
> >> `airflow.sdk._vendor` etc.
> >>
> >> In terms of repo layout, this likely means we would end up with:
> >>
> >> airflow-core/pyproject.toml
> >> airflow-core/src/
> >> airflow-core/tests/
> >> task-sdk/pyproject.toml
> >> task-sdk/src/
> >> task-sdk/tests/
> >> airflow-common/src
> >> airflow-common/tests/
> >> # Possibly no airflow-common/pyproject.toml, as deps would be included
> in
> >> the downstream projects. TBD.
> >>
> >> Thoughts and feedback welcomed.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Jens Scheffler <j_...@gmx.de.INVALID>.
I'd also rather prefer option 2 - reason here is it is rather pragmatic 
and we no not need to cut another package and have less package counts 
and dependencies.

I remember some time ago I was checking (together with Jarek, I am not 
sure anymore...) if the usage of symlinks would be possible. To keep the 
source in one package but "symlink" it into another. If then at point of 
packaging/release the files are materialized we have 1 set of code.

Otherwise if not possible still the redundancy could be solved by a 
pre-commit hook - and in Git the files are de-duplicated anyway based on 
content hash, so this does not hurt.

On 02.07.25 18:49, Shahar Epstein wrote:
> I support option 2 with proper automation & CI - the reasonings you've
> shown for that make sense to me.
>
>
> Shahar
>
>
> On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> Hello everyone,
>>
>> As we work on finishing off the code-level separation of Task SDK and Core
>> (scheduler etc) we have come across some situations where we would like to
>> share code between these.
>>
>> However it’s not as straight forward of “just put it in a common dist they
>> both depend upon” because one of the goals of the Task SDK separation was
>> to have 100% complete version independence between the two, ideally even if
>> they are built into the same image and venv. Most of the reason why this
>> isn’t straight forward comes down to backwards compatibility - if we make
>> an change to the common/shared distribution
>>
>>
>> We’ve listed the options we have thought about in
>> https://github.com/apache/airflow/issues/51545 (but that covers some more
>> things that I don’t want to get in to in this discussion such as possibly
>> separating operators and executors out of a single provider dist.)
>>
>> To give a concrete example of some code I would like to share
>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>> — logging config. Another thing we will want to share will be the
>> AirflowConfigParser class from airflow.configuration (but notably: only the
>> parser class, _not_ the default config values, again, lets not dwell on the
>> specifics of that)
>>
>> So to bring the options listed in the issue here for discussion, broadly
>> speaking there are two high-level approaches:
>>
>> 1. A single shared distribution
>> 2. No shared package and copy/duplicate code
>>
>> The advantage of Approach 1 is that we only have the code in one place.
>> However for me, at least in this specific case of Logging config or
>> AirflowConfigParser class is that backwards compatibility is much much
>> harder.
>>
>> The main advantage of Approach 2 is the the code is released with/embedded
>> in the dist (i.e. apache-airflow-task-sdk would contain the right version
>> of the logging config and ConfigParser etc). The downside is that either
>> the code will need to be duplicated in the repo, or better yet it would
>> live in a single place in the repo, but some tooling (TBD) will
>> automatically handle the duplication, either at commit time, or my
>> preference, at release time.
>>
>> For this kind of shared “utility” code I am very strongly leaning towards
>> option 2 with automation, as otherwise I think the backwards compatibility
>> requirements would make it unworkable (very quickly over time the
>> combinations we would have to test would just be unreasonable) and I don’t
>> feel confident we can have things as stable as we need to really deliver
>> the version separation/independency I want to delivery with AIP-72.
>>
>> So unless someone feels very strongly about this, I will come up with a
>> draft PR for further discussion that will implement code sharing via
>> “vendoring” it at build time. I have an idea of how I can achieve this so
>> we have a single version in the repo and it’ll work there, but at runtime
>> we vendor it in to the shipped dist so it lives at something like
>> `airflow.sdk._vendor` etc.
>>
>> In terms of repo layout, this likely means we would end up with:
>>
>> airflow-core/pyproject.toml
>> airflow-core/src/
>> airflow-core/tests/
>> task-sdk/pyproject.toml
>> task-sdk/src/
>> task-sdk/tests/
>> airflow-common/src
>> airflow-common/tests/
>> # Possibly no airflow-common/pyproject.toml, as deps would be included in
>> the downstream projects. TBD.
>>
>> Thoughts and feedback welcomed.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org


Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Shahar Epstein <sh...@apache.org>.
I support option 2 with proper automation & CI - the reasonings you've
shown for that make sense to me.


Shahar


On Wed, Jul 2, 2025 at 3:36 PM Ash Berlin-Taylor <as...@apache.org> wrote:

> Hello everyone,
>
> As we work on finishing off the code-level separation of Task SDK and Core
> (scheduler etc) we have come across some situations where we would like to
> share code between these.
>
> However it’s not as straight forward of “just put it in a common dist they
> both depend upon” because one of the goals of the Task SDK separation was
> to have 100% complete version independence between the two, ideally even if
> they are built into the same image and venv. Most of the reason why this
> isn’t straight forward comes down to backwards compatibility - if we make
> an change to the common/shared distribution
>
>
> We’ve listed the options we have thought about in
> https://github.com/apache/airflow/issues/51545 (but that covers some more
> things that I don’t want to get in to in this discussion such as possibly
> separating operators and executors out of a single provider dist.)
>
> To give a concrete example of some code I would like to share
> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
> — logging config. Another thing we will want to share will be the
> AirflowConfigParser class from airflow.configuration (but notably: only the
> parser class, _not_ the default config values, again, lets not dwell on the
> specifics of that)
>
> So to bring the options listed in the issue here for discussion, broadly
> speaking there are two high-level approaches:
>
> 1. A single shared distribution
> 2. No shared package and copy/duplicate code
>
> The advantage of Approach 1 is that we only have the code in one place.
> However for me, at least in this specific case of Logging config or
> AirflowConfigParser class is that backwards compatibility is much much
> harder.
>
> The main advantage of Approach 2 is the the code is released with/embedded
> in the dist (i.e. apache-airflow-task-sdk would contain the right version
> of the logging config and ConfigParser etc). The downside is that either
> the code will need to be duplicated in the repo, or better yet it would
> live in a single place in the repo, but some tooling (TBD) will
> automatically handle the duplication, either at commit time, or my
> preference, at release time.
>
> For this kind of shared “utility” code I am very strongly leaning towards
> option 2 with automation, as otherwise I think the backwards compatibility
> requirements would make it unworkable (very quickly over time the
> combinations we would have to test would just be unreasonable) and I don’t
> feel confident we can have things as stable as we need to really deliver
> the version separation/independency I want to delivery with AIP-72.
>
> So unless someone feels very strongly about this, I will come up with a
> draft PR for further discussion that will implement code sharing via
> “vendoring” it at build time. I have an idea of how I can achieve this so
> we have a single version in the repo and it’ll work there, but at runtime
> we vendor it in to the shipped dist so it lives at something like
> `airflow.sdk._vendor` etc.
>
> In terms of repo layout, this likely means we would end up with:
>
> airflow-core/pyproject.toml
> airflow-core/src/
> airflow-core/tests/
> task-sdk/pyproject.toml
> task-sdk/src/
> task-sdk/tests/
> airflow-common/src
> airflow-common/tests/
> # Possibly no airflow-common/pyproject.toml, as deps would be included in
> the downstream projects. TBD.
>
> Thoughts and feedback welcomed.

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Jarek Potiuk <ja...@potiuk.com>.
> My discussion was not so much about this specific use case of the logging
config, but of sharing code in general which we know we need to do, as we
currently have imports from core to sdk and from sdk to core, and we need
to break that up.

Yep. I understand that - and my answer is (and this is what I tried to
explain) that it depends on what we are trying to share and that answer
might be different in different cases. That was essentially what I wrote.

On Wed, Jul 2, 2025 at 6:32 PM Ash Berlin-Taylor <as...@apache.org> wrote:

> My discussion was not so much about this specific use case of the logging
> config, but of sharing code in general which we know we need to do, as we
> currently have imports from core to sdk and from sdk to core, and we need
> to break that up.
>
> -a
>
> > On 2 Jul 2025, at 14:44, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > I think the answer **might** be different for different functionalities.
> > For example I see that "config" is likely a better candidate for "shared
> > distribution" than "logging" and for config we could use 1 where for
> > logging we could use a form of 2.
> >
> > On a high level (And looking at the code). How much of the logging code
> > needs to be actually shared?  I do not think much.
> >
> > Some assumptions:
> >
> > 1) the regular scheduler, triggerer, api_server etc. "application" logs
> - i
> > would say we should absolutely minimise our own code. There are plenty of
> > logging utilities that enhance standard logging framework and I have a
> > doubt we can do better work than them - even in integration with external
> > logging systems. We do not **really** need a shared logging code for
> "task
> > logs" and "application logs". We already cannot do it now - the "log
> > handler" of ours implemented in various providers and "remote logging" of
> > ours is only "task oriented". And there is - I think - no particular
> reason
> > to change it. So I'd say both configuration and implementation of "task
> > logs" and "application logs" can be implemented using completely
> different
> > libraries and even have completely different configuration - and there is
> > virtually no overlap and shared code there as far as I can see.
> >
> > 2) Task logs are a completely different story. Here we want to have
> custom
> > code and full control, StructLog is a good choice and we should bank more
> > on it. But when you look closely we really need two things there:
> >
> > a) ability to write logs from workers in triggerers (where task.sdk will
> be
> > (and should be) the only dependency)
> > b) ability to read logs by the UI (where more / airflow dependencies are
> > needed)
> >
> > And while those two have a "data structure" in common, the reading and
> > writing code can be completely separate.  I will base it on amazon
> provider
> > that also provides CloudWatch integration (one of log handlers).
> >
> > 1) We could have log writing (current write part of log_handler) in
> > "task-sdk" + "amazon-provider"
> > 2) We could keep writing logs in providers, and extract reading logs to
> new
> > distributions for remote logs - say "providers-airlfow-amazon-logging"
> > distribution which will have minimal number of dependencies that "ui"
> needs
> > to read amazon logs (and do the same with other log handlers).
> >
> > Very little code from those two implementations would have to be copied
> and
> > we could define very strict rules to make sure that the data
> structs/paths
> > etc. between the readers and writers remain compatible. Also it has the
> > nice effect that we could make sure "ui" does not need providers at all.
> If
> > we do it this way, pass rendered links (we already do) via database, and
> > figure out simple way of sharing connection definitions (via database) -
> we
> > could very easily make "api_server" "provider-less".
> >
> > The problem is - of course - Triggerer and Dag Processor- because if we
> > have single "airflow-core" package - that should also run triggerer,
> > triggerrer already has to "also" depend on "task.sdk" and "providers"
> (and
> > that one is not as easy to get rid of). And both have to have some ways
> of
> > sending logs that will be "task" and "dag" oriented and visible in the UI
> > (Triggerer more than Dag Processor I think).
> >
> > But that can be done in one of two ways:
> >
> > * (option 1) - separating out the Triggerer / Dag processor to separate
> > packages, assuming that a lot of triggerer / dag processor code is
> > independent from other parts of Airflow code. Which I am not sure if we
> > want and is doable now (it would likely necessitate another common
> package
> > that is shared between airflow-core and airflow-triggerer). Maybe that
> can
> > be a long term solution. But not likely something we want to invest now
> in.
> > * (option 2) - continuing what we do today - making sure that
> > "airflow-core" + "task.sdk" + "providers" + (new) "log providers" can all
> > be installed together (option 2). Option 2 will require even more
> > importance of our  CI  / dependency scrutiny - including constraints
> > generation (but we have made great progress on that and I think with the
> > new tooling it's quite a possible option).
> >
> > That would be my proposal for logging. Note that my thinking is different
> > for configuration. I think configuration could be rather easily moved to
> a
> > "common.config" package - we could likely agree and take care of
> > back-compatibility there etc. And also for other "parts" of code that we
> > want to share, the answer might be different as well.
> >
> > J.
> >
> >
> >
> > On Wed, Jul 2, 2025 at 2:36 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> >> Hello everyone,
> >>
> >> As we work on finishing off the code-level separation of Task SDK and
> Core
> >> (scheduler etc) we have come across some situations where we would like
> to
> >> share code between these.
> >>
> >> However it’s not as straight forward of “just put it in a common dist
> they
> >> both depend upon” because one of the goals of the Task SDK separation
> was
> >> to have 100% complete version independence between the two, ideally
> even if
> >> they are built into the same image and venv. Most of the reason why this
> >> isn’t straight forward comes down to backwards compatibility - if we
> make
> >> an change to the common/shared distribution
> >>
> >>
> >> We’ve listed the options we have thought about in
> >> https://github.com/apache/airflow/issues/51545 (but that covers some
> more
> >> things that I don’t want to get in to in this discussion such as
> possibly
> >> separating operators and executors out of a single provider dist.)
> >>
> >> To give a concrete example of some code I would like to share
> >>
> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
> >> — logging config. Another thing we will want to share will be the
> >> AirflowConfigParser class from airflow.configuration (but notably: only
> the
> >> parser class, _not_ the default config values, again, lets not dwell on
> the
> >> specifics of that)
> >>
> >> So to bring the options listed in the issue here for discussion, broadly
> >> speaking there are two high-level approaches:
> >>
> >> 1. A single shared distribution
> >> 2. No shared package and copy/duplicate code
> >>
> >> The advantage of Approach 1 is that we only have the code in one place.
> >> However for me, at least in this specific case of Logging config or
> >> AirflowConfigParser class is that backwards compatibility is much much
> >> harder.
> >>
> >> The main advantage of Approach 2 is the the code is released
> with/embedded
> >> in the dist (i.e. apache-airflow-task-sdk would contain the right
> version
> >> of the logging config and ConfigParser etc). The downside is that either
> >> the code will need to be duplicated in the repo, or better yet it would
> >> live in a single place in the repo, but some tooling (TBD) will
> >> automatically handle the duplication, either at commit time, or my
> >> preference, at release time.
> >>
> >> For this kind of shared “utility” code I am very strongly leaning
> towards
> >> option 2 with automation, as otherwise I think the backwards
> compatibility
> >> requirements would make it unworkable (very quickly over time the
> >> combinations we would have to test would just be unreasonable) and I
> don’t
> >> feel confident we can have things as stable as we need to really deliver
> >> the version separation/independency I want to delivery with AIP-72.
> >>
> >> So unless someone feels very strongly about this, I will come up with a
> >> draft PR for further discussion that will implement code sharing via
> >> “vendoring” it at build time. I have an idea of how I can achieve this
> so
> >> we have a single version in the repo and it’ll work there, but at
> runtime
> >> we vendor it in to the shipped dist so it lives at something like
> >> `airflow.sdk._vendor` etc.
> >>
> >> In terms of repo layout, this likely means we would end up with:
> >>
> >> airflow-core/pyproject.toml
> >> airflow-core/src/
> >> airflow-core/tests/
> >> task-sdk/pyproject.toml
> >> task-sdk/src/
> >> task-sdk/tests/
> >> airflow-common/src
> >> airflow-common/tests/
> >> # Possibly no airflow-common/pyproject.toml, as deps would be included
> in
> >> the downstream projects. TBD.
> >>
> >> Thoughts and feedback welcomed.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Ash Berlin-Taylor <as...@apache.org>.
My discussion was not so much about this specific use case of the logging config, but of sharing code in general which we know we need to do, as we currently have imports from core to sdk and from sdk to core, and we need to break that up.

-a

> On 2 Jul 2025, at 14:44, Jarek Potiuk <ja...@potiuk.com> wrote:
> 
> I think the answer **might** be different for different functionalities.
> For example I see that "config" is likely a better candidate for "shared
> distribution" than "logging" and for config we could use 1 where for
> logging we could use a form of 2.
> 
> On a high level (And looking at the code). How much of the logging code
> needs to be actually shared?  I do not think much.
> 
> Some assumptions:
> 
> 1) the regular scheduler, triggerer, api_server etc. "application" logs - i
> would say we should absolutely minimise our own code. There are plenty of
> logging utilities that enhance standard logging framework and I have a
> doubt we can do better work than them - even in integration with external
> logging systems. We do not **really** need a shared logging code for "task
> logs" and "application logs". We already cannot do it now - the "log
> handler" of ours implemented in various providers and "remote logging" of
> ours is only "task oriented". And there is - I think - no particular reason
> to change it. So I'd say both configuration and implementation of "task
> logs" and "application logs" can be implemented using completely different
> libraries and even have completely different configuration - and there is
> virtually no overlap and shared code there as far as I can see.
> 
> 2) Task logs are a completely different story. Here we want to have custom
> code and full control, StructLog is a good choice and we should bank more
> on it. But when you look closely we really need two things there:
> 
> a) ability to write logs from workers in triggerers (where task.sdk will be
> (and should be) the only dependency)
> b) ability to read logs by the UI (where more / airflow dependencies are
> needed)
> 
> And while those two have a "data structure" in common, the reading and
> writing code can be completely separate.  I will base it on amazon provider
> that also provides CloudWatch integration (one of log handlers).
> 
> 1) We could have log writing (current write part of log_handler) in
> "task-sdk" + "amazon-provider"
> 2) We could keep writing logs in providers, and extract reading logs to new
> distributions for remote logs - say "providers-airlfow-amazon-logging"
> distribution which will have minimal number of dependencies that "ui" needs
> to read amazon logs (and do the same with other log handlers).
> 
> Very little code from those two implementations would have to be copied and
> we could define very strict rules to make sure that the data structs/paths
> etc. between the readers and writers remain compatible. Also it has the
> nice effect that we could make sure "ui" does not need providers at all. If
> we do it this way, pass rendered links (we already do) via database, and
> figure out simple way of sharing connection definitions (via database) - we
> could very easily make "api_server" "provider-less".
> 
> The problem is - of course - Triggerer and Dag Processor- because if we
> have single "airflow-core" package - that should also run triggerer,
> triggerrer already has to "also" depend on "task.sdk" and "providers" (and
> that one is not as easy to get rid of). And both have to have some ways of
> sending logs that will be "task" and "dag" oriented and visible in the UI
> (Triggerer more than Dag Processor I think).
> 
> But that can be done in one of two ways:
> 
> * (option 1) - separating out the Triggerer / Dag processor to separate
> packages, assuming that a lot of triggerer / dag processor code is
> independent from other parts of Airflow code. Which I am not sure if we
> want and is doable now (it would likely necessitate another common package
> that is shared between airflow-core and airflow-triggerer). Maybe that can
> be a long term solution. But not likely something we want to invest now in.
> * (option 2) - continuing what we do today - making sure that
> "airflow-core" + "task.sdk" + "providers" + (new) "log providers" can all
> be installed together (option 2). Option 2 will require even more
> importance of our  CI  / dependency scrutiny - including constraints
> generation (but we have made great progress on that and I think with the
> new tooling it's quite a possible option).
> 
> That would be my proposal for logging. Note that my thinking is different
> for configuration. I think configuration could be rather easily moved to a
> "common.config" package - we could likely agree and take care of
> back-compatibility there etc. And also for other "parts" of code that we
> want to share, the answer might be different as well.
> 
> J.
> 
> 
> 
> On Wed, Jul 2, 2025 at 2:36 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> 
>> Hello everyone,
>> 
>> As we work on finishing off the code-level separation of Task SDK and Core
>> (scheduler etc) we have come across some situations where we would like to
>> share code between these.
>> 
>> However it’s not as straight forward of “just put it in a common dist they
>> both depend upon” because one of the goals of the Task SDK separation was
>> to have 100% complete version independence between the two, ideally even if
>> they are built into the same image and venv. Most of the reason why this
>> isn’t straight forward comes down to backwards compatibility - if we make
>> an change to the common/shared distribution
>> 
>> 
>> We’ve listed the options we have thought about in
>> https://github.com/apache/airflow/issues/51545 (but that covers some more
>> things that I don’t want to get in to in this discussion such as possibly
>> separating operators and executors out of a single provider dist.)
>> 
>> To give a concrete example of some code I would like to share
>> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
>> — logging config. Another thing we will want to share will be the
>> AirflowConfigParser class from airflow.configuration (but notably: only the
>> parser class, _not_ the default config values, again, lets not dwell on the
>> specifics of that)
>> 
>> So to bring the options listed in the issue here for discussion, broadly
>> speaking there are two high-level approaches:
>> 
>> 1. A single shared distribution
>> 2. No shared package and copy/duplicate code
>> 
>> The advantage of Approach 1 is that we only have the code in one place.
>> However for me, at least in this specific case of Logging config or
>> AirflowConfigParser class is that backwards compatibility is much much
>> harder.
>> 
>> The main advantage of Approach 2 is the the code is released with/embedded
>> in the dist (i.e. apache-airflow-task-sdk would contain the right version
>> of the logging config and ConfigParser etc). The downside is that either
>> the code will need to be duplicated in the repo, or better yet it would
>> live in a single place in the repo, but some tooling (TBD) will
>> automatically handle the duplication, either at commit time, or my
>> preference, at release time.
>> 
>> For this kind of shared “utility” code I am very strongly leaning towards
>> option 2 with automation, as otherwise I think the backwards compatibility
>> requirements would make it unworkable (very quickly over time the
>> combinations we would have to test would just be unreasonable) and I don’t
>> feel confident we can have things as stable as we need to really deliver
>> the version separation/independency I want to delivery with AIP-72.
>> 
>> So unless someone feels very strongly about this, I will come up with a
>> draft PR for further discussion that will implement code sharing via
>> “vendoring” it at build time. I have an idea of how I can achieve this so
>> we have a single version in the repo and it’ll work there, but at runtime
>> we vendor it in to the shipped dist so it lives at something like
>> `airflow.sdk._vendor` etc.
>> 
>> In terms of repo layout, this likely means we would end up with:
>> 
>> airflow-core/pyproject.toml
>> airflow-core/src/
>> airflow-core/tests/
>> task-sdk/pyproject.toml
>> task-sdk/src/
>> task-sdk/tests/
>> airflow-common/src
>> airflow-common/tests/
>> # Possibly no airflow-common/pyproject.toml, as deps would be included in
>> the downstream projects. TBD.
>> 
>> Thoughts and feedback welcomed.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org


Re: Code sharing between Airflow Core and Task SDK - how do we achieve it

Posted by Jarek Potiuk <ja...@potiuk.com>.
I think the answer **might** be different for different functionalities.
For example I see that "config" is likely a better candidate for "shared
distribution" than "logging" and for config we could use 1 where for
logging we could use a form of 2.

On a high level (And looking at the code). How much of the logging code
needs to be actually shared?  I do not think much.

Some assumptions:

1) the regular scheduler, triggerer, api_server etc. "application" logs - i
would say we should absolutely minimise our own code. There are plenty of
logging utilities that enhance standard logging framework and I have a
doubt we can do better work than them - even in integration with external
logging systems. We do not **really** need a shared logging code for "task
logs" and "application logs". We already cannot do it now - the "log
handler" of ours implemented in various providers and "remote logging" of
ours is only "task oriented". And there is - I think - no particular reason
to change it. So I'd say both configuration and implementation of "task
logs" and "application logs" can be implemented using completely different
libraries and even have completely different configuration - and there is
virtually no overlap and shared code there as far as I can see.

2) Task logs are a completely different story. Here we want to have custom
code and full control, StructLog is a good choice and we should bank more
on it. But when you look closely we really need two things there:

a) ability to write logs from workers in triggerers (where task.sdk will be
(and should be) the only dependency)
b) ability to read logs by the UI (where more / airflow dependencies are
needed)

And while those two have a "data structure" in common, the reading and
writing code can be completely separate.  I will base it on amazon provider
that also provides CloudWatch integration (one of log handlers).

1) We could have log writing (current write part of log_handler) in
"task-sdk" + "amazon-provider"
2) We could keep writing logs in providers, and extract reading logs to new
distributions for remote logs - say "providers-airlfow-amazon-logging"
distribution which will have minimal number of dependencies that "ui" needs
to read amazon logs (and do the same with other log handlers).

Very little code from those two implementations would have to be copied and
we could define very strict rules to make sure that the data structs/paths
etc. between the readers and writers remain compatible. Also it has the
nice effect that we could make sure "ui" does not need providers at all. If
we do it this way, pass rendered links (we already do) via database, and
figure out simple way of sharing connection definitions (via database) - we
could very easily make "api_server" "provider-less".

The problem is - of course - Triggerer and Dag Processor- because if we
have single "airflow-core" package - that should also run triggerer,
triggerrer already has to "also" depend on "task.sdk" and "providers" (and
that one is not as easy to get rid of). And both have to have some ways of
sending logs that will be "task" and "dag" oriented and visible in the UI
(Triggerer more than Dag Processor I think).

But that can be done in one of two ways:

* (option 1) - separating out the Triggerer / Dag processor to separate
packages, assuming that a lot of triggerer / dag processor code is
independent from other parts of Airflow code. Which I am not sure if we
want and is doable now (it would likely necessitate another common package
that is shared between airflow-core and airflow-triggerer). Maybe that can
be a long term solution. But not likely something we want to invest now in.
* (option 2) - continuing what we do today - making sure that
"airflow-core" + "task.sdk" + "providers" + (new) "log providers" can all
be installed together (option 2). Option 2 will require even more
importance of our  CI  / dependency scrutiny - including constraints
generation (but we have made great progress on that and I think with the
new tooling it's quite a possible option).

That would be my proposal for logging. Note that my thinking is different
for configuration. I think configuration could be rather easily moved to a
"common.config" package - we could likely agree and take care of
back-compatibility there etc. And also for other "parts" of code that we
want to share, the answer might be different as well.

J.



On Wed, Jul 2, 2025 at 2:36 PM Ash Berlin-Taylor <as...@apache.org> wrote:

> Hello everyone,
>
> As we work on finishing off the code-level separation of Task SDK and Core
> (scheduler etc) we have come across some situations where we would like to
> share code between these.
>
> However it’s not as straight forward of “just put it in a common dist they
> both depend upon” because one of the goals of the Task SDK separation was
> to have 100% complete version independence between the two, ideally even if
> they are built into the same image and venv. Most of the reason why this
> isn’t straight forward comes down to backwards compatibility - if we make
> an change to the common/shared distribution
>
>
> We’ve listed the options we have thought about in
> https://github.com/apache/airflow/issues/51545 (but that covers some more
> things that I don’t want to get in to in this discussion such as possibly
> separating operators and executors out of a single provider dist.)
>
> To give a concrete example of some code I would like to share
> https://github.com/apache/airflow/blob/84897570bf7e438afb157ba4700768ea74824295/airflow-core/src/airflow/_logging/structlog.py
> — logging config. Another thing we will want to share will be the
> AirflowConfigParser class from airflow.configuration (but notably: only the
> parser class, _not_ the default config values, again, lets not dwell on the
> specifics of that)
>
> So to bring the options listed in the issue here for discussion, broadly
> speaking there are two high-level approaches:
>
> 1. A single shared distribution
> 2. No shared package and copy/duplicate code
>
> The advantage of Approach 1 is that we only have the code in one place.
> However for me, at least in this specific case of Logging config or
> AirflowConfigParser class is that backwards compatibility is much much
> harder.
>
> The main advantage of Approach 2 is the the code is released with/embedded
> in the dist (i.e. apache-airflow-task-sdk would contain the right version
> of the logging config and ConfigParser etc). The downside is that either
> the code will need to be duplicated in the repo, or better yet it would
> live in a single place in the repo, but some tooling (TBD) will
> automatically handle the duplication, either at commit time, or my
> preference, at release time.
>
> For this kind of shared “utility” code I am very strongly leaning towards
> option 2 with automation, as otherwise I think the backwards compatibility
> requirements would make it unworkable (very quickly over time the
> combinations we would have to test would just be unreasonable) and I don’t
> feel confident we can have things as stable as we need to really deliver
> the version separation/independency I want to delivery with AIP-72.
>
> So unless someone feels very strongly about this, I will come up with a
> draft PR for further discussion that will implement code sharing via
> “vendoring” it at build time. I have an idea of how I can achieve this so
> we have a single version in the repo and it’ll work there, but at runtime
> we vendor it in to the shipped dist so it lives at something like
> `airflow.sdk._vendor` etc.
>
> In terms of repo layout, this likely means we would end up with:
>
> airflow-core/pyproject.toml
> airflow-core/src/
> airflow-core/tests/
> task-sdk/pyproject.toml
> task-sdk/src/
> task-sdk/tests/
> airflow-common/src
> airflow-common/tests/
> # Possibly no airflow-common/pyproject.toml, as deps would be included in
> the downstream projects. TBD.
>
> Thoughts and feedback welcomed.