You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Amogh Desai <am...@apache.org> on 2026/05/05 08:51:33 UTC
Re: [DISCUSS] Add Variable.list() to Task SDK as a replacement for Variable.all()
Hi Jun Yeong,
Valid gap. There is no GET /variables list endpoint at all, but that is
true for any
other Airflow artefacts too. So variables isn't the only one missing it,
its all of the
Airflow resources: connections, xcoms, etc.
One suggestion worth considering: rather than Variable.list() returning a
full list,
a lazy iterator might be a better fit. Something like:
for key in Variable.keys(prefix="team_a_config_"):
val = Variable.get(key)
Happy to review when a PR is ready.
Thanks & Regards,
Amogh Desai
On Fri, May 1, 2026 at 3:02 AM 김준영 <ju...@gmail.com> wrote:
> Hi Jens,
>
> That is a valid alternative, and using a single JSON variable is
> indeed a great pattern for centralized or static configurations to
> reduce I/O.
>
> However, for the use cases I'm targeting, the "Single JSON Variable"
> approach has a few significant drawbacks:
>
> 1.Concurrency & Atomic Updates: When multiple external systems or
> independent CI/CD pipelines need to update their own configurations, a
> single JSON variable creates a race condition. They would have to
> "Read-Modify-Write" the entire blob, which risks overwriting each
> other's changes. Separate variables allow for atomic, independent
> updates.
>
> 2.Integration Complexity: Many users integrate Airflow with external
> tools that independently push values to the Airflow Metadata DB or
> Secret Backend. Forcing these decoupled systems to coordinate and
> maintain a single shared JSON structure adds significant integration
> overhead.
>
> 3.Data Modeling Flexibility: While a JSON blob works for some,
> Variable.list(prefix=) allows Airflow to be unopinionated about how
> users model their data. It provides a standard "Key-Value store"
> experience (similar to AWS SSM or Redis) where prefix-based discovery
> is a first-class citizen.
>
> In short, while the JSON approach is a good workaround for specific
> cases, Variable.list() provides the necessary flexibility for highly
> dynamic and decoupled environments without forcing a specific data
> structure on the user.
>
> What do you think?
>
> Best regards, Jun Yeong
>
> 2026년 5월 1일 (금) 오전 6:19, Jens Scheffler <js...@apache.org>님이 작성:
> >
> > Hi Jun Yeong,
> >
> > one thought on this: We had similar.
> >
> > Our use case: Implemented a custom archiving (Dag) that needed to take
> > care of different retention times in different Dags (maybe bad example
> > because this archival itself accesses the database for archiving...
> > haha) and we wanted to have different retention times per Dag.
> >
> > What we did is we made a JSON structure with all parameters into a
> > single Variable. Then we did not need to have many IO operations and
> > Variables but could store all in a single Variable.
> >
> > Would this be a viable solution for your case as well?
> >
> > Jens
> >
> > On 30.04.26 22:47, 김준영 wrote:
> > > Hi Jens,
> > >
> > > Thank you for the thoughtful feedback!
> > >
> > > The primary demand for this feature comes from workflows that require
> > > dynamic configuration discovery. A common pattern is grouping related
> > > variables under a shared prefix (e.g., team_a_config_ or
> > > pipeline_x_param_). In many cases, these keys are generated or updated
> > > dynamically by external systems, meaning the exact set of keys isn't
> > > known at DAG authoring time.
> > >
> > > While users in Airflow 2.x relied on session.query(Variable).all() as
> > > a workaround, Airflow 3’s move toward the Task SDK aims to abstract
> > > away direct ORM/DB access for better security and stability.
> > > Variable.list(prefix=) provides a supported, clean way to achieve this
> > > discovery within that new architecture.
> > >
> > > Regarding the secrets backend limitation, I completely agree. It’s
> > > important to manage expectations, so I will update the PyDoc to
> > > explicitly state that this method only lists variables stored in the
> > > metadata database.
> > >
> > > Best regards, Jun Yeong
> > >
> > > 2026년 5월 1일 (금) 오전 5:23, Jens Scheffler <js...@apache.org>님이 작성:
> > >> Hi!
> > >>
> > >> thanks for the discussion. While I am not against this I would say in
> > >> Airflow 2 it was also not a "public API" but the DB connecton "just
> > >> used" to list and have a missing API compensated.
> > >>
> > >> Can you express what the demand for the missing feature is? What
> > >> business function did you implement based on listing all Variables?
> > >>
> > >> As you already stated and also highlighted in the PR the list() might
> > >> not tell about all Variables as the list is not provided from secret
> > >> managers. So it might (small risk thoug) lead to some confusion.
> Should
> > >> be explicitly documented in the PyDoc. But this is a nit.
> > >>
> > >> Jens
> > >>
> > >> On 30.04.26 11:47, 김준영 wrote:
> > >>> Hi all,
> > >>>
> > >>> I'd like to propose adding Variable.list() to the Task SDK to address
> > >>> the gap left by the removal of direct ORM access in Airflow 3.
> > >>>
> > >>> Background:
> > >>> In Airflow 2.x, users could list all variables via:
> > >>>
> > >>> from airflow.models import Variable
> > >>> from airflow.utils.session import create_session
> > >>>
> > >>> with create_session() as session:
> > >>> variables = session.query(Variable).all()
> > >>>
> > >>> In Airflow 3.x, this pattern raises:
> > >>> RuntimeError: Direct database access via the ORM is not
> allowed in Airflow
> > >>> 3.0
> > >>>
> > >>> There is currently no supported way to discover variable keys
> dynamically
> > >>> when they are not known at DAG authoring time.
> > >>>
> > >>> Proposal:
> > >>> - Add Variable.list(prefix=None) to the Task SDK
> > >>> - Scope is limited to the metadata database only (same as the
> old ORM pattern)
> > >>> - Secrets backend support is intentionally out of scope, as it
> would
> > >>> require a broader interface contract change and separate
> community
> > >>> discussion
> > >>>
> > >>> Related issue: https://github.com/apache/airflow/issues/61166
> > >>> Draft PR: https://github.com/apache/airflow/pull/66022
> > >>>
> > >>> I would appreciate any feedback or concerns from the community
> before
> > >>> this moves forward.
> > >>>
> > >>> Best Regards,
> > >>> Jun Yeong Kim
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >>> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] Add Variable.list() to Task SDK as a replacement for Variable.all()
Posted by 김준영 <ju...@gmail.com>.
Hi Amogh,
Quick closing note on this — PR #66022 (Variable.keys via lazy proxy)
merged yesterday. Thanks again for steering the design toward the lazy
iterator shape.
On your wider point that the listing gap exists for connections/xcoms/
etc. too: I scanned GitHub issues for equivalent reports and didn't
find concrete user demand for those resources (Variable.all() had a
direct report at #61166; Connection / Pool / XCom keys turn up
nothing similar). Reading that as a signal that Variable was the
unique case driven by a real DAG-author pattern, rather than a generic
API symmetry gap.
So I'm calling the scope done at Variable.keys for now and happy to
revisit if a concrete case shows up for another resource.
Thanks,
Jun Yeong
2026년 5월 6일 (수) 오전 1:35, 김준영 <ju...@gmail.com>님이 작성:
>
> Agreed with the design direction. I've pushed the change to return a
> lazy_object_proxy.Proxy. Would appreciate it if you could take a look.
>
> Best regards,
> Jun Yeong
>
> 2026년 5월 6일 (수) 오전 12:59, Amogh Desai <am...@apache.org>님이 작성:
> >
> > Small correction on the design: Variable.keys() returning list[str] is
> > eager. What I had in
> > mind was a lazy proxy object: fetch the keys from the execution API once on
> > first access, cache
> > them, then serve iteration/indexing from that cache. Something like:
> >
> > keys = Variable.keys(prefix="team_a_config_")
> > for key in keys:
> > val = Variable.get(key)
> >
> > Similar to how lazy_object_proxy.Proxy is used in the template context
> > today. Keeps the call
> > site clean without pulling everything upfront.
> >
> > Re Ash's point about AIP 103 -- task state is for tasks persisting state
> > across retries, not a shared
> > config store. So I think you read it right.
> >
> > And forgot to comment earlier, but re Jens' workaround -- probably better
> > suited for a centralized static
> > config, but it will likely fall apart when multiple external systems are
> > independently writing their own keys.
> > The use case you have is legitimate and the workaround is honestly awkward
> > for it.
> >
> > Regarding secrets backend scoping, I think I am ok with it being on the
> > metadata db backend for the time
> > being as long as we document it well.
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Tue, May 5, 2026 at 7:40 PM 김준영 <ju...@gmail.com> wrote:
> >
> > > Hi Ash,
> > >
> > > Regarding secrets backends—since they don't currently support listing
> > > in Airflow, I’ve scoped this feature to the metastore only and will
> > > document that explicitly. As for AIP-103, Task State is designed for
> > > tasks to persist their own state across retries, whereas this PR
> > > focuses on DAGs fetching externally-managed config at runtime, keeping
> > > the boundary between them clean.
> > >
> > > Best regards,
> > > Jun Yeong
> > >
> > > 2026년 5월 5일 (화) 오후 10:48, Ash Berlin-Taylor <as...@apache.org>님이 작성:
> > > >
> > > > Some of these use cases might be better suited to AIP-103 (the State
> > > storage AIP)
> > > >
> > > > One possible issue with exposing Variables.list() (and connections) — do
> > > the secrets backends support listing variables?
> > > >
> > > > -ash
> > > >
> > > > > On 5 May 2026, at 11:41, 김준영 <ju...@gmail.com> wrote:
> > > > >
> > > > > Hi Amogh,
> > > > >
> > > > > Thanks for the feedback! I've updated the PR (#66022) to implement the
> > > > > lazy iterator pattern you suggested:
> > > > >
> > > > > for key in Variable.keys(prefix="team_a_config_"):
> > > > > val = Variable.get(key)
> > > > >
> > > > > Now, Variable.keys(prefix=None) returns a list[str] of matching keys,
> > > > > supported by a new GET /variables/keys?prefix= endpoint in the
> > > > > Execution API. This allows callers to fetch only the specific values
> > > > > they need via Variable.get(key).
> > > > >
> > > > > Regarding your point about other Airflow resources (Connections,
> > > > > XComs, etc.) having the same gap: once this PR is merged and the
> > > > > pattern is established, I'd be happy to extend this keys() approach to
> > > > > those resources in follow-up PRs. Would that be a direction you'd
> > > > > encourage?
> > > > >
> > > > > The PR is ready for another review whenever you have a moment.
> > > > >
> > > > > Best regards,
> > > > > Jun Yeong
> > > > >
> > > > > 2026년 5월 5일 (화) 오후 5:51, Amogh Desai <am...@apache.org>님이 작성:
> > > > >>
> > > > >> Hi Jun Yeong,
> > > > >>
> > > > >> Valid gap. There is no GET /variables list endpoint at all, but that
> > > is
> > > > >> true for any
> > > > >> other Airflow artefacts too. So variables isn't the only one missing
> > > it,
> > > > >> its all of the
> > > > >> Airflow resources: connections, xcoms, etc.
> > > > >>
> > > > >> One suggestion worth considering: rather than Variable.list()
> > > returning a
> > > > >> full list,
> > > > >>
> > > > >> a lazy iterator might be a better fit. Something like:
> > > > >>
> > > > >>
> > > > >> for key in Variable.keys(prefix="team_a_config_"):
> > > > >> val = Variable.get(key)
> > > > >>
> > > > >> Happy to review when a PR is ready.
> > > > >>
> > > > >> Thanks & Regards,
> > > > >> Amogh Desai
> > > > >>
> > > > >>
> > > > >> On Fri, May 1, 2026 at 3:02 AM 김준영 <ju...@gmail.com> wrote:
> > > > >>
> > > > >>> Hi Jens,
> > > > >>>
> > > > >>> That is a valid alternative, and using a single JSON variable is
> > > > >>> indeed a great pattern for centralized or static configurations to
> > > > >>> reduce I/O.
> > > > >>>
> > > > >>> However, for the use cases I'm targeting, the "Single JSON Variable"
> > > > >>> approach has a few significant drawbacks:
> > > > >>>
> > > > >>> 1.Concurrency & Atomic Updates: When multiple external systems or
> > > > >>> independent CI/CD pipelines need to update their own configurations,
> > > a
> > > > >>> single JSON variable creates a race condition. They would have to
> > > > >>> "Read-Modify-Write" the entire blob, which risks overwriting each
> > > > >>> other's changes. Separate variables allow for atomic, independent
> > > > >>> updates.
> > > > >>>
> > > > >>> 2.Integration Complexity: Many users integrate Airflow with external
> > > > >>> tools that independently push values to the Airflow Metadata DB or
> > > > >>> Secret Backend. Forcing these decoupled systems to coordinate and
> > > > >>> maintain a single shared JSON structure adds significant integration
> > > > >>> overhead.
> > > > >>>
> > > > >>> 3.Data Modeling Flexibility: While a JSON blob works for some,
> > > > >>> Variable.list(prefix=) allows Airflow to be unopinionated about how
> > > > >>> users model their data. It provides a standard "Key-Value store"
> > > > >>> experience (similar to AWS SSM or Redis) where prefix-based discovery
> > > > >>> is a first-class citizen.
> > > > >>>
> > > > >>> In short, while the JSON approach is a good workaround for specific
> > > > >>> cases, Variable.list() provides the necessary flexibility for highly
> > > > >>> dynamic and decoupled environments without forcing a specific data
> > > > >>> structure on the user.
> > > > >>>
> > > > >>> What do you think?
> > > > >>>
> > > > >>> Best regards, Jun Yeong
> > > > >>>
> > > > >>> 2026년 5월 1일 (금) 오전 6:19, Jens Scheffler <js...@apache.org>님이 작성:
> > > > >>>>
> > > > >>>> Hi Jun Yeong,
> > > > >>>>
> > > > >>>> one thought on this: We had similar.
> > > > >>>>
> > > > >>>> Our use case: Implemented a custom archiving (Dag) that needed to
> > > take
> > > > >>>> care of different retention times in different Dags (maybe bad
> > > example
> > > > >>>> because this archival itself accesses the database for archiving...
> > > > >>>> haha) and we wanted to have different retention times per Dag.
> > > > >>>>
> > > > >>>> What we did is we made a JSON structure with all parameters into a
> > > > >>>> single Variable. Then we did not need to have many IO operations and
> > > > >>>> Variables but could store all in a single Variable.
> > > > >>>>
> > > > >>>> Would this be a viable solution for your case as well?
> > > > >>>>
> > > > >>>> Jens
> > > > >>>>
> > > > >>>> On 30.04.26 22:47, 김준영 wrote:
> > > > >>>>> Hi Jens,
> > > > >>>>>
> > > > >>>>> Thank you for the thoughtful feedback!
> > > > >>>>>
> > > > >>>>> The primary demand for this feature comes from workflows that
> > > require
> > > > >>>>> dynamic configuration discovery. A common pattern is grouping
> > > related
> > > > >>>>> variables under a shared prefix (e.g., team_a_config_ or
> > > > >>>>> pipeline_x_param_). In many cases, these keys are generated or
> > > updated
> > > > >>>>> dynamically by external systems, meaning the exact set of keys
> > > isn't
> > > > >>>>> known at DAG authoring time.
> > > > >>>>>
> > > > >>>>> While users in Airflow 2.x relied on session.query(Variable).all()
> > > as
> > > > >>>>> a workaround, Airflow 3’s move toward the Task SDK aims to abstract
> > > > >>>>> away direct ORM/DB access for better security and stability.
> > > > >>>>> Variable.list(prefix=) provides a supported, clean way to achieve
> > > this
> > > > >>>>> discovery within that new architecture.
> > > > >>>>>
> > > > >>>>> Regarding the secrets backend limitation, I completely agree. It’s
> > > > >>>>> important to manage expectations, so I will update the PyDoc to
> > > > >>>>> explicitly state that this method only lists variables stored in
> > > the
> > > > >>>>> metadata database.
> > > > >>>>>
> > > > >>>>> Best regards, Jun Yeong
> > > > >>>>>
> > > > >>>>> 2026년 5월 1일 (금) 오전 5:23, Jens Scheffler <js...@apache.org>님이
> > > 작성:
> > > > >>>>>> Hi!
> > > > >>>>>>
> > > > >>>>>> thanks for the discussion. While I am not against this I would
> > > say in
> > > > >>>>>> Airflow 2 it was also not a "public API" but the DB connecton
> > > "just
> > > > >>>>>> used" to list and have a missing API compensated.
> > > > >>>>>>
> > > > >>>>>> Can you express what the demand for the missing feature is? What
> > > > >>>>>> business function did you implement based on listing all
> > > Variables?
> > > > >>>>>>
> > > > >>>>>> As you already stated and also highlighted in the PR the list()
> > > might
> > > > >>>>>> not tell about all Variables as the list is not provided from
> > > secret
> > > > >>>>>> managers. So it might (small risk thoug) lead to some confusion.
> > > > >>> Should
> > > > >>>>>> be explicitly documented in the PyDoc. But this is a nit.
> > > > >>>>>>
> > > > >>>>>> Jens
> > > > >>>>>>
> > > > >>>>>> On 30.04.26 11:47, 김준영 wrote:
> > > > >>>>>>> Hi all,
> > > > >>>>>>>
> > > > >>>>>>> I'd like to propose adding Variable.list() to the Task SDK to
> > > address
> > > > >>>>>>> the gap left by the removal of direct ORM access in Airflow 3.
> > > > >>>>>>>
> > > > >>>>>>> Background:
> > > > >>>>>>> In Airflow 2.x, users could list all variables via:
> > > > >>>>>>>
> > > > >>>>>>> from airflow.models import Variable
> > > > >>>>>>> from airflow.utils.session import create_session
> > > > >>>>>>>
> > > > >>>>>>> with create_session() as session:
> > > > >>>>>>> variables = session.query(Variable).all()
> > > > >>>>>>>
> > > > >>>>>>> In Airflow 3.x, this pattern raises:
> > > > >>>>>>> RuntimeError: Direct database access via the ORM is not
> > > > >>> allowed in Airflow
> > > > >>>>>>> 3.0
> > > > >>>>>>>
> > > > >>>>>>> There is currently no supported way to discover variable keys
> > > > >>> dynamically
> > > > >>>>>>> when they are not known at DAG authoring time.
> > > > >>>>>>>
> > > > >>>>>>> Proposal:
> > > > >>>>>>> - Add Variable.list(prefix=None) to the Task SDK
> > > > >>>>>>> - Scope is limited to the metadata database only (same as the
> > > > >>> old ORM pattern)
> > > > >>>>>>> - Secrets backend support is intentionally out of scope, as it
> > > > >>> would
> > > > >>>>>>> require a broader interface contract change and separate
> > > > >>> community
> > > > >>>>>>> discussion
> > > > >>>>>>>
> > > > >>>>>>> Related issue: https://github.com/apache/airflow/issues/61166
> > > > >>>>>>> Draft PR: https://github.com/apache/airflow/pull/66022
> > > > >>>>>>>
> > > > >>>>>>> I would appreciate any feedback or concerns from the community
> > > > >>> before
> > > > >>>>>>> this moves forward.
> > > > >>>>>>>
> > > > >>>>>>> Best Regards,
> > > > >>>>>>> Jun Yeong Kim
> > > > >>>>>>>
> > > > >>>>>>>
> > > ---------------------------------------------------------------------
> > > > >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > >>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >>>>>>>
> > > > >>>>>>
> > > ---------------------------------------------------------------------
> > > > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >>>>>>
> > > > >>>>>
> > > ---------------------------------------------------------------------
> > > > >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > ---------------------------------------------------------------------
> > > > >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > >>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >>>>
> > > > >>>
> > > > >>> ---------------------------------------------------------------------
> > > > >>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > >>> For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >>>
> > > > >>>
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Add Variable.list() to Task SDK as a replacement for Variable.all()
Posted by 김준영 <ju...@gmail.com>.
Agreed with the design direction. I've pushed the change to return a
lazy_object_proxy.Proxy. Would appreciate it if you could take a look.
Best regards,
Jun Yeong
2026년 5월 6일 (수) 오전 12:59, Amogh Desai <am...@apache.org>님이 작성:
>
> Small correction on the design: Variable.keys() returning list[str] is
> eager. What I had in
> mind was a lazy proxy object: fetch the keys from the execution API once on
> first access, cache
> them, then serve iteration/indexing from that cache. Something like:
>
> keys = Variable.keys(prefix="team_a_config_")
> for key in keys:
> val = Variable.get(key)
>
> Similar to how lazy_object_proxy.Proxy is used in the template context
> today. Keeps the call
> site clean without pulling everything upfront.
>
> Re Ash's point about AIP 103 -- task state is for tasks persisting state
> across retries, not a shared
> config store. So I think you read it right.
>
> And forgot to comment earlier, but re Jens' workaround -- probably better
> suited for a centralized static
> config, but it will likely fall apart when multiple external systems are
> independently writing their own keys.
> The use case you have is legitimate and the workaround is honestly awkward
> for it.
>
> Regarding secrets backend scoping, I think I am ok with it being on the
> metadata db backend for the time
> being as long as we document it well.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Tue, May 5, 2026 at 7:40 PM 김준영 <ju...@gmail.com> wrote:
>
> > Hi Ash,
> >
> > Regarding secrets backends—since they don't currently support listing
> > in Airflow, I’ve scoped this feature to the metastore only and will
> > document that explicitly. As for AIP-103, Task State is designed for
> > tasks to persist their own state across retries, whereas this PR
> > focuses on DAGs fetching externally-managed config at runtime, keeping
> > the boundary between them clean.
> >
> > Best regards,
> > Jun Yeong
> >
> > 2026년 5월 5일 (화) 오후 10:48, Ash Berlin-Taylor <as...@apache.org>님이 작성:
> > >
> > > Some of these use cases might be better suited to AIP-103 (the State
> > storage AIP)
> > >
> > > One possible issue with exposing Variables.list() (and connections) — do
> > the secrets backends support listing variables?
> > >
> > > -ash
> > >
> > > > On 5 May 2026, at 11:41, 김준영 <ju...@gmail.com> wrote:
> > > >
> > > > Hi Amogh,
> > > >
> > > > Thanks for the feedback! I've updated the PR (#66022) to implement the
> > > > lazy iterator pattern you suggested:
> > > >
> > > > for key in Variable.keys(prefix="team_a_config_"):
> > > > val = Variable.get(key)
> > > >
> > > > Now, Variable.keys(prefix=None) returns a list[str] of matching keys,
> > > > supported by a new GET /variables/keys?prefix= endpoint in the
> > > > Execution API. This allows callers to fetch only the specific values
> > > > they need via Variable.get(key).
> > > >
> > > > Regarding your point about other Airflow resources (Connections,
> > > > XComs, etc.) having the same gap: once this PR is merged and the
> > > > pattern is established, I'd be happy to extend this keys() approach to
> > > > those resources in follow-up PRs. Would that be a direction you'd
> > > > encourage?
> > > >
> > > > The PR is ready for another review whenever you have a moment.
> > > >
> > > > Best regards,
> > > > Jun Yeong
> > > >
> > > > 2026년 5월 5일 (화) 오후 5:51, Amogh Desai <am...@apache.org>님이 작성:
> > > >>
> > > >> Hi Jun Yeong,
> > > >>
> > > >> Valid gap. There is no GET /variables list endpoint at all, but that
> > is
> > > >> true for any
> > > >> other Airflow artefacts too. So variables isn't the only one missing
> > it,
> > > >> its all of the
> > > >> Airflow resources: connections, xcoms, etc.
> > > >>
> > > >> One suggestion worth considering: rather than Variable.list()
> > returning a
> > > >> full list,
> > > >>
> > > >> a lazy iterator might be a better fit. Something like:
> > > >>
> > > >>
> > > >> for key in Variable.keys(prefix="team_a_config_"):
> > > >> val = Variable.get(key)
> > > >>
> > > >> Happy to review when a PR is ready.
> > > >>
> > > >> Thanks & Regards,
> > > >> Amogh Desai
> > > >>
> > > >>
> > > >> On Fri, May 1, 2026 at 3:02 AM 김준영 <ju...@gmail.com> wrote:
> > > >>
> > > >>> Hi Jens,
> > > >>>
> > > >>> That is a valid alternative, and using a single JSON variable is
> > > >>> indeed a great pattern for centralized or static configurations to
> > > >>> reduce I/O.
> > > >>>
> > > >>> However, for the use cases I'm targeting, the "Single JSON Variable"
> > > >>> approach has a few significant drawbacks:
> > > >>>
> > > >>> 1.Concurrency & Atomic Updates: When multiple external systems or
> > > >>> independent CI/CD pipelines need to update their own configurations,
> > a
> > > >>> single JSON variable creates a race condition. They would have to
> > > >>> "Read-Modify-Write" the entire blob, which risks overwriting each
> > > >>> other's changes. Separate variables allow for atomic, independent
> > > >>> updates.
> > > >>>
> > > >>> 2.Integration Complexity: Many users integrate Airflow with external
> > > >>> tools that independently push values to the Airflow Metadata DB or
> > > >>> Secret Backend. Forcing these decoupled systems to coordinate and
> > > >>> maintain a single shared JSON structure adds significant integration
> > > >>> overhead.
> > > >>>
> > > >>> 3.Data Modeling Flexibility: While a JSON blob works for some,
> > > >>> Variable.list(prefix=) allows Airflow to be unopinionated about how
> > > >>> users model their data. It provides a standard "Key-Value store"
> > > >>> experience (similar to AWS SSM or Redis) where prefix-based discovery
> > > >>> is a first-class citizen.
> > > >>>
> > > >>> In short, while the JSON approach is a good workaround for specific
> > > >>> cases, Variable.list() provides the necessary flexibility for highly
> > > >>> dynamic and decoupled environments without forcing a specific data
> > > >>> structure on the user.
> > > >>>
> > > >>> What do you think?
> > > >>>
> > > >>> Best regards, Jun Yeong
> > > >>>
> > > >>> 2026년 5월 1일 (금) 오전 6:19, Jens Scheffler <js...@apache.org>님이 작성:
> > > >>>>
> > > >>>> Hi Jun Yeong,
> > > >>>>
> > > >>>> one thought on this: We had similar.
> > > >>>>
> > > >>>> Our use case: Implemented a custom archiving (Dag) that needed to
> > take
> > > >>>> care of different retention times in different Dags (maybe bad
> > example
> > > >>>> because this archival itself accesses the database for archiving...
> > > >>>> haha) and we wanted to have different retention times per Dag.
> > > >>>>
> > > >>>> What we did is we made a JSON structure with all parameters into a
> > > >>>> single Variable. Then we did not need to have many IO operations and
> > > >>>> Variables but could store all in a single Variable.
> > > >>>>
> > > >>>> Would this be a viable solution for your case as well?
> > > >>>>
> > > >>>> Jens
> > > >>>>
> > > >>>> On 30.04.26 22:47, 김준영 wrote:
> > > >>>>> Hi Jens,
> > > >>>>>
> > > >>>>> Thank you for the thoughtful feedback!
> > > >>>>>
> > > >>>>> The primary demand for this feature comes from workflows that
> > require
> > > >>>>> dynamic configuration discovery. A common pattern is grouping
> > related
> > > >>>>> variables under a shared prefix (e.g., team_a_config_ or
> > > >>>>> pipeline_x_param_). In many cases, these keys are generated or
> > updated
> > > >>>>> dynamically by external systems, meaning the exact set of keys
> > isn't
> > > >>>>> known at DAG authoring time.
> > > >>>>>
> > > >>>>> While users in Airflow 2.x relied on session.query(Variable).all()
> > as
> > > >>>>> a workaround, Airflow 3’s move toward the Task SDK aims to abstract
> > > >>>>> away direct ORM/DB access for better security and stability.
> > > >>>>> Variable.list(prefix=) provides a supported, clean way to achieve
> > this
> > > >>>>> discovery within that new architecture.
> > > >>>>>
> > > >>>>> Regarding the secrets backend limitation, I completely agree. It’s
> > > >>>>> important to manage expectations, so I will update the PyDoc to
> > > >>>>> explicitly state that this method only lists variables stored in
> > the
> > > >>>>> metadata database.
> > > >>>>>
> > > >>>>> Best regards, Jun Yeong
> > > >>>>>
> > > >>>>> 2026년 5월 1일 (금) 오전 5:23, Jens Scheffler <js...@apache.org>님이
> > 작성:
> > > >>>>>> Hi!
> > > >>>>>>
> > > >>>>>> thanks for the discussion. While I am not against this I would
> > say in
> > > >>>>>> Airflow 2 it was also not a "public API" but the DB connecton
> > "just
> > > >>>>>> used" to list and have a missing API compensated.
> > > >>>>>>
> > > >>>>>> Can you express what the demand for the missing feature is? What
> > > >>>>>> business function did you implement based on listing all
> > Variables?
> > > >>>>>>
> > > >>>>>> As you already stated and also highlighted in the PR the list()
> > might
> > > >>>>>> not tell about all Variables as the list is not provided from
> > secret
> > > >>>>>> managers. So it might (small risk thoug) lead to some confusion.
> > > >>> Should
> > > >>>>>> be explicitly documented in the PyDoc. But this is a nit.
> > > >>>>>>
> > > >>>>>> Jens
> > > >>>>>>
> > > >>>>>> On 30.04.26 11:47, 김준영 wrote:
> > > >>>>>>> Hi all,
> > > >>>>>>>
> > > >>>>>>> I'd like to propose adding Variable.list() to the Task SDK to
> > address
> > > >>>>>>> the gap left by the removal of direct ORM access in Airflow 3.
> > > >>>>>>>
> > > >>>>>>> Background:
> > > >>>>>>> In Airflow 2.x, users could list all variables via:
> > > >>>>>>>
> > > >>>>>>> from airflow.models import Variable
> > > >>>>>>> from airflow.utils.session import create_session
> > > >>>>>>>
> > > >>>>>>> with create_session() as session:
> > > >>>>>>> variables = session.query(Variable).all()
> > > >>>>>>>
> > > >>>>>>> In Airflow 3.x, this pattern raises:
> > > >>>>>>> RuntimeError: Direct database access via the ORM is not
> > > >>> allowed in Airflow
> > > >>>>>>> 3.0
> > > >>>>>>>
> > > >>>>>>> There is currently no supported way to discover variable keys
> > > >>> dynamically
> > > >>>>>>> when they are not known at DAG authoring time.
> > > >>>>>>>
> > > >>>>>>> Proposal:
> > > >>>>>>> - Add Variable.list(prefix=None) to the Task SDK
> > > >>>>>>> - Scope is limited to the metadata database only (same as the
> > > >>> old ORM pattern)
> > > >>>>>>> - Secrets backend support is intentionally out of scope, as it
> > > >>> would
> > > >>>>>>> require a broader interface contract change and separate
> > > >>> community
> > > >>>>>>> discussion
> > > >>>>>>>
> > > >>>>>>> Related issue: https://github.com/apache/airflow/issues/61166
> > > >>>>>>> Draft PR: https://github.com/apache/airflow/pull/66022
> > > >>>>>>>
> > > >>>>>>> I would appreciate any feedback or concerns from the community
> > > >>> before
> > > >>>>>>> this moves forward.
> > > >>>>>>>
> > > >>>>>>> Best Regards,
> > > >>>>>>> Jun Yeong Kim
> > > >>>>>>>
> > > >>>>>>>
> > ---------------------------------------------------------------------
> > > >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > > >>>>>>>
> > > >>>>>>
> > ---------------------------------------------------------------------
> > > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > > >>>>>>
> > > >>>>>
> > ---------------------------------------------------------------------
> > > >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > > >>>>>
> > > >>>>
> > > >>>>
> > ---------------------------------------------------------------------
> > > >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > > >>>>
> > > >>>
> > > >>> ---------------------------------------------------------------------
> > > >>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >>> For additional commands, e-mail: dev-help@airflow.apache.org
> > > >>>
> > > >>>
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Add Variable.list() to Task SDK as a replacement for Variable.all()
Posted by Amogh Desai <am...@apache.org>.
Small correction on the design: Variable.keys() returning list[str] is
eager. What I had in
mind was a lazy proxy object: fetch the keys from the execution API once on
first access, cache
them, then serve iteration/indexing from that cache. Something like:
keys = Variable.keys(prefix="team_a_config_")
for key in keys:
val = Variable.get(key)
Similar to how lazy_object_proxy.Proxy is used in the template context
today. Keeps the call
site clean without pulling everything upfront.
Re Ash's point about AIP 103 -- task state is for tasks persisting state
across retries, not a shared
config store. So I think you read it right.
And forgot to comment earlier, but re Jens' workaround -- probably better
suited for a centralized static
config, but it will likely fall apart when multiple external systems are
independently writing their own keys.
The use case you have is legitimate and the workaround is honestly awkward
for it.
Regarding secrets backend scoping, I think I am ok with it being on the
metadata db backend for the time
being as long as we document it well.
Thanks & Regards,
Amogh Desai
On Tue, May 5, 2026 at 7:40 PM 김준영 <ju...@gmail.com> wrote:
> Hi Ash,
>
> Regarding secrets backends—since they don't currently support listing
> in Airflow, I’ve scoped this feature to the metastore only and will
> document that explicitly. As for AIP-103, Task State is designed for
> tasks to persist their own state across retries, whereas this PR
> focuses on DAGs fetching externally-managed config at runtime, keeping
> the boundary between them clean.
>
> Best regards,
> Jun Yeong
>
> 2026년 5월 5일 (화) 오후 10:48, Ash Berlin-Taylor <as...@apache.org>님이 작성:
> >
> > Some of these use cases might be better suited to AIP-103 (the State
> storage AIP)
> >
> > One possible issue with exposing Variables.list() (and connections) — do
> the secrets backends support listing variables?
> >
> > -ash
> >
> > > On 5 May 2026, at 11:41, 김준영 <ju...@gmail.com> wrote:
> > >
> > > Hi Amogh,
> > >
> > > Thanks for the feedback! I've updated the PR (#66022) to implement the
> > > lazy iterator pattern you suggested:
> > >
> > > for key in Variable.keys(prefix="team_a_config_"):
> > > val = Variable.get(key)
> > >
> > > Now, Variable.keys(prefix=None) returns a list[str] of matching keys,
> > > supported by a new GET /variables/keys?prefix= endpoint in the
> > > Execution API. This allows callers to fetch only the specific values
> > > they need via Variable.get(key).
> > >
> > > Regarding your point about other Airflow resources (Connections,
> > > XComs, etc.) having the same gap: once this PR is merged and the
> > > pattern is established, I'd be happy to extend this keys() approach to
> > > those resources in follow-up PRs. Would that be a direction you'd
> > > encourage?
> > >
> > > The PR is ready for another review whenever you have a moment.
> > >
> > > Best regards,
> > > Jun Yeong
> > >
> > > 2026년 5월 5일 (화) 오후 5:51, Amogh Desai <am...@apache.org>님이 작성:
> > >>
> > >> Hi Jun Yeong,
> > >>
> > >> Valid gap. There is no GET /variables list endpoint at all, but that
> is
> > >> true for any
> > >> other Airflow artefacts too. So variables isn't the only one missing
> it,
> > >> its all of the
> > >> Airflow resources: connections, xcoms, etc.
> > >>
> > >> One suggestion worth considering: rather than Variable.list()
> returning a
> > >> full list,
> > >>
> > >> a lazy iterator might be a better fit. Something like:
> > >>
> > >>
> > >> for key in Variable.keys(prefix="team_a_config_"):
> > >> val = Variable.get(key)
> > >>
> > >> Happy to review when a PR is ready.
> > >>
> > >> Thanks & Regards,
> > >> Amogh Desai
> > >>
> > >>
> > >> On Fri, May 1, 2026 at 3:02 AM 김준영 <ju...@gmail.com> wrote:
> > >>
> > >>> Hi Jens,
> > >>>
> > >>> That is a valid alternative, and using a single JSON variable is
> > >>> indeed a great pattern for centralized or static configurations to
> > >>> reduce I/O.
> > >>>
> > >>> However, for the use cases I'm targeting, the "Single JSON Variable"
> > >>> approach has a few significant drawbacks:
> > >>>
> > >>> 1.Concurrency & Atomic Updates: When multiple external systems or
> > >>> independent CI/CD pipelines need to update their own configurations,
> a
> > >>> single JSON variable creates a race condition. They would have to
> > >>> "Read-Modify-Write" the entire blob, which risks overwriting each
> > >>> other's changes. Separate variables allow for atomic, independent
> > >>> updates.
> > >>>
> > >>> 2.Integration Complexity: Many users integrate Airflow with external
> > >>> tools that independently push values to the Airflow Metadata DB or
> > >>> Secret Backend. Forcing these decoupled systems to coordinate and
> > >>> maintain a single shared JSON structure adds significant integration
> > >>> overhead.
> > >>>
> > >>> 3.Data Modeling Flexibility: While a JSON blob works for some,
> > >>> Variable.list(prefix=) allows Airflow to be unopinionated about how
> > >>> users model their data. It provides a standard "Key-Value store"
> > >>> experience (similar to AWS SSM or Redis) where prefix-based discovery
> > >>> is a first-class citizen.
> > >>>
> > >>> In short, while the JSON approach is a good workaround for specific
> > >>> cases, Variable.list() provides the necessary flexibility for highly
> > >>> dynamic and decoupled environments without forcing a specific data
> > >>> structure on the user.
> > >>>
> > >>> What do you think?
> > >>>
> > >>> Best regards, Jun Yeong
> > >>>
> > >>> 2026년 5월 1일 (금) 오전 6:19, Jens Scheffler <js...@apache.org>님이 작성:
> > >>>>
> > >>>> Hi Jun Yeong,
> > >>>>
> > >>>> one thought on this: We had similar.
> > >>>>
> > >>>> Our use case: Implemented a custom archiving (Dag) that needed to
> take
> > >>>> care of different retention times in different Dags (maybe bad
> example
> > >>>> because this archival itself accesses the database for archiving...
> > >>>> haha) and we wanted to have different retention times per Dag.
> > >>>>
> > >>>> What we did is we made a JSON structure with all parameters into a
> > >>>> single Variable. Then we did not need to have many IO operations and
> > >>>> Variables but could store all in a single Variable.
> > >>>>
> > >>>> Would this be a viable solution for your case as well?
> > >>>>
> > >>>> Jens
> > >>>>
> > >>>> On 30.04.26 22:47, 김준영 wrote:
> > >>>>> Hi Jens,
> > >>>>>
> > >>>>> Thank you for the thoughtful feedback!
> > >>>>>
> > >>>>> The primary demand for this feature comes from workflows that
> require
> > >>>>> dynamic configuration discovery. A common pattern is grouping
> related
> > >>>>> variables under a shared prefix (e.g., team_a_config_ or
> > >>>>> pipeline_x_param_). In many cases, these keys are generated or
> updated
> > >>>>> dynamically by external systems, meaning the exact set of keys
> isn't
> > >>>>> known at DAG authoring time.
> > >>>>>
> > >>>>> While users in Airflow 2.x relied on session.query(Variable).all()
> as
> > >>>>> a workaround, Airflow 3’s move toward the Task SDK aims to abstract
> > >>>>> away direct ORM/DB access for better security and stability.
> > >>>>> Variable.list(prefix=) provides a supported, clean way to achieve
> this
> > >>>>> discovery within that new architecture.
> > >>>>>
> > >>>>> Regarding the secrets backend limitation, I completely agree. It’s
> > >>>>> important to manage expectations, so I will update the PyDoc to
> > >>>>> explicitly state that this method only lists variables stored in
> the
> > >>>>> metadata database.
> > >>>>>
> > >>>>> Best regards, Jun Yeong
> > >>>>>
> > >>>>> 2026년 5월 1일 (금) 오전 5:23, Jens Scheffler <js...@apache.org>님이
> 작성:
> > >>>>>> Hi!
> > >>>>>>
> > >>>>>> thanks for the discussion. While I am not against this I would
> say in
> > >>>>>> Airflow 2 it was also not a "public API" but the DB connecton
> "just
> > >>>>>> used" to list and have a missing API compensated.
> > >>>>>>
> > >>>>>> Can you express what the demand for the missing feature is? What
> > >>>>>> business function did you implement based on listing all
> Variables?
> > >>>>>>
> > >>>>>> As you already stated and also highlighted in the PR the list()
> might
> > >>>>>> not tell about all Variables as the list is not provided from
> secret
> > >>>>>> managers. So it might (small risk thoug) lead to some confusion.
> > >>> Should
> > >>>>>> be explicitly documented in the PyDoc. But this is a nit.
> > >>>>>>
> > >>>>>> Jens
> > >>>>>>
> > >>>>>> On 30.04.26 11:47, 김준영 wrote:
> > >>>>>>> Hi all,
> > >>>>>>>
> > >>>>>>> I'd like to propose adding Variable.list() to the Task SDK to
> address
> > >>>>>>> the gap left by the removal of direct ORM access in Airflow 3.
> > >>>>>>>
> > >>>>>>> Background:
> > >>>>>>> In Airflow 2.x, users could list all variables via:
> > >>>>>>>
> > >>>>>>> from airflow.models import Variable
> > >>>>>>> from airflow.utils.session import create_session
> > >>>>>>>
> > >>>>>>> with create_session() as session:
> > >>>>>>> variables = session.query(Variable).all()
> > >>>>>>>
> > >>>>>>> In Airflow 3.x, this pattern raises:
> > >>>>>>> RuntimeError: Direct database access via the ORM is not
> > >>> allowed in Airflow
> > >>>>>>> 3.0
> > >>>>>>>
> > >>>>>>> There is currently no supported way to discover variable keys
> > >>> dynamically
> > >>>>>>> when they are not known at DAG authoring time.
> > >>>>>>>
> > >>>>>>> Proposal:
> > >>>>>>> - Add Variable.list(prefix=None) to the Task SDK
> > >>>>>>> - Scope is limited to the metadata database only (same as the
> > >>> old ORM pattern)
> > >>>>>>> - Secrets backend support is intentionally out of scope, as it
> > >>> would
> > >>>>>>> require a broader interface contract change and separate
> > >>> community
> > >>>>>>> discussion
> > >>>>>>>
> > >>>>>>> Related issue: https://github.com/apache/airflow/issues/61166
> > >>>>>>> Draft PR: https://github.com/apache/airflow/pull/66022
> > >>>>>>>
> > >>>>>>> I would appreciate any feedback or concerns from the community
> > >>> before
> > >>>>>>> this moves forward.
> > >>>>>>>
> > >>>>>>> Best Regards,
> > >>>>>>> Jun Yeong Kim
> > >>>>>>>
> > >>>>>>>
> ---------------------------------------------------------------------
> > >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>>>>>>
> > >>>>>>
> ---------------------------------------------------------------------
> > >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>>>>>
> > >>>>>
> ---------------------------------------------------------------------
> > >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>>>>
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>>>
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >>> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>>
> > >>>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] Add Variable.list() to Task SDK as a replacement for Variable.all()
Posted by 김준영 <ju...@gmail.com>.
Hi Ash,
Regarding secrets backends—since they don't currently support listing
in Airflow, I’ve scoped this feature to the metastore only and will
document that explicitly. As for AIP-103, Task State is designed for
tasks to persist their own state across retries, whereas this PR
focuses on DAGs fetching externally-managed config at runtime, keeping
the boundary between them clean.
Best regards,
Jun Yeong
2026년 5월 5일 (화) 오후 10:48, Ash Berlin-Taylor <as...@apache.org>님이 작성:
>
> Some of these use cases might be better suited to AIP-103 (the State storage AIP)
>
> One possible issue with exposing Variables.list() (and connections) — do the secrets backends support listing variables?
>
> -ash
>
> > On 5 May 2026, at 11:41, 김준영 <ju...@gmail.com> wrote:
> >
> > Hi Amogh,
> >
> > Thanks for the feedback! I've updated the PR (#66022) to implement the
> > lazy iterator pattern you suggested:
> >
> > for key in Variable.keys(prefix="team_a_config_"):
> > val = Variable.get(key)
> >
> > Now, Variable.keys(prefix=None) returns a list[str] of matching keys,
> > supported by a new GET /variables/keys?prefix= endpoint in the
> > Execution API. This allows callers to fetch only the specific values
> > they need via Variable.get(key).
> >
> > Regarding your point about other Airflow resources (Connections,
> > XComs, etc.) having the same gap: once this PR is merged and the
> > pattern is established, I'd be happy to extend this keys() approach to
> > those resources in follow-up PRs. Would that be a direction you'd
> > encourage?
> >
> > The PR is ready for another review whenever you have a moment.
> >
> > Best regards,
> > Jun Yeong
> >
> > 2026년 5월 5일 (화) 오후 5:51, Amogh Desai <am...@apache.org>님이 작성:
> >>
> >> Hi Jun Yeong,
> >>
> >> Valid gap. There is no GET /variables list endpoint at all, but that is
> >> true for any
> >> other Airflow artefacts too. So variables isn't the only one missing it,
> >> its all of the
> >> Airflow resources: connections, xcoms, etc.
> >>
> >> One suggestion worth considering: rather than Variable.list() returning a
> >> full list,
> >>
> >> a lazy iterator might be a better fit. Something like:
> >>
> >>
> >> for key in Variable.keys(prefix="team_a_config_"):
> >> val = Variable.get(key)
> >>
> >> Happy to review when a PR is ready.
> >>
> >> Thanks & Regards,
> >> Amogh Desai
> >>
> >>
> >> On Fri, May 1, 2026 at 3:02 AM 김준영 <ju...@gmail.com> wrote:
> >>
> >>> Hi Jens,
> >>>
> >>> That is a valid alternative, and using a single JSON variable is
> >>> indeed a great pattern for centralized or static configurations to
> >>> reduce I/O.
> >>>
> >>> However, for the use cases I'm targeting, the "Single JSON Variable"
> >>> approach has a few significant drawbacks:
> >>>
> >>> 1.Concurrency & Atomic Updates: When multiple external systems or
> >>> independent CI/CD pipelines need to update their own configurations, a
> >>> single JSON variable creates a race condition. They would have to
> >>> "Read-Modify-Write" the entire blob, which risks overwriting each
> >>> other's changes. Separate variables allow for atomic, independent
> >>> updates.
> >>>
> >>> 2.Integration Complexity: Many users integrate Airflow with external
> >>> tools that independently push values to the Airflow Metadata DB or
> >>> Secret Backend. Forcing these decoupled systems to coordinate and
> >>> maintain a single shared JSON structure adds significant integration
> >>> overhead.
> >>>
> >>> 3.Data Modeling Flexibility: While a JSON blob works for some,
> >>> Variable.list(prefix=) allows Airflow to be unopinionated about how
> >>> users model their data. It provides a standard "Key-Value store"
> >>> experience (similar to AWS SSM or Redis) where prefix-based discovery
> >>> is a first-class citizen.
> >>>
> >>> In short, while the JSON approach is a good workaround for specific
> >>> cases, Variable.list() provides the necessary flexibility for highly
> >>> dynamic and decoupled environments without forcing a specific data
> >>> structure on the user.
> >>>
> >>> What do you think?
> >>>
> >>> Best regards, Jun Yeong
> >>>
> >>> 2026년 5월 1일 (금) 오전 6:19, Jens Scheffler <js...@apache.org>님이 작성:
> >>>>
> >>>> Hi Jun Yeong,
> >>>>
> >>>> one thought on this: We had similar.
> >>>>
> >>>> Our use case: Implemented a custom archiving (Dag) that needed to take
> >>>> care of different retention times in different Dags (maybe bad example
> >>>> because this archival itself accesses the database for archiving...
> >>>> haha) and we wanted to have different retention times per Dag.
> >>>>
> >>>> What we did is we made a JSON structure with all parameters into a
> >>>> single Variable. Then we did not need to have many IO operations and
> >>>> Variables but could store all in a single Variable.
> >>>>
> >>>> Would this be a viable solution for your case as well?
> >>>>
> >>>> Jens
> >>>>
> >>>> On 30.04.26 22:47, 김준영 wrote:
> >>>>> Hi Jens,
> >>>>>
> >>>>> Thank you for the thoughtful feedback!
> >>>>>
> >>>>> The primary demand for this feature comes from workflows that require
> >>>>> dynamic configuration discovery. A common pattern is grouping related
> >>>>> variables under a shared prefix (e.g., team_a_config_ or
> >>>>> pipeline_x_param_). In many cases, these keys are generated or updated
> >>>>> dynamically by external systems, meaning the exact set of keys isn't
> >>>>> known at DAG authoring time.
> >>>>>
> >>>>> While users in Airflow 2.x relied on session.query(Variable).all() as
> >>>>> a workaround, Airflow 3’s move toward the Task SDK aims to abstract
> >>>>> away direct ORM/DB access for better security and stability.
> >>>>> Variable.list(prefix=) provides a supported, clean way to achieve this
> >>>>> discovery within that new architecture.
> >>>>>
> >>>>> Regarding the secrets backend limitation, I completely agree. It’s
> >>>>> important to manage expectations, so I will update the PyDoc to
> >>>>> explicitly state that this method only lists variables stored in the
> >>>>> metadata database.
> >>>>>
> >>>>> Best regards, Jun Yeong
> >>>>>
> >>>>> 2026년 5월 1일 (금) 오전 5:23, Jens Scheffler <js...@apache.org>님이 작성:
> >>>>>> Hi!
> >>>>>>
> >>>>>> thanks for the discussion. While I am not against this I would say in
> >>>>>> Airflow 2 it was also not a "public API" but the DB connecton "just
> >>>>>> used" to list and have a missing API compensated.
> >>>>>>
> >>>>>> Can you express what the demand for the missing feature is? What
> >>>>>> business function did you implement based on listing all Variables?
> >>>>>>
> >>>>>> As you already stated and also highlighted in the PR the list() might
> >>>>>> not tell about all Variables as the list is not provided from secret
> >>>>>> managers. So it might (small risk thoug) lead to some confusion.
> >>> Should
> >>>>>> be explicitly documented in the PyDoc. But this is a nit.
> >>>>>>
> >>>>>> Jens
> >>>>>>
> >>>>>> On 30.04.26 11:47, 김준영 wrote:
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> I'd like to propose adding Variable.list() to the Task SDK to address
> >>>>>>> the gap left by the removal of direct ORM access in Airflow 3.
> >>>>>>>
> >>>>>>> Background:
> >>>>>>> In Airflow 2.x, users could list all variables via:
> >>>>>>>
> >>>>>>> from airflow.models import Variable
> >>>>>>> from airflow.utils.session import create_session
> >>>>>>>
> >>>>>>> with create_session() as session:
> >>>>>>> variables = session.query(Variable).all()
> >>>>>>>
> >>>>>>> In Airflow 3.x, this pattern raises:
> >>>>>>> RuntimeError: Direct database access via the ORM is not
> >>> allowed in Airflow
> >>>>>>> 3.0
> >>>>>>>
> >>>>>>> There is currently no supported way to discover variable keys
> >>> dynamically
> >>>>>>> when they are not known at DAG authoring time.
> >>>>>>>
> >>>>>>> Proposal:
> >>>>>>> - Add Variable.list(prefix=None) to the Task SDK
> >>>>>>> - Scope is limited to the metadata database only (same as the
> >>> old ORM pattern)
> >>>>>>> - Secrets backend support is intentionally out of scope, as it
> >>> would
> >>>>>>> require a broader interface contract change and separate
> >>> community
> >>>>>>> discussion
> >>>>>>>
> >>>>>>> Related issue: https://github.com/apache/airflow/issues/61166
> >>>>>>> Draft PR: https://github.com/apache/airflow/pull/66022
> >>>>>>>
> >>>>>>> I would appreciate any feedback or concerns from the community
> >>> before
> >>>>>>> this moves forward.
> >>>>>>>
> >>>>>>> Best Regards,
> >>>>>>> Jun Yeong Kim
> >>>>>>>
> >>>>>>> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>
> >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Add Variable.list() to Task SDK as a replacement for Variable.all()
Posted by Ash Berlin-Taylor <as...@apache.org>.
Some of these use cases might be better suited to AIP-103 (the State storage AIP)
One possible issue with exposing Variables.list() (and connections) — do the secrets backends support listing variables?
-ash
> On 5 May 2026, at 11:41, 김준영 <ju...@gmail.com> wrote:
>
> Hi Amogh,
>
> Thanks for the feedback! I've updated the PR (#66022) to implement the
> lazy iterator pattern you suggested:
>
> for key in Variable.keys(prefix="team_a_config_"):
> val = Variable.get(key)
>
> Now, Variable.keys(prefix=None) returns a list[str] of matching keys,
> supported by a new GET /variables/keys?prefix= endpoint in the
> Execution API. This allows callers to fetch only the specific values
> they need via Variable.get(key).
>
> Regarding your point about other Airflow resources (Connections,
> XComs, etc.) having the same gap: once this PR is merged and the
> pattern is established, I'd be happy to extend this keys() approach to
> those resources in follow-up PRs. Would that be a direction you'd
> encourage?
>
> The PR is ready for another review whenever you have a moment.
>
> Best regards,
> Jun Yeong
>
> 2026년 5월 5일 (화) 오후 5:51, Amogh Desai <am...@apache.org>님이 작성:
>>
>> Hi Jun Yeong,
>>
>> Valid gap. There is no GET /variables list endpoint at all, but that is
>> true for any
>> other Airflow artefacts too. So variables isn't the only one missing it,
>> its all of the
>> Airflow resources: connections, xcoms, etc.
>>
>> One suggestion worth considering: rather than Variable.list() returning a
>> full list,
>>
>> a lazy iterator might be a better fit. Something like:
>>
>>
>> for key in Variable.keys(prefix="team_a_config_"):
>> val = Variable.get(key)
>>
>> Happy to review when a PR is ready.
>>
>> Thanks & Regards,
>> Amogh Desai
>>
>>
>> On Fri, May 1, 2026 at 3:02 AM 김준영 <ju...@gmail.com> wrote:
>>
>>> Hi Jens,
>>>
>>> That is a valid alternative, and using a single JSON variable is
>>> indeed a great pattern for centralized or static configurations to
>>> reduce I/O.
>>>
>>> However, for the use cases I'm targeting, the "Single JSON Variable"
>>> approach has a few significant drawbacks:
>>>
>>> 1.Concurrency & Atomic Updates: When multiple external systems or
>>> independent CI/CD pipelines need to update their own configurations, a
>>> single JSON variable creates a race condition. They would have to
>>> "Read-Modify-Write" the entire blob, which risks overwriting each
>>> other's changes. Separate variables allow for atomic, independent
>>> updates.
>>>
>>> 2.Integration Complexity: Many users integrate Airflow with external
>>> tools that independently push values to the Airflow Metadata DB or
>>> Secret Backend. Forcing these decoupled systems to coordinate and
>>> maintain a single shared JSON structure adds significant integration
>>> overhead.
>>>
>>> 3.Data Modeling Flexibility: While a JSON blob works for some,
>>> Variable.list(prefix=) allows Airflow to be unopinionated about how
>>> users model their data. It provides a standard "Key-Value store"
>>> experience (similar to AWS SSM or Redis) where prefix-based discovery
>>> is a first-class citizen.
>>>
>>> In short, while the JSON approach is a good workaround for specific
>>> cases, Variable.list() provides the necessary flexibility for highly
>>> dynamic and decoupled environments without forcing a specific data
>>> structure on the user.
>>>
>>> What do you think?
>>>
>>> Best regards, Jun Yeong
>>>
>>> 2026년 5월 1일 (금) 오전 6:19, Jens Scheffler <js...@apache.org>님이 작성:
>>>>
>>>> Hi Jun Yeong,
>>>>
>>>> one thought on this: We had similar.
>>>>
>>>> Our use case: Implemented a custom archiving (Dag) that needed to take
>>>> care of different retention times in different Dags (maybe bad example
>>>> because this archival itself accesses the database for archiving...
>>>> haha) and we wanted to have different retention times per Dag.
>>>>
>>>> What we did is we made a JSON structure with all parameters into a
>>>> single Variable. Then we did not need to have many IO operations and
>>>> Variables but could store all in a single Variable.
>>>>
>>>> Would this be a viable solution for your case as well?
>>>>
>>>> Jens
>>>>
>>>> On 30.04.26 22:47, 김준영 wrote:
>>>>> Hi Jens,
>>>>>
>>>>> Thank you for the thoughtful feedback!
>>>>>
>>>>> The primary demand for this feature comes from workflows that require
>>>>> dynamic configuration discovery. A common pattern is grouping related
>>>>> variables under a shared prefix (e.g., team_a_config_ or
>>>>> pipeline_x_param_). In many cases, these keys are generated or updated
>>>>> dynamically by external systems, meaning the exact set of keys isn't
>>>>> known at DAG authoring time.
>>>>>
>>>>> While users in Airflow 2.x relied on session.query(Variable).all() as
>>>>> a workaround, Airflow 3’s move toward the Task SDK aims to abstract
>>>>> away direct ORM/DB access for better security and stability.
>>>>> Variable.list(prefix=) provides a supported, clean way to achieve this
>>>>> discovery within that new architecture.
>>>>>
>>>>> Regarding the secrets backend limitation, I completely agree. It’s
>>>>> important to manage expectations, so I will update the PyDoc to
>>>>> explicitly state that this method only lists variables stored in the
>>>>> metadata database.
>>>>>
>>>>> Best regards, Jun Yeong
>>>>>
>>>>> 2026년 5월 1일 (금) 오전 5:23, Jens Scheffler <js...@apache.org>님이 작성:
>>>>>> Hi!
>>>>>>
>>>>>> thanks for the discussion. While I am not against this I would say in
>>>>>> Airflow 2 it was also not a "public API" but the DB connecton "just
>>>>>> used" to list and have a missing API compensated.
>>>>>>
>>>>>> Can you express what the demand for the missing feature is? What
>>>>>> business function did you implement based on listing all Variables?
>>>>>>
>>>>>> As you already stated and also highlighted in the PR the list() might
>>>>>> not tell about all Variables as the list is not provided from secret
>>>>>> managers. So it might (small risk thoug) lead to some confusion.
>>> Should
>>>>>> be explicitly documented in the PyDoc. But this is a nit.
>>>>>>
>>>>>> Jens
>>>>>>
>>>>>> On 30.04.26 11:47, 김준영 wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I'd like to propose adding Variable.list() to the Task SDK to address
>>>>>>> the gap left by the removal of direct ORM access in Airflow 3.
>>>>>>>
>>>>>>> Background:
>>>>>>> In Airflow 2.x, users could list all variables via:
>>>>>>>
>>>>>>> from airflow.models import Variable
>>>>>>> from airflow.utils.session import create_session
>>>>>>>
>>>>>>> with create_session() as session:
>>>>>>> variables = session.query(Variable).all()
>>>>>>>
>>>>>>> In Airflow 3.x, this pattern raises:
>>>>>>> RuntimeError: Direct database access via the ORM is not
>>> allowed in Airflow
>>>>>>> 3.0
>>>>>>>
>>>>>>> There is currently no supported way to discover variable keys
>>> dynamically
>>>>>>> when they are not known at DAG authoring time.
>>>>>>>
>>>>>>> Proposal:
>>>>>>> - Add Variable.list(prefix=None) to the Task SDK
>>>>>>> - Scope is limited to the metadata database only (same as the
>>> old ORM pattern)
>>>>>>> - Secrets backend support is intentionally out of scope, as it
>>> would
>>>>>>> require a broader interface contract change and separate
>>> community
>>>>>>> discussion
>>>>>>>
>>>>>>> Related issue: https://github.com/apache/airflow/issues/61166
>>>>>>> Draft PR: https://github.com/apache/airflow/pull/66022
>>>>>>>
>>>>>>> I would appreciate any feedback or concerns from the community
>>> before
>>>>>>> this moves forward.
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Jun Yeong Kim
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Add Variable.list() to Task SDK as a replacement for Variable.all()
Posted by 김준영 <ju...@gmail.com>.
Hi Amogh,
Thanks for the feedback! I've updated the PR (#66022) to implement the
lazy iterator pattern you suggested:
for key in Variable.keys(prefix="team_a_config_"):
val = Variable.get(key)
Now, Variable.keys(prefix=None) returns a list[str] of matching keys,
supported by a new GET /variables/keys?prefix= endpoint in the
Execution API. This allows callers to fetch only the specific values
they need via Variable.get(key).
Regarding your point about other Airflow resources (Connections,
XComs, etc.) having the same gap: once this PR is merged and the
pattern is established, I'd be happy to extend this keys() approach to
those resources in follow-up PRs. Would that be a direction you'd
encourage?
The PR is ready for another review whenever you have a moment.
Best regards,
Jun Yeong
2026년 5월 5일 (화) 오후 5:51, Amogh Desai <am...@apache.org>님이 작성:
>
> Hi Jun Yeong,
>
> Valid gap. There is no GET /variables list endpoint at all, but that is
> true for any
> other Airflow artefacts too. So variables isn't the only one missing it,
> its all of the
> Airflow resources: connections, xcoms, etc.
>
> One suggestion worth considering: rather than Variable.list() returning a
> full list,
>
> a lazy iterator might be a better fit. Something like:
>
>
> for key in Variable.keys(prefix="team_a_config_"):
> val = Variable.get(key)
>
> Happy to review when a PR is ready.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Fri, May 1, 2026 at 3:02 AM 김준영 <ju...@gmail.com> wrote:
>
> > Hi Jens,
> >
> > That is a valid alternative, and using a single JSON variable is
> > indeed a great pattern for centralized or static configurations to
> > reduce I/O.
> >
> > However, for the use cases I'm targeting, the "Single JSON Variable"
> > approach has a few significant drawbacks:
> >
> > 1.Concurrency & Atomic Updates: When multiple external systems or
> > independent CI/CD pipelines need to update their own configurations, a
> > single JSON variable creates a race condition. They would have to
> > "Read-Modify-Write" the entire blob, which risks overwriting each
> > other's changes. Separate variables allow for atomic, independent
> > updates.
> >
> > 2.Integration Complexity: Many users integrate Airflow with external
> > tools that independently push values to the Airflow Metadata DB or
> > Secret Backend. Forcing these decoupled systems to coordinate and
> > maintain a single shared JSON structure adds significant integration
> > overhead.
> >
> > 3.Data Modeling Flexibility: While a JSON blob works for some,
> > Variable.list(prefix=) allows Airflow to be unopinionated about how
> > users model their data. It provides a standard "Key-Value store"
> > experience (similar to AWS SSM or Redis) where prefix-based discovery
> > is a first-class citizen.
> >
> > In short, while the JSON approach is a good workaround for specific
> > cases, Variable.list() provides the necessary flexibility for highly
> > dynamic and decoupled environments without forcing a specific data
> > structure on the user.
> >
> > What do you think?
> >
> > Best regards, Jun Yeong
> >
> > 2026년 5월 1일 (금) 오전 6:19, Jens Scheffler <js...@apache.org>님이 작성:
> > >
> > > Hi Jun Yeong,
> > >
> > > one thought on this: We had similar.
> > >
> > > Our use case: Implemented a custom archiving (Dag) that needed to take
> > > care of different retention times in different Dags (maybe bad example
> > > because this archival itself accesses the database for archiving...
> > > haha) and we wanted to have different retention times per Dag.
> > >
> > > What we did is we made a JSON structure with all parameters into a
> > > single Variable. Then we did not need to have many IO operations and
> > > Variables but could store all in a single Variable.
> > >
> > > Would this be a viable solution for your case as well?
> > >
> > > Jens
> > >
> > > On 30.04.26 22:47, 김준영 wrote:
> > > > Hi Jens,
> > > >
> > > > Thank you for the thoughtful feedback!
> > > >
> > > > The primary demand for this feature comes from workflows that require
> > > > dynamic configuration discovery. A common pattern is grouping related
> > > > variables under a shared prefix (e.g., team_a_config_ or
> > > > pipeline_x_param_). In many cases, these keys are generated or updated
> > > > dynamically by external systems, meaning the exact set of keys isn't
> > > > known at DAG authoring time.
> > > >
> > > > While users in Airflow 2.x relied on session.query(Variable).all() as
> > > > a workaround, Airflow 3’s move toward the Task SDK aims to abstract
> > > > away direct ORM/DB access for better security and stability.
> > > > Variable.list(prefix=) provides a supported, clean way to achieve this
> > > > discovery within that new architecture.
> > > >
> > > > Regarding the secrets backend limitation, I completely agree. It’s
> > > > important to manage expectations, so I will update the PyDoc to
> > > > explicitly state that this method only lists variables stored in the
> > > > metadata database.
> > > >
> > > > Best regards, Jun Yeong
> > > >
> > > > 2026년 5월 1일 (금) 오전 5:23, Jens Scheffler <js...@apache.org>님이 작성:
> > > >> Hi!
> > > >>
> > > >> thanks for the discussion. While I am not against this I would say in
> > > >> Airflow 2 it was also not a "public API" but the DB connecton "just
> > > >> used" to list and have a missing API compensated.
> > > >>
> > > >> Can you express what the demand for the missing feature is? What
> > > >> business function did you implement based on listing all Variables?
> > > >>
> > > >> As you already stated and also highlighted in the PR the list() might
> > > >> not tell about all Variables as the list is not provided from secret
> > > >> managers. So it might (small risk thoug) lead to some confusion.
> > Should
> > > >> be explicitly documented in the PyDoc. But this is a nit.
> > > >>
> > > >> Jens
> > > >>
> > > >> On 30.04.26 11:47, 김준영 wrote:
> > > >>> Hi all,
> > > >>>
> > > >>> I'd like to propose adding Variable.list() to the Task SDK to address
> > > >>> the gap left by the removal of direct ORM access in Airflow 3.
> > > >>>
> > > >>> Background:
> > > >>> In Airflow 2.x, users could list all variables via:
> > > >>>
> > > >>> from airflow.models import Variable
> > > >>> from airflow.utils.session import create_session
> > > >>>
> > > >>> with create_session() as session:
> > > >>> variables = session.query(Variable).all()
> > > >>>
> > > >>> In Airflow 3.x, this pattern raises:
> > > >>> RuntimeError: Direct database access via the ORM is not
> > allowed in Airflow
> > > >>> 3.0
> > > >>>
> > > >>> There is currently no supported way to discover variable keys
> > dynamically
> > > >>> when they are not known at DAG authoring time.
> > > >>>
> > > >>> Proposal:
> > > >>> - Add Variable.list(prefix=None) to the Task SDK
> > > >>> - Scope is limited to the metadata database only (same as the
> > old ORM pattern)
> > > >>> - Secrets backend support is intentionally out of scope, as it
> > would
> > > >>> require a broader interface contract change and separate
> > community
> > > >>> discussion
> > > >>>
> > > >>> Related issue: https://github.com/apache/airflow/issues/61166
> > > >>> Draft PR: https://github.com/apache/airflow/pull/66022
> > > >>>
> > > >>> I would appreciate any feedback or concerns from the community
> > before
> > > >>> this moves forward.
> > > >>>
> > > >>> Best Regards,
> > > >>> Jun Yeong Kim
> > > >>>
> > > >>> ---------------------------------------------------------------------
> > > >>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >>> For additional commands, e-mail: dev-help@airflow.apache.org
> > > >>>
> > > >> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > > >>
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org