You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by "Ghaeli, Sean" <gh...@amazon.com.INVALID> on 2025/08/12 22:38:27 UTC
Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
Valkey checks off many boxes that align with Airflow’s provider criteria:
* Truly open-source
* Dedicated maintainers responsive to community questions
* Not a commercial service that could maintain their own provider
* Very lean dependencies: https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt
* Popular and actively maintained with broad community support: https://github.com/valkey-io/valkey
Given all of these, I believe the Valkey provider makes sense as an official Airflow provider.
Sean
On 2025-07-15, 1:22 PM, "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock <dheilbock@gmx.net.inva <ma...@gmx.net.inva>lid>
wrote:
> As of last year, the redis Python client was still fully compatible with
> Valkey, has this changed? This would make it even easier. I think they also
> forked redis-py
Yep. I think that's part of the solution to make the dependency tree for
the provider independent of redis-controlled dependencies.
Governance, decision making and release process of valley should be
separate from redis (the company). That's for me one of the criteria and
requirements for having a separate valley provider. Very much like
"opensearch" is separate from "elasticsearch".
>
> Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr
> > Von: "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com>>
> > An: dev@airflow.apache.org <ma...@airflow.apache.org>
> > Cc: ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid
> > Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
> >
> > > Can you please explain why it should be a provider managed by Airflow
> > rather than on your own?
> >
> > Just to explain it Elad - and yes, I know I am usually the one that is
> > sceptical -> valley is truly community driven replacement for redis
> > https://valkey.io/participants/ <https://valkey.io/participants/> - you see a lot of well-known community
> > recognised entities, and there is no single entity behind it that could
> > take "responsibility" - while there are enough stakeholders to make sure
> > that it will be maintained here in Airflow community. Also it's a
> "drop-in"
> > replacement for redis generally - and I do not expect a lot of problems
> > when we start with both redis provider and redis celery integration.
> >
> > I've heard and saw a lot of praise from the community for Valkey being
> good
> > redis replacement, and - even we at the PMC and devlist discussions
> > discussed about the dangers Redis licence creates for that - even now we
> > limited Redis support in Helm chart of ours to the "really free" version
> > and we discussed that Valkey might be a good way forward - especially if
> > someone wants to contribute support for it. so if someone wants to
> > contribute to Airflow AND make sure that Redis broker support in Celery
> is
> > covered - I am all for it.
> >
> >
> > On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <eladkal@gmail.com <ma...@gmail.com>> wrote:
> >
> > > > * Helm Chart support to run valley as Celery Broker instead of Redis
> > >
> > > Oh yes! didn't consider its usage as a broker but is it related to the
> > > provider package?
> > > Amazon SQS can be used as broker but it's not related to the SQS
> > > integrations in the Airflow amazon provider (hook,operator,sensor)
> > > I also don't see ValKey listed as option for brokers in
> > >
> > >
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview <https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview>
> > >
> > > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>>
> wrote:
> > >
> > > > Yes. Valkey is a good replacement/parallel implementation to Redis.
> One
> > > > thing however is important - I would see this even more useful if it
> also
> > > > allows for the Celery Redis replacement for broker - means that it is
> > > fully
> > > > supported in our:
> > > >
> > > > * unit tests (of course)
> > > > * integration tests with ("valkey") integration - including running
> > > > docker-compose integration to run in CI
> > > > * Helm Chart support to run valley as Celery Broker instead of Redis
> > > >
> > > > I think I'd only consider valley as "successful" provider
> integration if
> > > > all that above is completed.
> > > >
> > > > J.
> > > >
> > > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean
> <ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > >
> > > > >
> > > > > I’d like to propose the addition of a new provider to support
> Valkey<
> > > > > https://valkey.io/> <https://valkey.io/>>: a fully open-source, community-governed fork
> of
> > > > > Redis 7.2, now maintained under the Linux Foundation.
> > > > >
> > > > >
> > > > >
> > > > > Valkey has been gaining traction recently, in part due to changes
> by
> > > > Redis
> > > > > moving their license to one unrecognized by the Open Source
> Initiative.
> > > > > Valkey represents a future-proof alternative. The proposed provider
> > > would
> > > > > closely mirror the existing Redis provider in functionality and
> serve
> > > as
> > > > a
> > > > > drop-in, OSS-aligned option for workflows that depend on key-value
> > > > storage,
> > > > > pub/sub, or caching patterns.
> > > > >
> > > > >
> > > > >
> > > > > Tentatively, here are the components I plan on implementing
> (mirroring
> > > > the
> > > > > Redis provider):
> > > > >
> > > > >
> > > > >
> > > > > * ValkeyHook – Manage connections and interactions with the
> Valkey
> > > > > server, enabling reuse across operators and sensors.
> > > > > * ValkeyPublishOperator – Allows tasks to publish messages to
> > > Valkey
> > > > > pub/sub channels, useful for event-driven workflows or inter-task
> > > > signaling.
> > > > > * ValkeyKeySensor – Waits for a specific key to exist or reach
> a
> > > > > value, supporting synchronization patterns where one task's output
> > > > controls
> > > > > downstream logic.
> > > > > * ValkeyPubSubSensor – Subscribes to a Valkey channel and
> triggers
> > > on
> > > > > incoming messages, enabling real-time DAG triggering.
> > > > >
> > > > >
> > > > >
> > > > > Here's an example of how we could use the components to publish a
> > > message
> > > > > and then trigger a DAG:
> > > > >
> > > > >
> > > > >
> > > > > `
> > > > >
> > > > > publish_message = ValkeyPublishOperator(
> > > > >
> > > > > task_id="publish_ready_signal",
> > > > >
> > > > > channel="processing_status",
> > > > >
> > > > > message="READY",
> > > > >
> > > > > valkey_conn_id="valkey_default",
> > > > >
> > > > > )
> > > > >
> > > > > `
> > > > >
> > > > >
> > > > >
> > > > > This task publishes the message "READY" to the processing_status
> > > channel
> > > > > on a Valkey server. Other processes listening to this channel will
> > > > receive
> > > > > the message and proceed with their execution.
> > > > >
> > > > >
> > > > >
> > > > > `
> > > > >
> > > > > wait_for_ready = ValkeyPubSubSensor(
> > > > >
> > > > > task_id="wait_for_ready_message",
> > > > >
> > > > > channel="processing_status",
> > > > >
> > > > > valkey_conn_id="valkey_default",
> > > > >
> > > > > timeout=600,
> > > > >
> > > > > )
> > > > >
> > > > > `
> > > > >
> > > > >
> > > > >
> > > > > Here, a sensor waits for the message on the same channel before
> > > > > continuing. Once the message is received, the DAG can continue with
> > > > further
> > > > > tasks—such as triggering another DAG—in an event-driven manner.
> > > > >
> > > > >
> > > > >
> > > > > Given the community's interest in event-driven scheduling (AIP 82<
> > > > > https://github.com/apache/airflow/issues/52712> <https://github.com/apache/airflow/issues/52712>>) as well as
> concerns
> > > > > about limited provider optionality (PR #52712<
> > > > > https://github.com/apache/airflow/issues/52712> <https://github.com/apache/airflow/issues/52712>>), a Valkey
> provider
> > > > would
> > > > > expand user choice.
> > > > >
> > > > >
> > > > >
> > > > > I’d be happy to begin working on the initial PR and documentation.
> > > > >
> > > > >
> > > > >
> > > > > Looking forward to feedback.
> > > > >
> > > > > Sean
> > > > >
> > > > >
> > > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>
> For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org>
>
>
Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
Posted by Jarek Potiuk <ja...@potiuk.com>.
Just to clarify - for the "Open Governance" argument part - it is important
to me, mostly because there is no single entity that can exclusively
benefit from us maintaining the provider. So I simply do not buy the
argument that "some 3rd party provider can release it". I just think this
argument could ONLY be used when such a 3rd-party provider actually exists.
And I suggest that in future discussions we simply take into account the
"governance" of such proposed providers, and do not use the argument when
it is not really valid. That was my main "argument" and "disagreement" here.
On the other argument - popularity, and whether we actually "need" a
separate provider for it - indeed it would be great to have some
information from people who use "redis" already (and how). And have some
numbers - though - we know currently, having reliable numbers is not easy -
or even not possible and currently even asking for such numbers is really
asking someone to complete an impossible task. Yes. We can "ask" for
numbers, but if we know that getting those numbers cannot be provided, this
is really equivalent to "NO".
And I am also not strongly for "yes we should accept it now" - I think we
should consider (and this is more of an intuition, hearing
though grapevines and leap of faith than hard data) that we - as a
community - can make by believing that it **might** be useful.
To be honest - of course redis/valkey support is not useful for ETL - for
sure. But at least my intuition tells me that in-memory key value storage
might be very useful to keep context for all kinds of agentic workflows in
the near future. We currently do not provide any "real" solution for that
(we discuss state management for watermarks [1] but we don't provide our
workflows a way to do so. Having a provider, where people can easily use
Valkey Hooks for such state management in their task-flow tasks seems like
awfully good idea I think. And from "grapevine" discussions i know many
companies turn their back to redis and replace it with Valkey as we speak
due to the licensing changes of Redis. Even we ourselves decided that we
stick to the old version of redis in our helm chart because of that [2].
So I think this discussion is more about the "future" than the past and
what we "believe" might be useful. And yes - we can decide to wait, but
also we have to remember that we also have great powers of influencing data
engineering space. If **we** decide that Valkey is first-class-citizen in
Airflow - people might follow. And I think we should carefully and
deliberately use that power as a community.
So I would love to hear what others think about it as well ?
J
[1] Watermarks state AIP draft
https://cwiki.apache.org/confluence/display/AIRFLOW/%5BDRAFT%5D+AIP-93+Watermarks
[2] Discussion on our redis approach
https://lists.apache.org/thread/4mjs2mqcgxswk96wwvw1rjc969lq5rl7
On Thu, Aug 14, 2025 at 12:02 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> I’m with Elad on this, -1. The open nature/source/core or not of the
> project isn’t the key aspect for this, but simply the usefulness of it at
> all
>
>
> 1. Do we need a separate provider for it? It already bugs me that we have
> separate ElasticSearch and OpenSearch providers and this feels like the
> same thing all over — we’re having to maintain two copies of a provider
> that is otherwise identical due to upstream OSS vs commercial spats.
>
> 2. Is it even worth having a redis provider? I have a feeling the
> popularity of the redis provider is a side-effect of using reds for Celery
> broker `apache-airflow[celery,redis]` being installed. We have no way of
> knowing but I would make the case that 99% of cases don’t use the hooks,
> operators or sensors.
>
> -ash
>
>
> > On 13 Aug 2025, at 11:06, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >> Since Valkey 8.1.x (latest serie) is still 100% compatible with Redis,
> >> wouldn't be easier to just state in docs Redis provider is also
> >> compatible with Valkey backend?
> >
> > Yep, that's one of the options. We discussed it above - that we could
> have
> > valley/redis switch for example in the current redis provider - it will
> > make it more coupled and future features might diverge, but yes, this is
> > indeed a valid option, I think it's more of the question of name in this
> > case. If for example we rename redis provider to "kv-storage" or
> > "common-kv-storage" for example or something similar, that might be
> > actually quite a viable option for us.
> >
> > J.
> >
> >
> > On Wed, Aug 13, 2025 at 11:44 AM Josef Šimánek <jo...@gmail.com>
> > wrote:
> >
> >> st 13. 8. 2025 v 0:38 odesílatel Ghaeli, Sean
> >> <gh...@amazon.com.invalid> napsal:
> >>>
> >>> Valkey checks off many boxes that align with Airflow’s provider
> criteria:
> >>>
> >>> * Truly open-source
> >>> * Dedicated maintainers responsive to community questions
> >>> * Not a commercial service that could maintain their own provider
> >>> * Very lean dependencies:
> >> https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt
> >>> * Popular and actively maintained with broad community support:
> >> https://github.com/valkey-io/valkey
> >>>
> >>> Given all of these, I believe the Valkey provider makes sense as an
> >> official Airflow provider.
> >>
> >> Since Valkey 8.1.x (latest serie) is still 100% compatible with Redis,
> >> wouldn't be easier to just state in docs Redis provider is also
> >> compatible with Valkey backend?
> >>
> >>> Sean
> >>>
> >>> On 2025-07-15, 1:22 PM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> >> jarek@potiuk.com>> wrote:
> >>>
> >>>
> >>> CAUTION: This email originated from outside of the organization. Do not
> >> click links or open attachments unless you can confirm the sender and
> know
> >> the content is safe.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> >> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si
> vous
> >> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> >> certain que le contenu ne présente aucun risque.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock
> <dheilbock@gmx.net.inva
> >> <ma...@gmx.net.inva>lid>
> >>> wrote:
> >>>
> >>>
> >>>> As of last year, the redis Python client was still fully compatible
> >> with
> >>>> Valkey, has this changed? This would make it even easier. I think they
> >> also
> >>>> forked redis-py
> >>>
> >>>
> >>>
> >>>
> >>> Yep. I think that's part of the solution to make the dependency tree
> for
> >>> the provider independent of redis-controlled dependencies.
> >>>
> >>>
> >>> Governance, decision making and release process of valley should be
> >>> separate from redis (the company). That's for me one of the criteria
> and
> >>> requirements for having a separate valley provider. Very much like
> >>> "opensearch" is separate from "elasticsearch".
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>
> >>>
> >>>
> >>>> Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr
> >>>>> Von: "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com>>
> >>>>> An: dev@airflow.apache.org <ma...@airflow.apache.org>
> >>>>> Cc: ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid
> >>>>> Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
> >>>>>
> >>>>>> Can you please explain why it should be a provider managed by
> >> Airflow
> >>>>> rather than on your own?
> >>>>>
> >>>>> Just to explain it Elad - and yes, I know I am usually the one that
> >> is
> >>>>> sceptical -> valley is truly community driven replacement for redis
> >>>>> https://valkey.io/participants/ <https://valkey.io/participants/> -
> >> you see a lot of well-known community
> >>>>> recognised entities, and there is no single entity behind it that
> >> could
> >>>>> take "responsibility" - while there are enough stakeholders to make
> >> sure
> >>>>> that it will be maintained here in Airflow community. Also it's a
> >>>> "drop-in"
> >>>>> replacement for redis generally - and I do not expect a lot of
> >> problems
> >>>>> when we start with both redis provider and redis celery integration.
> >>>>>
> >>>>> I've heard and saw a lot of praise from the community for Valkey
> >> being
> >>>> good
> >>>>> redis replacement, and - even we at the PMC and devlist discussions
> >>>>> discussed about the dangers Redis licence creates for that - even
> >> now we
> >>>>> limited Redis support in Helm chart of ours to the "really free"
> >> version
> >>>>> and we discussed that Valkey might be a good way forward -
> >> especially if
> >>>>> someone wants to contribute support for it. so if someone wants to
> >>>>> contribute to Airflow AND make sure that Redis broker support in
> >> Celery
> >>>> is
> >>>>> covered - I am all for it.
> >>>>>
> >>>>>
> >>>>> On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <eladkal@gmail.com
> >> <ma...@gmail.com>> wrote:
> >>>>>
> >>>>>>> * Helm Chart support to run valley as Celery Broker instead of
> >> Redis
> >>>>>>
> >>>>>> Oh yes! didn't consider its usage as a broker but is it related to
> >> the
> >>>>>> provider package?
> >>>>>> Amazon SQS can be used as broker but it's not related to the SQS
> >>>>>> integrations in the Airflow amazon provider (hook,operator,sensor)
> >>>>>> I also don't see ValKey listed as option for brokers in
> >>>>>>
> >>>>>>
> >>>>
> >>
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> >> <
> >>
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> >>>
> >>>>>>
> >>>>>> On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <jarek@potiuk.com
> >> <ma...@potiuk.com>>
> >>>> wrote:
> >>>>>>
> >>>>>>> Yes. Valkey is a good replacement/parallel implementation to
> >> Redis.
> >>>> One
> >>>>>>> thing however is important - I would see this even more useful
> >> if it
> >>>> also
> >>>>>>> allows for the Celery Redis replacement for broker - means that
> >> it is
> >>>>>> fully
> >>>>>>> supported in our:
> >>>>>>>
> >>>>>>> * unit tests (of course)
> >>>>>>> * integration tests with ("valkey") integration - including
> >> running
> >>>>>>> docker-compose integration to run in CI
> >>>>>>> * Helm Chart support to run valley as Celery Broker instead of
> >> Redis
> >>>>>>>
> >>>>>>> I think I'd only consider valley as "successful" provider
> >>>> integration if
> >>>>>>> all that above is completed.
> >>>>>>>
> >>>>>>> J.
> >>>>>>>
> >>>>>>> On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean
> >>>> <ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I’d like to propose the addition of a new provider to support
> >>>> Valkey<
> >>>>>>>> https://valkey.io/> <https://valkey.io/>>: a fully
> >> open-source, community-governed fork
> >>>> of
> >>>>>>>> Redis 7.2, now maintained under the Linux Foundation.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Valkey has been gaining traction recently, in part due to
> >> changes
> >>>> by
> >>>>>>> Redis
> >>>>>>>> moving their license to one unrecognized by the Open Source
> >>>> Initiative.
> >>>>>>>> Valkey represents a future-proof alternative. The proposed
> >> provider
> >>>>>> would
> >>>>>>>> closely mirror the existing Redis provider in functionality and
> >>>> serve
> >>>>>> as
> >>>>>>> a
> >>>>>>>> drop-in, OSS-aligned option for workflows that depend on
> >> key-value
> >>>>>>> storage,
> >>>>>>>> pub/sub, or caching patterns.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Tentatively, here are the components I plan on implementing
> >>>> (mirroring
> >>>>>>> the
> >>>>>>>> Redis provider):
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> * ValkeyHook – Manage connections and interactions with the
> >>>> Valkey
> >>>>>>>> server, enabling reuse across operators and sensors.
> >>>>>>>> * ValkeyPublishOperator – Allows tasks to publish messages to
> >>>>>> Valkey
> >>>>>>>> pub/sub channels, useful for event-driven workflows or
> >> inter-task
> >>>>>>> signaling.
> >>>>>>>> * ValkeyKeySensor – Waits for a specific key to exist or reach
> >>>> a
> >>>>>>>> value, supporting synchronization patterns where one task's
> >> output
> >>>>>>> controls
> >>>>>>>> downstream logic.
> >>>>>>>> * ValkeyPubSubSensor – Subscribes to a Valkey channel and
> >>>> triggers
> >>>>>> on
> >>>>>>>> incoming messages, enabling real-time DAG triggering.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Here's an example of how we could use the components to
> >> publish a
> >>>>>> message
> >>>>>>>> and then trigger a DAG:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> `
> >>>>>>>>
> >>>>>>>> publish_message = ValkeyPublishOperator(
> >>>>>>>>
> >>>>>>>> task_id="publish_ready_signal",
> >>>>>>>>
> >>>>>>>> channel="processing_status",
> >>>>>>>>
> >>>>>>>> message="READY",
> >>>>>>>>
> >>>>>>>> valkey_conn_id="valkey_default",
> >>>>>>>>
> >>>>>>>> )
> >>>>>>>>
> >>>>>>>> `
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> This task publishes the message "READY" to the
> >> processing_status
> >>>>>> channel
> >>>>>>>> on a Valkey server. Other processes listening to this channel
> >> will
> >>>>>>> receive
> >>>>>>>> the message and proceed with their execution.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> `
> >>>>>>>>
> >>>>>>>> wait_for_ready = ValkeyPubSubSensor(
> >>>>>>>>
> >>>>>>>> task_id="wait_for_ready_message",
> >>>>>>>>
> >>>>>>>> channel="processing_status",
> >>>>>>>>
> >>>>>>>> valkey_conn_id="valkey_default",
> >>>>>>>>
> >>>>>>>> timeout=600,
> >>>>>>>>
> >>>>>>>> )
> >>>>>>>>
> >>>>>>>> `
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Here, a sensor waits for the message on the same channel before
> >>>>>>>> continuing. Once the message is received, the DAG can continue
> >> with
> >>>>>>> further
> >>>>>>>> tasks—such as triggering another DAG—in an event-driven manner.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Given the community's interest in event-driven scheduling (AIP
> >> 82<
> >>>>>>>> https://github.com/apache/airflow/issues/52712> <
> >> https://github.com/apache/airflow/issues/52712>>) as well as
> >>>> concerns
> >>>>>>>> about limited provider optionality (PR #52712<
> >>>>>>>> https://github.com/apache/airflow/issues/52712> <
> >> https://github.com/apache/airflow/issues/52712>>), a Valkey
> >>>> provider
> >>>>>>> would
> >>>>>>>> expand user choice.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I’d be happy to begin working on the initial PR and
> >> documentation.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Looking forward to feedback.
> >>>>>>>>
> >>>>>>>> Sean
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> >> dev-unsubscribe@airflow.apache.org>
> >>>> For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> >> dev-help@airflow.apache.org>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> For additional commands, e-mail: dev-help@airflow.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
Posted by Ash Berlin-Taylor <as...@apache.org>.
I’m with Elad on this, -1. The open nature/source/core or not of the project isn’t the key aspect for this, but simply the usefulness of it at all
1. Do we need a separate provider for it? It already bugs me that we have separate ElasticSearch and OpenSearch providers and this feels like the same thing all over — we’re having to maintain two copies of a provider that is otherwise identical due to upstream OSS vs commercial spats.
2. Is it even worth having a redis provider? I have a feeling the popularity of the redis provider is a side-effect of using reds for Celery broker `apache-airflow[celery,redis]` being installed. We have no way of knowing but I would make the case that 99% of cases don’t use the hooks, operators or sensors.
-ash
> On 13 Aug 2025, at 11:06, Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Since Valkey 8.1.x (latest serie) is still 100% compatible with Redis,
>> wouldn't be easier to just state in docs Redis provider is also
>> compatible with Valkey backend?
>
> Yep, that's one of the options. We discussed it above - that we could have
> valley/redis switch for example in the current redis provider - it will
> make it more coupled and future features might diverge, but yes, this is
> indeed a valid option, I think it's more of the question of name in this
> case. If for example we rename redis provider to "kv-storage" or
> "common-kv-storage" for example or something similar, that might be
> actually quite a viable option for us.
>
> J.
>
>
> On Wed, Aug 13, 2025 at 11:44 AM Josef Šimánek <jo...@gmail.com>
> wrote:
>
>> st 13. 8. 2025 v 0:38 odesílatel Ghaeli, Sean
>> <gh...@amazon.com.invalid> napsal:
>>>
>>> Valkey checks off many boxes that align with Airflow’s provider criteria:
>>>
>>> * Truly open-source
>>> * Dedicated maintainers responsive to community questions
>>> * Not a commercial service that could maintain their own provider
>>> * Very lean dependencies:
>> https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt
>>> * Popular and actively maintained with broad community support:
>> https://github.com/valkey-io/valkey
>>>
>>> Given all of these, I believe the Valkey provider makes sense as an
>> official Airflow provider.
>>
>> Since Valkey 8.1.x (latest serie) is still 100% compatible with Redis,
>> wouldn't be easier to just state in docs Redis provider is also
>> compatible with Valkey backend?
>>
>>> Sean
>>>
>>> On 2025-07-15, 1:22 PM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
>> jarek@potiuk.com>> wrote:
>>>
>>>
>>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>>
>>>
>>>
>>>
>>>
>>>
>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
>> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
>> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
>> certain que le contenu ne présente aucun risque.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock <dheilbock@gmx.net.inva
>> <ma...@gmx.net.inva>lid>
>>> wrote:
>>>
>>>
>>>> As of last year, the redis Python client was still fully compatible
>> with
>>>> Valkey, has this changed? This would make it even easier. I think they
>> also
>>>> forked redis-py
>>>
>>>
>>>
>>>
>>> Yep. I think that's part of the solution to make the dependency tree for
>>> the provider independent of redis-controlled dependencies.
>>>
>>>
>>> Governance, decision making and release process of valley should be
>>> separate from redis (the company). That's for me one of the criteria and
>>> requirements for having a separate valley provider. Very much like
>>> "opensearch" is separate from "elasticsearch".
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>
>>>
>>>> Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr
>>>>> Von: "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com>>
>>>>> An: dev@airflow.apache.org <ma...@airflow.apache.org>
>>>>> Cc: ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid
>>>>> Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
>>>>>
>>>>>> Can you please explain why it should be a provider managed by
>> Airflow
>>>>> rather than on your own?
>>>>>
>>>>> Just to explain it Elad - and yes, I know I am usually the one that
>> is
>>>>> sceptical -> valley is truly community driven replacement for redis
>>>>> https://valkey.io/participants/ <https://valkey.io/participants/> -
>> you see a lot of well-known community
>>>>> recognised entities, and there is no single entity behind it that
>> could
>>>>> take "responsibility" - while there are enough stakeholders to make
>> sure
>>>>> that it will be maintained here in Airflow community. Also it's a
>>>> "drop-in"
>>>>> replacement for redis generally - and I do not expect a lot of
>> problems
>>>>> when we start with both redis provider and redis celery integration.
>>>>>
>>>>> I've heard and saw a lot of praise from the community for Valkey
>> being
>>>> good
>>>>> redis replacement, and - even we at the PMC and devlist discussions
>>>>> discussed about the dangers Redis licence creates for that - even
>> now we
>>>>> limited Redis support in Helm chart of ours to the "really free"
>> version
>>>>> and we discussed that Valkey might be a good way forward -
>> especially if
>>>>> someone wants to contribute support for it. so if someone wants to
>>>>> contribute to Airflow AND make sure that Redis broker support in
>> Celery
>>>> is
>>>>> covered - I am all for it.
>>>>>
>>>>>
>>>>> On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <eladkal@gmail.com
>> <ma...@gmail.com>> wrote:
>>>>>
>>>>>>> * Helm Chart support to run valley as Celery Broker instead of
>> Redis
>>>>>>
>>>>>> Oh yes! didn't consider its usage as a broker but is it related to
>> the
>>>>>> provider package?
>>>>>> Amazon SQS can be used as broker but it's not related to the SQS
>>>>>> integrations in the Airflow amazon provider (hook,operator,sensor)
>>>>>> I also don't see ValKey listed as option for brokers in
>>>>>>
>>>>>>
>>>>
>> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
>> <
>> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
>>>
>>>>>>
>>>>>> On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <jarek@potiuk.com
>> <ma...@potiuk.com>>
>>>> wrote:
>>>>>>
>>>>>>> Yes. Valkey is a good replacement/parallel implementation to
>> Redis.
>>>> One
>>>>>>> thing however is important - I would see this even more useful
>> if it
>>>> also
>>>>>>> allows for the Celery Redis replacement for broker - means that
>> it is
>>>>>> fully
>>>>>>> supported in our:
>>>>>>>
>>>>>>> * unit tests (of course)
>>>>>>> * integration tests with ("valkey") integration - including
>> running
>>>>>>> docker-compose integration to run in CI
>>>>>>> * Helm Chart support to run valley as Celery Broker instead of
>> Redis
>>>>>>>
>>>>>>> I think I'd only consider valley as "successful" provider
>>>> integration if
>>>>>>> all that above is completed.
>>>>>>>
>>>>>>> J.
>>>>>>>
>>>>>>> On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean
>>>> <ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I’d like to propose the addition of a new provider to support
>>>> Valkey<
>>>>>>>> https://valkey.io/> <https://valkey.io/>>: a fully
>> open-source, community-governed fork
>>>> of
>>>>>>>> Redis 7.2, now maintained under the Linux Foundation.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Valkey has been gaining traction recently, in part due to
>> changes
>>>> by
>>>>>>> Redis
>>>>>>>> moving their license to one unrecognized by the Open Source
>>>> Initiative.
>>>>>>>> Valkey represents a future-proof alternative. The proposed
>> provider
>>>>>> would
>>>>>>>> closely mirror the existing Redis provider in functionality and
>>>> serve
>>>>>> as
>>>>>>> a
>>>>>>>> drop-in, OSS-aligned option for workflows that depend on
>> key-value
>>>>>>> storage,
>>>>>>>> pub/sub, or caching patterns.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Tentatively, here are the components I plan on implementing
>>>> (mirroring
>>>>>>> the
>>>>>>>> Redis provider):
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> * ValkeyHook – Manage connections and interactions with the
>>>> Valkey
>>>>>>>> server, enabling reuse across operators and sensors.
>>>>>>>> * ValkeyPublishOperator – Allows tasks to publish messages to
>>>>>> Valkey
>>>>>>>> pub/sub channels, useful for event-driven workflows or
>> inter-task
>>>>>>> signaling.
>>>>>>>> * ValkeyKeySensor – Waits for a specific key to exist or reach
>>>> a
>>>>>>>> value, supporting synchronization patterns where one task's
>> output
>>>>>>> controls
>>>>>>>> downstream logic.
>>>>>>>> * ValkeyPubSubSensor – Subscribes to a Valkey channel and
>>>> triggers
>>>>>> on
>>>>>>>> incoming messages, enabling real-time DAG triggering.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Here's an example of how we could use the components to
>> publish a
>>>>>> message
>>>>>>>> and then trigger a DAG:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> `
>>>>>>>>
>>>>>>>> publish_message = ValkeyPublishOperator(
>>>>>>>>
>>>>>>>> task_id="publish_ready_signal",
>>>>>>>>
>>>>>>>> channel="processing_status",
>>>>>>>>
>>>>>>>> message="READY",
>>>>>>>>
>>>>>>>> valkey_conn_id="valkey_default",
>>>>>>>>
>>>>>>>> )
>>>>>>>>
>>>>>>>> `
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> This task publishes the message "READY" to the
>> processing_status
>>>>>> channel
>>>>>>>> on a Valkey server. Other processes listening to this channel
>> will
>>>>>>> receive
>>>>>>>> the message and proceed with their execution.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> `
>>>>>>>>
>>>>>>>> wait_for_ready = ValkeyPubSubSensor(
>>>>>>>>
>>>>>>>> task_id="wait_for_ready_message",
>>>>>>>>
>>>>>>>> channel="processing_status",
>>>>>>>>
>>>>>>>> valkey_conn_id="valkey_default",
>>>>>>>>
>>>>>>>> timeout=600,
>>>>>>>>
>>>>>>>> )
>>>>>>>>
>>>>>>>> `
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Here, a sensor waits for the message on the same channel before
>>>>>>>> continuing. Once the message is received, the DAG can continue
>> with
>>>>>>> further
>>>>>>>> tasks—such as triggering another DAG—in an event-driven manner.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Given the community's interest in event-driven scheduling (AIP
>> 82<
>>>>>>>> https://github.com/apache/airflow/issues/52712> <
>> https://github.com/apache/airflow/issues/52712>>) as well as
>>>> concerns
>>>>>>>> about limited provider optionality (PR #52712<
>>>>>>>> https://github.com/apache/airflow/issues/52712> <
>> https://github.com/apache/airflow/issues/52712>>), a Valkey
>>>> provider
>>>>>>> would
>>>>>>>> expand user choice.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I’d be happy to begin working on the initial PR and
>> documentation.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Looking forward to feedback.
>>>>>>>>
>>>>>>>> Sean
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
>> dev-unsubscribe@airflow.apache.org>
>>>> For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
>> dev-help@airflow.apache.org>
>>>>
>>>>
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> For additional commands, e-mail: dev-help@airflow.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
Posted by Jarek Potiuk <ja...@potiuk.com>.
> Since Valkey 8.1.x (latest serie) is still 100% compatible with Redis,
> wouldn't be easier to just state in docs Redis provider is also
> compatible with Valkey backend?
Yep, that's one of the options. We discussed it above - that we could have
valley/redis switch for example in the current redis provider - it will
make it more coupled and future features might diverge, but yes, this is
indeed a valid option, I think it's more of the question of name in this
case. If for example we rename redis provider to "kv-storage" or
"common-kv-storage" for example or something similar, that might be
actually quite a viable option for us.
J.
On Wed, Aug 13, 2025 at 11:44 AM Josef Šimánek <jo...@gmail.com>
wrote:
> st 13. 8. 2025 v 0:38 odesílatel Ghaeli, Sean
> <gh...@amazon.com.invalid> napsal:
> >
> > Valkey checks off many boxes that align with Airflow’s provider criteria:
> >
> > * Truly open-source
> > * Dedicated maintainers responsive to community questions
> > * Not a commercial service that could maintain their own provider
> > * Very lean dependencies:
> https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt
> > * Popular and actively maintained with broad community support:
> https://github.com/valkey-io/valkey
> >
> > Given all of these, I believe the Valkey provider makes sense as an
> official Airflow provider.
>
> Since Valkey 8.1.x (latest serie) is still 100% compatible with Redis,
> wouldn't be easier to just state in docs Redis provider is also
> compatible with Valkey backend?
>
> > Sean
> >
> > On 2025-07-15, 1:22 PM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> jarek@potiuk.com>> wrote:
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
> >
> >
> >
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> certain que le contenu ne présente aucun risque.
> >
> >
> >
> >
> >
> >
> > On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock <dheilbock@gmx.net.inva
> <ma...@gmx.net.inva>lid>
> > wrote:
> >
> >
> > > As of last year, the redis Python client was still fully compatible
> with
> > > Valkey, has this changed? This would make it even easier. I think they
> also
> > > forked redis-py
> >
> >
> >
> >
> > Yep. I think that's part of the solution to make the dependency tree for
> > the provider independent of redis-controlled dependencies.
> >
> >
> > Governance, decision making and release process of valley should be
> > separate from redis (the company). That's for me one of the criteria and
> > requirements for having a separate valley provider. Very much like
> > "opensearch" is separate from "elasticsearch".
> >
> >
> >
> >
> >
> >
> > >
> >
> >
> > > Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr
> > > > Von: "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com>>
> > > > An: dev@airflow.apache.org <ma...@airflow.apache.org>
> > > > Cc: ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid
> > > > Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
> > > >
> > > > > Can you please explain why it should be a provider managed by
> Airflow
> > > > rather than on your own?
> > > >
> > > > Just to explain it Elad - and yes, I know I am usually the one that
> is
> > > > sceptical -> valley is truly community driven replacement for redis
> > > > https://valkey.io/participants/ <https://valkey.io/participants/> -
> you see a lot of well-known community
> > > > recognised entities, and there is no single entity behind it that
> could
> > > > take "responsibility" - while there are enough stakeholders to make
> sure
> > > > that it will be maintained here in Airflow community. Also it's a
> > > "drop-in"
> > > > replacement for redis generally - and I do not expect a lot of
> problems
> > > > when we start with both redis provider and redis celery integration.
> > > >
> > > > I've heard and saw a lot of praise from the community for Valkey
> being
> > > good
> > > > redis replacement, and - even we at the PMC and devlist discussions
> > > > discussed about the dangers Redis licence creates for that - even
> now we
> > > > limited Redis support in Helm chart of ours to the "really free"
> version
> > > > and we discussed that Valkey might be a good way forward -
> especially if
> > > > someone wants to contribute support for it. so if someone wants to
> > > > contribute to Airflow AND make sure that Redis broker support in
> Celery
> > > is
> > > > covered - I am all for it.
> > > >
> > > >
> > > > On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <eladkal@gmail.com
> <ma...@gmail.com>> wrote:
> > > >
> > > > > > * Helm Chart support to run valley as Celery Broker instead of
> Redis
> > > > >
> > > > > Oh yes! didn't consider its usage as a broker but is it related to
> the
> > > > > provider package?
> > > > > Amazon SQS can be used as broker but it's not related to the SQS
> > > > > integrations in the Airflow amazon provider (hook,operator,sensor)
> > > > > I also don't see ValKey listed as option for brokers in
> > > > >
> > > > >
> > >
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> <
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> >
> > > > >
> > > > > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <jarek@potiuk.com
> <ma...@potiuk.com>>
> > > wrote:
> > > > >
> > > > > > Yes. Valkey is a good replacement/parallel implementation to
> Redis.
> > > One
> > > > > > thing however is important - I would see this even more useful
> if it
> > > also
> > > > > > allows for the Celery Redis replacement for broker - means that
> it is
> > > > > fully
> > > > > > supported in our:
> > > > > >
> > > > > > * unit tests (of course)
> > > > > > * integration tests with ("valkey") integration - including
> running
> > > > > > docker-compose integration to run in CI
> > > > > > * Helm Chart support to run valley as Celery Broker instead of
> Redis
> > > > > >
> > > > > > I think I'd only consider valley as "successful" provider
> > > integration if
> > > > > > all that above is completed.
> > > > > >
> > > > > > J.
> > > > > >
> > > > > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean
> > > <ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I’d like to propose the addition of a new provider to support
> > > Valkey<
> > > > > > > https://valkey.io/> <https://valkey.io/>>: a fully
> open-source, community-governed fork
> > > of
> > > > > > > Redis 7.2, now maintained under the Linux Foundation.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Valkey has been gaining traction recently, in part due to
> changes
> > > by
> > > > > > Redis
> > > > > > > moving their license to one unrecognized by the Open Source
> > > Initiative.
> > > > > > > Valkey represents a future-proof alternative. The proposed
> provider
> > > > > would
> > > > > > > closely mirror the existing Redis provider in functionality and
> > > serve
> > > > > as
> > > > > > a
> > > > > > > drop-in, OSS-aligned option for workflows that depend on
> key-value
> > > > > > storage,
> > > > > > > pub/sub, or caching patterns.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Tentatively, here are the components I plan on implementing
> > > (mirroring
> > > > > > the
> > > > > > > Redis provider):
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > * ValkeyHook – Manage connections and interactions with the
> > > Valkey
> > > > > > > server, enabling reuse across operators and sensors.
> > > > > > > * ValkeyPublishOperator – Allows tasks to publish messages to
> > > > > Valkey
> > > > > > > pub/sub channels, useful for event-driven workflows or
> inter-task
> > > > > > signaling.
> > > > > > > * ValkeyKeySensor – Waits for a specific key to exist or reach
> > > a
> > > > > > > value, supporting synchronization patterns where one task's
> output
> > > > > > controls
> > > > > > > downstream logic.
> > > > > > > * ValkeyPubSubSensor – Subscribes to a Valkey channel and
> > > triggers
> > > > > on
> > > > > > > incoming messages, enabling real-time DAG triggering.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Here's an example of how we could use the components to
> publish a
> > > > > message
> > > > > > > and then trigger a DAG:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > `
> > > > > > >
> > > > > > > publish_message = ValkeyPublishOperator(
> > > > > > >
> > > > > > > task_id="publish_ready_signal",
> > > > > > >
> > > > > > > channel="processing_status",
> > > > > > >
> > > > > > > message="READY",
> > > > > > >
> > > > > > > valkey_conn_id="valkey_default",
> > > > > > >
> > > > > > > )
> > > > > > >
> > > > > > > `
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > This task publishes the message "READY" to the
> processing_status
> > > > > channel
> > > > > > > on a Valkey server. Other processes listening to this channel
> will
> > > > > > receive
> > > > > > > the message and proceed with their execution.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > `
> > > > > > >
> > > > > > > wait_for_ready = ValkeyPubSubSensor(
> > > > > > >
> > > > > > > task_id="wait_for_ready_message",
> > > > > > >
> > > > > > > channel="processing_status",
> > > > > > >
> > > > > > > valkey_conn_id="valkey_default",
> > > > > > >
> > > > > > > timeout=600,
> > > > > > >
> > > > > > > )
> > > > > > >
> > > > > > > `
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Here, a sensor waits for the message on the same channel before
> > > > > > > continuing. Once the message is received, the DAG can continue
> with
> > > > > > further
> > > > > > > tasks—such as triggering another DAG—in an event-driven manner.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Given the community's interest in event-driven scheduling (AIP
> 82<
> > > > > > > https://github.com/apache/airflow/issues/52712> <
> https://github.com/apache/airflow/issues/52712>>) as well as
> > > concerns
> > > > > > > about limited provider optionality (PR #52712<
> > > > > > > https://github.com/apache/airflow/issues/52712> <
> https://github.com/apache/airflow/issues/52712>>), a Valkey
> > > provider
> > > > > > would
> > > > > > > expand user choice.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I’d be happy to begin working on the initial PR and
> documentation.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Looking forward to feedback.
> > > > > > >
> > > > > > > Sean
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> dev-unsubscribe@airflow.apache.org>
> > > For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> dev-help@airflow.apache.org>
> > >
> > >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
Posted by Josef Šimánek <jo...@gmail.com>.
st 13. 8. 2025 v 0:38 odesílatel Ghaeli, Sean
<gh...@amazon.com.invalid> napsal:
>
> Valkey checks off many boxes that align with Airflow’s provider criteria:
>
> * Truly open-source
> * Dedicated maintainers responsive to community questions
> * Not a commercial service that could maintain their own provider
> * Very lean dependencies: https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt
> * Popular and actively maintained with broad community support: https://github.com/valkey-io/valkey
>
> Given all of these, I believe the Valkey provider makes sense as an official Airflow provider.
Since Valkey 8.1.x (latest serie) is still 100% compatible with Redis,
wouldn't be easier to just state in docs Redis provider is also
compatible with Valkey backend?
> Sean
>
> On 2025-07-15, 1:22 PM, "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com>> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
>
>
>
>
>
>
> On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock <dheilbock@gmx.net.inva <ma...@gmx.net.inva>lid>
> wrote:
>
>
> > As of last year, the redis Python client was still fully compatible with
> > Valkey, has this changed? This would make it even easier. I think they also
> > forked redis-py
>
>
>
>
> Yep. I think that's part of the solution to make the dependency tree for
> the provider independent of redis-controlled dependencies.
>
>
> Governance, decision making and release process of valley should be
> separate from redis (the company). That's for me one of the criteria and
> requirements for having a separate valley provider. Very much like
> "opensearch" is separate from "elasticsearch".
>
>
>
>
>
>
> >
>
>
> > Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr
> > > Von: "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com>>
> > > An: dev@airflow.apache.org <ma...@airflow.apache.org>
> > > Cc: ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid
> > > Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
> > >
> > > > Can you please explain why it should be a provider managed by Airflow
> > > rather than on your own?
> > >
> > > Just to explain it Elad - and yes, I know I am usually the one that is
> > > sceptical -> valley is truly community driven replacement for redis
> > > https://valkey.io/participants/ <https://valkey.io/participants/> - you see a lot of well-known community
> > > recognised entities, and there is no single entity behind it that could
> > > take "responsibility" - while there are enough stakeholders to make sure
> > > that it will be maintained here in Airflow community. Also it's a
> > "drop-in"
> > > replacement for redis generally - and I do not expect a lot of problems
> > > when we start with both redis provider and redis celery integration.
> > >
> > > I've heard and saw a lot of praise from the community for Valkey being
> > good
> > > redis replacement, and - even we at the PMC and devlist discussions
> > > discussed about the dangers Redis licence creates for that - even now we
> > > limited Redis support in Helm chart of ours to the "really free" version
> > > and we discussed that Valkey might be a good way forward - especially if
> > > someone wants to contribute support for it. so if someone wants to
> > > contribute to Airflow AND make sure that Redis broker support in Celery
> > is
> > > covered - I am all for it.
> > >
> > >
> > > On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <eladkal@gmail.com <ma...@gmail.com>> wrote:
> > >
> > > > > * Helm Chart support to run valley as Celery Broker instead of Redis
> > > >
> > > > Oh yes! didn't consider its usage as a broker but is it related to the
> > > > provider package?
> > > > Amazon SQS can be used as broker but it's not related to the SQS
> > > > integrations in the Airflow amazon provider (hook,operator,sensor)
> > > > I also don't see ValKey listed as option for brokers in
> > > >
> > > >
> > https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview <https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview>
> > > >
> > > > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>>
> > wrote:
> > > >
> > > > > Yes. Valkey is a good replacement/parallel implementation to Redis.
> > One
> > > > > thing however is important - I would see this even more useful if it
> > also
> > > > > allows for the Celery Redis replacement for broker - means that it is
> > > > fully
> > > > > supported in our:
> > > > >
> > > > > * unit tests (of course)
> > > > > * integration tests with ("valkey") integration - including running
> > > > > docker-compose integration to run in CI
> > > > > * Helm Chart support to run valley as Celery Broker instead of Redis
> > > > >
> > > > > I think I'd only consider valley as "successful" provider
> > integration if
> > > > > all that above is completed.
> > > > >
> > > > > J.
> > > > >
> > > > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean
> > <ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > >
> > > > > >
> > > > > > I’d like to propose the addition of a new provider to support
> > Valkey<
> > > > > > https://valkey.io/> <https://valkey.io/>>: a fully open-source, community-governed fork
> > of
> > > > > > Redis 7.2, now maintained under the Linux Foundation.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Valkey has been gaining traction recently, in part due to changes
> > by
> > > > > Redis
> > > > > > moving their license to one unrecognized by the Open Source
> > Initiative.
> > > > > > Valkey represents a future-proof alternative. The proposed provider
> > > > would
> > > > > > closely mirror the existing Redis provider in functionality and
> > serve
> > > > as
> > > > > a
> > > > > > drop-in, OSS-aligned option for workflows that depend on key-value
> > > > > storage,
> > > > > > pub/sub, or caching patterns.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Tentatively, here are the components I plan on implementing
> > (mirroring
> > > > > the
> > > > > > Redis provider):
> > > > > >
> > > > > >
> > > > > >
> > > > > > * ValkeyHook – Manage connections and interactions with the
> > Valkey
> > > > > > server, enabling reuse across operators and sensors.
> > > > > > * ValkeyPublishOperator – Allows tasks to publish messages to
> > > > Valkey
> > > > > > pub/sub channels, useful for event-driven workflows or inter-task
> > > > > signaling.
> > > > > > * ValkeyKeySensor – Waits for a specific key to exist or reach
> > a
> > > > > > value, supporting synchronization patterns where one task's output
> > > > > controls
> > > > > > downstream logic.
> > > > > > * ValkeyPubSubSensor – Subscribes to a Valkey channel and
> > triggers
> > > > on
> > > > > > incoming messages, enabling real-time DAG triggering.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Here's an example of how we could use the components to publish a
> > > > message
> > > > > > and then trigger a DAG:
> > > > > >
> > > > > >
> > > > > >
> > > > > > `
> > > > > >
> > > > > > publish_message = ValkeyPublishOperator(
> > > > > >
> > > > > > task_id="publish_ready_signal",
> > > > > >
> > > > > > channel="processing_status",
> > > > > >
> > > > > > message="READY",
> > > > > >
> > > > > > valkey_conn_id="valkey_default",
> > > > > >
> > > > > > )
> > > > > >
> > > > > > `
> > > > > >
> > > > > >
> > > > > >
> > > > > > This task publishes the message "READY" to the processing_status
> > > > channel
> > > > > > on a Valkey server. Other processes listening to this channel will
> > > > > receive
> > > > > > the message and proceed with their execution.
> > > > > >
> > > > > >
> > > > > >
> > > > > > `
> > > > > >
> > > > > > wait_for_ready = ValkeyPubSubSensor(
> > > > > >
> > > > > > task_id="wait_for_ready_message",
> > > > > >
> > > > > > channel="processing_status",
> > > > > >
> > > > > > valkey_conn_id="valkey_default",
> > > > > >
> > > > > > timeout=600,
> > > > > >
> > > > > > )
> > > > > >
> > > > > > `
> > > > > >
> > > > > >
> > > > > >
> > > > > > Here, a sensor waits for the message on the same channel before
> > > > > > continuing. Once the message is received, the DAG can continue with
> > > > > further
> > > > > > tasks—such as triggering another DAG—in an event-driven manner.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Given the community's interest in event-driven scheduling (AIP 82<
> > > > > > https://github.com/apache/airflow/issues/52712> <https://github.com/apache/airflow/issues/52712>>) as well as
> > concerns
> > > > > > about limited provider optionality (PR #52712<
> > > > > > https://github.com/apache/airflow/issues/52712> <https://github.com/apache/airflow/issues/52712>>), a Valkey
> > provider
> > > > > would
> > > > > > expand user choice.
> > > > > >
> > > > > >
> > > > > >
> > > > > > I’d be happy to begin working on the initial PR and documentation.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Looking forward to feedback.
> > > > > >
> > > > > > Sean
> > > > > >
> > > > > >
> > > > >
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>
> > For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org>
> >
> >
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
Posted by Jarek Potiuk <ja...@potiuk.com>.
> Great Expectation is also true open source and they maintained their own
provider till they donated it to Astronomer.
Great expectations is "open core" - there is a single company behind it and
governance is "Great Expectations company decides everything".
> 1. How popular we expect it to be with dag authors and explain the
reasoning for the number/estimation.
Valid question - hopefully Ghaeli can answer it.
> 2. Is the popularity cross the threshold of accepting it as
official Airflow provider or leave it as 3rd party and I'm saying again...
Great Expectation is not yet an official provider so...
Please - don't compare "open core" to "open source". Governance and
decision making, not the licence is what matters to distinguish "true" open
source from something that has an "open source licence for client".
> 3. Maintainability commitment that includes also issue triage and on-going
support for users.
That would also be good to hear. I agree.
J
On Wed, Aug 13, 2025 at 11:23 AM Elad Kalif <el...@apache.org> wrote:
> I already stated my position but I will mention the key aspects.
>
> I don't think the fact of ValKey being a "true" open source is not about
> that.
> Great Expectation is also true open source and they maintained their own
> provider till they donated it to Astronomer.
> I still haven't heard why ValKey provider can be hosted on ValKey repo or
> under AWS.
>
> I am waiting for any users on this mailing list to present a real life use
> case for using ValKey with offline processing. My stand is that it's very
> very low demand (similar to Redis!)
> ValKey can be extremely useful as a broker for Celery and I welcome AWS (or
> anyone else) to add integration for this in Helm Chart (That doesn't
> require a provider!)
>
> The three answers I am seeking are:
> 1. How popular we expect it to be with dag authors and explain the
> reasoning for the number/estimation.
> 2. Is the popularity cross the threshold of accepting it as
> official Airflow provider or leave it as 3rd party and I'm saying again...
> Great Expectation is not yet an official provider so...
> 3. Maintainability commitment that includes also issue triage and on-going
> support for users.
>
>
> On Wed, Aug 13, 2025 at 12:04 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > To make it easier -> https://valkey.io/participants/ here is the list of
> > companies to participate in valley development. Similarly to Airflow and
> > other ASF projects - Valkey seem to be "truly" community driven, and
> when I
> > look at the list, it's hard to pin-point that single entity who would be
> > taking it on (at least it's hard for me).
> >
> > On Wed, Aug 13, 2025 at 11:01 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > > at this point I am still -1
> > >
> > > > What is the downside of running it as a 3rd party provider and
> if/when
> > it
> > > > gains popularity donate it to airflow?
> > > Just out of curiosity - since Valkey is a community project, not
> > > controlled by a single entity nor having money or team to actually
> > release
> > > and manage Airflow integration. Who do you think is the best 3rd-party
> to
> > > release the provider?
> > > Usually those arguments about the 3rd-party we raise when there is -
> > > clearly - such a 3rd-party - but I fail to see who that 3rd-party would
> > be
> > > - what do you think Elad? Can you point such a 3rd-party?
> > >
> > > J.
> > >
> > >
> > > On Wed, Aug 13, 2025 at 10:55 AM Elad Kalif <el...@apache.org>
> wrote:
> > >
> > >> at this point I am still -1
> > >>
> > >> On Wed, Aug 13, 2025 at 1:43 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >>
> > >> > If you would like to contribute - start a PR and get a VOTE on it.
> > >> That's
> > >> > what most people do when they want to add a new provider or call for
> > >> LAZY
> > >> > CONSENSUS if you think consensus is reached and you can get it by
> the
> > >> lazy
> > >> > consensus https://www.apache.org/foundation/voting.html
> > >> >
> > >> > On Wed, Aug 13, 2025 at 12:38 AM Ghaeli, Sean
> > <ghaeli@amazon.com.invalid
> > >> >
> > >> > wrote:
> > >> >
> > >> > > Valkey checks off many boxes that align with Airflow’s provider
> > >> criteria:
> > >> > >
> > >> > > * Truly open-source
> > >> > > * Dedicated maintainers responsive to community questions
> > >> > > * Not a commercial service that could maintain their own provider
> > >> > > * Very lean dependencies:
> > >> > >
> > https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt
> > >> > > * Popular and actively maintained with broad community support:
> > >> > > https://github.com/valkey-io/valkey
> > >> > >
> > >> > > Given all of these, I believe the Valkey provider makes sense as
> an
> > >> > > official Airflow provider.
> > >> > >
> > >> > > Sean
> > >> > >
> > >> > > On 2025-07-15, 1:22 PM, "Jarek Potiuk" <jarek@potiuk.com
> <mailto:
> > >> > > jarek@potiuk.com>> wrote:
> > >> > >
> > >> > >
> > >> > > CAUTION: This email originated from outside of the organization.
> Do
> > >> not
> > >> > > click links or open attachments unless you can confirm the sender
> > and
> > >> > know
> > >> > > the content is safe.
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > >> externe.
> > >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
> ne
> > >> > pouvez
> > >> > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> > certain
> > >> > que
> > >> > > le contenu ne présente aucun risque.
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock
> > >> <dheilbock@gmx.net.inva
> > >> > > <ma...@gmx.net.inva>lid>
> > >> > > wrote:
> > >> > >
> > >> > >
> > >> > > > As of last year, the redis Python client was still fully
> > compatible
> > >> > with
> > >> > > > Valkey, has this changed? This would make it even easier. I
> think
> > >> they
> > >> > > also
> > >> > > > forked redis-py
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > Yep. I think that's part of the solution to make the dependency
> tree
> > >> for
> > >> > > the provider independent of redis-controlled dependencies.
> > >> > >
> > >> > >
> > >> > > Governance, decision making and release process of valley should
> be
> > >> > > separate from redis (the company). That's for me one of the
> criteria
> > >> and
> > >> > > requirements for having a separate valley provider. Very much like
> > >> > > "opensearch" is separate from "elasticsearch".
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > >
> > >> > >
> > >> > >
> > >> > > > Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr
> > >> > > > > Von: "Jarek Potiuk" <jarek@potiuk.com <mailto:
> jarek@potiuk.com
> > >>
> > >> > > > > An: dev@airflow.apache.org <ma...@airflow.apache.org>
> > >> > > > > Cc: ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid
> > >> > > > > Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS
> Fork)
> > >> > > > >
> > >> > > > > > Can you please explain why it should be a provider managed
> by
> > >> > Airflow
> > >> > > > > rather than on your own?
> > >> > > > >
> > >> > > > > Just to explain it Elad - and yes, I know I am usually the one
> > >> that
> > >> > is
> > >> > > > > sceptical -> valley is truly community driven replacement for
> > >> redis
> > >> > > > > https://valkey.io/participants/ <
> > https://valkey.io/participants/>
> > >> -
> > >> > > you see a lot of well-known community
> > >> > > > > recognised entities, and there is no single entity behind it
> > that
> > >> > could
> > >> > > > > take "responsibility" - while there are enough stakeholders to
> > >> make
> > >> > > sure
> > >> > > > > that it will be maintained here in Airflow community. Also
> it's
> > a
> > >> > > > "drop-in"
> > >> > > > > replacement for redis generally - and I do not expect a lot of
> > >> > problems
> > >> > > > > when we start with both redis provider and redis celery
> > >> integration.
> > >> > > > >
> > >> > > > > I've heard and saw a lot of praise from the community for
> Valkey
> > >> > being
> > >> > > > good
> > >> > > > > redis replacement, and - even we at the PMC and devlist
> > >> discussions
> > >> > > > > discussed about the dangers Redis licence creates for that -
> > even
> > >> now
> > >> > > we
> > >> > > > > limited Redis support in Helm chart of ours to the "really
> free"
> > >> > > version
> > >> > > > > and we discussed that Valkey might be a good way forward -
> > >> especially
> > >> > > if
> > >> > > > > someone wants to contribute support for it. so if someone
> wants
> > to
> > >> > > > > contribute to Airflow AND make sure that Redis broker support
> in
> > >> > Celery
> > >> > > > is
> > >> > > > > covered - I am all for it.
> > >> > > > >
> > >> > > > >
> > >> > > > > On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <eladkal@gmail.com
> > >> > <mailto:
> > >> > > eladkal@gmail.com>> wrote:
> > >> > > > >
> > >> > > > > > > * Helm Chart support to run valley as Celery Broker
> instead
> > of
> > >> > > Redis
> > >> > > > > >
> > >> > > > > > Oh yes! didn't consider its usage as a broker but is it
> > related
> > >> to
> > >> > > the
> > >> > > > > > provider package?
> > >> > > > > > Amazon SQS can be used as broker but it's not related to the
> > SQS
> > >> > > > > > integrations in the Airflow amazon provider
> > >> (hook,operator,sensor)
> > >> > > > > > I also don't see ValKey listed as option for brokers in
> > >> > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> > >> > > <
> > >> > >
> > >> >
> > >>
> >
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> > >> > > >
> > >> > > > > >
> > >> > > > > > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <
> > jarek@potiuk.com
> > >> > > <ma...@potiuk.com>>
> > >> > > > wrote:
> > >> > > > > >
> > >> > > > > > > Yes. Valkey is a good replacement/parallel implementation
> to
> > >> > Redis.
> > >> > > > One
> > >> > > > > > > thing however is important - I would see this even more
> > >> useful if
> > >> > > it
> > >> > > > also
> > >> > > > > > > allows for the Celery Redis replacement for broker - means
> > >> that
> > >> > it
> > >> > > is
> > >> > > > > > fully
> > >> > > > > > > supported in our:
> > >> > > > > > >
> > >> > > > > > > * unit tests (of course)
> > >> > > > > > > * integration tests with ("valkey") integration -
> including
> > >> > running
> > >> > > > > > > docker-compose integration to run in CI
> > >> > > > > > > * Helm Chart support to run valley as Celery Broker
> instead
> > of
> > >> > > Redis
> > >> > > > > > >
> > >> > > > > > > I think I'd only consider valley as "successful" provider
> > >> > > > integration if
> > >> > > > > > > all that above is completed.
> > >> > > > > > >
> > >> > > > > > > J.
> > >> > > > > > >
> > >> > > > > > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean
> > >> > > > <ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi all,
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > I’d like to propose the addition of a new provider to
> > >> support
> > >> > > > Valkey<
> > >> > > > > > > > https://valkey.io/> <https://valkey.io/>>: a fully
> > >> > > open-source, community-governed fork
> > >> > > > of
> > >> > > > > > > > Redis 7.2, now maintained under the Linux Foundation.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Valkey has been gaining traction recently, in part due
> to
> > >> > changes
> > >> > > > by
> > >> > > > > > > Redis
> > >> > > > > > > > moving their license to one unrecognized by the Open
> > Source
> > >> > > > Initiative.
> > >> > > > > > > > Valkey represents a future-proof alternative. The
> proposed
> > >> > > provider
> > >> > > > > > would
> > >> > > > > > > > closely mirror the existing Redis provider in
> > functionality
> > >> and
> > >> > > > serve
> > >> > > > > > as
> > >> > > > > > > a
> > >> > > > > > > > drop-in, OSS-aligned option for workflows that depend on
> > >> > > key-value
> > >> > > > > > > storage,
> > >> > > > > > > > pub/sub, or caching patterns.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Tentatively, here are the components I plan on
> > implementing
> > >> > > > (mirroring
> > >> > > > > > > the
> > >> > > > > > > > Redis provider):
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > * ValkeyHook – Manage connections and interactions with
> > the
> > >> > > > Valkey
> > >> > > > > > > > server, enabling reuse across operators and sensors.
> > >> > > > > > > > * ValkeyPublishOperator – Allows tasks to publish
> messages
> > >> to
> > >> > > > > > Valkey
> > >> > > > > > > > pub/sub channels, useful for event-driven workflows or
> > >> > inter-task
> > >> > > > > > > signaling.
> > >> > > > > > > > * ValkeyKeySensor – Waits for a specific key to exist or
> > >> reach
> > >> > > > a
> > >> > > > > > > > value, supporting synchronization patterns where one
> > task's
> > >> > > output
> > >> > > > > > > controls
> > >> > > > > > > > downstream logic.
> > >> > > > > > > > * ValkeyPubSubSensor – Subscribes to a Valkey channel
> and
> > >> > > > triggers
> > >> > > > > > on
> > >> > > > > > > > incoming messages, enabling real-time DAG triggering.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Here's an example of how we could use the components to
> > >> > publish a
> > >> > > > > > message
> > >> > > > > > > > and then trigger a DAG:
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > `
> > >> > > > > > > >
> > >> > > > > > > > publish_message = ValkeyPublishOperator(
> > >> > > > > > > >
> > >> > > > > > > > task_id="publish_ready_signal",
> > >> > > > > > > >
> > >> > > > > > > > channel="processing_status",
> > >> > > > > > > >
> > >> > > > > > > > message="READY",
> > >> > > > > > > >
> > >> > > > > > > > valkey_conn_id="valkey_default",
> > >> > > > > > > >
> > >> > > > > > > > )
> > >> > > > > > > >
> > >> > > > > > > > `
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > This task publishes the message "READY" to the
> > >> > processing_status
> > >> > > > > > channel
> > >> > > > > > > > on a Valkey server. Other processes listening to this
> > >> channel
> > >> > > will
> > >> > > > > > > receive
> > >> > > > > > > > the message and proceed with their execution.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > `
> > >> > > > > > > >
> > >> > > > > > > > wait_for_ready = ValkeyPubSubSensor(
> > >> > > > > > > >
> > >> > > > > > > > task_id="wait_for_ready_message",
> > >> > > > > > > >
> > >> > > > > > > > channel="processing_status",
> > >> > > > > > > >
> > >> > > > > > > > valkey_conn_id="valkey_default",
> > >> > > > > > > >
> > >> > > > > > > > timeout=600,
> > >> > > > > > > >
> > >> > > > > > > > )
> > >> > > > > > > >
> > >> > > > > > > > `
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Here, a sensor waits for the message on the same channel
> > >> before
> > >> > > > > > > > continuing. Once the message is received, the DAG can
> > >> continue
> > >> > > with
> > >> > > > > > > further
> > >> > > > > > > > tasks—such as triggering another DAG—in an event-driven
> > >> manner.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Given the community's interest in event-driven
> scheduling
> > >> (AIP
> > >> > > 82<
> > >> > > > > > > > https://github.com/apache/airflow/issues/52712> <
> > >> > > https://github.com/apache/airflow/issues/52712>>) as well as
> > >> > > > concerns
> > >> > > > > > > > about limited provider optionality (PR #52712<
> > >> > > > > > > > https://github.com/apache/airflow/issues/52712> <
> > >> > > https://github.com/apache/airflow/issues/52712>>), a Valkey
> > >> > > > provider
> > >> > > > > > > would
> > >> > > > > > > > expand user choice.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > I’d be happy to begin working on the initial PR and
> > >> > > documentation.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Looking forward to feedback.
> > >> > > > > > > >
> > >> > > > > > > > Sean
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > > >
> > >> ---------------------------------------------------------------------
> > >> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > <mailto:
> > >> > > dev-unsubscribe@airflow.apache.org>
> > >> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >> <mailto:
> > >> > > dev-help@airflow.apache.org>
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
Posted by Elad Kalif <el...@apache.org>.
I already stated my position but I will mention the key aspects.
I don't think the fact of ValKey being a "true" open source is not about
that.
Great Expectation is also true open source and they maintained their own
provider till they donated it to Astronomer.
I still haven't heard why ValKey provider can be hosted on ValKey repo or
under AWS.
I am waiting for any users on this mailing list to present a real life use
case for using ValKey with offline processing. My stand is that it's very
very low demand (similar to Redis!)
ValKey can be extremely useful as a broker for Celery and I welcome AWS (or
anyone else) to add integration for this in Helm Chart (That doesn't
require a provider!)
The three answers I am seeking are:
1. How popular we expect it to be with dag authors and explain the
reasoning for the number/estimation.
2. Is the popularity cross the threshold of accepting it as
official Airflow provider or leave it as 3rd party and I'm saying again...
Great Expectation is not yet an official provider so...
3. Maintainability commitment that includes also issue triage and on-going
support for users.
On Wed, Aug 13, 2025 at 12:04 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> To make it easier -> https://valkey.io/participants/ here is the list of
> companies to participate in valley development. Similarly to Airflow and
> other ASF projects - Valkey seem to be "truly" community driven, and when I
> look at the list, it's hard to pin-point that single entity who would be
> taking it on (at least it's hard for me).
>
> On Wed, Aug 13, 2025 at 11:01 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > at this point I am still -1
> >
> > > What is the downside of running it as a 3rd party provider and if/when
> it
> > > gains popularity donate it to airflow?
> > Just out of curiosity - since Valkey is a community project, not
> > controlled by a single entity nor having money or team to actually
> release
> > and manage Airflow integration. Who do you think is the best 3rd-party to
> > release the provider?
> > Usually those arguments about the 3rd-party we raise when there is -
> > clearly - such a 3rd-party - but I fail to see who that 3rd-party would
> be
> > - what do you think Elad? Can you point such a 3rd-party?
> >
> > J.
> >
> >
> > On Wed, Aug 13, 2025 at 10:55 AM Elad Kalif <el...@apache.org> wrote:
> >
> >> at this point I am still -1
> >>
> >> On Wed, Aug 13, 2025 at 1:43 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >> > If you would like to contribute - start a PR and get a VOTE on it.
> >> That's
> >> > what most people do when they want to add a new provider or call for
> >> LAZY
> >> > CONSENSUS if you think consensus is reached and you can get it by the
> >> lazy
> >> > consensus https://www.apache.org/foundation/voting.html
> >> >
> >> > On Wed, Aug 13, 2025 at 12:38 AM Ghaeli, Sean
> <ghaeli@amazon.com.invalid
> >> >
> >> > wrote:
> >> >
> >> > > Valkey checks off many boxes that align with Airflow’s provider
> >> criteria:
> >> > >
> >> > > * Truly open-source
> >> > > * Dedicated maintainers responsive to community questions
> >> > > * Not a commercial service that could maintain their own provider
> >> > > * Very lean dependencies:
> >> > >
> https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt
> >> > > * Popular and actively maintained with broad community support:
> >> > > https://github.com/valkey-io/valkey
> >> > >
> >> > > Given all of these, I believe the Valkey provider makes sense as an
> >> > > official Airflow provider.
> >> > >
> >> > > Sean
> >> > >
> >> > > On 2025-07-15, 1:22 PM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> >> > > jarek@potiuk.com>> wrote:
> >> > >
> >> > >
> >> > > CAUTION: This email originated from outside of the organization. Do
> >> not
> >> > > click links or open attachments unless you can confirm the sender
> and
> >> > know
> >> > > the content is safe.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> >> externe.
> >> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> >> > pouvez
> >> > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> certain
> >> > que
> >> > > le contenu ne présente aucun risque.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock
> >> <dheilbock@gmx.net.inva
> >> > > <ma...@gmx.net.inva>lid>
> >> > > wrote:
> >> > >
> >> > >
> >> > > > As of last year, the redis Python client was still fully
> compatible
> >> > with
> >> > > > Valkey, has this changed? This would make it even easier. I think
> >> they
> >> > > also
> >> > > > forked redis-py
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > Yep. I think that's part of the solution to make the dependency tree
> >> for
> >> > > the provider independent of redis-controlled dependencies.
> >> > >
> >> > >
> >> > > Governance, decision making and release process of valley should be
> >> > > separate from redis (the company). That's for me one of the criteria
> >> and
> >> > > requirements for having a separate valley provider. Very much like
> >> > > "opensearch" is separate from "elasticsearch".
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > >
> >> > >
> >> > >
> >> > > > Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr
> >> > > > > Von: "Jarek Potiuk" <jarek@potiuk.com <mailto:jarek@potiuk.com
> >>
> >> > > > > An: dev@airflow.apache.org <ma...@airflow.apache.org>
> >> > > > > Cc: ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid
> >> > > > > Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
> >> > > > >
> >> > > > > > Can you please explain why it should be a provider managed by
> >> > Airflow
> >> > > > > rather than on your own?
> >> > > > >
> >> > > > > Just to explain it Elad - and yes, I know I am usually the one
> >> that
> >> > is
> >> > > > > sceptical -> valley is truly community driven replacement for
> >> redis
> >> > > > > https://valkey.io/participants/ <
> https://valkey.io/participants/>
> >> -
> >> > > you see a lot of well-known community
> >> > > > > recognised entities, and there is no single entity behind it
> that
> >> > could
> >> > > > > take "responsibility" - while there are enough stakeholders to
> >> make
> >> > > sure
> >> > > > > that it will be maintained here in Airflow community. Also it's
> a
> >> > > > "drop-in"
> >> > > > > replacement for redis generally - and I do not expect a lot of
> >> > problems
> >> > > > > when we start with both redis provider and redis celery
> >> integration.
> >> > > > >
> >> > > > > I've heard and saw a lot of praise from the community for Valkey
> >> > being
> >> > > > good
> >> > > > > redis replacement, and - even we at the PMC and devlist
> >> discussions
> >> > > > > discussed about the dangers Redis licence creates for that -
> even
> >> now
> >> > > we
> >> > > > > limited Redis support in Helm chart of ours to the "really free"
> >> > > version
> >> > > > > and we discussed that Valkey might be a good way forward -
> >> especially
> >> > > if
> >> > > > > someone wants to contribute support for it. so if someone wants
> to
> >> > > > > contribute to Airflow AND make sure that Redis broker support in
> >> > Celery
> >> > > > is
> >> > > > > covered - I am all for it.
> >> > > > >
> >> > > > >
> >> > > > > On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <eladkal@gmail.com
> >> > <mailto:
> >> > > eladkal@gmail.com>> wrote:
> >> > > > >
> >> > > > > > > * Helm Chart support to run valley as Celery Broker instead
> of
> >> > > Redis
> >> > > > > >
> >> > > > > > Oh yes! didn't consider its usage as a broker but is it
> related
> >> to
> >> > > the
> >> > > > > > provider package?
> >> > > > > > Amazon SQS can be used as broker but it's not related to the
> SQS
> >> > > > > > integrations in the Airflow amazon provider
> >> (hook,operator,sensor)
> >> > > > > > I also don't see ValKey listed as option for brokers in
> >> > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> >> > > <
> >> > >
> >> >
> >>
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> >> > > >
> >> > > > > >
> >> > > > > > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <
> jarek@potiuk.com
> >> > > <ma...@potiuk.com>>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > Yes. Valkey is a good replacement/parallel implementation to
> >> > Redis.
> >> > > > One
> >> > > > > > > thing however is important - I would see this even more
> >> useful if
> >> > > it
> >> > > > also
> >> > > > > > > allows for the Celery Redis replacement for broker - means
> >> that
> >> > it
> >> > > is
> >> > > > > > fully
> >> > > > > > > supported in our:
> >> > > > > > >
> >> > > > > > > * unit tests (of course)
> >> > > > > > > * integration tests with ("valkey") integration - including
> >> > running
> >> > > > > > > docker-compose integration to run in CI
> >> > > > > > > * Helm Chart support to run valley as Celery Broker instead
> of
> >> > > Redis
> >> > > > > > >
> >> > > > > > > I think I'd only consider valley as "successful" provider
> >> > > > integration if
> >> > > > > > > all that above is completed.
> >> > > > > > >
> >> > > > > > > J.
> >> > > > > > >
> >> > > > > > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean
> >> > > > <ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hi all,
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > I’d like to propose the addition of a new provider to
> >> support
> >> > > > Valkey<
> >> > > > > > > > https://valkey.io/> <https://valkey.io/>>: a fully
> >> > > open-source, community-governed fork
> >> > > > of
> >> > > > > > > > Redis 7.2, now maintained under the Linux Foundation.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Valkey has been gaining traction recently, in part due to
> >> > changes
> >> > > > by
> >> > > > > > > Redis
> >> > > > > > > > moving their license to one unrecognized by the Open
> Source
> >> > > > Initiative.
> >> > > > > > > > Valkey represents a future-proof alternative. The proposed
> >> > > provider
> >> > > > > > would
> >> > > > > > > > closely mirror the existing Redis provider in
> functionality
> >> and
> >> > > > serve
> >> > > > > > as
> >> > > > > > > a
> >> > > > > > > > drop-in, OSS-aligned option for workflows that depend on
> >> > > key-value
> >> > > > > > > storage,
> >> > > > > > > > pub/sub, or caching patterns.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Tentatively, here are the components I plan on
> implementing
> >> > > > (mirroring
> >> > > > > > > the
> >> > > > > > > > Redis provider):
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > * ValkeyHook – Manage connections and interactions with
> the
> >> > > > Valkey
> >> > > > > > > > server, enabling reuse across operators and sensors.
> >> > > > > > > > * ValkeyPublishOperator – Allows tasks to publish messages
> >> to
> >> > > > > > Valkey
> >> > > > > > > > pub/sub channels, useful for event-driven workflows or
> >> > inter-task
> >> > > > > > > signaling.
> >> > > > > > > > * ValkeyKeySensor – Waits for a specific key to exist or
> >> reach
> >> > > > a
> >> > > > > > > > value, supporting synchronization patterns where one
> task's
> >> > > output
> >> > > > > > > controls
> >> > > > > > > > downstream logic.
> >> > > > > > > > * ValkeyPubSubSensor – Subscribes to a Valkey channel and
> >> > > > triggers
> >> > > > > > on
> >> > > > > > > > incoming messages, enabling real-time DAG triggering.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Here's an example of how we could use the components to
> >> > publish a
> >> > > > > > message
> >> > > > > > > > and then trigger a DAG:
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > `
> >> > > > > > > >
> >> > > > > > > > publish_message = ValkeyPublishOperator(
> >> > > > > > > >
> >> > > > > > > > task_id="publish_ready_signal",
> >> > > > > > > >
> >> > > > > > > > channel="processing_status",
> >> > > > > > > >
> >> > > > > > > > message="READY",
> >> > > > > > > >
> >> > > > > > > > valkey_conn_id="valkey_default",
> >> > > > > > > >
> >> > > > > > > > )
> >> > > > > > > >
> >> > > > > > > > `
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > This task publishes the message "READY" to the
> >> > processing_status
> >> > > > > > channel
> >> > > > > > > > on a Valkey server. Other processes listening to this
> >> channel
> >> > > will
> >> > > > > > > receive
> >> > > > > > > > the message and proceed with their execution.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > `
> >> > > > > > > >
> >> > > > > > > > wait_for_ready = ValkeyPubSubSensor(
> >> > > > > > > >
> >> > > > > > > > task_id="wait_for_ready_message",
> >> > > > > > > >
> >> > > > > > > > channel="processing_status",
> >> > > > > > > >
> >> > > > > > > > valkey_conn_id="valkey_default",
> >> > > > > > > >
> >> > > > > > > > timeout=600,
> >> > > > > > > >
> >> > > > > > > > )
> >> > > > > > > >
> >> > > > > > > > `
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Here, a sensor waits for the message on the same channel
> >> before
> >> > > > > > > > continuing. Once the message is received, the DAG can
> >> continue
> >> > > with
> >> > > > > > > further
> >> > > > > > > > tasks—such as triggering another DAG—in an event-driven
> >> manner.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Given the community's interest in event-driven scheduling
> >> (AIP
> >> > > 82<
> >> > > > > > > > https://github.com/apache/airflow/issues/52712> <
> >> > > https://github.com/apache/airflow/issues/52712>>) as well as
> >> > > > concerns
> >> > > > > > > > about limited provider optionality (PR #52712<
> >> > > > > > > > https://github.com/apache/airflow/issues/52712> <
> >> > > https://github.com/apache/airflow/issues/52712>>), a Valkey
> >> > > > provider
> >> > > > > > > would
> >> > > > > > > > expand user choice.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > I’d be happy to begin working on the initial PR and
> >> > > documentation.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Looking forward to feedback.
> >> > > > > > > >
> >> > > > > > > > Sean
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > > >
> >> ---------------------------------------------------------------------
> >> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> <mailto:
> >> > > dev-unsubscribe@airflow.apache.org>
> >> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> >> <mailto:
> >> > > dev-help@airflow.apache.org>
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >>
> >
>
Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
Posted by Jarek Potiuk <ja...@potiuk.com>.
To make it easier -> https://valkey.io/participants/ here is the list of
companies to participate in valley development. Similarly to Airflow and
other ASF projects - Valkey seem to be "truly" community driven, and when I
look at the list, it's hard to pin-point that single entity who would be
taking it on (at least it's hard for me).
On Wed, Aug 13, 2025 at 11:01 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > at this point I am still -1
>
> > What is the downside of running it as a 3rd party provider and if/when it
> > gains popularity donate it to airflow?
> Just out of curiosity - since Valkey is a community project, not
> controlled by a single entity nor having money or team to actually release
> and manage Airflow integration. Who do you think is the best 3rd-party to
> release the provider?
> Usually those arguments about the 3rd-party we raise when there is -
> clearly - such a 3rd-party - but I fail to see who that 3rd-party would be
> - what do you think Elad? Can you point such a 3rd-party?
>
> J.
>
>
> On Wed, Aug 13, 2025 at 10:55 AM Elad Kalif <el...@apache.org> wrote:
>
>> at this point I am still -1
>>
>> On Wed, Aug 13, 2025 at 1:43 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> > If you would like to contribute - start a PR and get a VOTE on it.
>> That's
>> > what most people do when they want to add a new provider or call for
>> LAZY
>> > CONSENSUS if you think consensus is reached and you can get it by the
>> lazy
>> > consensus https://www.apache.org/foundation/voting.html
>> >
>> > On Wed, Aug 13, 2025 at 12:38 AM Ghaeli, Sean <ghaeli@amazon.com.invalid
>> >
>> > wrote:
>> >
>> > > Valkey checks off many boxes that align with Airflow’s provider
>> criteria:
>> > >
>> > > * Truly open-source
>> > > * Dedicated maintainers responsive to community questions
>> > > * Not a commercial service that could maintain their own provider
>> > > * Very lean dependencies:
>> > > https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt
>> > > * Popular and actively maintained with broad community support:
>> > > https://github.com/valkey-io/valkey
>> > >
>> > > Given all of these, I believe the Valkey provider makes sense as an
>> > > official Airflow provider.
>> > >
>> > > Sean
>> > >
>> > > On 2025-07-15, 1:22 PM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
>> > > jarek@potiuk.com>> wrote:
>> > >
>> > >
>> > > CAUTION: This email originated from outside of the organization. Do
>> not
>> > > click links or open attachments unless you can confirm the sender and
>> > know
>> > > the content is safe.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
>> externe.
>> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
>> > pouvez
>> > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
>> > que
>> > > le contenu ne présente aucun risque.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock
>> <dheilbock@gmx.net.inva
>> > > <ma...@gmx.net.inva>lid>
>> > > wrote:
>> > >
>> > >
>> > > > As of last year, the redis Python client was still fully compatible
>> > with
>> > > > Valkey, has this changed? This would make it even easier. I think
>> they
>> > > also
>> > > > forked redis-py
>> > >
>> > >
>> > >
>> > >
>> > > Yep. I think that's part of the solution to make the dependency tree
>> for
>> > > the provider independent of redis-controlled dependencies.
>> > >
>> > >
>> > > Governance, decision making and release process of valley should be
>> > > separate from redis (the company). That's for me one of the criteria
>> and
>> > > requirements for having a separate valley provider. Very much like
>> > > "opensearch" is separate from "elasticsearch".
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > >
>> > >
>> > >
>> > > > Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr
>> > > > > Von: "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com>>
>> > > > > An: dev@airflow.apache.org <ma...@airflow.apache.org>
>> > > > > Cc: ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid
>> > > > > Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
>> > > > >
>> > > > > > Can you please explain why it should be a provider managed by
>> > Airflow
>> > > > > rather than on your own?
>> > > > >
>> > > > > Just to explain it Elad - and yes, I know I am usually the one
>> that
>> > is
>> > > > > sceptical -> valley is truly community driven replacement for
>> redis
>> > > > > https://valkey.io/participants/ <https://valkey.io/participants/>
>> -
>> > > you see a lot of well-known community
>> > > > > recognised entities, and there is no single entity behind it that
>> > could
>> > > > > take "responsibility" - while there are enough stakeholders to
>> make
>> > > sure
>> > > > > that it will be maintained here in Airflow community. Also it's a
>> > > > "drop-in"
>> > > > > replacement for redis generally - and I do not expect a lot of
>> > problems
>> > > > > when we start with both redis provider and redis celery
>> integration.
>> > > > >
>> > > > > I've heard and saw a lot of praise from the community for Valkey
>> > being
>> > > > good
>> > > > > redis replacement, and - even we at the PMC and devlist
>> discussions
>> > > > > discussed about the dangers Redis licence creates for that - even
>> now
>> > > we
>> > > > > limited Redis support in Helm chart of ours to the "really free"
>> > > version
>> > > > > and we discussed that Valkey might be a good way forward -
>> especially
>> > > if
>> > > > > someone wants to contribute support for it. so if someone wants to
>> > > > > contribute to Airflow AND make sure that Redis broker support in
>> > Celery
>> > > > is
>> > > > > covered - I am all for it.
>> > > > >
>> > > > >
>> > > > > On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <eladkal@gmail.com
>> > <mailto:
>> > > eladkal@gmail.com>> wrote:
>> > > > >
>> > > > > > > * Helm Chart support to run valley as Celery Broker instead of
>> > > Redis
>> > > > > >
>> > > > > > Oh yes! didn't consider its usage as a broker but is it related
>> to
>> > > the
>> > > > > > provider package?
>> > > > > > Amazon SQS can be used as broker but it's not related to the SQS
>> > > > > > integrations in the Airflow amazon provider
>> (hook,operator,sensor)
>> > > > > > I also don't see ValKey listed as option for brokers in
>> > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
>> > > <
>> > >
>> >
>> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
>> > > >
>> > > > > >
>> > > > > > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <jarek@potiuk.com
>> > > <ma...@potiuk.com>>
>> > > > wrote:
>> > > > > >
>> > > > > > > Yes. Valkey is a good replacement/parallel implementation to
>> > Redis.
>> > > > One
>> > > > > > > thing however is important - I would see this even more
>> useful if
>> > > it
>> > > > also
>> > > > > > > allows for the Celery Redis replacement for broker - means
>> that
>> > it
>> > > is
>> > > > > > fully
>> > > > > > > supported in our:
>> > > > > > >
>> > > > > > > * unit tests (of course)
>> > > > > > > * integration tests with ("valkey") integration - including
>> > running
>> > > > > > > docker-compose integration to run in CI
>> > > > > > > * Helm Chart support to run valley as Celery Broker instead of
>> > > Redis
>> > > > > > >
>> > > > > > > I think I'd only consider valley as "successful" provider
>> > > > integration if
>> > > > > > > all that above is completed.
>> > > > > > >
>> > > > > > > J.
>> > > > > > >
>> > > > > > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean
>> > > > <ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi all,
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I’d like to propose the addition of a new provider to
>> support
>> > > > Valkey<
>> > > > > > > > https://valkey.io/> <https://valkey.io/>>: a fully
>> > > open-source, community-governed fork
>> > > > of
>> > > > > > > > Redis 7.2, now maintained under the Linux Foundation.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Valkey has been gaining traction recently, in part due to
>> > changes
>> > > > by
>> > > > > > > Redis
>> > > > > > > > moving their license to one unrecognized by the Open Source
>> > > > Initiative.
>> > > > > > > > Valkey represents a future-proof alternative. The proposed
>> > > provider
>> > > > > > would
>> > > > > > > > closely mirror the existing Redis provider in functionality
>> and
>> > > > serve
>> > > > > > as
>> > > > > > > a
>> > > > > > > > drop-in, OSS-aligned option for workflows that depend on
>> > > key-value
>> > > > > > > storage,
>> > > > > > > > pub/sub, or caching patterns.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Tentatively, here are the components I plan on implementing
>> > > > (mirroring
>> > > > > > > the
>> > > > > > > > Redis provider):
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > * ValkeyHook – Manage connections and interactions with the
>> > > > Valkey
>> > > > > > > > server, enabling reuse across operators and sensors.
>> > > > > > > > * ValkeyPublishOperator – Allows tasks to publish messages
>> to
>> > > > > > Valkey
>> > > > > > > > pub/sub channels, useful for event-driven workflows or
>> > inter-task
>> > > > > > > signaling.
>> > > > > > > > * ValkeyKeySensor – Waits for a specific key to exist or
>> reach
>> > > > a
>> > > > > > > > value, supporting synchronization patterns where one task's
>> > > output
>> > > > > > > controls
>> > > > > > > > downstream logic.
>> > > > > > > > * ValkeyPubSubSensor – Subscribes to a Valkey channel and
>> > > > triggers
>> > > > > > on
>> > > > > > > > incoming messages, enabling real-time DAG triggering.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Here's an example of how we could use the components to
>> > publish a
>> > > > > > message
>> > > > > > > > and then trigger a DAG:
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > `
>> > > > > > > >
>> > > > > > > > publish_message = ValkeyPublishOperator(
>> > > > > > > >
>> > > > > > > > task_id="publish_ready_signal",
>> > > > > > > >
>> > > > > > > > channel="processing_status",
>> > > > > > > >
>> > > > > > > > message="READY",
>> > > > > > > >
>> > > > > > > > valkey_conn_id="valkey_default",
>> > > > > > > >
>> > > > > > > > )
>> > > > > > > >
>> > > > > > > > `
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > This task publishes the message "READY" to the
>> > processing_status
>> > > > > > channel
>> > > > > > > > on a Valkey server. Other processes listening to this
>> channel
>> > > will
>> > > > > > > receive
>> > > > > > > > the message and proceed with their execution.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > `
>> > > > > > > >
>> > > > > > > > wait_for_ready = ValkeyPubSubSensor(
>> > > > > > > >
>> > > > > > > > task_id="wait_for_ready_message",
>> > > > > > > >
>> > > > > > > > channel="processing_status",
>> > > > > > > >
>> > > > > > > > valkey_conn_id="valkey_default",
>> > > > > > > >
>> > > > > > > > timeout=600,
>> > > > > > > >
>> > > > > > > > )
>> > > > > > > >
>> > > > > > > > `
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Here, a sensor waits for the message on the same channel
>> before
>> > > > > > > > continuing. Once the message is received, the DAG can
>> continue
>> > > with
>> > > > > > > further
>> > > > > > > > tasks—such as triggering another DAG—in an event-driven
>> manner.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Given the community's interest in event-driven scheduling
>> (AIP
>> > > 82<
>> > > > > > > > https://github.com/apache/airflow/issues/52712> <
>> > > https://github.com/apache/airflow/issues/52712>>) as well as
>> > > > concerns
>> > > > > > > > about limited provider optionality (PR #52712<
>> > > > > > > > https://github.com/apache/airflow/issues/52712> <
>> > > https://github.com/apache/airflow/issues/52712>>), a Valkey
>> > > > provider
>> > > > > > > would
>> > > > > > > > expand user choice.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I’d be happy to begin working on the initial PR and
>> > > documentation.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Looking forward to feedback.
>> > > > > > > >
>> > > > > > > > Sean
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
>> > > dev-unsubscribe@airflow.apache.org>
>> > > > For additional commands, e-mail: dev-help@airflow.apache.org
>> <mailto:
>> > > dev-help@airflow.apache.org>
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> > >
>> >
>>
>
Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
Posted by Jarek Potiuk <ja...@potiuk.com>.
> at this point I am still -1
> What is the downside of running it as a 3rd party provider and if/when it
> gains popularity donate it to airflow?
Just out of curiosity - since Valkey is a community project, not
controlled by a single entity nor having money or team to actually release
and manage Airflow integration. Who do you think is the best 3rd-party to
release the provider?
Usually those arguments about the 3rd-party we raise when there is -
clearly - such a 3rd-party - but I fail to see who that 3rd-party would be
- what do you think Elad? Can you point such a 3rd-party?
J.
On Wed, Aug 13, 2025 at 10:55 AM Elad Kalif <el...@apache.org> wrote:
> at this point I am still -1
>
> On Wed, Aug 13, 2025 at 1:43 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > If you would like to contribute - start a PR and get a VOTE on it. That's
> > what most people do when they want to add a new provider or call for LAZY
> > CONSENSUS if you think consensus is reached and you can get it by the
> lazy
> > consensus https://www.apache.org/foundation/voting.html
> >
> > On Wed, Aug 13, 2025 at 12:38 AM Ghaeli, Sean <ghaeli@amazon.com.invalid
> >
> > wrote:
> >
> > > Valkey checks off many boxes that align with Airflow’s provider
> criteria:
> > >
> > > * Truly open-source
> > > * Dedicated maintainers responsive to community questions
> > > * Not a commercial service that could maintain their own provider
> > > * Very lean dependencies:
> > > https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt
> > > * Popular and actively maintained with broad community support:
> > > https://github.com/valkey-io/valkey
> > >
> > > Given all of these, I believe the Valkey provider makes sense as an
> > > official Airflow provider.
> > >
> > > Sean
> > >
> > > On 2025-07-15, 1:22 PM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> > > jarek@potiuk.com>> wrote:
> > >
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > > click links or open attachments unless you can confirm the sender and
> > know
> > > the content is safe.
> > >
> > >
> > >
> > >
> > >
> > >
> > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe.
> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> > pouvez
> > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> > que
> > > le contenu ne présente aucun risque.
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock
> <dheilbock@gmx.net.inva
> > > <ma...@gmx.net.inva>lid>
> > > wrote:
> > >
> > >
> > > > As of last year, the redis Python client was still fully compatible
> > with
> > > > Valkey, has this changed? This would make it even easier. I think
> they
> > > also
> > > > forked redis-py
> > >
> > >
> > >
> > >
> > > Yep. I think that's part of the solution to make the dependency tree
> for
> > > the provider independent of redis-controlled dependencies.
> > >
> > >
> > > Governance, decision making and release process of valley should be
> > > separate from redis (the company). That's for me one of the criteria
> and
> > > requirements for having a separate valley provider. Very much like
> > > "opensearch" is separate from "elasticsearch".
> > >
> > >
> > >
> > >
> > >
> > >
> > > >
> > >
> > >
> > > > Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr
> > > > > Von: "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com>>
> > > > > An: dev@airflow.apache.org <ma...@airflow.apache.org>
> > > > > Cc: ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid
> > > > > Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
> > > > >
> > > > > > Can you please explain why it should be a provider managed by
> > Airflow
> > > > > rather than on your own?
> > > > >
> > > > > Just to explain it Elad - and yes, I know I am usually the one that
> > is
> > > > > sceptical -> valley is truly community driven replacement for redis
> > > > > https://valkey.io/participants/ <https://valkey.io/participants/>
> -
> > > you see a lot of well-known community
> > > > > recognised entities, and there is no single entity behind it that
> > could
> > > > > take "responsibility" - while there are enough stakeholders to make
> > > sure
> > > > > that it will be maintained here in Airflow community. Also it's a
> > > > "drop-in"
> > > > > replacement for redis generally - and I do not expect a lot of
> > problems
> > > > > when we start with both redis provider and redis celery
> integration.
> > > > >
> > > > > I've heard and saw a lot of praise from the community for Valkey
> > being
> > > > good
> > > > > redis replacement, and - even we at the PMC and devlist discussions
> > > > > discussed about the dangers Redis licence creates for that - even
> now
> > > we
> > > > > limited Redis support in Helm chart of ours to the "really free"
> > > version
> > > > > and we discussed that Valkey might be a good way forward -
> especially
> > > if
> > > > > someone wants to contribute support for it. so if someone wants to
> > > > > contribute to Airflow AND make sure that Redis broker support in
> > Celery
> > > > is
> > > > > covered - I am all for it.
> > > > >
> > > > >
> > > > > On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <eladkal@gmail.com
> > <mailto:
> > > eladkal@gmail.com>> wrote:
> > > > >
> > > > > > > * Helm Chart support to run valley as Celery Broker instead of
> > > Redis
> > > > > >
> > > > > > Oh yes! didn't consider its usage as a broker but is it related
> to
> > > the
> > > > > > provider package?
> > > > > > Amazon SQS can be used as broker but it's not related to the SQS
> > > > > > integrations in the Airflow amazon provider
> (hook,operator,sensor)
> > > > > > I also don't see ValKey listed as option for brokers in
> > > > > >
> > > > > >
> > > >
> > >
> >
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> > > <
> > >
> >
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> > > >
> > > > > >
> > > > > > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <jarek@potiuk.com
> > > <ma...@potiuk.com>>
> > > > wrote:
> > > > > >
> > > > > > > Yes. Valkey is a good replacement/parallel implementation to
> > Redis.
> > > > One
> > > > > > > thing however is important - I would see this even more useful
> if
> > > it
> > > > also
> > > > > > > allows for the Celery Redis replacement for broker - means that
> > it
> > > is
> > > > > > fully
> > > > > > > supported in our:
> > > > > > >
> > > > > > > * unit tests (of course)
> > > > > > > * integration tests with ("valkey") integration - including
> > running
> > > > > > > docker-compose integration to run in CI
> > > > > > > * Helm Chart support to run valley as Celery Broker instead of
> > > Redis
> > > > > > >
> > > > > > > I think I'd only consider valley as "successful" provider
> > > > integration if
> > > > > > > all that above is completed.
> > > > > > >
> > > > > > > J.
> > > > > > >
> > > > > > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean
> > > > <ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I’d like to propose the addition of a new provider to support
> > > > Valkey<
> > > > > > > > https://valkey.io/> <https://valkey.io/>>: a fully
> > > open-source, community-governed fork
> > > > of
> > > > > > > > Redis 7.2, now maintained under the Linux Foundation.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Valkey has been gaining traction recently, in part due to
> > changes
> > > > by
> > > > > > > Redis
> > > > > > > > moving their license to one unrecognized by the Open Source
> > > > Initiative.
> > > > > > > > Valkey represents a future-proof alternative. The proposed
> > > provider
> > > > > > would
> > > > > > > > closely mirror the existing Redis provider in functionality
> and
> > > > serve
> > > > > > as
> > > > > > > a
> > > > > > > > drop-in, OSS-aligned option for workflows that depend on
> > > key-value
> > > > > > > storage,
> > > > > > > > pub/sub, or caching patterns.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Tentatively, here are the components I plan on implementing
> > > > (mirroring
> > > > > > > the
> > > > > > > > Redis provider):
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > * ValkeyHook – Manage connections and interactions with the
> > > > Valkey
> > > > > > > > server, enabling reuse across operators and sensors.
> > > > > > > > * ValkeyPublishOperator – Allows tasks to publish messages to
> > > > > > Valkey
> > > > > > > > pub/sub channels, useful for event-driven workflows or
> > inter-task
> > > > > > > signaling.
> > > > > > > > * ValkeyKeySensor – Waits for a specific key to exist or
> reach
> > > > a
> > > > > > > > value, supporting synchronization patterns where one task's
> > > output
> > > > > > > controls
> > > > > > > > downstream logic.
> > > > > > > > * ValkeyPubSubSensor – Subscribes to a Valkey channel and
> > > > triggers
> > > > > > on
> > > > > > > > incoming messages, enabling real-time DAG triggering.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Here's an example of how we could use the components to
> > publish a
> > > > > > message
> > > > > > > > and then trigger a DAG:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > `
> > > > > > > >
> > > > > > > > publish_message = ValkeyPublishOperator(
> > > > > > > >
> > > > > > > > task_id="publish_ready_signal",
> > > > > > > >
> > > > > > > > channel="processing_status",
> > > > > > > >
> > > > > > > > message="READY",
> > > > > > > >
> > > > > > > > valkey_conn_id="valkey_default",
> > > > > > > >
> > > > > > > > )
> > > > > > > >
> > > > > > > > `
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > This task publishes the message "READY" to the
> > processing_status
> > > > > > channel
> > > > > > > > on a Valkey server. Other processes listening to this channel
> > > will
> > > > > > > receive
> > > > > > > > the message and proceed with their execution.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > `
> > > > > > > >
> > > > > > > > wait_for_ready = ValkeyPubSubSensor(
> > > > > > > >
> > > > > > > > task_id="wait_for_ready_message",
> > > > > > > >
> > > > > > > > channel="processing_status",
> > > > > > > >
> > > > > > > > valkey_conn_id="valkey_default",
> > > > > > > >
> > > > > > > > timeout=600,
> > > > > > > >
> > > > > > > > )
> > > > > > > >
> > > > > > > > `
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Here, a sensor waits for the message on the same channel
> before
> > > > > > > > continuing. Once the message is received, the DAG can
> continue
> > > with
> > > > > > > further
> > > > > > > > tasks—such as triggering another DAG—in an event-driven
> manner.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Given the community's interest in event-driven scheduling
> (AIP
> > > 82<
> > > > > > > > https://github.com/apache/airflow/issues/52712> <
> > > https://github.com/apache/airflow/issues/52712>>) as well as
> > > > concerns
> > > > > > > > about limited provider optionality (PR #52712<
> > > > > > > > https://github.com/apache/airflow/issues/52712> <
> > > https://github.com/apache/airflow/issues/52712>>), a Valkey
> > > > provider
> > > > > > > would
> > > > > > > > expand user choice.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I’d be happy to begin working on the initial PR and
> > > documentation.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Looking forward to feedback.
> > > > > > > >
> > > > > > > > Sean
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> > > dev-unsubscribe@airflow.apache.org>
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> <mailto:
> > > dev-help@airflow.apache.org>
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
>
Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
Posted by Elad Kalif <el...@apache.org>.
at this point I am still -1
On Wed, Aug 13, 2025 at 1:43 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> If you would like to contribute - start a PR and get a VOTE on it. That's
> what most people do when they want to add a new provider or call for LAZY
> CONSENSUS if you think consensus is reached and you can get it by the lazy
> consensus https://www.apache.org/foundation/voting.html
>
> On Wed, Aug 13, 2025 at 12:38 AM Ghaeli, Sean <gh...@amazon.com.invalid>
> wrote:
>
> > Valkey checks off many boxes that align with Airflow’s provider criteria:
> >
> > * Truly open-source
> > * Dedicated maintainers responsive to community questions
> > * Not a commercial service that could maintain their own provider
> > * Very lean dependencies:
> > https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt
> > * Popular and actively maintained with broad community support:
> > https://github.com/valkey-io/valkey
> >
> > Given all of these, I believe the Valkey provider makes sense as an
> > official Airflow provider.
> >
> > Sean
> >
> > On 2025-07-15, 1:22 PM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> > jarek@potiuk.com>> wrote:
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> > le contenu ne présente aucun risque.
> >
> >
> >
> >
> >
> >
> > On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock <dheilbock@gmx.net.inva
> > <ma...@gmx.net.inva>lid>
> > wrote:
> >
> >
> > > As of last year, the redis Python client was still fully compatible
> with
> > > Valkey, has this changed? This would make it even easier. I think they
> > also
> > > forked redis-py
> >
> >
> >
> >
> > Yep. I think that's part of the solution to make the dependency tree for
> > the provider independent of redis-controlled dependencies.
> >
> >
> > Governance, decision making and release process of valley should be
> > separate from redis (the company). That's for me one of the criteria and
> > requirements for having a separate valley provider. Very much like
> > "opensearch" is separate from "elasticsearch".
> >
> >
> >
> >
> >
> >
> > >
> >
> >
> > > Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr
> > > > Von: "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com>>
> > > > An: dev@airflow.apache.org <ma...@airflow.apache.org>
> > > > Cc: ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid
> > > > Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
> > > >
> > > > > Can you please explain why it should be a provider managed by
> Airflow
> > > > rather than on your own?
> > > >
> > > > Just to explain it Elad - and yes, I know I am usually the one that
> is
> > > > sceptical -> valley is truly community driven replacement for redis
> > > > https://valkey.io/participants/ <https://valkey.io/participants/> -
> > you see a lot of well-known community
> > > > recognised entities, and there is no single entity behind it that
> could
> > > > take "responsibility" - while there are enough stakeholders to make
> > sure
> > > > that it will be maintained here in Airflow community. Also it's a
> > > "drop-in"
> > > > replacement for redis generally - and I do not expect a lot of
> problems
> > > > when we start with both redis provider and redis celery integration.
> > > >
> > > > I've heard and saw a lot of praise from the community for Valkey
> being
> > > good
> > > > redis replacement, and - even we at the PMC and devlist discussions
> > > > discussed about the dangers Redis licence creates for that - even now
> > we
> > > > limited Redis support in Helm chart of ours to the "really free"
> > version
> > > > and we discussed that Valkey might be a good way forward - especially
> > if
> > > > someone wants to contribute support for it. so if someone wants to
> > > > contribute to Airflow AND make sure that Redis broker support in
> Celery
> > > is
> > > > covered - I am all for it.
> > > >
> > > >
> > > > On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <eladkal@gmail.com
> <mailto:
> > eladkal@gmail.com>> wrote:
> > > >
> > > > > > * Helm Chart support to run valley as Celery Broker instead of
> > Redis
> > > > >
> > > > > Oh yes! didn't consider its usage as a broker but is it related to
> > the
> > > > > provider package?
> > > > > Amazon SQS can be used as broker but it's not related to the SQS
> > > > > integrations in the Airflow amazon provider (hook,operator,sensor)
> > > > > I also don't see ValKey listed as option for brokers in
> > > > >
> > > > >
> > >
> >
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> > <
> >
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> > >
> > > > >
> > > > > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <jarek@potiuk.com
> > <ma...@potiuk.com>>
> > > wrote:
> > > > >
> > > > > > Yes. Valkey is a good replacement/parallel implementation to
> Redis.
> > > One
> > > > > > thing however is important - I would see this even more useful if
> > it
> > > also
> > > > > > allows for the Celery Redis replacement for broker - means that
> it
> > is
> > > > > fully
> > > > > > supported in our:
> > > > > >
> > > > > > * unit tests (of course)
> > > > > > * integration tests with ("valkey") integration - including
> running
> > > > > > docker-compose integration to run in CI
> > > > > > * Helm Chart support to run valley as Celery Broker instead of
> > Redis
> > > > > >
> > > > > > I think I'd only consider valley as "successful" provider
> > > integration if
> > > > > > all that above is completed.
> > > > > >
> > > > > > J.
> > > > > >
> > > > > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean
> > > <ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I’d like to propose the addition of a new provider to support
> > > Valkey<
> > > > > > > https://valkey.io/> <https://valkey.io/>>: a fully
> > open-source, community-governed fork
> > > of
> > > > > > > Redis 7.2, now maintained under the Linux Foundation.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Valkey has been gaining traction recently, in part due to
> changes
> > > by
> > > > > > Redis
> > > > > > > moving their license to one unrecognized by the Open Source
> > > Initiative.
> > > > > > > Valkey represents a future-proof alternative. The proposed
> > provider
> > > > > would
> > > > > > > closely mirror the existing Redis provider in functionality and
> > > serve
> > > > > as
> > > > > > a
> > > > > > > drop-in, OSS-aligned option for workflows that depend on
> > key-value
> > > > > > storage,
> > > > > > > pub/sub, or caching patterns.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Tentatively, here are the components I plan on implementing
> > > (mirroring
> > > > > > the
> > > > > > > Redis provider):
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > * ValkeyHook – Manage connections and interactions with the
> > > Valkey
> > > > > > > server, enabling reuse across operators and sensors.
> > > > > > > * ValkeyPublishOperator – Allows tasks to publish messages to
> > > > > Valkey
> > > > > > > pub/sub channels, useful for event-driven workflows or
> inter-task
> > > > > > signaling.
> > > > > > > * ValkeyKeySensor – Waits for a specific key to exist or reach
> > > a
> > > > > > > value, supporting synchronization patterns where one task's
> > output
> > > > > > controls
> > > > > > > downstream logic.
> > > > > > > * ValkeyPubSubSensor – Subscribes to a Valkey channel and
> > > triggers
> > > > > on
> > > > > > > incoming messages, enabling real-time DAG triggering.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Here's an example of how we could use the components to
> publish a
> > > > > message
> > > > > > > and then trigger a DAG:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > `
> > > > > > >
> > > > > > > publish_message = ValkeyPublishOperator(
> > > > > > >
> > > > > > > task_id="publish_ready_signal",
> > > > > > >
> > > > > > > channel="processing_status",
> > > > > > >
> > > > > > > message="READY",
> > > > > > >
> > > > > > > valkey_conn_id="valkey_default",
> > > > > > >
> > > > > > > )
> > > > > > >
> > > > > > > `
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > This task publishes the message "READY" to the
> processing_status
> > > > > channel
> > > > > > > on a Valkey server. Other processes listening to this channel
> > will
> > > > > > receive
> > > > > > > the message and proceed with their execution.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > `
> > > > > > >
> > > > > > > wait_for_ready = ValkeyPubSubSensor(
> > > > > > >
> > > > > > > task_id="wait_for_ready_message",
> > > > > > >
> > > > > > > channel="processing_status",
> > > > > > >
> > > > > > > valkey_conn_id="valkey_default",
> > > > > > >
> > > > > > > timeout=600,
> > > > > > >
> > > > > > > )
> > > > > > >
> > > > > > > `
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Here, a sensor waits for the message on the same channel before
> > > > > > > continuing. Once the message is received, the DAG can continue
> > with
> > > > > > further
> > > > > > > tasks—such as triggering another DAG—in an event-driven manner.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Given the community's interest in event-driven scheduling (AIP
> > 82<
> > > > > > > https://github.com/apache/airflow/issues/52712> <
> > https://github.com/apache/airflow/issues/52712>>) as well as
> > > concerns
> > > > > > > about limited provider optionality (PR #52712<
> > > > > > > https://github.com/apache/airflow/issues/52712> <
> > https://github.com/apache/airflow/issues/52712>>), a Valkey
> > > provider
> > > > > > would
> > > > > > > expand user choice.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I’d be happy to begin working on the initial PR and
> > documentation.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Looking forward to feedback.
> > > > > > >
> > > > > > > Sean
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> > dev-unsubscribe@airflow.apache.org>
> > > For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> > dev-help@airflow.apache.org>
> > >
> > >
> >
> >
> >
> >
>
Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
Posted by Jarek Potiuk <ja...@potiuk.com>.
If you would like to contribute - start a PR and get a VOTE on it. That's
what most people do when they want to add a new provider or call for LAZY
CONSENSUS if you think consensus is reached and you can get it by the lazy
consensus https://www.apache.org/foundation/voting.html
On Wed, Aug 13, 2025 at 12:38 AM Ghaeli, Sean <gh...@amazon.com.invalid>
wrote:
> Valkey checks off many boxes that align with Airflow’s provider criteria:
>
> * Truly open-source
> * Dedicated maintainers responsive to community questions
> * Not a commercial service that could maintain their own provider
> * Very lean dependencies:
> https://github.com/valkey-io/valkey/blob/unstable/src/CMakeLists.txt
> * Popular and actively maintained with broad community support:
> https://github.com/valkey-io/valkey
>
> Given all of these, I believe the Valkey provider makes sense as an
> official Airflow provider.
>
> Sean
>
> On 2025-07-15, 1:22 PM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> jarek@potiuk.com>> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
>
>
>
> On Tue, Jul 15, 2025 at 9:52 PM Dominik Heilbock <dheilbock@gmx.net.inva
> <ma...@gmx.net.inva>lid>
> wrote:
>
>
> > As of last year, the redis Python client was still fully compatible with
> > Valkey, has this changed? This would make it even easier. I think they
> also
> > forked redis-py
>
>
>
>
> Yep. I think that's part of the solution to make the dependency tree for
> the provider independent of redis-controlled dependencies.
>
>
> Governance, decision making and release process of valley should be
> separate from redis (the company). That's for me one of the criteria and
> requirements for having a separate valley provider. Very much like
> "opensearch" is separate from "elasticsearch".
>
>
>
>
>
>
> >
>
>
> > Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr
> > > Von: "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com>>
> > > An: dev@airflow.apache.org <ma...@airflow.apache.org>
> > > Cc: ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid
> > > Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
> > >
> > > > Can you please explain why it should be a provider managed by Airflow
> > > rather than on your own?
> > >
> > > Just to explain it Elad - and yes, I know I am usually the one that is
> > > sceptical -> valley is truly community driven replacement for redis
> > > https://valkey.io/participants/ <https://valkey.io/participants/> -
> you see a lot of well-known community
> > > recognised entities, and there is no single entity behind it that could
> > > take "responsibility" - while there are enough stakeholders to make
> sure
> > > that it will be maintained here in Airflow community. Also it's a
> > "drop-in"
> > > replacement for redis generally - and I do not expect a lot of problems
> > > when we start with both redis provider and redis celery integration.
> > >
> > > I've heard and saw a lot of praise from the community for Valkey being
> > good
> > > redis replacement, and - even we at the PMC and devlist discussions
> > > discussed about the dangers Redis licence creates for that - even now
> we
> > > limited Redis support in Helm chart of ours to the "really free"
> version
> > > and we discussed that Valkey might be a good way forward - especially
> if
> > > someone wants to contribute support for it. so if someone wants to
> > > contribute to Airflow AND make sure that Redis broker support in Celery
> > is
> > > covered - I am all for it.
> > >
> > >
> > > On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <eladkal@gmail.com <mailto:
> eladkal@gmail.com>> wrote:
> > >
> > > > > * Helm Chart support to run valley as Celery Broker instead of
> Redis
> > > >
> > > > Oh yes! didn't consider its usage as a broker but is it related to
> the
> > > > provider package?
> > > > Amazon SQS can be used as broker but it's not related to the SQS
> > > > integrations in the Airflow amazon provider (hook,operator,sensor)
> > > > I also don't see ValKey listed as option for brokers in
> > > >
> > > >
> >
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> <
> https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> >
> > > >
> > > > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <jarek@potiuk.com
> <ma...@potiuk.com>>
> > wrote:
> > > >
> > > > > Yes. Valkey is a good replacement/parallel implementation to Redis.
> > One
> > > > > thing however is important - I would see this even more useful if
> it
> > also
> > > > > allows for the Celery Redis replacement for broker - means that it
> is
> > > > fully
> > > > > supported in our:
> > > > >
> > > > > * unit tests (of course)
> > > > > * integration tests with ("valkey") integration - including running
> > > > > docker-compose integration to run in CI
> > > > > * Helm Chart support to run valley as Celery Broker instead of
> Redis
> > > > >
> > > > > I think I'd only consider valley as "successful" provider
> > integration if
> > > > > all that above is completed.
> > > > >
> > > > > J.
> > > > >
> > > > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean
> > <ghaeli@amazon.com.inva <ma...@amazon.com.inva>lid>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > >
> > > > > >
> > > > > > I’d like to propose the addition of a new provider to support
> > Valkey<
> > > > > > https://valkey.io/> <https://valkey.io/>>: a fully
> open-source, community-governed fork
> > of
> > > > > > Redis 7.2, now maintained under the Linux Foundation.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Valkey has been gaining traction recently, in part due to changes
> > by
> > > > > Redis
> > > > > > moving their license to one unrecognized by the Open Source
> > Initiative.
> > > > > > Valkey represents a future-proof alternative. The proposed
> provider
> > > > would
> > > > > > closely mirror the existing Redis provider in functionality and
> > serve
> > > > as
> > > > > a
> > > > > > drop-in, OSS-aligned option for workflows that depend on
> key-value
> > > > > storage,
> > > > > > pub/sub, or caching patterns.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Tentatively, here are the components I plan on implementing
> > (mirroring
> > > > > the
> > > > > > Redis provider):
> > > > > >
> > > > > >
> > > > > >
> > > > > > * ValkeyHook – Manage connections and interactions with the
> > Valkey
> > > > > > server, enabling reuse across operators and sensors.
> > > > > > * ValkeyPublishOperator – Allows tasks to publish messages to
> > > > Valkey
> > > > > > pub/sub channels, useful for event-driven workflows or inter-task
> > > > > signaling.
> > > > > > * ValkeyKeySensor – Waits for a specific key to exist or reach
> > a
> > > > > > value, supporting synchronization patterns where one task's
> output
> > > > > controls
> > > > > > downstream logic.
> > > > > > * ValkeyPubSubSensor – Subscribes to a Valkey channel and
> > triggers
> > > > on
> > > > > > incoming messages, enabling real-time DAG triggering.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Here's an example of how we could use the components to publish a
> > > > message
> > > > > > and then trigger a DAG:
> > > > > >
> > > > > >
> > > > > >
> > > > > > `
> > > > > >
> > > > > > publish_message = ValkeyPublishOperator(
> > > > > >
> > > > > > task_id="publish_ready_signal",
> > > > > >
> > > > > > channel="processing_status",
> > > > > >
> > > > > > message="READY",
> > > > > >
> > > > > > valkey_conn_id="valkey_default",
> > > > > >
> > > > > > )
> > > > > >
> > > > > > `
> > > > > >
> > > > > >
> > > > > >
> > > > > > This task publishes the message "READY" to the processing_status
> > > > channel
> > > > > > on a Valkey server. Other processes listening to this channel
> will
> > > > > receive
> > > > > > the message and proceed with their execution.
> > > > > >
> > > > > >
> > > > > >
> > > > > > `
> > > > > >
> > > > > > wait_for_ready = ValkeyPubSubSensor(
> > > > > >
> > > > > > task_id="wait_for_ready_message",
> > > > > >
> > > > > > channel="processing_status",
> > > > > >
> > > > > > valkey_conn_id="valkey_default",
> > > > > >
> > > > > > timeout=600,
> > > > > >
> > > > > > )
> > > > > >
> > > > > > `
> > > > > >
> > > > > >
> > > > > >
> > > > > > Here, a sensor waits for the message on the same channel before
> > > > > > continuing. Once the message is received, the DAG can continue
> with
> > > > > further
> > > > > > tasks—such as triggering another DAG—in an event-driven manner.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Given the community's interest in event-driven scheduling (AIP
> 82<
> > > > > > https://github.com/apache/airflow/issues/52712> <
> https://github.com/apache/airflow/issues/52712>>) as well as
> > concerns
> > > > > > about limited provider optionality (PR #52712<
> > > > > > https://github.com/apache/airflow/issues/52712> <
> https://github.com/apache/airflow/issues/52712>>), a Valkey
> > provider
> > > > > would
> > > > > > expand user choice.
> > > > > >
> > > > > >
> > > > > >
> > > > > > I’d be happy to begin working on the initial PR and
> documentation.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Looking forward to feedback.
> > > > > >
> > > > > > Sean
> > > > > >
> > > > > >
> > > > >
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> dev-unsubscribe@airflow.apache.org>
> > For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> dev-help@airflow.apache.org>
> >
> >
>
>
>
>