You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Pavankumar Gopidesu <go...@gmail.com> on 2026/07/05 14:15:46 UTC
New Community Provider: Data Quality
Hi Airflow community,
I would like to start a discussion regarding a new provider:
apache-airflow-providers-dq.
While Airflow already includes SQL check operators that many users rely on
for data quality, this new provider builds on that foundation by
introducing DQRule and RuleSet objects, stable rule identity, persisted
history, and direct connections to Airflow assets. This approach makes
quality results easier to inspect over time, allows downstream consumers to
gate tasks based on recent quality results, and provides a unified schema
for LLM-assisted workflows. Execution will continue to utilize existing
DbApiHook connections.
The initial version of the provider is intentionally focused:
- Declarative DQRule and RuleSet objects.
- DQCheckOperator and @task.dq_check.
- DbApiHook-based SQL checks, including built-in checks and custom_sql.
- Persisted results for tasks, runs, and rules.
- A minimal Airflow UI plugin for viewing results and rule history.
- Experimental asset helpers such as asset_quality() and
require_quality().
Regarding scope, this first iteration uses object storage only to persist
DQ results and history; checks are executed via database connections.
Future iterations may include file or object-store based checks (e.g., S3,
GCS) where Airflow runs quality rules against data directly.
This proposal does not require changes to Airflow core. Asset support is
currently provider-owned metadata, with static configuration stored on the
asset and runtime summaries stored on asset events. If the provider gains
traction, we can discuss making Data Quality a first-class component of
Airflow assets.
This work also serves as a practical follow-up to the data quality
direction mentioned in AIP-99. Persisted history is valuable for users and
future LLM-assisted workflows, such as those from Anthropic or common.ai,
to understand rule performance and generate candidate rules based on schema
context.
A rough pseudo-flow is provided below:
seed_rules = RuleSet(
name="orders_quality",
rules=[
DQRule(name="order_id_not_null", check="null_count",
column="order_id", condition={"equal_to": 0}),
DQRule(name="amount_valid", check="min", column="amount",
condition={"geq_to": 0}),
],
)
orders_asset = asset_quality(
Asset("orders"),
conn_id="warehouse",
table="orders",
ruleset=seed_rules,
)
# Optional: common.ai / Anthropic provider can generate a RuleSet from
schema context.
generated_rules = generate_rules_from_schema(...)
@task.dq_check(asset=orders_asset)
def check_orders(ruleset):
return ruleset
checked_orders = check_orders(generated_rules)
with DAG("orders_consumer", schedule=orders_asset):
require_quality(orders_asset, min_score=0.95) >> consume_orders()
The UI remains deliberately minimal for this initial release, focusing on
result and history inspection.
You can view examples [1] of how it's integrated with assets/llms.
currently i named it providers `apache-airflow-providers-dq`. if any other
preference likely with `dataquality`. Please let me know if you have a
preference. naming is hard :)
[1]:
https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
Thanks,
Pavan
Re: New Community Provider: Data Quality
Posted by Pavankumar Gopidesu <go...@gmail.com>.
Yes, the primary target is platform-leaning data practitioners: operational
checks (nulls, uniqueness, volume, freshness) at pipeline boundaries, with
the built-in catalog and defaults designed around that use case.
Analytics-leaning folks can still consume it through YAML rulesets and
custom_sql, but domain-rich business validation and complex derivations
stay out of scope — we won't build a catalog for them. That said, if
someone can express those derivations purely as custom_sql rules, I
wouldn't mind at all, the provider just provides the helpers, and with
agents these days, folding queries that join multiple tables and carry
their business logic is getting pretty easy :)
Regards,
Pavan.
On Fri, Jul 17, 2026 at 1:15 AM Sameer Mesiah <sm...@gmail.com> wrote:
> Okay. I think you addressed points 1) and 3) very well.
>
> I believe for 2), it seems like we were already aligned. So this is
> fundamentally a provider intended for platform-leaning data practitioners?
> Is that correct? In that case, I would flip my vote to +1.
>
> Thanks for taking out the time to address my questions. It was interesting
> hearing about your future plans for this provider (especially DataFusion).
>
> On Fri, 17 Jul 2026 at 01:00, Pavankumar Gopidesu <gopidesupavan@gmail.com
> >
> wrote:
>
> > Hi Sameer,
> >
> > Thanks for taking the time to go through the PR, glad you have questions,
> > let's discuss. My responses are below, sorry for the long response.
> >
> > Firstly, a side note: this is not an AI response; I typed it completely
> but
> > used AI for rephrasing. I am not a native English speaker. :)
> >
> > 1) Provider philosophy — framework within Airflow vs.
> > integration/abstraction
> >
> > I agree providers come with various levels of integration and
> abstraction,
> > but I wouldn't call this a framework at all. All we are doing is defining
> > rules as declarative data, providing a few helper functions, and
> persisting
> > the results. The core execution sits entirely on the common.sql DbApiHook
> > interface, so any provider that implements that interface can execute the
> > checks, nothing is reimplemented inside this provider. In that sense,
> this
> > provider is a consumer of common.sql, exactly the way many providers
> today
> > already are.
> >
> > On why a separate provider rather than folding it into an existing common
> > one, a few concrete reasons:
> >
> > - It will grow beyond SQL execution. The next engine I'm working on is
> > DataFusion, for running checks directly against ingested data in
> object
> > stores (a skeleton PR is already in progress). DataFusion is an
> > excellent
> > fit for exactly this workload — I use it day to day, and it routinely
> > returns results over millions of records in milliseconds — so quality
> > checks on raw ingested data need no heavy machinery: no Spark cluster,
> > no
> > separate compute job, just a fast scan right after ingestion, before
> the
> > data ever reaches a warehouse. That engine would be an odd thing to
> > graft
> > onto common.sql, whose scope is the DB-API interface.
> > - It composes with the AI providers. Because rules are declarative
> > specs, the anthropic and common.ai providers can generate rule specs
> > against a schema — an integration that belongs with the quality
> domain,
> > not
> > inside any of the providers it draws on.
> > - The boundary is the domain, not a service. Everything here is
> > quality-related work — rules, results, scoring, gating — hence the
> name.
> > Placing it in its own package keeps that domain in one place, evolving
> > on
> > its own cadence, without disrupting existing common providers that
> > nearly
> > everything depends on.
> >
> > So the shape is: one small quality-domain provider that consumes existing
> > abstractions (common.sql today, DataFusion next, object storage for
> > results, AI providers for rule generation) rather than a framework that
> > reimplements any of them.
> >
> > 2) Target user ?
> >
> > Fair point on being explicit about the audience. The provider ships with
> a
> > set of built-in operational checks (nulls, uniqueness, volume, min/max)
> > plus custom_sql on top — and I'd note that the declarative API you're
> > describing already exists: rules are data, not code. A ruleset can be a
> > plain YAML file loaded at parse time, and extending checks means adding a
> > custom_sql rule in your Dag repo — never modifying provider internals.
> >
> > On platform-leaning vs. analytics-leaning: I take the distinction, though
> > in the teams I've worked with, the line keeps blurring — engineers end up
> > owning more of their stack end to end. The way I'd frame the question is
> > fit for purpose: does this help me do the task without investing in a
> > larger job or integration? If I have simple queries and declarative
> checks,
> > this provider is that fit.
> >
> > On domain-rich validation and dbt: everyone has their own way of
> > implementing business logic, and where it runs is their choice — if dbt,
> > GX, or Soda is the right fit there, use them; we are not promising
> > domain-rich validation and we are not competing with those tools. But
> > consider the case where I'm a consumer and you're a producer who promised
> > to send certain fields with certain values. To verify that contract with
> a
> > few simple checks, I wouldn't stand up dbt or GX. Airflow already ships
> > everything needed to run those checks — what's been missing is a unified
> > entry point to define them. That's what this provider builds.
> >
> > And once rules have a declarative definition and durable per-rule
> execution
> > history, something else opens up: agents are very good at generating
> rules
> > against a schema, and at answering questions like "how has my producer's
> > feed quality looked over the last month?" — which is not straightforward
> to
> > answer today. With the stored execution history per asset, an agent can
> > answer that and even suggest additional rules. The provider ships that
> > durable view.
> >
> > So overall: it's about fit for purpose. We're promising simple,
> declarative
> > operational checks over Airflow's existing provider interfaces — execute
> > rules, get a score, gate your business flow on it. And if someone finds
> it
> > useful for checks after their transformations too, there's no harm in
> that.
> >
> > 3) Maintenance surface:
> >
> > This follows from (1): what we'd own is a small declarative layer, not a
> > framework. Rules are plain models with no behavior, results go through
> the
> > existing object storage abstraction, scoring is simple arithmetic — no
> new
> > tables, no migrations, no scheduler, API server, or UI changes. And "does
> > the value justify the scope" is the right test: everything here reuses
> > machinery Airflow already ships, so if users genuinely benefit, this thin
> > layer is a cost worth carrying. If they don't, we'll learn that quickly
> and
> > cheaply — the provider ships in the incubation state, and our provider
> > governance model lets us suspend or remove it entirely if it doesn't
> prove
> > useful.
> >
> > Happy to discuss any of this further — and thanks again for the detailed
> > review, it genuinely helps sharpen the proposal.
> >
> > Regards,
> > Pavan
> >
> > On Thu, Jul 16, 2026 at 10:21 PM Sameer Mesiah <sm...@gmail.com>
> > wrote:
> >
> > > Hi Pavan,
> > >
> > > First, I would like to say that this new provider definitely adds value
> > and
> > > I liked the brief walkthrough you did today during the call. But after
> > > going through the PR (and hearing Vikram's thoughts), there are a few
> > > concerns that popped up:
> > >
> > > 1) I'm struggling to reconcile this with the existing philosophy of
> > Airflow
> > > providers. Most providers integrate with an external service or expose
> a
> > > common abstraction over multiple services. This proposal instead
> > introduces
> > > a fairly comprehensive data quality framework that is implemented
> > entirely
> > > within Airflow. I am just wondering whether this should be a
> > > standalone provider versus additional functionality that could be added
> > to
> > > the common provider.
> > >
> > > 2) I'm not yet sure who the primary target user is. If the intended
> > > audience is Analytics Engineers, I'd expect the framework to expose a
> > > simple declarative API for defining and extending reusable checks
> without
> > > requiring changes to provider internals (you can correct me here if you
> > > intend on exposing a simplified API eventually). In many organisations,
> > > Airflow providers are owned by platform teams and are treated as
> > > third-party dependencies, so modifying provider code to introduce new
> > > reusable checks is often not a practical workflow. Arguably, the case
> for
> > > this new framework is stronger if you are not going too deep into
> > > domain-rich business rules but just want to limit it to operational
> > checks
> > > data platform owners might be interested in. To ground my critique
> with a
> > > very straightforward question: if you intend for this new provider to
> > > support domain-rich data validation, what benefit would it have over
> > tools
> > > like DBT which are more accessible to practitioners of this specific
> area
> > > i.e Analytics Engineers. Vikram did share the perspective of a Data
> > > Engineer who is building data pipelines but I just wanted to expand on
> > that
> > > a bit more by differentiating between the needs of platform-leaning and
> > > analytics-leaning Data Engineers. These 2 roles are increasingly
> > bifurcated
> > > and I think we should really think about which group we are targeting
> > here.
> > >
> > > 3) This proposal introduces a significant amount of new framework
> surface
> > > area that Airflow will own and maintain going forward. Beyond the
> > operator
> > > itself, we're introducing rule definitions, persistence, history,
> > scoring,
> > > asset integration, decorators, configuration, etc. That represents a
> > > substantial long-term maintenance commitment, so I think it's worth
> > > discussing whether the value provided justifies that scope and whether
> > this
> > > is the right architectural boundary for Airflow.
> > >
> > > Overall, I would say I am +0.5. It's a good proposal but it needs more
> > > refinement because we do not want to introduce a provider that our
> users
> > > are just not going to want to use.
> > >
> > > On Wed, 15 Jul 2026 at 21:27, Pavankumar Gopidesu <
> > gopidesupavan@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi Vikram,
> > > >
> > > > Thanks for the feedback. I agree that helpers would be useful for
> users
> > > who
> > > > want to run data quality checks from existing tasks or operators,
> > without
> > > > always needing to use a dedicated DQ operator.
> > > >
> > > > I updated the PR to include
> > > >
> > > > - importable helpers for running and persisting checks inside
> tasks
> > or
> > > > any existing operators.
> > > >
> > > > For asset integration, this PR does not require Airflow core changes.
> > The
> > > > provider adds two asset-oriented helpers:
> > > >
> > > > - asset_quality() to attach quality configuration to an asset.
> This
> > > lets
> > > > the asset carry the ruleset that is expected to be executed for
> it.
> > In
> > > > the
> > > > current implementation this is stored as provider-owned metadata
> in
> > > the
> > > > asset extra field.
> > > > - require_quality() to let downstream tasks gate on the latest
> > quality
> > > > result from the triggering asset event.
> > > >
> > > > Below is an example view of how assets can use data quality with
> > > > partition-aware producer and consumer Dags.
> > > >
> > > > ```
> > > >
> > > > *Example ruleset:*
> > > >
> > > > orders_quality_rules = RuleSet(
> > > > name="orders_quality",
> > > > rules=[
> > > > DQRule(
> > > > id="orders.amount_non_negative",
> > > > name="amount_non_negative",
> > > > check="min",
> > > > column="amount",
> > > > condition={"geq_to": 0},
> > > > ),
> > > > ],
> > > > )
> > > >
> > > > raw_orders = asset_quality(
> > > > Asset("daily_raw_orders"),
> > > > ruleset=orders_quality_rules,
> > > > conn_id="warehouse",
> > > > table="raw_orders",
> > > > )
> > > >
> > > > *Producer Dag:*
> > > >
> > > > with DAG(
> > > > dag_id="daily_raw_orders",
> > > > schedule=CronPartitionTimetable("0 2 * * *", timezone="UTC"),
> > > > ):
> > > > @task(outlets=[raw_orders])
> > > > def load_and_check():
> > > > context = get_current_context()
> > > > partition_key = context["partition_key"]
> > > >
> > > > load_raw_orders(partition_key)
> > > >
> > > > result = run_quality_checks(
> > > > ruleset=orders_quality_rules,
> > > > table="raw_orders",
> > > > partition_clause=f"ds = '{partition_key}'",
> > > > )
> > > >
> > > > persist_quality_results(
> > > > result,
> > > > context=context,
> > > > outlets=[raw_orders],
> > > > )
> > > >
> > > > load_and_check()
> > > >
> > > >
> > > > *Consumer Dag:*
> > > >
> > > > with DAG(
> > > > dag_id="curate_daily_orders",
> > > > schedule=PartitionedAssetTimetable(assets=raw_orders),
> > > > ):
> > > > quality_gate = require_quality(raw_orders, min_score=1.0)
> > > >
> > > > @task
> > > > def curate():
> > > > context = get_current_context()
> > > > partition_key = context["partition_key"]
> > > >
> > > > curate_orders(partition_key)
> > > >
> > > > quality_gate >> curate()
> > > >
> > > > ```
> > > >
> > > > The flow is:
> > > >
> > > > 1. A partitioned producer Dag runs and loads data for the current
> > > > partition.
> > > > 2. The producer runs run_quality_checks() with a partition-scoped
> > > > filter.
> > > > 3. persist_quality_results() records the result and adds a DQ
> > summary
> > > to
> > > > the produced asset event.
> > > > 4. PartitionedAssetTimetable triggers the consumer Dag for that
> > > > partition.
> > > > 5. require_quality() reads the DQ summary from the triggering
> asset
> > > > event.
> > > > 6. The consumer processes the partition only if quality passes.
> > > >
> > > > Hope this makes sense?
> > > >
> > > > Regards,
> > > >
> > > > Pavan
> > > >
> > > >
> > > > On Wed, Jul 15, 2026 at 6:47 PM Vikram Koka via dev <
> > > > dev@airflow.apache.org>
> > > > wrote:
> > > >
> > > > > Hey Pavan,
> > > > >
> > > > > Thanks for bringing this up and working on it. I strongly support
> the
> > > > > overall concept of the Data Quality provider.
> > > > >
> > > > > However, I have two concerns / questions:
> > > > > 1. I am unsure of the Operator approach shown in the PR.
> > > > > I am not certain that these SHOULD be separate Operators which
> result
> > > in
> > > > > separate tasks.
> > > > > If these are made available as Helper functions, they could be
> > included
> > > > in
> > > > > existing Operators.
> > > > > I am not saying they shouldn't be separate Tasks in Dags, but the
> > > Helper
> > > > > function approach would provide more optionality.
> > > > >
> > > > > 2. I am wondering how to integrate these with Asset Partitions
> > > > > I do believe this can be integrated with the Asset Partitions
> > concept,
> > > > but
> > > > > I am personally unclear on exactly how.
> > > > > I think adding an example illustrating this would be very helpful.
> > > > >
> > > > > Best regards,
> > > > > Vikram
> > > > >
> > > > >
> > > > > On Mon, Jul 13, 2026 at 3:26 PM Pavankumar Gopidesu <
> > > > > gopidesupavan@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks Kaxil..
> > > > > >
> > > > > > Thanks Everyone, consensus looks positive; I will merge the
> > skeleton
> > > PR
> > > > > > first.
> > > > > >
> > > > > > Regards,
> > > > > > Pavan
> > > > > >
> > > > > > On Mon, Jul 13, 2026 at 10:05 PM Kaxil Naik <kaxilnaik@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > +1 for the high-level need for such data quality checks. Thanks
> > > > Pavan,
> > > > > > will
> > > > > > > review the PRs shortly.
> > > > > > >
> > > > > > > On Fri, 10 Jul 2026 at 01:58, Pavankumar Gopidesu <
> > > > > > gopidesupavan@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Bugra,
> > > > > > > >
> > > > > > > > I have updated the naming convention now.
> > > > > > > >
> > > > > > > > Pavan
> > > > > > > >
> > > > > > > > On Wed, Jul 8, 2026 at 7:07 PM Buğra Öztürk <
> > > > ozturkbugra93@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Pavan for bringing this together and starting the
> > > > > discussion!
> > > > > > > > > Sounds good! +1 on the idea.
> > > > > > > > >
> > > > > > > > > Harder than solving problems. Not a strong suggestion, but
> > > > > > > > > `common-dataquality` sounds more reasonable to me. It also
> > adds
> > > > the
> > > > > > > value
> > > > > > > > > of the `common` part, which provides the separation pattern
> > > Jarek
> > > > > > > > > mentioned. It gives a better understanding that it is a
> > common
> > > > > > > offering.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Bugra Ozturk
> > > > > > > > >
> > > > > > > > > On Wed, Jul 8, 2026 at 7:15 PM Pavankumar Gopidesu <
> > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks Jarek, I agree that the separate provider approach
> > > > offers
> > > > > > much
> > > > > > > > > more
> > > > > > > > > > flexibility for iterating on features and fixes.
> > > > > > > > > >
> > > > > > > > > > Naming is always hard :)
> > > > > > > > > >
> > > > > > > > > > Option 1: apache-airflow-providers-dataquality
> > > > > > > > > > Option 2: apache-airflow-providers-common-dataquality
> (This
> > > > goes
> > > > > > > inside
> > > > > > > > > the
> > > > > > > > > > common providers folder we already have)
> > > > > > > > > >
> > > > > > > > > > So, I am up for either option :)
> > > > > > > > > >
> > > > > > > > > > have removed first short name
> > `apache-airflow-providers-dq`.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Pavan
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk <
> > > jarek@potiuk.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 Good design/ idea. No objections - dataquality is a
> > good
> > > > > name
> > > > > > -
> > > > > > > > but
> > > > > > > > > I
> > > > > > > > > > > would also consider `common-dataquality" - even if it's
> > > > longer,
> > > > > > it
> > > > > > > > > builds
> > > > > > > > > > > on the pattern we have already with common-ai. But not
> a
> > > > > blocker.
> > > > > > > > > > >
> > > > > > > > > > > I also think it's good to have it as a separate
> provider,
> > > > even
> > > > > if
> > > > > > > it
> > > > > > > > > > gains
> > > > > > > > > > > traction for two reasons:
> > > > > > > > > > >
> > > > > > > > > > > a) ability to add features or fix issues independently
> > from
> > > > the
> > > > > > > core
> > > > > > > > > > > b) an explicit "optional" feature that is easy to
> promote
> > > > > > > > > > >
> > > > > > > > > > > I think what we saw with common is that people see
> > airflow
> > > > > > already
> > > > > > > as
> > > > > > > > > too
> > > > > > > > > > > heavy - and "too many releases" sometimes, so quite
> > > > > > > > > counter-intuitively -
> > > > > > > > > > > by having separate providers adding features that "hook
> > in"
> > > > > > > existing
> > > > > > > > > > > functionalities of core - we do not make airflow
> > "heavier"
> > > > and
> > > > > we
> > > > > > > do
> > > > > > > > > not
> > > > > > > > > > > force people to migrating to future newer versions to
> use
> > > new
> > > > > > > > features.
> > > > > > > > > > >
> > > > > > > > > > > J.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu <
> > > > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Amogh,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the feedback.
> > > > > > > > > > > >
> > > > > > > > > > > > I am happy to change the provider name to
> dataquality.
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding the LLM-assisted features, the current PR
> > does
> > > > not
> > > > > > > > include
> > > > > > > > > > any
> > > > > > > > > > > > implementation. It only adds the SKILLS [1 ]and the
> > > > reference
> > > > > > > > schema
> > > > > > > > > > for
> > > > > > > > > > > > the DQ Rule structure. Are you suggesting that I move
> > > this
> > > > > > SKILL
> > > > > > > > > > > > documentation to a separate PR?
> > > > > > > > > > > >
> > > > > > > > > > > > [1]:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Pavan
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai <
> > > > > > > amoghdesai@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Pavan,
> > > > > > > > > > > > >
> > > > > > > > > > > > > First of all, +1 to this.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Now, few things:
> > > > > > > > > > > > >
> > > > > > > > > > > > > * On naming: dataquality over dq for me honestly.
> Our
> > > > > > existing
> > > > > > > > > > provider
> > > > > > > > > > > > > names spell things out
> > > > > > > > > > > > > (common.sql, openlineage, not abbreviated forms)
> and
> > dq
> > > > is
> > > > > > > > > genuinely
> > > > > > > > > > > > > ambiguous outside context.
> > > > > > > > > > > > >
> > > > > > > > > > > > > * On scope: I also agree with Niko that #69413 is
> too
> > > > large
> > > > > > for
> > > > > > > > one
> > > > > > > > > > > pass
> > > > > > > > > > > > &
> > > > > > > > > > > > > I am glad to see the
> > > > > > > > > > > > > backend/UI split already happening in #69575. Would
> > > also
> > > > > > > suggest
> > > > > > > > > > > keeping
> > > > > > > > > > > > > the LLM assisted rule
> > > > > > > > > > > > > generation pieces (*schema-based
> > > > > generate_rules_from_schema*)
> > > > > > > out
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > initial provider PR entirely
> > > > > > > > > > > > > cos as I see it, its a separable capability and
> > > bundling
> > > > it
> > > > > > > will
> > > > > > > > > slow
> > > > > > > > > > > > > review of the core DQRule or
> > > > > > > > > > > > > RuleSet or operator surface, which is the part that
> > > > > actually
> > > > > > > > needs
> > > > > > > > > > the
> > > > > > > > > > > > most
> > > > > > > > > > > > > detailed review.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In short: go for it!
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks & Regards,
> > > > > > > > > > > > > Amogh Desai
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu
> <
> > > > > > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > In the meantime, the PR is ready for review. Feel
> > > free
> > > > to
> > > > > > > > review
> > > > > > > > > > and
> > > > > > > > > > > > > > provide any feedback.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > Pavan
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar
> Gopidesu
> > <
> > > > > > > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Sorry, I forgot to add: the draft PR is here
> > > > > > > > > > > > > > > https://github.com/apache/airflow/pull/69413;
> > it's
> > > > > > still a
> > > > > > > > > WIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > some screenshots
> > > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468
> > > > > > > > > > > > > :)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Pavan
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar
> > Gopidesu
> > > <
> > > > > > > > > > > > > > > gopidesupavan@gmail.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> Hi Airflow community,
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> I would like to start a discussion regarding a
> > new
> > > > > > > provider:
> > > > > > > > > > > > > > >> apache-airflow-providers-dq.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> While Airflow already includes SQL check
> > operators
> > > > > that
> > > > > > > many
> > > > > > > > > > users
> > > > > > > > > > > > > rely
> > > > > > > > > > > > > > >> on for data quality, this new provider builds
> on
> > > > that
> > > > > > > > > foundation
> > > > > > > > > > > by
> > > > > > > > > > > > > > >> introducing DQRule and RuleSet objects, stable
> > > rule
> > > > > > > > identity,
> > > > > > > > > > > > > persisted
> > > > > > > > > > > > > > >> history, and direct connections to Airflow
> > assets.
> > > > > This
> > > > > > > > > approach
> > > > > > > > > > > > makes
> > > > > > > > > > > > > > >> quality results easier to inspect over time,
> > > allows
> > > > > > > > downstream
> > > > > > > > > > > > > > consumers to
> > > > > > > > > > > > > > >> gate tasks based on recent quality results,
> and
> > > > > > provides a
> > > > > > > > > > unified
> > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > >> for LLM-assisted workflows. Execution will
> > > continue
> > > > to
> > > > > > > > utilize
> > > > > > > > > > > > > existing
> > > > > > > > > > > > > > >> DbApiHook connections.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> The initial version of the provider is
> > > intentionally
> > > > > > > > focused:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> - Declarative DQRule and RuleSet objects.
> > > > > > > > > > > > > > >> - DQCheckOperator and @task.dq_check.
> > > > > > > > > > > > > > >> - DbApiHook-based SQL checks, including
> > built-in
> > > > > > checks
> > > > > > > > and
> > > > > > > > > > > > > > custom_sql.
> > > > > > > > > > > > > > >> - Persisted results for tasks, runs, and
> > rules.
> > > > > > > > > > > > > > >> - A minimal Airflow UI plugin for viewing
> > > results
> > > > > and
> > > > > > > rule
> > > > > > > > > > > > history.
> > > > > > > > > > > > > > >> - Experimental asset helpers such as
> > > > asset_quality()
> > > > > > and
> > > > > > > > > > > > > > >> require_quality().
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Regarding scope, this first iteration uses
> > object
> > > > > > storage
> > > > > > > > only
> > > > > > > > > > to
> > > > > > > > > > > > > > persist
> > > > > > > > > > > > > > >> DQ results and history; checks are executed
> via
> > > > > database
> > > > > > > > > > > > connections.
> > > > > > > > > > > > > > >> Future iterations may include file or
> > object-store
> > > > > based
> > > > > > > > > checks
> > > > > > > > > > > > (e.g.,
> > > > > > > > > > > > > > S3,
> > > > > > > > > > > > > > >> GCS) where Airflow runs quality rules against
> > data
> > > > > > > directly.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> This proposal does not require changes to
> > Airflow
> > > > > core.
> > > > > > > > Asset
> > > > > > > > > > > > support
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > >> currently provider-owned metadata, with static
> > > > > > > configuration
> > > > > > > > > > > stored
> > > > > > > > > > > > on
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > >> asset and runtime summaries stored on asset
> > > events.
> > > > If
> > > > > > the
> > > > > > > > > > > provider
> > > > > > > > > > > > > > gains
> > > > > > > > > > > > > > >> traction, we can discuss making Data Quality a
> > > > > > first-class
> > > > > > > > > > > component
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > >> Airflow assets.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> This work also serves as a practical follow-up
> > to
> > > > the
> > > > > > data
> > > > > > > > > > quality
> > > > > > > > > > > > > > >> direction mentioned in AIP-99. Persisted
> history
> > > is
> > > > > > > valuable
> > > > > > > > > for
> > > > > > > > > > > > users
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > >> future LLM-assisted workflows, such as those
> > from
> > > > > > > Anthropic
> > > > > > > > or
> > > > > > > > > > > > > > common.ai,
> > > > > > > > > > > > > > >> to understand rule performance and generate
> > > > candidate
> > > > > > > rules
> > > > > > > > > > based
> > > > > > > > > > > on
> > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > >> context.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> A rough pseudo-flow is provided below:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> seed_rules = RuleSet(
> > > > > > > > > > > > > > >> name="orders_quality",
> > > > > > > > > > > > > > >> rules=[
> > > > > > > > > > > > > > >> DQRule(name="order_id_not_null",
> > > > > > > check="null_count",
> > > > > > > > > > > > > > >> column="order_id", condition={"equal_to": 0}),
> > > > > > > > > > > > > > >> DQRule(name="amount_valid",
> check="min",
> > > > > > > > > > column="amount",
> > > > > > > > > > > > > > >> condition={"geq_to": 0}),
> > > > > > > > > > > > > > >> ],
> > > > > > > > > > > > > > >> )
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> orders_asset = asset_quality(
> > > > > > > > > > > > > > >> Asset("orders"),
> > > > > > > > > > > > > > >> conn_id="warehouse",
> > > > > > > > > > > > > > >> table="orders",
> > > > > > > > > > > > > > >> ruleset=seed_rules,
> > > > > > > > > > > > > > >> )
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> # Optional: common.ai / Anthropic provider
> can
> > > > > > generate a
> > > > > > > > > > RuleSet
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > >> schema context.
> > > > > > > > > > > > > > >> generated_rules =
> > generate_rules_from_schema(...)
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> @task.dq_check(asset=orders_asset)
> > > > > > > > > > > > > > >> def check_orders(ruleset):
> > > > > > > > > > > > > > >> return ruleset
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> checked_orders = check_orders(generated_rules)
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> with DAG("orders_consumer",
> > > schedule=orders_asset):
> > > > > > > > > > > > > > >> require_quality(orders_asset,
> > min_score=0.95)
> > > >>
> > > > > > > > > > > > consume_orders()
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> The UI remains deliberately minimal for this
> > > initial
> > > > > > > > release,
> > > > > > > > > > > > focusing
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > >> result and history inspection.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> You can view examples [1] of how it's
> integrated
> > > > with
> > > > > > > > > > assets/llms.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> currently i named it providers
> > > > > > > > `apache-airflow-providers-dq`.
> > > > > > > > > if
> > > > > > > > > > > any
> > > > > > > > > > > > > > >> other preference likely with `dataquality`.
> > Please
> > > > let
> > > > > > me
> > > > > > > > know
> > > > > > > > > > if
> > > > > > > > > > > > you
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > >> a preference. naming is hard :)
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> [1]:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > >> Pavan
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: New Community Provider: Data Quality
Posted by Sameer Mesiah <sm...@gmail.com>.
Okay. I think you addressed points 1) and 3) very well.
I believe for 2), it seems like we were already aligned. So this is
fundamentally a provider intended for platform-leaning data practitioners?
Is that correct? In that case, I would flip my vote to +1.
Thanks for taking out the time to address my questions. It was interesting
hearing about your future plans for this provider (especially DataFusion).
On Fri, 17 Jul 2026 at 01:00, Pavankumar Gopidesu <go...@gmail.com>
wrote:
> Hi Sameer,
>
> Thanks for taking the time to go through the PR, glad you have questions,
> let's discuss. My responses are below, sorry for the long response.
>
> Firstly, a side note: this is not an AI response; I typed it completely but
> used AI for rephrasing. I am not a native English speaker. :)
>
> 1) Provider philosophy — framework within Airflow vs.
> integration/abstraction
>
> I agree providers come with various levels of integration and abstraction,
> but I wouldn't call this a framework at all. All we are doing is defining
> rules as declarative data, providing a few helper functions, and persisting
> the results. The core execution sits entirely on the common.sql DbApiHook
> interface, so any provider that implements that interface can execute the
> checks, nothing is reimplemented inside this provider. In that sense, this
> provider is a consumer of common.sql, exactly the way many providers today
> already are.
>
> On why a separate provider rather than folding it into an existing common
> one, a few concrete reasons:
>
> - It will grow beyond SQL execution. The next engine I'm working on is
> DataFusion, for running checks directly against ingested data in object
> stores (a skeleton PR is already in progress). DataFusion is an
> excellent
> fit for exactly this workload — I use it day to day, and it routinely
> returns results over millions of records in milliseconds — so quality
> checks on raw ingested data need no heavy machinery: no Spark cluster,
> no
> separate compute job, just a fast scan right after ingestion, before the
> data ever reaches a warehouse. That engine would be an odd thing to
> graft
> onto common.sql, whose scope is the DB-API interface.
> - It composes with the AI providers. Because rules are declarative
> specs, the anthropic and common.ai providers can generate rule specs
> against a schema — an integration that belongs with the quality domain,
> not
> inside any of the providers it draws on.
> - The boundary is the domain, not a service. Everything here is
> quality-related work — rules, results, scoring, gating — hence the name.
> Placing it in its own package keeps that domain in one place, evolving
> on
> its own cadence, without disrupting existing common providers that
> nearly
> everything depends on.
>
> So the shape is: one small quality-domain provider that consumes existing
> abstractions (common.sql today, DataFusion next, object storage for
> results, AI providers for rule generation) rather than a framework that
> reimplements any of them.
>
> 2) Target user ?
>
> Fair point on being explicit about the audience. The provider ships with a
> set of built-in operational checks (nulls, uniqueness, volume, min/max)
> plus custom_sql on top — and I'd note that the declarative API you're
> describing already exists: rules are data, not code. A ruleset can be a
> plain YAML file loaded at parse time, and extending checks means adding a
> custom_sql rule in your Dag repo — never modifying provider internals.
>
> On platform-leaning vs. analytics-leaning: I take the distinction, though
> in the teams I've worked with, the line keeps blurring — engineers end up
> owning more of their stack end to end. The way I'd frame the question is
> fit for purpose: does this help me do the task without investing in a
> larger job or integration? If I have simple queries and declarative checks,
> this provider is that fit.
>
> On domain-rich validation and dbt: everyone has their own way of
> implementing business logic, and where it runs is their choice — if dbt,
> GX, or Soda is the right fit there, use them; we are not promising
> domain-rich validation and we are not competing with those tools. But
> consider the case where I'm a consumer and you're a producer who promised
> to send certain fields with certain values. To verify that contract with a
> few simple checks, I wouldn't stand up dbt or GX. Airflow already ships
> everything needed to run those checks — what's been missing is a unified
> entry point to define them. That's what this provider builds.
>
> And once rules have a declarative definition and durable per-rule execution
> history, something else opens up: agents are very good at generating rules
> against a schema, and at answering questions like "how has my producer's
> feed quality looked over the last month?" — which is not straightforward to
> answer today. With the stored execution history per asset, an agent can
> answer that and even suggest additional rules. The provider ships that
> durable view.
>
> So overall: it's about fit for purpose. We're promising simple, declarative
> operational checks over Airflow's existing provider interfaces — execute
> rules, get a score, gate your business flow on it. And if someone finds it
> useful for checks after their transformations too, there's no harm in that.
>
> 3) Maintenance surface:
>
> This follows from (1): what we'd own is a small declarative layer, not a
> framework. Rules are plain models with no behavior, results go through the
> existing object storage abstraction, scoring is simple arithmetic — no new
> tables, no migrations, no scheduler, API server, or UI changes. And "does
> the value justify the scope" is the right test: everything here reuses
> machinery Airflow already ships, so if users genuinely benefit, this thin
> layer is a cost worth carrying. If they don't, we'll learn that quickly and
> cheaply — the provider ships in the incubation state, and our provider
> governance model lets us suspend or remove it entirely if it doesn't prove
> useful.
>
> Happy to discuss any of this further — and thanks again for the detailed
> review, it genuinely helps sharpen the proposal.
>
> Regards,
> Pavan
>
> On Thu, Jul 16, 2026 at 10:21 PM Sameer Mesiah <sm...@gmail.com>
> wrote:
>
> > Hi Pavan,
> >
> > First, I would like to say that this new provider definitely adds value
> and
> > I liked the brief walkthrough you did today during the call. But after
> > going through the PR (and hearing Vikram's thoughts), there are a few
> > concerns that popped up:
> >
> > 1) I'm struggling to reconcile this with the existing philosophy of
> Airflow
> > providers. Most providers integrate with an external service or expose a
> > common abstraction over multiple services. This proposal instead
> introduces
> > a fairly comprehensive data quality framework that is implemented
> entirely
> > within Airflow. I am just wondering whether this should be a
> > standalone provider versus additional functionality that could be added
> to
> > the common provider.
> >
> > 2) I'm not yet sure who the primary target user is. If the intended
> > audience is Analytics Engineers, I'd expect the framework to expose a
> > simple declarative API for defining and extending reusable checks without
> > requiring changes to provider internals (you can correct me here if you
> > intend on exposing a simplified API eventually). In many organisations,
> > Airflow providers are owned by platform teams and are treated as
> > third-party dependencies, so modifying provider code to introduce new
> > reusable checks is often not a practical workflow. Arguably, the case for
> > this new framework is stronger if you are not going too deep into
> > domain-rich business rules but just want to limit it to operational
> checks
> > data platform owners might be interested in. To ground my critique with a
> > very straightforward question: if you intend for this new provider to
> > support domain-rich data validation, what benefit would it have over
> tools
> > like DBT which are more accessible to practitioners of this specific area
> > i.e Analytics Engineers. Vikram did share the perspective of a Data
> > Engineer who is building data pipelines but I just wanted to expand on
> that
> > a bit more by differentiating between the needs of platform-leaning and
> > analytics-leaning Data Engineers. These 2 roles are increasingly
> bifurcated
> > and I think we should really think about which group we are targeting
> here.
> >
> > 3) This proposal introduces a significant amount of new framework surface
> > area that Airflow will own and maintain going forward. Beyond the
> operator
> > itself, we're introducing rule definitions, persistence, history,
> scoring,
> > asset integration, decorators, configuration, etc. That represents a
> > substantial long-term maintenance commitment, so I think it's worth
> > discussing whether the value provided justifies that scope and whether
> this
> > is the right architectural boundary for Airflow.
> >
> > Overall, I would say I am +0.5. It's a good proposal but it needs more
> > refinement because we do not want to introduce a provider that our users
> > are just not going to want to use.
> >
> > On Wed, 15 Jul 2026 at 21:27, Pavankumar Gopidesu <
> gopidesupavan@gmail.com
> > >
> > wrote:
> >
> > > Hi Vikram,
> > >
> > > Thanks for the feedback. I agree that helpers would be useful for users
> > who
> > > want to run data quality checks from existing tasks or operators,
> without
> > > always needing to use a dedicated DQ operator.
> > >
> > > I updated the PR to include
> > >
> > > - importable helpers for running and persisting checks inside tasks
> or
> > > any existing operators.
> > >
> > > For asset integration, this PR does not require Airflow core changes.
> The
> > > provider adds two asset-oriented helpers:
> > >
> > > - asset_quality() to attach quality configuration to an asset. This
> > lets
> > > the asset carry the ruleset that is expected to be executed for it.
> In
> > > the
> > > current implementation this is stored as provider-owned metadata in
> > the
> > > asset extra field.
> > > - require_quality() to let downstream tasks gate on the latest
> quality
> > > result from the triggering asset event.
> > >
> > > Below is an example view of how assets can use data quality with
> > > partition-aware producer and consumer Dags.
> > >
> > > ```
> > >
> > > *Example ruleset:*
> > >
> > > orders_quality_rules = RuleSet(
> > > name="orders_quality",
> > > rules=[
> > > DQRule(
> > > id="orders.amount_non_negative",
> > > name="amount_non_negative",
> > > check="min",
> > > column="amount",
> > > condition={"geq_to": 0},
> > > ),
> > > ],
> > > )
> > >
> > > raw_orders = asset_quality(
> > > Asset("daily_raw_orders"),
> > > ruleset=orders_quality_rules,
> > > conn_id="warehouse",
> > > table="raw_orders",
> > > )
> > >
> > > *Producer Dag:*
> > >
> > > with DAG(
> > > dag_id="daily_raw_orders",
> > > schedule=CronPartitionTimetable("0 2 * * *", timezone="UTC"),
> > > ):
> > > @task(outlets=[raw_orders])
> > > def load_and_check():
> > > context = get_current_context()
> > > partition_key = context["partition_key"]
> > >
> > > load_raw_orders(partition_key)
> > >
> > > result = run_quality_checks(
> > > ruleset=orders_quality_rules,
> > > table="raw_orders",
> > > partition_clause=f"ds = '{partition_key}'",
> > > )
> > >
> > > persist_quality_results(
> > > result,
> > > context=context,
> > > outlets=[raw_orders],
> > > )
> > >
> > > load_and_check()
> > >
> > >
> > > *Consumer Dag:*
> > >
> > > with DAG(
> > > dag_id="curate_daily_orders",
> > > schedule=PartitionedAssetTimetable(assets=raw_orders),
> > > ):
> > > quality_gate = require_quality(raw_orders, min_score=1.0)
> > >
> > > @task
> > > def curate():
> > > context = get_current_context()
> > > partition_key = context["partition_key"]
> > >
> > > curate_orders(partition_key)
> > >
> > > quality_gate >> curate()
> > >
> > > ```
> > >
> > > The flow is:
> > >
> > > 1. A partitioned producer Dag runs and loads data for the current
> > > partition.
> > > 2. The producer runs run_quality_checks() with a partition-scoped
> > > filter.
> > > 3. persist_quality_results() records the result and adds a DQ
> summary
> > to
> > > the produced asset event.
> > > 4. PartitionedAssetTimetable triggers the consumer Dag for that
> > > partition.
> > > 5. require_quality() reads the DQ summary from the triggering asset
> > > event.
> > > 6. The consumer processes the partition only if quality passes.
> > >
> > > Hope this makes sense?
> > >
> > > Regards,
> > >
> > > Pavan
> > >
> > >
> > > On Wed, Jul 15, 2026 at 6:47 PM Vikram Koka via dev <
> > > dev@airflow.apache.org>
> > > wrote:
> > >
> > > > Hey Pavan,
> > > >
> > > > Thanks for bringing this up and working on it. I strongly support the
> > > > overall concept of the Data Quality provider.
> > > >
> > > > However, I have two concerns / questions:
> > > > 1. I am unsure of the Operator approach shown in the PR.
> > > > I am not certain that these SHOULD be separate Operators which result
> > in
> > > > separate tasks.
> > > > If these are made available as Helper functions, they could be
> included
> > > in
> > > > existing Operators.
> > > > I am not saying they shouldn't be separate Tasks in Dags, but the
> > Helper
> > > > function approach would provide more optionality.
> > > >
> > > > 2. I am wondering how to integrate these with Asset Partitions
> > > > I do believe this can be integrated with the Asset Partitions
> concept,
> > > but
> > > > I am personally unclear on exactly how.
> > > > I think adding an example illustrating this would be very helpful.
> > > >
> > > > Best regards,
> > > > Vikram
> > > >
> > > >
> > > > On Mon, Jul 13, 2026 at 3:26 PM Pavankumar Gopidesu <
> > > > gopidesupavan@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Kaxil..
> > > > >
> > > > > Thanks Everyone, consensus looks positive; I will merge the
> skeleton
> > PR
> > > > > first.
> > > > >
> > > > > Regards,
> > > > > Pavan
> > > > >
> > > > > On Mon, Jul 13, 2026 at 10:05 PM Kaxil Naik <ka...@gmail.com>
> > > wrote:
> > > > >
> > > > > > +1 for the high-level need for such data quality checks. Thanks
> > > Pavan,
> > > > > will
> > > > > > review the PRs shortly.
> > > > > >
> > > > > > On Fri, 10 Jul 2026 at 01:58, Pavankumar Gopidesu <
> > > > > gopidesupavan@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Bugra,
> > > > > > >
> > > > > > > I have updated the naming convention now.
> > > > > > >
> > > > > > > Pavan
> > > > > > >
> > > > > > > On Wed, Jul 8, 2026 at 7:07 PM Buğra Öztürk <
> > > ozturkbugra93@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Pavan for bringing this together and starting the
> > > > discussion!
> > > > > > > > Sounds good! +1 on the idea.
> > > > > > > >
> > > > > > > > Harder than solving problems. Not a strong suggestion, but
> > > > > > > > `common-dataquality` sounds more reasonable to me. It also
> adds
> > > the
> > > > > > value
> > > > > > > > of the `common` part, which provides the separation pattern
> > Jarek
> > > > > > > > mentioned. It gives a better understanding that it is a
> common
> > > > > > offering.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Bugra Ozturk
> > > > > > > >
> > > > > > > > On Wed, Jul 8, 2026 at 7:15 PM Pavankumar Gopidesu <
> > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Jarek, I agree that the separate provider approach
> > > offers
> > > > > much
> > > > > > > > more
> > > > > > > > > flexibility for iterating on features and fixes.
> > > > > > > > >
> > > > > > > > > Naming is always hard :)
> > > > > > > > >
> > > > > > > > > Option 1: apache-airflow-providers-dataquality
> > > > > > > > > Option 2: apache-airflow-providers-common-dataquality (This
> > > goes
> > > > > > inside
> > > > > > > > the
> > > > > > > > > common providers folder we already have)
> > > > > > > > >
> > > > > > > > > So, I am up for either option :)
> > > > > > > > >
> > > > > > > > > have removed first short name
> `apache-airflow-providers-dq`.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Pavan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk <
> > jarek@potiuk.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 Good design/ idea. No objections - dataquality is a
> good
> > > > name
> > > > > -
> > > > > > > but
> > > > > > > > I
> > > > > > > > > > would also consider `common-dataquality" - even if it's
> > > longer,
> > > > > it
> > > > > > > > builds
> > > > > > > > > > on the pattern we have already with common-ai. But not a
> > > > blocker.
> > > > > > > > > >
> > > > > > > > > > I also think it's good to have it as a separate provider,
> > > even
> > > > if
> > > > > > it
> > > > > > > > > gains
> > > > > > > > > > traction for two reasons:
> > > > > > > > > >
> > > > > > > > > > a) ability to add features or fix issues independently
> from
> > > the
> > > > > > core
> > > > > > > > > > b) an explicit "optional" feature that is easy to promote
> > > > > > > > > >
> > > > > > > > > > I think what we saw with common is that people see
> airflow
> > > > > already
> > > > > > as
> > > > > > > > too
> > > > > > > > > > heavy - and "too many releases" sometimes, so quite
> > > > > > > > counter-intuitively -
> > > > > > > > > > by having separate providers adding features that "hook
> in"
> > > > > > existing
> > > > > > > > > > functionalities of core - we do not make airflow
> "heavier"
> > > and
> > > > we
> > > > > > do
> > > > > > > > not
> > > > > > > > > > force people to migrating to future newer versions to use
> > new
> > > > > > > features.
> > > > > > > > > >
> > > > > > > > > > J.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu <
> > > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Amogh,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the feedback.
> > > > > > > > > > >
> > > > > > > > > > > I am happy to change the provider name to dataquality.
> > > > > > > > > > >
> > > > > > > > > > > Regarding the LLM-assisted features, the current PR
> does
> > > not
> > > > > > > include
> > > > > > > > > any
> > > > > > > > > > > implementation. It only adds the SKILLS [1 ]and the
> > > reference
> > > > > > > schema
> > > > > > > > > for
> > > > > > > > > > > the DQ Rule structure. Are you suggesting that I move
> > this
> > > > > SKILL
> > > > > > > > > > > documentation to a separate PR?
> > > > > > > > > > >
> > > > > > > > > > > [1]:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Pavan
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai <
> > > > > > amoghdesai@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Pavan,
> > > > > > > > > > > >
> > > > > > > > > > > > First of all, +1 to this.
> > > > > > > > > > > >
> > > > > > > > > > > > Now, few things:
> > > > > > > > > > > >
> > > > > > > > > > > > * On naming: dataquality over dq for me honestly. Our
> > > > > existing
> > > > > > > > > provider
> > > > > > > > > > > > names spell things out
> > > > > > > > > > > > (common.sql, openlineage, not abbreviated forms) and
> dq
> > > is
> > > > > > > > genuinely
> > > > > > > > > > > > ambiguous outside context.
> > > > > > > > > > > >
> > > > > > > > > > > > * On scope: I also agree with Niko that #69413 is too
> > > large
> > > > > for
> > > > > > > one
> > > > > > > > > > pass
> > > > > > > > > > > &
> > > > > > > > > > > > I am glad to see the
> > > > > > > > > > > > backend/UI split already happening in #69575. Would
> > also
> > > > > > suggest
> > > > > > > > > > keeping
> > > > > > > > > > > > the LLM assisted rule
> > > > > > > > > > > > generation pieces (*schema-based
> > > > generate_rules_from_schema*)
> > > > > > out
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > initial provider PR entirely
> > > > > > > > > > > > cos as I see it, its a separable capability and
> > bundling
> > > it
> > > > > > will
> > > > > > > > slow
> > > > > > > > > > > > review of the core DQRule or
> > > > > > > > > > > > RuleSet or operator surface, which is the part that
> > > > actually
> > > > > > > needs
> > > > > > > > > the
> > > > > > > > > > > most
> > > > > > > > > > > > detailed review.
> > > > > > > > > > > >
> > > > > > > > > > > > In short: go for it!
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks & Regards,
> > > > > > > > > > > > Amogh Desai
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <
> > > > > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > In the meantime, the PR is ready for review. Feel
> > free
> > > to
> > > > > > > review
> > > > > > > > > and
> > > > > > > > > > > > > provide any feedback.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > Pavan
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu
> <
> > > > > > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Sorry, I forgot to add: the draft PR is here
> > > > > > > > > > > > > > https://github.com/apache/airflow/pull/69413;
> it's
> > > > > still a
> > > > > > > > WIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > some screenshots
> > > > > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468
> > > > > > > > > > > > :)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Pavan
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar
> Gopidesu
> > <
> > > > > > > > > > > > > > gopidesupavan@gmail.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Hi Airflow community,
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> I would like to start a discussion regarding a
> new
> > > > > > provider:
> > > > > > > > > > > > > >> apache-airflow-providers-dq.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> While Airflow already includes SQL check
> operators
> > > > that
> > > > > > many
> > > > > > > > > users
> > > > > > > > > > > > rely
> > > > > > > > > > > > > >> on for data quality, this new provider builds on
> > > that
> > > > > > > > foundation
> > > > > > > > > > by
> > > > > > > > > > > > > >> introducing DQRule and RuleSet objects, stable
> > rule
> > > > > > > identity,
> > > > > > > > > > > > persisted
> > > > > > > > > > > > > >> history, and direct connections to Airflow
> assets.
> > > > This
> > > > > > > > approach
> > > > > > > > > > > makes
> > > > > > > > > > > > > >> quality results easier to inspect over time,
> > allows
> > > > > > > downstream
> > > > > > > > > > > > > consumers to
> > > > > > > > > > > > > >> gate tasks based on recent quality results, and
> > > > > provides a
> > > > > > > > > unified
> > > > > > > > > > > > > schema
> > > > > > > > > > > > > >> for LLM-assisted workflows. Execution will
> > continue
> > > to
> > > > > > > utilize
> > > > > > > > > > > > existing
> > > > > > > > > > > > > >> DbApiHook connections.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> The initial version of the provider is
> > intentionally
> > > > > > > focused:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> - Declarative DQRule and RuleSet objects.
> > > > > > > > > > > > > >> - DQCheckOperator and @task.dq_check.
> > > > > > > > > > > > > >> - DbApiHook-based SQL checks, including
> built-in
> > > > > checks
> > > > > > > and
> > > > > > > > > > > > > custom_sql.
> > > > > > > > > > > > > >> - Persisted results for tasks, runs, and
> rules.
> > > > > > > > > > > > > >> - A minimal Airflow UI plugin for viewing
> > results
> > > > and
> > > > > > rule
> > > > > > > > > > > history.
> > > > > > > > > > > > > >> - Experimental asset helpers such as
> > > asset_quality()
> > > > > and
> > > > > > > > > > > > > >> require_quality().
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Regarding scope, this first iteration uses
> object
> > > > > storage
> > > > > > > only
> > > > > > > > > to
> > > > > > > > > > > > > persist
> > > > > > > > > > > > > >> DQ results and history; checks are executed via
> > > > database
> > > > > > > > > > > connections.
> > > > > > > > > > > > > >> Future iterations may include file or
> object-store
> > > > based
> > > > > > > > checks
> > > > > > > > > > > (e.g.,
> > > > > > > > > > > > > S3,
> > > > > > > > > > > > > >> GCS) where Airflow runs quality rules against
> data
> > > > > > directly.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> This proposal does not require changes to
> Airflow
> > > > core.
> > > > > > > Asset
> > > > > > > > > > > support
> > > > > > > > > > > > is
> > > > > > > > > > > > > >> currently provider-owned metadata, with static
> > > > > > configuration
> > > > > > > > > > stored
> > > > > > > > > > > on
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> asset and runtime summaries stored on asset
> > events.
> > > If
> > > > > the
> > > > > > > > > > provider
> > > > > > > > > > > > > gains
> > > > > > > > > > > > > >> traction, we can discuss making Data Quality a
> > > > > first-class
> > > > > > > > > > component
> > > > > > > > > > > > of
> > > > > > > > > > > > > >> Airflow assets.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> This work also serves as a practical follow-up
> to
> > > the
> > > > > data
> > > > > > > > > quality
> > > > > > > > > > > > > >> direction mentioned in AIP-99. Persisted history
> > is
> > > > > > valuable
> > > > > > > > for
> > > > > > > > > > > users
> > > > > > > > > > > > > and
> > > > > > > > > > > > > >> future LLM-assisted workflows, such as those
> from
> > > > > > Anthropic
> > > > > > > or
> > > > > > > > > > > > > common.ai,
> > > > > > > > > > > > > >> to understand rule performance and generate
> > > candidate
> > > > > > rules
> > > > > > > > > based
> > > > > > > > > > on
> > > > > > > > > > > > > schema
> > > > > > > > > > > > > >> context.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> A rough pseudo-flow is provided below:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> seed_rules = RuleSet(
> > > > > > > > > > > > > >> name="orders_quality",
> > > > > > > > > > > > > >> rules=[
> > > > > > > > > > > > > >> DQRule(name="order_id_not_null",
> > > > > > check="null_count",
> > > > > > > > > > > > > >> column="order_id", condition={"equal_to": 0}),
> > > > > > > > > > > > > >> DQRule(name="amount_valid", check="min",
> > > > > > > > > column="amount",
> > > > > > > > > > > > > >> condition={"geq_to": 0}),
> > > > > > > > > > > > > >> ],
> > > > > > > > > > > > > >> )
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> orders_asset = asset_quality(
> > > > > > > > > > > > > >> Asset("orders"),
> > > > > > > > > > > > > >> conn_id="warehouse",
> > > > > > > > > > > > > >> table="orders",
> > > > > > > > > > > > > >> ruleset=seed_rules,
> > > > > > > > > > > > > >> )
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> # Optional: common.ai / Anthropic provider can
> > > > > generate a
> > > > > > > > > RuleSet
> > > > > > > > > > > > from
> > > > > > > > > > > > > >> schema context.
> > > > > > > > > > > > > >> generated_rules =
> generate_rules_from_schema(...)
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> @task.dq_check(asset=orders_asset)
> > > > > > > > > > > > > >> def check_orders(ruleset):
> > > > > > > > > > > > > >> return ruleset
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> checked_orders = check_orders(generated_rules)
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> with DAG("orders_consumer",
> > schedule=orders_asset):
> > > > > > > > > > > > > >> require_quality(orders_asset,
> min_score=0.95)
> > >>
> > > > > > > > > > > consume_orders()
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> The UI remains deliberately minimal for this
> > initial
> > > > > > > release,
> > > > > > > > > > > focusing
> > > > > > > > > > > > > on
> > > > > > > > > > > > > >> result and history inspection.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> You can view examples [1] of how it's integrated
> > > with
> > > > > > > > > assets/llms.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> currently i named it providers
> > > > > > > `apache-airflow-providers-dq`.
> > > > > > > > if
> > > > > > > > > > any
> > > > > > > > > > > > > >> other preference likely with `dataquality`.
> Please
> > > let
> > > > > me
> > > > > > > know
> > > > > > > > > if
> > > > > > > > > > > you
> > > > > > > > > > > > > have
> > > > > > > > > > > > > >> a preference. naming is hard :)
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> [1]:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > >> Pavan
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: New Community Provider: Data Quality
Posted by Pavankumar Gopidesu <go...@gmail.com>.
Hi Sameer,
Thanks for taking the time to go through the PR, glad you have questions,
let's discuss. My responses are below, sorry for the long response.
Firstly, a side note: this is not an AI response; I typed it completely but
used AI for rephrasing. I am not a native English speaker. :)
1) Provider philosophy — framework within Airflow vs.
integration/abstraction
I agree providers come with various levels of integration and abstraction,
but I wouldn't call this a framework at all. All we are doing is defining
rules as declarative data, providing a few helper functions, and persisting
the results. The core execution sits entirely on the common.sql DbApiHook
interface, so any provider that implements that interface can execute the
checks, nothing is reimplemented inside this provider. In that sense, this
provider is a consumer of common.sql, exactly the way many providers today
already are.
On why a separate provider rather than folding it into an existing common
one, a few concrete reasons:
- It will grow beyond SQL execution. The next engine I'm working on is
DataFusion, for running checks directly against ingested data in object
stores (a skeleton PR is already in progress). DataFusion is an excellent
fit for exactly this workload — I use it day to day, and it routinely
returns results over millions of records in milliseconds — so quality
checks on raw ingested data need no heavy machinery: no Spark cluster, no
separate compute job, just a fast scan right after ingestion, before the
data ever reaches a warehouse. That engine would be an odd thing to graft
onto common.sql, whose scope is the DB-API interface.
- It composes with the AI providers. Because rules are declarative
specs, the anthropic and common.ai providers can generate rule specs
against a schema — an integration that belongs with the quality domain, not
inside any of the providers it draws on.
- The boundary is the domain, not a service. Everything here is
quality-related work — rules, results, scoring, gating — hence the name.
Placing it in its own package keeps that domain in one place, evolving on
its own cadence, without disrupting existing common providers that nearly
everything depends on.
So the shape is: one small quality-domain provider that consumes existing
abstractions (common.sql today, DataFusion next, object storage for
results, AI providers for rule generation) rather than a framework that
reimplements any of them.
2) Target user ?
Fair point on being explicit about the audience. The provider ships with a
set of built-in operational checks (nulls, uniqueness, volume, min/max)
plus custom_sql on top — and I'd note that the declarative API you're
describing already exists: rules are data, not code. A ruleset can be a
plain YAML file loaded at parse time, and extending checks means adding a
custom_sql rule in your Dag repo — never modifying provider internals.
On platform-leaning vs. analytics-leaning: I take the distinction, though
in the teams I've worked with, the line keeps blurring — engineers end up
owning more of their stack end to end. The way I'd frame the question is
fit for purpose: does this help me do the task without investing in a
larger job or integration? If I have simple queries and declarative checks,
this provider is that fit.
On domain-rich validation and dbt: everyone has their own way of
implementing business logic, and where it runs is their choice — if dbt,
GX, or Soda is the right fit there, use them; we are not promising
domain-rich validation and we are not competing with those tools. But
consider the case where I'm a consumer and you're a producer who promised
to send certain fields with certain values. To verify that contract with a
few simple checks, I wouldn't stand up dbt or GX. Airflow already ships
everything needed to run those checks — what's been missing is a unified
entry point to define them. That's what this provider builds.
And once rules have a declarative definition and durable per-rule execution
history, something else opens up: agents are very good at generating rules
against a schema, and at answering questions like "how has my producer's
feed quality looked over the last month?" — which is not straightforward to
answer today. With the stored execution history per asset, an agent can
answer that and even suggest additional rules. The provider ships that
durable view.
So overall: it's about fit for purpose. We're promising simple, declarative
operational checks over Airflow's existing provider interfaces — execute
rules, get a score, gate your business flow on it. And if someone finds it
useful for checks after their transformations too, there's no harm in that.
3) Maintenance surface:
This follows from (1): what we'd own is a small declarative layer, not a
framework. Rules are plain models with no behavior, results go through the
existing object storage abstraction, scoring is simple arithmetic — no new
tables, no migrations, no scheduler, API server, or UI changes. And "does
the value justify the scope" is the right test: everything here reuses
machinery Airflow already ships, so if users genuinely benefit, this thin
layer is a cost worth carrying. If they don't, we'll learn that quickly and
cheaply — the provider ships in the incubation state, and our provider
governance model lets us suspend or remove it entirely if it doesn't prove
useful.
Happy to discuss any of this further — and thanks again for the detailed
review, it genuinely helps sharpen the proposal.
Regards,
Pavan
On Thu, Jul 16, 2026 at 10:21 PM Sameer Mesiah <sm...@gmail.com> wrote:
> Hi Pavan,
>
> First, I would like to say that this new provider definitely adds value and
> I liked the brief walkthrough you did today during the call. But after
> going through the PR (and hearing Vikram's thoughts), there are a few
> concerns that popped up:
>
> 1) I'm struggling to reconcile this with the existing philosophy of Airflow
> providers. Most providers integrate with an external service or expose a
> common abstraction over multiple services. This proposal instead introduces
> a fairly comprehensive data quality framework that is implemented entirely
> within Airflow. I am just wondering whether this should be a
> standalone provider versus additional functionality that could be added to
> the common provider.
>
> 2) I'm not yet sure who the primary target user is. If the intended
> audience is Analytics Engineers, I'd expect the framework to expose a
> simple declarative API for defining and extending reusable checks without
> requiring changes to provider internals (you can correct me here if you
> intend on exposing a simplified API eventually). In many organisations,
> Airflow providers are owned by platform teams and are treated as
> third-party dependencies, so modifying provider code to introduce new
> reusable checks is often not a practical workflow. Arguably, the case for
> this new framework is stronger if you are not going too deep into
> domain-rich business rules but just want to limit it to operational checks
> data platform owners might be interested in. To ground my critique with a
> very straightforward question: if you intend for this new provider to
> support domain-rich data validation, what benefit would it have over tools
> like DBT which are more accessible to practitioners of this specific area
> i.e Analytics Engineers. Vikram did share the perspective of a Data
> Engineer who is building data pipelines but I just wanted to expand on that
> a bit more by differentiating between the needs of platform-leaning and
> analytics-leaning Data Engineers. These 2 roles are increasingly bifurcated
> and I think we should really think about which group we are targeting here.
>
> 3) This proposal introduces a significant amount of new framework surface
> area that Airflow will own and maintain going forward. Beyond the operator
> itself, we're introducing rule definitions, persistence, history, scoring,
> asset integration, decorators, configuration, etc. That represents a
> substantial long-term maintenance commitment, so I think it's worth
> discussing whether the value provided justifies that scope and whether this
> is the right architectural boundary for Airflow.
>
> Overall, I would say I am +0.5. It's a good proposal but it needs more
> refinement because we do not want to introduce a provider that our users
> are just not going to want to use.
>
> On Wed, 15 Jul 2026 at 21:27, Pavankumar Gopidesu <gopidesupavan@gmail.com
> >
> wrote:
>
> > Hi Vikram,
> >
> > Thanks for the feedback. I agree that helpers would be useful for users
> who
> > want to run data quality checks from existing tasks or operators, without
> > always needing to use a dedicated DQ operator.
> >
> > I updated the PR to include
> >
> > - importable helpers for running and persisting checks inside tasks or
> > any existing operators.
> >
> > For asset integration, this PR does not require Airflow core changes. The
> > provider adds two asset-oriented helpers:
> >
> > - asset_quality() to attach quality configuration to an asset. This
> lets
> > the asset carry the ruleset that is expected to be executed for it. In
> > the
> > current implementation this is stored as provider-owned metadata in
> the
> > asset extra field.
> > - require_quality() to let downstream tasks gate on the latest quality
> > result from the triggering asset event.
> >
> > Below is an example view of how assets can use data quality with
> > partition-aware producer and consumer Dags.
> >
> > ```
> >
> > *Example ruleset:*
> >
> > orders_quality_rules = RuleSet(
> > name="orders_quality",
> > rules=[
> > DQRule(
> > id="orders.amount_non_negative",
> > name="amount_non_negative",
> > check="min",
> > column="amount",
> > condition={"geq_to": 0},
> > ),
> > ],
> > )
> >
> > raw_orders = asset_quality(
> > Asset("daily_raw_orders"),
> > ruleset=orders_quality_rules,
> > conn_id="warehouse",
> > table="raw_orders",
> > )
> >
> > *Producer Dag:*
> >
> > with DAG(
> > dag_id="daily_raw_orders",
> > schedule=CronPartitionTimetable("0 2 * * *", timezone="UTC"),
> > ):
> > @task(outlets=[raw_orders])
> > def load_and_check():
> > context = get_current_context()
> > partition_key = context["partition_key"]
> >
> > load_raw_orders(partition_key)
> >
> > result = run_quality_checks(
> > ruleset=orders_quality_rules,
> > table="raw_orders",
> > partition_clause=f"ds = '{partition_key}'",
> > )
> >
> > persist_quality_results(
> > result,
> > context=context,
> > outlets=[raw_orders],
> > )
> >
> > load_and_check()
> >
> >
> > *Consumer Dag:*
> >
> > with DAG(
> > dag_id="curate_daily_orders",
> > schedule=PartitionedAssetTimetable(assets=raw_orders),
> > ):
> > quality_gate = require_quality(raw_orders, min_score=1.0)
> >
> > @task
> > def curate():
> > context = get_current_context()
> > partition_key = context["partition_key"]
> >
> > curate_orders(partition_key)
> >
> > quality_gate >> curate()
> >
> > ```
> >
> > The flow is:
> >
> > 1. A partitioned producer Dag runs and loads data for the current
> > partition.
> > 2. The producer runs run_quality_checks() with a partition-scoped
> > filter.
> > 3. persist_quality_results() records the result and adds a DQ summary
> to
> > the produced asset event.
> > 4. PartitionedAssetTimetable triggers the consumer Dag for that
> > partition.
> > 5. require_quality() reads the DQ summary from the triggering asset
> > event.
> > 6. The consumer processes the partition only if quality passes.
> >
> > Hope this makes sense?
> >
> > Regards,
> >
> > Pavan
> >
> >
> > On Wed, Jul 15, 2026 at 6:47 PM Vikram Koka via dev <
> > dev@airflow.apache.org>
> > wrote:
> >
> > > Hey Pavan,
> > >
> > > Thanks for bringing this up and working on it. I strongly support the
> > > overall concept of the Data Quality provider.
> > >
> > > However, I have two concerns / questions:
> > > 1. I am unsure of the Operator approach shown in the PR.
> > > I am not certain that these SHOULD be separate Operators which result
> in
> > > separate tasks.
> > > If these are made available as Helper functions, they could be included
> > in
> > > existing Operators.
> > > I am not saying they shouldn't be separate Tasks in Dags, but the
> Helper
> > > function approach would provide more optionality.
> > >
> > > 2. I am wondering how to integrate these with Asset Partitions
> > > I do believe this can be integrated with the Asset Partitions concept,
> > but
> > > I am personally unclear on exactly how.
> > > I think adding an example illustrating this would be very helpful.
> > >
> > > Best regards,
> > > Vikram
> > >
> > >
> > > On Mon, Jul 13, 2026 at 3:26 PM Pavankumar Gopidesu <
> > > gopidesupavan@gmail.com>
> > > wrote:
> > >
> > > > Thanks Kaxil..
> > > >
> > > > Thanks Everyone, consensus looks positive; I will merge the skeleton
> PR
> > > > first.
> > > >
> > > > Regards,
> > > > Pavan
> > > >
> > > > On Mon, Jul 13, 2026 at 10:05 PM Kaxil Naik <ka...@gmail.com>
> > wrote:
> > > >
> > > > > +1 for the high-level need for such data quality checks. Thanks
> > Pavan,
> > > > will
> > > > > review the PRs shortly.
> > > > >
> > > > > On Fri, 10 Jul 2026 at 01:58, Pavankumar Gopidesu <
> > > > gopidesupavan@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks Bugra,
> > > > > >
> > > > > > I have updated the naming convention now.
> > > > > >
> > > > > > Pavan
> > > > > >
> > > > > > On Wed, Jul 8, 2026 at 7:07 PM Buğra Öztürk <
> > ozturkbugra93@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Pavan for bringing this together and starting the
> > > discussion!
> > > > > > > Sounds good! +1 on the idea.
> > > > > > >
> > > > > > > Harder than solving problems. Not a strong suggestion, but
> > > > > > > `common-dataquality` sounds more reasonable to me. It also adds
> > the
> > > > > value
> > > > > > > of the `common` part, which provides the separation pattern
> Jarek
> > > > > > > mentioned. It gives a better understanding that it is a common
> > > > > offering.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Bugra Ozturk
> > > > > > >
> > > > > > > On Wed, Jul 8, 2026 at 7:15 PM Pavankumar Gopidesu <
> > > > > > > gopidesupavan@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Jarek, I agree that the separate provider approach
> > offers
> > > > much
> > > > > > > more
> > > > > > > > flexibility for iterating on features and fixes.
> > > > > > > >
> > > > > > > > Naming is always hard :)
> > > > > > > >
> > > > > > > > Option 1: apache-airflow-providers-dataquality
> > > > > > > > Option 2: apache-airflow-providers-common-dataquality (This
> > goes
> > > > > inside
> > > > > > > the
> > > > > > > > common providers folder we already have)
> > > > > > > >
> > > > > > > > So, I am up for either option :)
> > > > > > > >
> > > > > > > > have removed first short name `apache-airflow-providers-dq`.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Pavan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk <
> jarek@potiuk.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 Good design/ idea. No objections - dataquality is a good
> > > name
> > > > -
> > > > > > but
> > > > > > > I
> > > > > > > > > would also consider `common-dataquality" - even if it's
> > longer,
> > > > it
> > > > > > > builds
> > > > > > > > > on the pattern we have already with common-ai. But not a
> > > blocker.
> > > > > > > > >
> > > > > > > > > I also think it's good to have it as a separate provider,
> > even
> > > if
> > > > > it
> > > > > > > > gains
> > > > > > > > > traction for two reasons:
> > > > > > > > >
> > > > > > > > > a) ability to add features or fix issues independently from
> > the
> > > > > core
> > > > > > > > > b) an explicit "optional" feature that is easy to promote
> > > > > > > > >
> > > > > > > > > I think what we saw with common is that people see airflow
> > > > already
> > > > > as
> > > > > > > too
> > > > > > > > > heavy - and "too many releases" sometimes, so quite
> > > > > > > counter-intuitively -
> > > > > > > > > by having separate providers adding features that "hook in"
> > > > > existing
> > > > > > > > > functionalities of core - we do not make airflow "heavier"
> > and
> > > we
> > > > > do
> > > > > > > not
> > > > > > > > > force people to migrating to future newer versions to use
> new
> > > > > > features.
> > > > > > > > >
> > > > > > > > > J.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu <
> > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Amogh,
> > > > > > > > > >
> > > > > > > > > > Thanks for the feedback.
> > > > > > > > > >
> > > > > > > > > > I am happy to change the provider name to dataquality.
> > > > > > > > > >
> > > > > > > > > > Regarding the LLM-assisted features, the current PR does
> > not
> > > > > > include
> > > > > > > > any
> > > > > > > > > > implementation. It only adds the SKILLS [1 ]and the
> > reference
> > > > > > schema
> > > > > > > > for
> > > > > > > > > > the DQ Rule structure. Are you suggesting that I move
> this
> > > > SKILL
> > > > > > > > > > documentation to a separate PR?
> > > > > > > > > >
> > > > > > > > > > [1]:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Pavan
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai <
> > > > > amoghdesai@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Pavan,
> > > > > > > > > > >
> > > > > > > > > > > First of all, +1 to this.
> > > > > > > > > > >
> > > > > > > > > > > Now, few things:
> > > > > > > > > > >
> > > > > > > > > > > * On naming: dataquality over dq for me honestly. Our
> > > > existing
> > > > > > > > provider
> > > > > > > > > > > names spell things out
> > > > > > > > > > > (common.sql, openlineage, not abbreviated forms) and dq
> > is
> > > > > > > genuinely
> > > > > > > > > > > ambiguous outside context.
> > > > > > > > > > >
> > > > > > > > > > > * On scope: I also agree with Niko that #69413 is too
> > large
> > > > for
> > > > > > one
> > > > > > > > > pass
> > > > > > > > > > &
> > > > > > > > > > > I am glad to see the
> > > > > > > > > > > backend/UI split already happening in #69575. Would
> also
> > > > > suggest
> > > > > > > > > keeping
> > > > > > > > > > > the LLM assisted rule
> > > > > > > > > > > generation pieces (*schema-based
> > > generate_rules_from_schema*)
> > > > > out
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > initial provider PR entirely
> > > > > > > > > > > cos as I see it, its a separable capability and
> bundling
> > it
> > > > > will
> > > > > > > slow
> > > > > > > > > > > review of the core DQRule or
> > > > > > > > > > > RuleSet or operator surface, which is the part that
> > > actually
> > > > > > needs
> > > > > > > > the
> > > > > > > > > > most
> > > > > > > > > > > detailed review.
> > > > > > > > > > >
> > > > > > > > > > > In short: go for it!
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks & Regards,
> > > > > > > > > > > Amogh Desai
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <
> > > > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > In the meantime, the PR is ready for review. Feel
> free
> > to
> > > > > > review
> > > > > > > > and
> > > > > > > > > > > > provide any feedback.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Pavan
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <
> > > > > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Sorry, I forgot to add: the draft PR is here
> > > > > > > > > > > > > https://github.com/apache/airflow/pull/69413; it's
> > > > still a
> > > > > > > WIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > some screenshots
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> https://github.com/apache/airflow/pull/69413#issuecomment-4886311468
> > > > > > > > > > > :)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Pavan
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu
> <
> > > > > > > > > > > > > gopidesupavan@gmail.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Hi Airflow community,
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> I would like to start a discussion regarding a new
> > > > > provider:
> > > > > > > > > > > > >> apache-airflow-providers-dq.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> While Airflow already includes SQL check operators
> > > that
> > > > > many
> > > > > > > > users
> > > > > > > > > > > rely
> > > > > > > > > > > > >> on for data quality, this new provider builds on
> > that
> > > > > > > foundation
> > > > > > > > > by
> > > > > > > > > > > > >> introducing DQRule and RuleSet objects, stable
> rule
> > > > > > identity,
> > > > > > > > > > > persisted
> > > > > > > > > > > > >> history, and direct connections to Airflow assets.
> > > This
> > > > > > > approach
> > > > > > > > > > makes
> > > > > > > > > > > > >> quality results easier to inspect over time,
> allows
> > > > > > downstream
> > > > > > > > > > > > consumers to
> > > > > > > > > > > > >> gate tasks based on recent quality results, and
> > > > provides a
> > > > > > > > unified
> > > > > > > > > > > > schema
> > > > > > > > > > > > >> for LLM-assisted workflows. Execution will
> continue
> > to
> > > > > > utilize
> > > > > > > > > > > existing
> > > > > > > > > > > > >> DbApiHook connections.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> The initial version of the provider is
> intentionally
> > > > > > focused:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> - Declarative DQRule and RuleSet objects.
> > > > > > > > > > > > >> - DQCheckOperator and @task.dq_check.
> > > > > > > > > > > > >> - DbApiHook-based SQL checks, including built-in
> > > > checks
> > > > > > and
> > > > > > > > > > > > custom_sql.
> > > > > > > > > > > > >> - Persisted results for tasks, runs, and rules.
> > > > > > > > > > > > >> - A minimal Airflow UI plugin for viewing
> results
> > > and
> > > > > rule
> > > > > > > > > > history.
> > > > > > > > > > > > >> - Experimental asset helpers such as
> > asset_quality()
> > > > and
> > > > > > > > > > > > >> require_quality().
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Regarding scope, this first iteration uses object
> > > > storage
> > > > > > only
> > > > > > > > to
> > > > > > > > > > > > persist
> > > > > > > > > > > > >> DQ results and history; checks are executed via
> > > database
> > > > > > > > > > connections.
> > > > > > > > > > > > >> Future iterations may include file or object-store
> > > based
> > > > > > > checks
> > > > > > > > > > (e.g.,
> > > > > > > > > > > > S3,
> > > > > > > > > > > > >> GCS) where Airflow runs quality rules against data
> > > > > directly.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> This proposal does not require changes to Airflow
> > > core.
> > > > > > Asset
> > > > > > > > > > support
> > > > > > > > > > > is
> > > > > > > > > > > > >> currently provider-owned metadata, with static
> > > > > configuration
> > > > > > > > > stored
> > > > > > > > > > on
> > > > > > > > > > > > the
> > > > > > > > > > > > >> asset and runtime summaries stored on asset
> events.
> > If
> > > > the
> > > > > > > > > provider
> > > > > > > > > > > > gains
> > > > > > > > > > > > >> traction, we can discuss making Data Quality a
> > > > first-class
> > > > > > > > > component
> > > > > > > > > > > of
> > > > > > > > > > > > >> Airflow assets.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> This work also serves as a practical follow-up to
> > the
> > > > data
> > > > > > > > quality
> > > > > > > > > > > > >> direction mentioned in AIP-99. Persisted history
> is
> > > > > valuable
> > > > > > > for
> > > > > > > > > > users
> > > > > > > > > > > > and
> > > > > > > > > > > > >> future LLM-assisted workflows, such as those from
> > > > > Anthropic
> > > > > > or
> > > > > > > > > > > > common.ai,
> > > > > > > > > > > > >> to understand rule performance and generate
> > candidate
> > > > > rules
> > > > > > > > based
> > > > > > > > > on
> > > > > > > > > > > > schema
> > > > > > > > > > > > >> context.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> A rough pseudo-flow is provided below:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> seed_rules = RuleSet(
> > > > > > > > > > > > >> name="orders_quality",
> > > > > > > > > > > > >> rules=[
> > > > > > > > > > > > >> DQRule(name="order_id_not_null",
> > > > > check="null_count",
> > > > > > > > > > > > >> column="order_id", condition={"equal_to": 0}),
> > > > > > > > > > > > >> DQRule(name="amount_valid", check="min",
> > > > > > > > column="amount",
> > > > > > > > > > > > >> condition={"geq_to": 0}),
> > > > > > > > > > > > >> ],
> > > > > > > > > > > > >> )
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> orders_asset = asset_quality(
> > > > > > > > > > > > >> Asset("orders"),
> > > > > > > > > > > > >> conn_id="warehouse",
> > > > > > > > > > > > >> table="orders",
> > > > > > > > > > > > >> ruleset=seed_rules,
> > > > > > > > > > > > >> )
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> # Optional: common.ai / Anthropic provider can
> > > > generate a
> > > > > > > > RuleSet
> > > > > > > > > > > from
> > > > > > > > > > > > >> schema context.
> > > > > > > > > > > > >> generated_rules = generate_rules_from_schema(...)
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> @task.dq_check(asset=orders_asset)
> > > > > > > > > > > > >> def check_orders(ruleset):
> > > > > > > > > > > > >> return ruleset
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> checked_orders = check_orders(generated_rules)
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> with DAG("orders_consumer",
> schedule=orders_asset):
> > > > > > > > > > > > >> require_quality(orders_asset, min_score=0.95)
> >>
> > > > > > > > > > consume_orders()
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> The UI remains deliberately minimal for this
> initial
> > > > > > release,
> > > > > > > > > > focusing
> > > > > > > > > > > > on
> > > > > > > > > > > > >> result and history inspection.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> You can view examples [1] of how it's integrated
> > with
> > > > > > > > assets/llms.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> currently i named it providers
> > > > > > `apache-airflow-providers-dq`.
> > > > > > > if
> > > > > > > > > any
> > > > > > > > > > > > >> other preference likely with `dataquality`. Please
> > let
> > > > me
> > > > > > know
> > > > > > > > if
> > > > > > > > > > you
> > > > > > > > > > > > have
> > > > > > > > > > > > >> a preference. naming is hard :)
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> [1]:
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > >> Pavan
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: New Community Provider: Data Quality
Posted by Sameer Mesiah <sm...@gmail.com>.
Hi Pavan,
First, I would like to say that this new provider definitely adds value and
I liked the brief walkthrough you did today during the call. But after
going through the PR (and hearing Vikram's thoughts), there are a few
concerns that popped up:
1) I'm struggling to reconcile this with the existing philosophy of Airflow
providers. Most providers integrate with an external service or expose a
common abstraction over multiple services. This proposal instead introduces
a fairly comprehensive data quality framework that is implemented entirely
within Airflow. I am just wondering whether this should be a
standalone provider versus additional functionality that could be added to
the common provider.
2) I'm not yet sure who the primary target user is. If the intended
audience is Analytics Engineers, I'd expect the framework to expose a
simple declarative API for defining and extending reusable checks without
requiring changes to provider internals (you can correct me here if you
intend on exposing a simplified API eventually). In many organisations,
Airflow providers are owned by platform teams and are treated as
third-party dependencies, so modifying provider code to introduce new
reusable checks is often not a practical workflow. Arguably, the case for
this new framework is stronger if you are not going too deep into
domain-rich business rules but just want to limit it to operational checks
data platform owners might be interested in. To ground my critique with a
very straightforward question: if you intend for this new provider to
support domain-rich data validation, what benefit would it have over tools
like DBT which are more accessible to practitioners of this specific area
i.e Analytics Engineers. Vikram did share the perspective of a Data
Engineer who is building data pipelines but I just wanted to expand on that
a bit more by differentiating between the needs of platform-leaning and
analytics-leaning Data Engineers. These 2 roles are increasingly bifurcated
and I think we should really think about which group we are targeting here.
3) This proposal introduces a significant amount of new framework surface
area that Airflow will own and maintain going forward. Beyond the operator
itself, we're introducing rule definitions, persistence, history, scoring,
asset integration, decorators, configuration, etc. That represents a
substantial long-term maintenance commitment, so I think it's worth
discussing whether the value provided justifies that scope and whether this
is the right architectural boundary for Airflow.
Overall, I would say I am +0.5. It's a good proposal but it needs more
refinement because we do not want to introduce a provider that our users
are just not going to want to use.
On Wed, 15 Jul 2026 at 21:27, Pavankumar Gopidesu <go...@gmail.com>
wrote:
> Hi Vikram,
>
> Thanks for the feedback. I agree that helpers would be useful for users who
> want to run data quality checks from existing tasks or operators, without
> always needing to use a dedicated DQ operator.
>
> I updated the PR to include
>
> - importable helpers for running and persisting checks inside tasks or
> any existing operators.
>
> For asset integration, this PR does not require Airflow core changes. The
> provider adds two asset-oriented helpers:
>
> - asset_quality() to attach quality configuration to an asset. This lets
> the asset carry the ruleset that is expected to be executed for it. In
> the
> current implementation this is stored as provider-owned metadata in the
> asset extra field.
> - require_quality() to let downstream tasks gate on the latest quality
> result from the triggering asset event.
>
> Below is an example view of how assets can use data quality with
> partition-aware producer and consumer Dags.
>
> ```
>
> *Example ruleset:*
>
> orders_quality_rules = RuleSet(
> name="orders_quality",
> rules=[
> DQRule(
> id="orders.amount_non_negative",
> name="amount_non_negative",
> check="min",
> column="amount",
> condition={"geq_to": 0},
> ),
> ],
> )
>
> raw_orders = asset_quality(
> Asset("daily_raw_orders"),
> ruleset=orders_quality_rules,
> conn_id="warehouse",
> table="raw_orders",
> )
>
> *Producer Dag:*
>
> with DAG(
> dag_id="daily_raw_orders",
> schedule=CronPartitionTimetable("0 2 * * *", timezone="UTC"),
> ):
> @task(outlets=[raw_orders])
> def load_and_check():
> context = get_current_context()
> partition_key = context["partition_key"]
>
> load_raw_orders(partition_key)
>
> result = run_quality_checks(
> ruleset=orders_quality_rules,
> table="raw_orders",
> partition_clause=f"ds = '{partition_key}'",
> )
>
> persist_quality_results(
> result,
> context=context,
> outlets=[raw_orders],
> )
>
> load_and_check()
>
>
> *Consumer Dag:*
>
> with DAG(
> dag_id="curate_daily_orders",
> schedule=PartitionedAssetTimetable(assets=raw_orders),
> ):
> quality_gate = require_quality(raw_orders, min_score=1.0)
>
> @task
> def curate():
> context = get_current_context()
> partition_key = context["partition_key"]
>
> curate_orders(partition_key)
>
> quality_gate >> curate()
>
> ```
>
> The flow is:
>
> 1. A partitioned producer Dag runs and loads data for the current
> partition.
> 2. The producer runs run_quality_checks() with a partition-scoped
> filter.
> 3. persist_quality_results() records the result and adds a DQ summary to
> the produced asset event.
> 4. PartitionedAssetTimetable triggers the consumer Dag for that
> partition.
> 5. require_quality() reads the DQ summary from the triggering asset
> event.
> 6. The consumer processes the partition only if quality passes.
>
> Hope this makes sense?
>
> Regards,
>
> Pavan
>
>
> On Wed, Jul 15, 2026 at 6:47 PM Vikram Koka via dev <
> dev@airflow.apache.org>
> wrote:
>
> > Hey Pavan,
> >
> > Thanks for bringing this up and working on it. I strongly support the
> > overall concept of the Data Quality provider.
> >
> > However, I have two concerns / questions:
> > 1. I am unsure of the Operator approach shown in the PR.
> > I am not certain that these SHOULD be separate Operators which result in
> > separate tasks.
> > If these are made available as Helper functions, they could be included
> in
> > existing Operators.
> > I am not saying they shouldn't be separate Tasks in Dags, but the Helper
> > function approach would provide more optionality.
> >
> > 2. I am wondering how to integrate these with Asset Partitions
> > I do believe this can be integrated with the Asset Partitions concept,
> but
> > I am personally unclear on exactly how.
> > I think adding an example illustrating this would be very helpful.
> >
> > Best regards,
> > Vikram
> >
> >
> > On Mon, Jul 13, 2026 at 3:26 PM Pavankumar Gopidesu <
> > gopidesupavan@gmail.com>
> > wrote:
> >
> > > Thanks Kaxil..
> > >
> > > Thanks Everyone, consensus looks positive; I will merge the skeleton PR
> > > first.
> > >
> > > Regards,
> > > Pavan
> > >
> > > On Mon, Jul 13, 2026 at 10:05 PM Kaxil Naik <ka...@gmail.com>
> wrote:
> > >
> > > > +1 for the high-level need for such data quality checks. Thanks
> Pavan,
> > > will
> > > > review the PRs shortly.
> > > >
> > > > On Fri, 10 Jul 2026 at 01:58, Pavankumar Gopidesu <
> > > gopidesupavan@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Thanks Bugra,
> > > > >
> > > > > I have updated the naming convention now.
> > > > >
> > > > > Pavan
> > > > >
> > > > > On Wed, Jul 8, 2026 at 7:07 PM Buğra Öztürk <
> ozturkbugra93@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Thanks Pavan for bringing this together and starting the
> > discussion!
> > > > > > Sounds good! +1 on the idea.
> > > > > >
> > > > > > Harder than solving problems. Not a strong suggestion, but
> > > > > > `common-dataquality` sounds more reasonable to me. It also adds
> the
> > > > value
> > > > > > of the `common` part, which provides the separation pattern Jarek
> > > > > > mentioned. It gives a better understanding that it is a common
> > > > offering.
> > > > > >
> > > > > > Best regards,
> > > > > > Bugra Ozturk
> > > > > >
> > > > > > On Wed, Jul 8, 2026 at 7:15 PM Pavankumar Gopidesu <
> > > > > > gopidesupavan@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Jarek, I agree that the separate provider approach
> offers
> > > much
> > > > > > more
> > > > > > > flexibility for iterating on features and fixes.
> > > > > > >
> > > > > > > Naming is always hard :)
> > > > > > >
> > > > > > > Option 1: apache-airflow-providers-dataquality
> > > > > > > Option 2: apache-airflow-providers-common-dataquality (This
> goes
> > > > inside
> > > > > > the
> > > > > > > common providers folder we already have)
> > > > > > >
> > > > > > > So, I am up for either option :)
> > > > > > >
> > > > > > > have removed first short name `apache-airflow-providers-dq`.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Pavan
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk <jarek@potiuk.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > +1 Good design/ idea. No objections - dataquality is a good
> > name
> > > -
> > > > > but
> > > > > > I
> > > > > > > > would also consider `common-dataquality" - even if it's
> longer,
> > > it
> > > > > > builds
> > > > > > > > on the pattern we have already with common-ai. But not a
> > blocker.
> > > > > > > >
> > > > > > > > I also think it's good to have it as a separate provider,
> even
> > if
> > > > it
> > > > > > > gains
> > > > > > > > traction for two reasons:
> > > > > > > >
> > > > > > > > a) ability to add features or fix issues independently from
> the
> > > > core
> > > > > > > > b) an explicit "optional" feature that is easy to promote
> > > > > > > >
> > > > > > > > I think what we saw with common is that people see airflow
> > > already
> > > > as
> > > > > > too
> > > > > > > > heavy - and "too many releases" sometimes, so quite
> > > > > > counter-intuitively -
> > > > > > > > by having separate providers adding features that "hook in"
> > > > existing
> > > > > > > > functionalities of core - we do not make airflow "heavier"
> and
> > we
> > > > do
> > > > > > not
> > > > > > > > force people to migrating to future newer versions to use new
> > > > > features.
> > > > > > > >
> > > > > > > > J.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu <
> > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Amogh,
> > > > > > > > >
> > > > > > > > > Thanks for the feedback.
> > > > > > > > >
> > > > > > > > > I am happy to change the provider name to dataquality.
> > > > > > > > >
> > > > > > > > > Regarding the LLM-assisted features, the current PR does
> not
> > > > > include
> > > > > > > any
> > > > > > > > > implementation. It only adds the SKILLS [1 ]and the
> reference
> > > > > schema
> > > > > > > for
> > > > > > > > > the DQ Rule structure. Are you suggesting that I move this
> > > SKILL
> > > > > > > > > documentation to a separate PR?
> > > > > > > > >
> > > > > > > > > [1]:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Pavan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai <
> > > > amoghdesai@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Pavan,
> > > > > > > > > >
> > > > > > > > > > First of all, +1 to this.
> > > > > > > > > >
> > > > > > > > > > Now, few things:
> > > > > > > > > >
> > > > > > > > > > * On naming: dataquality over dq for me honestly. Our
> > > existing
> > > > > > > provider
> > > > > > > > > > names spell things out
> > > > > > > > > > (common.sql, openlineage, not abbreviated forms) and dq
> is
> > > > > > genuinely
> > > > > > > > > > ambiguous outside context.
> > > > > > > > > >
> > > > > > > > > > * On scope: I also agree with Niko that #69413 is too
> large
> > > for
> > > > > one
> > > > > > > > pass
> > > > > > > > > &
> > > > > > > > > > I am glad to see the
> > > > > > > > > > backend/UI split already happening in #69575. Would also
> > > > suggest
> > > > > > > > keeping
> > > > > > > > > > the LLM assisted rule
> > > > > > > > > > generation pieces (*schema-based
> > generate_rules_from_schema*)
> > > > out
> > > > > > of
> > > > > > > > the
> > > > > > > > > > initial provider PR entirely
> > > > > > > > > > cos as I see it, its a separable capability and bundling
> it
> > > > will
> > > > > > slow
> > > > > > > > > > review of the core DQRule or
> > > > > > > > > > RuleSet or operator surface, which is the part that
> > actually
> > > > > needs
> > > > > > > the
> > > > > > > > > most
> > > > > > > > > > detailed review.
> > > > > > > > > >
> > > > > > > > > > In short: go for it!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks & Regards,
> > > > > > > > > > Amogh Desai
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <
> > > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > In the meantime, the PR is ready for review. Feel free
> to
> > > > > review
> > > > > > > and
> > > > > > > > > > > provide any feedback.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Pavan
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <
> > > > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Sorry, I forgot to add: the draft PR is here
> > > > > > > > > > > > https://github.com/apache/airflow/pull/69413; it's
> > > still a
> > > > > > WIP.
> > > > > > > > > > > >
> > > > > > > > > > > > some screenshots
> > > > > > > > > > > >
> > > > > > > >
> > > > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468
> > > > > > > > > > :)
> > > > > > > > > > > >
> > > > > > > > > > > > Pavan
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <
> > > > > > > > > > > > gopidesupavan@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Hi Airflow community,
> > > > > > > > > > > >>
> > > > > > > > > > > >> I would like to start a discussion regarding a new
> > > > provider:
> > > > > > > > > > > >> apache-airflow-providers-dq.
> > > > > > > > > > > >>
> > > > > > > > > > > >> While Airflow already includes SQL check operators
> > that
> > > > many
> > > > > > > users
> > > > > > > > > > rely
> > > > > > > > > > > >> on for data quality, this new provider builds on
> that
> > > > > > foundation
> > > > > > > > by
> > > > > > > > > > > >> introducing DQRule and RuleSet objects, stable rule
> > > > > identity,
> > > > > > > > > > persisted
> > > > > > > > > > > >> history, and direct connections to Airflow assets.
> > This
> > > > > > approach
> > > > > > > > > makes
> > > > > > > > > > > >> quality results easier to inspect over time, allows
> > > > > downstream
> > > > > > > > > > > consumers to
> > > > > > > > > > > >> gate tasks based on recent quality results, and
> > > provides a
> > > > > > > unified
> > > > > > > > > > > schema
> > > > > > > > > > > >> for LLM-assisted workflows. Execution will continue
> to
> > > > > utilize
> > > > > > > > > > existing
> > > > > > > > > > > >> DbApiHook connections.
> > > > > > > > > > > >>
> > > > > > > > > > > >> The initial version of the provider is intentionally
> > > > > focused:
> > > > > > > > > > > >>
> > > > > > > > > > > >> - Declarative DQRule and RuleSet objects.
> > > > > > > > > > > >> - DQCheckOperator and @task.dq_check.
> > > > > > > > > > > >> - DbApiHook-based SQL checks, including built-in
> > > checks
> > > > > and
> > > > > > > > > > > custom_sql.
> > > > > > > > > > > >> - Persisted results for tasks, runs, and rules.
> > > > > > > > > > > >> - A minimal Airflow UI plugin for viewing results
> > and
> > > > rule
> > > > > > > > > history.
> > > > > > > > > > > >> - Experimental asset helpers such as
> asset_quality()
> > > and
> > > > > > > > > > > >> require_quality().
> > > > > > > > > > > >>
> > > > > > > > > > > >> Regarding scope, this first iteration uses object
> > > storage
> > > > > only
> > > > > > > to
> > > > > > > > > > > persist
> > > > > > > > > > > >> DQ results and history; checks are executed via
> > database
> > > > > > > > > connections.
> > > > > > > > > > > >> Future iterations may include file or object-store
> > based
> > > > > > checks
> > > > > > > > > (e.g.,
> > > > > > > > > > > S3,
> > > > > > > > > > > >> GCS) where Airflow runs quality rules against data
> > > > directly.
> > > > > > > > > > > >>
> > > > > > > > > > > >> This proposal does not require changes to Airflow
> > core.
> > > > > Asset
> > > > > > > > > support
> > > > > > > > > > is
> > > > > > > > > > > >> currently provider-owned metadata, with static
> > > > configuration
> > > > > > > > stored
> > > > > > > > > on
> > > > > > > > > > > the
> > > > > > > > > > > >> asset and runtime summaries stored on asset events.
> If
> > > the
> > > > > > > > provider
> > > > > > > > > > > gains
> > > > > > > > > > > >> traction, we can discuss making Data Quality a
> > > first-class
> > > > > > > > component
> > > > > > > > > > of
> > > > > > > > > > > >> Airflow assets.
> > > > > > > > > > > >>
> > > > > > > > > > > >> This work also serves as a practical follow-up to
> the
> > > data
> > > > > > > quality
> > > > > > > > > > > >> direction mentioned in AIP-99. Persisted history is
> > > > valuable
> > > > > > for
> > > > > > > > > users
> > > > > > > > > > > and
> > > > > > > > > > > >> future LLM-assisted workflows, such as those from
> > > > Anthropic
> > > > > or
> > > > > > > > > > > common.ai,
> > > > > > > > > > > >> to understand rule performance and generate
> candidate
> > > > rules
> > > > > > > based
> > > > > > > > on
> > > > > > > > > > > schema
> > > > > > > > > > > >> context.
> > > > > > > > > > > >>
> > > > > > > > > > > >> A rough pseudo-flow is provided below:
> > > > > > > > > > > >>
> > > > > > > > > > > >> seed_rules = RuleSet(
> > > > > > > > > > > >> name="orders_quality",
> > > > > > > > > > > >> rules=[
> > > > > > > > > > > >> DQRule(name="order_id_not_null",
> > > > check="null_count",
> > > > > > > > > > > >> column="order_id", condition={"equal_to": 0}),
> > > > > > > > > > > >> DQRule(name="amount_valid", check="min",
> > > > > > > column="amount",
> > > > > > > > > > > >> condition={"geq_to": 0}),
> > > > > > > > > > > >> ],
> > > > > > > > > > > >> )
> > > > > > > > > > > >>
> > > > > > > > > > > >> orders_asset = asset_quality(
> > > > > > > > > > > >> Asset("orders"),
> > > > > > > > > > > >> conn_id="warehouse",
> > > > > > > > > > > >> table="orders",
> > > > > > > > > > > >> ruleset=seed_rules,
> > > > > > > > > > > >> )
> > > > > > > > > > > >>
> > > > > > > > > > > >> # Optional: common.ai / Anthropic provider can
> > > generate a
> > > > > > > RuleSet
> > > > > > > > > > from
> > > > > > > > > > > >> schema context.
> > > > > > > > > > > >> generated_rules = generate_rules_from_schema(...)
> > > > > > > > > > > >>
> > > > > > > > > > > >> @task.dq_check(asset=orders_asset)
> > > > > > > > > > > >> def check_orders(ruleset):
> > > > > > > > > > > >> return ruleset
> > > > > > > > > > > >>
> > > > > > > > > > > >> checked_orders = check_orders(generated_rules)
> > > > > > > > > > > >>
> > > > > > > > > > > >> with DAG("orders_consumer", schedule=orders_asset):
> > > > > > > > > > > >> require_quality(orders_asset, min_score=0.95) >>
> > > > > > > > > consume_orders()
> > > > > > > > > > > >>
> > > > > > > > > > > >> The UI remains deliberately minimal for this initial
> > > > > release,
> > > > > > > > > focusing
> > > > > > > > > > > on
> > > > > > > > > > > >> result and history inspection.
> > > > > > > > > > > >>
> > > > > > > > > > > >> You can view examples [1] of how it's integrated
> with
> > > > > > > assets/llms.
> > > > > > > > > > > >>
> > > > > > > > > > > >> currently i named it providers
> > > > > `apache-airflow-providers-dq`.
> > > > > > if
> > > > > > > > any
> > > > > > > > > > > >> other preference likely with `dataquality`. Please
> let
> > > me
> > > > > know
> > > > > > > if
> > > > > > > > > you
> > > > > > > > > > > have
> > > > > > > > > > > >> a preference. naming is hard :)
> > > > > > > > > > > >>
> > > > > > > > > > > >> [1]:
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> > > > > > > > > > > >>
> > > > > > > > > > > >> Thanks,
> > > > > > > > > > > >> Pavan
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: New Community Provider: Data Quality
Posted by Pavankumar Gopidesu <go...@gmail.com>.
Hi Vikram,
Thanks for the feedback. I agree that helpers would be useful for users who
want to run data quality checks from existing tasks or operators, without
always needing to use a dedicated DQ operator.
I updated the PR to include
- importable helpers for running and persisting checks inside tasks or
any existing operators.
For asset integration, this PR does not require Airflow core changes. The
provider adds two asset-oriented helpers:
- asset_quality() to attach quality configuration to an asset. This lets
the asset carry the ruleset that is expected to be executed for it. In the
current implementation this is stored as provider-owned metadata in the
asset extra field.
- require_quality() to let downstream tasks gate on the latest quality
result from the triggering asset event.
Below is an example view of how assets can use data quality with
partition-aware producer and consumer Dags.
```
*Example ruleset:*
orders_quality_rules = RuleSet(
name="orders_quality",
rules=[
DQRule(
id="orders.amount_non_negative",
name="amount_non_negative",
check="min",
column="amount",
condition={"geq_to": 0},
),
],
)
raw_orders = asset_quality(
Asset("daily_raw_orders"),
ruleset=orders_quality_rules,
conn_id="warehouse",
table="raw_orders",
)
*Producer Dag:*
with DAG(
dag_id="daily_raw_orders",
schedule=CronPartitionTimetable("0 2 * * *", timezone="UTC"),
):
@task(outlets=[raw_orders])
def load_and_check():
context = get_current_context()
partition_key = context["partition_key"]
load_raw_orders(partition_key)
result = run_quality_checks(
ruleset=orders_quality_rules,
table="raw_orders",
partition_clause=f"ds = '{partition_key}'",
)
persist_quality_results(
result,
context=context,
outlets=[raw_orders],
)
load_and_check()
*Consumer Dag:*
with DAG(
dag_id="curate_daily_orders",
schedule=PartitionedAssetTimetable(assets=raw_orders),
):
quality_gate = require_quality(raw_orders, min_score=1.0)
@task
def curate():
context = get_current_context()
partition_key = context["partition_key"]
curate_orders(partition_key)
quality_gate >> curate()
```
The flow is:
1. A partitioned producer Dag runs and loads data for the current
partition.
2. The producer runs run_quality_checks() with a partition-scoped filter.
3. persist_quality_results() records the result and adds a DQ summary to
the produced asset event.
4. PartitionedAssetTimetable triggers the consumer Dag for that
partition.
5. require_quality() reads the DQ summary from the triggering asset
event.
6. The consumer processes the partition only if quality passes.
Hope this makes sense?
Regards,
Pavan
On Wed, Jul 15, 2026 at 6:47 PM Vikram Koka via dev <de...@airflow.apache.org>
wrote:
> Hey Pavan,
>
> Thanks for bringing this up and working on it. I strongly support the
> overall concept of the Data Quality provider.
>
> However, I have two concerns / questions:
> 1. I am unsure of the Operator approach shown in the PR.
> I am not certain that these SHOULD be separate Operators which result in
> separate tasks.
> If these are made available as Helper functions, they could be included in
> existing Operators.
> I am not saying they shouldn't be separate Tasks in Dags, but the Helper
> function approach would provide more optionality.
>
> 2. I am wondering how to integrate these with Asset Partitions
> I do believe this can be integrated with the Asset Partitions concept, but
> I am personally unclear on exactly how.
> I think adding an example illustrating this would be very helpful.
>
> Best regards,
> Vikram
>
>
> On Mon, Jul 13, 2026 at 3:26 PM Pavankumar Gopidesu <
> gopidesupavan@gmail.com>
> wrote:
>
> > Thanks Kaxil..
> >
> > Thanks Everyone, consensus looks positive; I will merge the skeleton PR
> > first.
> >
> > Regards,
> > Pavan
> >
> > On Mon, Jul 13, 2026 at 10:05 PM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > +1 for the high-level need for such data quality checks. Thanks Pavan,
> > will
> > > review the PRs shortly.
> > >
> > > On Fri, 10 Jul 2026 at 01:58, Pavankumar Gopidesu <
> > gopidesupavan@gmail.com
> > > >
> > > wrote:
> > >
> > > > Thanks Bugra,
> > > >
> > > > I have updated the naming convention now.
> > > >
> > > > Pavan
> > > >
> > > > On Wed, Jul 8, 2026 at 7:07 PM Buğra Öztürk <ozturkbugra93@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Thanks Pavan for bringing this together and starting the
> discussion!
> > > > > Sounds good! +1 on the idea.
> > > > >
> > > > > Harder than solving problems. Not a strong suggestion, but
> > > > > `common-dataquality` sounds more reasonable to me. It also adds the
> > > value
> > > > > of the `common` part, which provides the separation pattern Jarek
> > > > > mentioned. It gives a better understanding that it is a common
> > > offering.
> > > > >
> > > > > Best regards,
> > > > > Bugra Ozturk
> > > > >
> > > > > On Wed, Jul 8, 2026 at 7:15 PM Pavankumar Gopidesu <
> > > > > gopidesupavan@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks Jarek, I agree that the separate provider approach offers
> > much
> > > > > more
> > > > > > flexibility for iterating on features and fixes.
> > > > > >
> > > > > > Naming is always hard :)
> > > > > >
> > > > > > Option 1: apache-airflow-providers-dataquality
> > > > > > Option 2: apache-airflow-providers-common-dataquality (This goes
> > > inside
> > > > > the
> > > > > > common providers folder we already have)
> > > > > >
> > > > > > So, I am up for either option :)
> > > > > >
> > > > > > have removed first short name `apache-airflow-providers-dq`.
> > > > > >
> > > > > > Thanks,
> > > > > > Pavan
> > > > > >
> > > > > >
> > > > > > On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > > > >
> > > > > > > +1 Good design/ idea. No objections - dataquality is a good
> name
> > -
> > > > but
> > > > > I
> > > > > > > would also consider `common-dataquality" - even if it's longer,
> > it
> > > > > builds
> > > > > > > on the pattern we have already with common-ai. But not a
> blocker.
> > > > > > >
> > > > > > > I also think it's good to have it as a separate provider, even
> if
> > > it
> > > > > > gains
> > > > > > > traction for two reasons:
> > > > > > >
> > > > > > > a) ability to add features or fix issues independently from the
> > > core
> > > > > > > b) an explicit "optional" feature that is easy to promote
> > > > > > >
> > > > > > > I think what we saw with common is that people see airflow
> > already
> > > as
> > > > > too
> > > > > > > heavy - and "too many releases" sometimes, so quite
> > > > > counter-intuitively -
> > > > > > > by having separate providers adding features that "hook in"
> > > existing
> > > > > > > functionalities of core - we do not make airflow "heavier" and
> we
> > > do
> > > > > not
> > > > > > > force people to migrating to future newer versions to use new
> > > > features.
> > > > > > >
> > > > > > > J.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu <
> > > > > > > gopidesupavan@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Amogh,
> > > > > > > >
> > > > > > > > Thanks for the feedback.
> > > > > > > >
> > > > > > > > I am happy to change the provider name to dataquality.
> > > > > > > >
> > > > > > > > Regarding the LLM-assisted features, the current PR does not
> > > > include
> > > > > > any
> > > > > > > > implementation. It only adds the SKILLS [1 ]and the reference
> > > > schema
> > > > > > for
> > > > > > > > the DQ Rule structure. Are you suggesting that I move this
> > SKILL
> > > > > > > > documentation to a separate PR?
> > > > > > > >
> > > > > > > > [1]:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Pavan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai <
> > > amoghdesai@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Pavan,
> > > > > > > > >
> > > > > > > > > First of all, +1 to this.
> > > > > > > > >
> > > > > > > > > Now, few things:
> > > > > > > > >
> > > > > > > > > * On naming: dataquality over dq for me honestly. Our
> > existing
> > > > > > provider
> > > > > > > > > names spell things out
> > > > > > > > > (common.sql, openlineage, not abbreviated forms) and dq is
> > > > > genuinely
> > > > > > > > > ambiguous outside context.
> > > > > > > > >
> > > > > > > > > * On scope: I also agree with Niko that #69413 is too large
> > for
> > > > one
> > > > > > > pass
> > > > > > > > &
> > > > > > > > > I am glad to see the
> > > > > > > > > backend/UI split already happening in #69575. Would also
> > > suggest
> > > > > > > keeping
> > > > > > > > > the LLM assisted rule
> > > > > > > > > generation pieces (*schema-based
> generate_rules_from_schema*)
> > > out
> > > > > of
> > > > > > > the
> > > > > > > > > initial provider PR entirely
> > > > > > > > > cos as I see it, its a separable capability and bundling it
> > > will
> > > > > slow
> > > > > > > > > review of the core DQRule or
> > > > > > > > > RuleSet or operator surface, which is the part that
> actually
> > > > needs
> > > > > > the
> > > > > > > > most
> > > > > > > > > detailed review.
> > > > > > > > >
> > > > > > > > > In short: go for it!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks & Regards,
> > > > > > > > > Amogh Desai
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <
> > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > In the meantime, the PR is ready for review. Feel free to
> > > > review
> > > > > > and
> > > > > > > > > > provide any feedback.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Pavan
> > > > > > > > > >
> > > > > > > > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <
> > > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Sorry, I forgot to add: the draft PR is here
> > > > > > > > > > > https://github.com/apache/airflow/pull/69413; it's
> > still a
> > > > > WIP.
> > > > > > > > > > >
> > > > > > > > > > > some screenshots
> > > > > > > > > > >
> > > > > > >
> > > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468
> > > > > > > > > :)
> > > > > > > > > > >
> > > > > > > > > > > Pavan
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <
> > > > > > > > > > > gopidesupavan@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi Airflow community,
> > > > > > > > > > >>
> > > > > > > > > > >> I would like to start a discussion regarding a new
> > > provider:
> > > > > > > > > > >> apache-airflow-providers-dq.
> > > > > > > > > > >>
> > > > > > > > > > >> While Airflow already includes SQL check operators
> that
> > > many
> > > > > > users
> > > > > > > > > rely
> > > > > > > > > > >> on for data quality, this new provider builds on that
> > > > > foundation
> > > > > > > by
> > > > > > > > > > >> introducing DQRule and RuleSet objects, stable rule
> > > > identity,
> > > > > > > > > persisted
> > > > > > > > > > >> history, and direct connections to Airflow assets.
> This
> > > > > approach
> > > > > > > > makes
> > > > > > > > > > >> quality results easier to inspect over time, allows
> > > > downstream
> > > > > > > > > > consumers to
> > > > > > > > > > >> gate tasks based on recent quality results, and
> > provides a
> > > > > > unified
> > > > > > > > > > schema
> > > > > > > > > > >> for LLM-assisted workflows. Execution will continue to
> > > > utilize
> > > > > > > > > existing
> > > > > > > > > > >> DbApiHook connections.
> > > > > > > > > > >>
> > > > > > > > > > >> The initial version of the provider is intentionally
> > > > focused:
> > > > > > > > > > >>
> > > > > > > > > > >> - Declarative DQRule and RuleSet objects.
> > > > > > > > > > >> - DQCheckOperator and @task.dq_check.
> > > > > > > > > > >> - DbApiHook-based SQL checks, including built-in
> > checks
> > > > and
> > > > > > > > > > custom_sql.
> > > > > > > > > > >> - Persisted results for tasks, runs, and rules.
> > > > > > > > > > >> - A minimal Airflow UI plugin for viewing results
> and
> > > rule
> > > > > > > > history.
> > > > > > > > > > >> - Experimental asset helpers such as asset_quality()
> > and
> > > > > > > > > > >> require_quality().
> > > > > > > > > > >>
> > > > > > > > > > >> Regarding scope, this first iteration uses object
> > storage
> > > > only
> > > > > > to
> > > > > > > > > > persist
> > > > > > > > > > >> DQ results and history; checks are executed via
> database
> > > > > > > > connections.
> > > > > > > > > > >> Future iterations may include file or object-store
> based
> > > > > checks
> > > > > > > > (e.g.,
> > > > > > > > > > S3,
> > > > > > > > > > >> GCS) where Airflow runs quality rules against data
> > > directly.
> > > > > > > > > > >>
> > > > > > > > > > >> This proposal does not require changes to Airflow
> core.
> > > > Asset
> > > > > > > > support
> > > > > > > > > is
> > > > > > > > > > >> currently provider-owned metadata, with static
> > > configuration
> > > > > > > stored
> > > > > > > > on
> > > > > > > > > > the
> > > > > > > > > > >> asset and runtime summaries stored on asset events. If
> > the
> > > > > > > provider
> > > > > > > > > > gains
> > > > > > > > > > >> traction, we can discuss making Data Quality a
> > first-class
> > > > > > > component
> > > > > > > > > of
> > > > > > > > > > >> Airflow assets.
> > > > > > > > > > >>
> > > > > > > > > > >> This work also serves as a practical follow-up to the
> > data
> > > > > > quality
> > > > > > > > > > >> direction mentioned in AIP-99. Persisted history is
> > > valuable
> > > > > for
> > > > > > > > users
> > > > > > > > > > and
> > > > > > > > > > >> future LLM-assisted workflows, such as those from
> > > Anthropic
> > > > or
> > > > > > > > > > common.ai,
> > > > > > > > > > >> to understand rule performance and generate candidate
> > > rules
> > > > > > based
> > > > > > > on
> > > > > > > > > > schema
> > > > > > > > > > >> context.
> > > > > > > > > > >>
> > > > > > > > > > >> A rough pseudo-flow is provided below:
> > > > > > > > > > >>
> > > > > > > > > > >> seed_rules = RuleSet(
> > > > > > > > > > >> name="orders_quality",
> > > > > > > > > > >> rules=[
> > > > > > > > > > >> DQRule(name="order_id_not_null",
> > > check="null_count",
> > > > > > > > > > >> column="order_id", condition={"equal_to": 0}),
> > > > > > > > > > >> DQRule(name="amount_valid", check="min",
> > > > > > column="amount",
> > > > > > > > > > >> condition={"geq_to": 0}),
> > > > > > > > > > >> ],
> > > > > > > > > > >> )
> > > > > > > > > > >>
> > > > > > > > > > >> orders_asset = asset_quality(
> > > > > > > > > > >> Asset("orders"),
> > > > > > > > > > >> conn_id="warehouse",
> > > > > > > > > > >> table="orders",
> > > > > > > > > > >> ruleset=seed_rules,
> > > > > > > > > > >> )
> > > > > > > > > > >>
> > > > > > > > > > >> # Optional: common.ai / Anthropic provider can
> > generate a
> > > > > > RuleSet
> > > > > > > > > from
> > > > > > > > > > >> schema context.
> > > > > > > > > > >> generated_rules = generate_rules_from_schema(...)
> > > > > > > > > > >>
> > > > > > > > > > >> @task.dq_check(asset=orders_asset)
> > > > > > > > > > >> def check_orders(ruleset):
> > > > > > > > > > >> return ruleset
> > > > > > > > > > >>
> > > > > > > > > > >> checked_orders = check_orders(generated_rules)
> > > > > > > > > > >>
> > > > > > > > > > >> with DAG("orders_consumer", schedule=orders_asset):
> > > > > > > > > > >> require_quality(orders_asset, min_score=0.95) >>
> > > > > > > > consume_orders()
> > > > > > > > > > >>
> > > > > > > > > > >> The UI remains deliberately minimal for this initial
> > > > release,
> > > > > > > > focusing
> > > > > > > > > > on
> > > > > > > > > > >> result and history inspection.
> > > > > > > > > > >>
> > > > > > > > > > >> You can view examples [1] of how it's integrated with
> > > > > > assets/llms.
> > > > > > > > > > >>
> > > > > > > > > > >> currently i named it providers
> > > > `apache-airflow-providers-dq`.
> > > > > if
> > > > > > > any
> > > > > > > > > > >> other preference likely with `dataquality`. Please let
> > me
> > > > know
> > > > > > if
> > > > > > > > you
> > > > > > > > > > have
> > > > > > > > > > >> a preference. naming is hard :)
> > > > > > > > > > >>
> > > > > > > > > > >> [1]:
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >> Pavan
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: New Community Provider: Data Quality
Posted by Vikram Koka via dev <de...@airflow.apache.org>.
Hey Pavan,
Thanks for bringing this up and working on it. I strongly support the
overall concept of the Data Quality provider.
However, I have two concerns / questions:
1. I am unsure of the Operator approach shown in the PR.
I am not certain that these SHOULD be separate Operators which result in
separate tasks.
If these are made available as Helper functions, they could be included in
existing Operators.
I am not saying they shouldn't be separate Tasks in Dags, but the Helper
function approach would provide more optionality.
2. I am wondering how to integrate these with Asset Partitions
I do believe this can be integrated with the Asset Partitions concept, but
I am personally unclear on exactly how.
I think adding an example illustrating this would be very helpful.
Best regards,
Vikram
On Mon, Jul 13, 2026 at 3:26 PM Pavankumar Gopidesu <go...@gmail.com>
wrote:
> Thanks Kaxil..
>
> Thanks Everyone, consensus looks positive; I will merge the skeleton PR
> first.
>
> Regards,
> Pavan
>
> On Mon, Jul 13, 2026 at 10:05 PM Kaxil Naik <ka...@gmail.com> wrote:
>
> > +1 for the high-level need for such data quality checks. Thanks Pavan,
> will
> > review the PRs shortly.
> >
> > On Fri, 10 Jul 2026 at 01:58, Pavankumar Gopidesu <
> gopidesupavan@gmail.com
> > >
> > wrote:
> >
> > > Thanks Bugra,
> > >
> > > I have updated the naming convention now.
> > >
> > > Pavan
> > >
> > > On Wed, Jul 8, 2026 at 7:07 PM Buğra Öztürk <oz...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Pavan for bringing this together and starting the discussion!
> > > > Sounds good! +1 on the idea.
> > > >
> > > > Harder than solving problems. Not a strong suggestion, but
> > > > `common-dataquality` sounds more reasonable to me. It also adds the
> > value
> > > > of the `common` part, which provides the separation pattern Jarek
> > > > mentioned. It gives a better understanding that it is a common
> > offering.
> > > >
> > > > Best regards,
> > > > Bugra Ozturk
> > > >
> > > > On Wed, Jul 8, 2026 at 7:15 PM Pavankumar Gopidesu <
> > > > gopidesupavan@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Jarek, I agree that the separate provider approach offers
> much
> > > > more
> > > > > flexibility for iterating on features and fixes.
> > > > >
> > > > > Naming is always hard :)
> > > > >
> > > > > Option 1: apache-airflow-providers-dataquality
> > > > > Option 2: apache-airflow-providers-common-dataquality (This goes
> > inside
> > > > the
> > > > > common providers folder we already have)
> > > > >
> > > > > So, I am up for either option :)
> > > > >
> > > > > have removed first short name `apache-airflow-providers-dq`.
> > > > >
> > > > > Thanks,
> > > > > Pavan
> > > > >
> > > > >
> > > > > On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > > >
> > > > > > +1 Good design/ idea. No objections - dataquality is a good name
> -
> > > but
> > > > I
> > > > > > would also consider `common-dataquality" - even if it's longer,
> it
> > > > builds
> > > > > > on the pattern we have already with common-ai. But not a blocker.
> > > > > >
> > > > > > I also think it's good to have it as a separate provider, even if
> > it
> > > > > gains
> > > > > > traction for two reasons:
> > > > > >
> > > > > > a) ability to add features or fix issues independently from the
> > core
> > > > > > b) an explicit "optional" feature that is easy to promote
> > > > > >
> > > > > > I think what we saw with common is that people see airflow
> already
> > as
> > > > too
> > > > > > heavy - and "too many releases" sometimes, so quite
> > > > counter-intuitively -
> > > > > > by having separate providers adding features that "hook in"
> > existing
> > > > > > functionalities of core - we do not make airflow "heavier" and we
> > do
> > > > not
> > > > > > force people to migrating to future newer versions to use new
> > > features.
> > > > > >
> > > > > > J.
> > > > > >
> > > > > >
> > > > > > On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu <
> > > > > > gopidesupavan@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Amogh,
> > > > > > >
> > > > > > > Thanks for the feedback.
> > > > > > >
> > > > > > > I am happy to change the provider name to dataquality.
> > > > > > >
> > > > > > > Regarding the LLM-assisted features, the current PR does not
> > > include
> > > > > any
> > > > > > > implementation. It only adds the SKILLS [1 ]and the reference
> > > schema
> > > > > for
> > > > > > > the DQ Rule structure. Are you suggesting that I move this
> SKILL
> > > > > > > documentation to a separate PR?
> > > > > > >
> > > > > > > [1]:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md
> > > > > > >
> > > > > > > Regards,
> > > > > > > Pavan
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai <
> > amoghdesai@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Pavan,
> > > > > > > >
> > > > > > > > First of all, +1 to this.
> > > > > > > >
> > > > > > > > Now, few things:
> > > > > > > >
> > > > > > > > * On naming: dataquality over dq for me honestly. Our
> existing
> > > > > provider
> > > > > > > > names spell things out
> > > > > > > > (common.sql, openlineage, not abbreviated forms) and dq is
> > > > genuinely
> > > > > > > > ambiguous outside context.
> > > > > > > >
> > > > > > > > * On scope: I also agree with Niko that #69413 is too large
> for
> > > one
> > > > > > pass
> > > > > > > &
> > > > > > > > I am glad to see the
> > > > > > > > backend/UI split already happening in #69575. Would also
> > suggest
> > > > > > keeping
> > > > > > > > the LLM assisted rule
> > > > > > > > generation pieces (*schema-based generate_rules_from_schema*)
> > out
> > > > of
> > > > > > the
> > > > > > > > initial provider PR entirely
> > > > > > > > cos as I see it, its a separable capability and bundling it
> > will
> > > > slow
> > > > > > > > review of the core DQRule or
> > > > > > > > RuleSet or operator surface, which is the part that actually
> > > needs
> > > > > the
> > > > > > > most
> > > > > > > > detailed review.
> > > > > > > >
> > > > > > > > In short: go for it!
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks & Regards,
> > > > > > > > Amogh Desai
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <
> > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > In the meantime, the PR is ready for review. Feel free to
> > > review
> > > > > and
> > > > > > > > > provide any feedback.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Pavan
> > > > > > > > >
> > > > > > > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <
> > > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Sorry, I forgot to add: the draft PR is here
> > > > > > > > > > https://github.com/apache/airflow/pull/69413; it's
> still a
> > > > WIP.
> > > > > > > > > >
> > > > > > > > > > some screenshots
> > > > > > > > > >
> > > > > >
> > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468
> > > > > > > > :)
> > > > > > > > > >
> > > > > > > > > > Pavan
> > > > > > > > > >
> > > > > > > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <
> > > > > > > > > > gopidesupavan@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi Airflow community,
> > > > > > > > > >>
> > > > > > > > > >> I would like to start a discussion regarding a new
> > provider:
> > > > > > > > > >> apache-airflow-providers-dq.
> > > > > > > > > >>
> > > > > > > > > >> While Airflow already includes SQL check operators that
> > many
> > > > > users
> > > > > > > > rely
> > > > > > > > > >> on for data quality, this new provider builds on that
> > > > foundation
> > > > > > by
> > > > > > > > > >> introducing DQRule and RuleSet objects, stable rule
> > > identity,
> > > > > > > > persisted
> > > > > > > > > >> history, and direct connections to Airflow assets. This
> > > > approach
> > > > > > > makes
> > > > > > > > > >> quality results easier to inspect over time, allows
> > > downstream
> > > > > > > > > consumers to
> > > > > > > > > >> gate tasks based on recent quality results, and
> provides a
> > > > > unified
> > > > > > > > > schema
> > > > > > > > > >> for LLM-assisted workflows. Execution will continue to
> > > utilize
> > > > > > > > existing
> > > > > > > > > >> DbApiHook connections.
> > > > > > > > > >>
> > > > > > > > > >> The initial version of the provider is intentionally
> > > focused:
> > > > > > > > > >>
> > > > > > > > > >> - Declarative DQRule and RuleSet objects.
> > > > > > > > > >> - DQCheckOperator and @task.dq_check.
> > > > > > > > > >> - DbApiHook-based SQL checks, including built-in
> checks
> > > and
> > > > > > > > > custom_sql.
> > > > > > > > > >> - Persisted results for tasks, runs, and rules.
> > > > > > > > > >> - A minimal Airflow UI plugin for viewing results and
> > rule
> > > > > > > history.
> > > > > > > > > >> - Experimental asset helpers such as asset_quality()
> and
> > > > > > > > > >> require_quality().
> > > > > > > > > >>
> > > > > > > > > >> Regarding scope, this first iteration uses object
> storage
> > > only
> > > > > to
> > > > > > > > > persist
> > > > > > > > > >> DQ results and history; checks are executed via database
> > > > > > > connections.
> > > > > > > > > >> Future iterations may include file or object-store based
> > > > checks
> > > > > > > (e.g.,
> > > > > > > > > S3,
> > > > > > > > > >> GCS) where Airflow runs quality rules against data
> > directly.
> > > > > > > > > >>
> > > > > > > > > >> This proposal does not require changes to Airflow core.
> > > Asset
> > > > > > > support
> > > > > > > > is
> > > > > > > > > >> currently provider-owned metadata, with static
> > configuration
> > > > > > stored
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > >> asset and runtime summaries stored on asset events. If
> the
> > > > > > provider
> > > > > > > > > gains
> > > > > > > > > >> traction, we can discuss making Data Quality a
> first-class
> > > > > > component
> > > > > > > > of
> > > > > > > > > >> Airflow assets.
> > > > > > > > > >>
> > > > > > > > > >> This work also serves as a practical follow-up to the
> data
> > > > > quality
> > > > > > > > > >> direction mentioned in AIP-99. Persisted history is
> > valuable
> > > > for
> > > > > > > users
> > > > > > > > > and
> > > > > > > > > >> future LLM-assisted workflows, such as those from
> > Anthropic
> > > or
> > > > > > > > > common.ai,
> > > > > > > > > >> to understand rule performance and generate candidate
> > rules
> > > > > based
> > > > > > on
> > > > > > > > > schema
> > > > > > > > > >> context.
> > > > > > > > > >>
> > > > > > > > > >> A rough pseudo-flow is provided below:
> > > > > > > > > >>
> > > > > > > > > >> seed_rules = RuleSet(
> > > > > > > > > >> name="orders_quality",
> > > > > > > > > >> rules=[
> > > > > > > > > >> DQRule(name="order_id_not_null",
> > check="null_count",
> > > > > > > > > >> column="order_id", condition={"equal_to": 0}),
> > > > > > > > > >> DQRule(name="amount_valid", check="min",
> > > > > column="amount",
> > > > > > > > > >> condition={"geq_to": 0}),
> > > > > > > > > >> ],
> > > > > > > > > >> )
> > > > > > > > > >>
> > > > > > > > > >> orders_asset = asset_quality(
> > > > > > > > > >> Asset("orders"),
> > > > > > > > > >> conn_id="warehouse",
> > > > > > > > > >> table="orders",
> > > > > > > > > >> ruleset=seed_rules,
> > > > > > > > > >> )
> > > > > > > > > >>
> > > > > > > > > >> # Optional: common.ai / Anthropic provider can
> generate a
> > > > > RuleSet
> > > > > > > > from
> > > > > > > > > >> schema context.
> > > > > > > > > >> generated_rules = generate_rules_from_schema(...)
> > > > > > > > > >>
> > > > > > > > > >> @task.dq_check(asset=orders_asset)
> > > > > > > > > >> def check_orders(ruleset):
> > > > > > > > > >> return ruleset
> > > > > > > > > >>
> > > > > > > > > >> checked_orders = check_orders(generated_rules)
> > > > > > > > > >>
> > > > > > > > > >> with DAG("orders_consumer", schedule=orders_asset):
> > > > > > > > > >> require_quality(orders_asset, min_score=0.95) >>
> > > > > > > consume_orders()
> > > > > > > > > >>
> > > > > > > > > >> The UI remains deliberately minimal for this initial
> > > release,
> > > > > > > focusing
> > > > > > > > > on
> > > > > > > > > >> result and history inspection.
> > > > > > > > > >>
> > > > > > > > > >> You can view examples [1] of how it's integrated with
> > > > > assets/llms.
> > > > > > > > > >>
> > > > > > > > > >> currently i named it providers
> > > `apache-airflow-providers-dq`.
> > > > if
> > > > > > any
> > > > > > > > > >> other preference likely with `dataquality`. Please let
> me
> > > know
> > > > > if
> > > > > > > you
> > > > > > > > > have
> > > > > > > > > >> a preference. naming is hard :)
> > > > > > > > > >>
> > > > > > > > > >> [1]:
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >> Pavan
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: New Community Provider: Data Quality
Posted by Pavankumar Gopidesu <go...@gmail.com>.
Thanks Kaxil..
Thanks Everyone, consensus looks positive; I will merge the skeleton PR
first.
Regards,
Pavan
On Mon, Jul 13, 2026 at 10:05 PM Kaxil Naik <ka...@gmail.com> wrote:
> +1 for the high-level need for such data quality checks. Thanks Pavan, will
> review the PRs shortly.
>
> On Fri, 10 Jul 2026 at 01:58, Pavankumar Gopidesu <gopidesupavan@gmail.com
> >
> wrote:
>
> > Thanks Bugra,
> >
> > I have updated the naming convention now.
> >
> > Pavan
> >
> > On Wed, Jul 8, 2026 at 7:07 PM Buğra Öztürk <oz...@gmail.com>
> > wrote:
> >
> > > Thanks Pavan for bringing this together and starting the discussion!
> > > Sounds good! +1 on the idea.
> > >
> > > Harder than solving problems. Not a strong suggestion, but
> > > `common-dataquality` sounds more reasonable to me. It also adds the
> value
> > > of the `common` part, which provides the separation pattern Jarek
> > > mentioned. It gives a better understanding that it is a common
> offering.
> > >
> > > Best regards,
> > > Bugra Ozturk
> > >
> > > On Wed, Jul 8, 2026 at 7:15 PM Pavankumar Gopidesu <
> > > gopidesupavan@gmail.com>
> > > wrote:
> > >
> > > > Thanks Jarek, I agree that the separate provider approach offers much
> > > more
> > > > flexibility for iterating on features and fixes.
> > > >
> > > > Naming is always hard :)
> > > >
> > > > Option 1: apache-airflow-providers-dataquality
> > > > Option 2: apache-airflow-providers-common-dataquality (This goes
> inside
> > > the
> > > > common providers folder we already have)
> > > >
> > > > So, I am up for either option :)
> > > >
> > > > have removed first short name `apache-airflow-providers-dq`.
> > > >
> > > > Thanks,
> > > > Pavan
> > > >
> > > >
> > > > On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > >
> > > > > +1 Good design/ idea. No objections - dataquality is a good name -
> > but
> > > I
> > > > > would also consider `common-dataquality" - even if it's longer, it
> > > builds
> > > > > on the pattern we have already with common-ai. But not a blocker.
> > > > >
> > > > > I also think it's good to have it as a separate provider, even if
> it
> > > > gains
> > > > > traction for two reasons:
> > > > >
> > > > > a) ability to add features or fix issues independently from the
> core
> > > > > b) an explicit "optional" feature that is easy to promote
> > > > >
> > > > > I think what we saw with common is that people see airflow already
> as
> > > too
> > > > > heavy - and "too many releases" sometimes, so quite
> > > counter-intuitively -
> > > > > by having separate providers adding features that "hook in"
> existing
> > > > > functionalities of core - we do not make airflow "heavier" and we
> do
> > > not
> > > > > force people to migrating to future newer versions to use new
> > features.
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > > On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu <
> > > > > gopidesupavan@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Amogh,
> > > > > >
> > > > > > Thanks for the feedback.
> > > > > >
> > > > > > I am happy to change the provider name to dataquality.
> > > > > >
> > > > > > Regarding the LLM-assisted features, the current PR does not
> > include
> > > > any
> > > > > > implementation. It only adds the SKILLS [1 ]and the reference
> > schema
> > > > for
> > > > > > the DQ Rule structure. Are you suggesting that I move this SKILL
> > > > > > documentation to a separate PR?
> > > > > >
> > > > > > [1]:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md
> > > > > >
> > > > > > Regards,
> > > > > > Pavan
> > > > > >
> > > > > >
> > > > > > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai <
> amoghdesai@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Pavan,
> > > > > > >
> > > > > > > First of all, +1 to this.
> > > > > > >
> > > > > > > Now, few things:
> > > > > > >
> > > > > > > * On naming: dataquality over dq for me honestly. Our existing
> > > > provider
> > > > > > > names spell things out
> > > > > > > (common.sql, openlineage, not abbreviated forms) and dq is
> > > genuinely
> > > > > > > ambiguous outside context.
> > > > > > >
> > > > > > > * On scope: I also agree with Niko that #69413 is too large for
> > one
> > > > > pass
> > > > > > &
> > > > > > > I am glad to see the
> > > > > > > backend/UI split already happening in #69575. Would also
> suggest
> > > > > keeping
> > > > > > > the LLM assisted rule
> > > > > > > generation pieces (*schema-based generate_rules_from_schema*)
> out
> > > of
> > > > > the
> > > > > > > initial provider PR entirely
> > > > > > > cos as I see it, its a separable capability and bundling it
> will
> > > slow
> > > > > > > review of the core DQRule or
> > > > > > > RuleSet or operator surface, which is the part that actually
> > needs
> > > > the
> > > > > > most
> > > > > > > detailed review.
> > > > > > >
> > > > > > > In short: go for it!
> > > > > > >
> > > > > > >
> > > > > > > Thanks & Regards,
> > > > > > > Amogh Desai
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <
> > > > > > > gopidesupavan@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > In the meantime, the PR is ready for review. Feel free to
> > review
> > > > and
> > > > > > > > provide any feedback.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Pavan
> > > > > > > >
> > > > > > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <
> > > > > > > > gopidesupavan@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Sorry, I forgot to add: the draft PR is here
> > > > > > > > > https://github.com/apache/airflow/pull/69413; it's still a
> > > WIP.
> > > > > > > > >
> > > > > > > > > some screenshots
> > > > > > > > >
> > > > >
> https://github.com/apache/airflow/pull/69413#issuecomment-4886311468
> > > > > > > :)
> > > > > > > > >
> > > > > > > > > Pavan
> > > > > > > > >
> > > > > > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <
> > > > > > > > > gopidesupavan@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > >> Hi Airflow community,
> > > > > > > > >>
> > > > > > > > >> I would like to start a discussion regarding a new
> provider:
> > > > > > > > >> apache-airflow-providers-dq.
> > > > > > > > >>
> > > > > > > > >> While Airflow already includes SQL check operators that
> many
> > > > users
> > > > > > > rely
> > > > > > > > >> on for data quality, this new provider builds on that
> > > foundation
> > > > > by
> > > > > > > > >> introducing DQRule and RuleSet objects, stable rule
> > identity,
> > > > > > > persisted
> > > > > > > > >> history, and direct connections to Airflow assets. This
> > > approach
> > > > > > makes
> > > > > > > > >> quality results easier to inspect over time, allows
> > downstream
> > > > > > > > consumers to
> > > > > > > > >> gate tasks based on recent quality results, and provides a
> > > > unified
> > > > > > > > schema
> > > > > > > > >> for LLM-assisted workflows. Execution will continue to
> > utilize
> > > > > > > existing
> > > > > > > > >> DbApiHook connections.
> > > > > > > > >>
> > > > > > > > >> The initial version of the provider is intentionally
> > focused:
> > > > > > > > >>
> > > > > > > > >> - Declarative DQRule and RuleSet objects.
> > > > > > > > >> - DQCheckOperator and @task.dq_check.
> > > > > > > > >> - DbApiHook-based SQL checks, including built-in checks
> > and
> > > > > > > > custom_sql.
> > > > > > > > >> - Persisted results for tasks, runs, and rules.
> > > > > > > > >> - A minimal Airflow UI plugin for viewing results and
> rule
> > > > > > history.
> > > > > > > > >> - Experimental asset helpers such as asset_quality() and
> > > > > > > > >> require_quality().
> > > > > > > > >>
> > > > > > > > >> Regarding scope, this first iteration uses object storage
> > only
> > > > to
> > > > > > > > persist
> > > > > > > > >> DQ results and history; checks are executed via database
> > > > > > connections.
> > > > > > > > >> Future iterations may include file or object-store based
> > > checks
> > > > > > (e.g.,
> > > > > > > > S3,
> > > > > > > > >> GCS) where Airflow runs quality rules against data
> directly.
> > > > > > > > >>
> > > > > > > > >> This proposal does not require changes to Airflow core.
> > Asset
> > > > > > support
> > > > > > > is
> > > > > > > > >> currently provider-owned metadata, with static
> configuration
> > > > > stored
> > > > > > on
> > > > > > > > the
> > > > > > > > >> asset and runtime summaries stored on asset events. If the
> > > > > provider
> > > > > > > > gains
> > > > > > > > >> traction, we can discuss making Data Quality a first-class
> > > > > component
> > > > > > > of
> > > > > > > > >> Airflow assets.
> > > > > > > > >>
> > > > > > > > >> This work also serves as a practical follow-up to the data
> > > > quality
> > > > > > > > >> direction mentioned in AIP-99. Persisted history is
> valuable
> > > for
> > > > > > users
> > > > > > > > and
> > > > > > > > >> future LLM-assisted workflows, such as those from
> Anthropic
> > or
> > > > > > > > common.ai,
> > > > > > > > >> to understand rule performance and generate candidate
> rules
> > > > based
> > > > > on
> > > > > > > > schema
> > > > > > > > >> context.
> > > > > > > > >>
> > > > > > > > >> A rough pseudo-flow is provided below:
> > > > > > > > >>
> > > > > > > > >> seed_rules = RuleSet(
> > > > > > > > >> name="orders_quality",
> > > > > > > > >> rules=[
> > > > > > > > >> DQRule(name="order_id_not_null",
> check="null_count",
> > > > > > > > >> column="order_id", condition={"equal_to": 0}),
> > > > > > > > >> DQRule(name="amount_valid", check="min",
> > > > column="amount",
> > > > > > > > >> condition={"geq_to": 0}),
> > > > > > > > >> ],
> > > > > > > > >> )
> > > > > > > > >>
> > > > > > > > >> orders_asset = asset_quality(
> > > > > > > > >> Asset("orders"),
> > > > > > > > >> conn_id="warehouse",
> > > > > > > > >> table="orders",
> > > > > > > > >> ruleset=seed_rules,
> > > > > > > > >> )
> > > > > > > > >>
> > > > > > > > >> # Optional: common.ai / Anthropic provider can generate a
> > > > RuleSet
> > > > > > > from
> > > > > > > > >> schema context.
> > > > > > > > >> generated_rules = generate_rules_from_schema(...)
> > > > > > > > >>
> > > > > > > > >> @task.dq_check(asset=orders_asset)
> > > > > > > > >> def check_orders(ruleset):
> > > > > > > > >> return ruleset
> > > > > > > > >>
> > > > > > > > >> checked_orders = check_orders(generated_rules)
> > > > > > > > >>
> > > > > > > > >> with DAG("orders_consumer", schedule=orders_asset):
> > > > > > > > >> require_quality(orders_asset, min_score=0.95) >>
> > > > > > consume_orders()
> > > > > > > > >>
> > > > > > > > >> The UI remains deliberately minimal for this initial
> > release,
> > > > > > focusing
> > > > > > > > on
> > > > > > > > >> result and history inspection.
> > > > > > > > >>
> > > > > > > > >> You can view examples [1] of how it's integrated with
> > > > assets/llms.
> > > > > > > > >>
> > > > > > > > >> currently i named it providers
> > `apache-airflow-providers-dq`.
> > > if
> > > > > any
> > > > > > > > >> other preference likely with `dataquality`. Please let me
> > know
> > > > if
> > > > > > you
> > > > > > > > have
> > > > > > > > >> a preference. naming is hard :)
> > > > > > > > >>
> > > > > > > > >> [1]:
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Pavan
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: New Community Provider: Data Quality
Posted by Kaxil Naik <ka...@gmail.com>.
+1 for the high-level need for such data quality checks. Thanks Pavan, will
review the PRs shortly.
On Fri, 10 Jul 2026 at 01:58, Pavankumar Gopidesu <go...@gmail.com>
wrote:
> Thanks Bugra,
>
> I have updated the naming convention now.
>
> Pavan
>
> On Wed, Jul 8, 2026 at 7:07 PM Buğra Öztürk <oz...@gmail.com>
> wrote:
>
> > Thanks Pavan for bringing this together and starting the discussion!
> > Sounds good! +1 on the idea.
> >
> > Harder than solving problems. Not a strong suggestion, but
> > `common-dataquality` sounds more reasonable to me. It also adds the value
> > of the `common` part, which provides the separation pattern Jarek
> > mentioned. It gives a better understanding that it is a common offering.
> >
> > Best regards,
> > Bugra Ozturk
> >
> > On Wed, Jul 8, 2026 at 7:15 PM Pavankumar Gopidesu <
> > gopidesupavan@gmail.com>
> > wrote:
> >
> > > Thanks Jarek, I agree that the separate provider approach offers much
> > more
> > > flexibility for iterating on features and fixes.
> > >
> > > Naming is always hard :)
> > >
> > > Option 1: apache-airflow-providers-dataquality
> > > Option 2: apache-airflow-providers-common-dataquality (This goes inside
> > the
> > > common providers folder we already have)
> > >
> > > So, I am up for either option :)
> > >
> > > have removed first short name `apache-airflow-providers-dq`.
> > >
> > > Thanks,
> > > Pavan
> > >
> > >
> > > On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > +1 Good design/ idea. No objections - dataquality is a good name -
> but
> > I
> > > > would also consider `common-dataquality" - even if it's longer, it
> > builds
> > > > on the pattern we have already with common-ai. But not a blocker.
> > > >
> > > > I also think it's good to have it as a separate provider, even if it
> > > gains
> > > > traction for two reasons:
> > > >
> > > > a) ability to add features or fix issues independently from the core
> > > > b) an explicit "optional" feature that is easy to promote
> > > >
> > > > I think what we saw with common is that people see airflow already as
> > too
> > > > heavy - and "too many releases" sometimes, so quite
> > counter-intuitively -
> > > > by having separate providers adding features that "hook in" existing
> > > > functionalities of core - we do not make airflow "heavier" and we do
> > not
> > > > force people to migrating to future newer versions to use new
> features.
> > > >
> > > > J.
> > > >
> > > >
> > > > On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu <
> > > > gopidesupavan@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Amogh,
> > > > >
> > > > > Thanks for the feedback.
> > > > >
> > > > > I am happy to change the provider name to dataquality.
> > > > >
> > > > > Regarding the LLM-assisted features, the current PR does not
> include
> > > any
> > > > > implementation. It only adds the SKILLS [1 ]and the reference
> schema
> > > for
> > > > > the DQ Rule structure. Are you suggesting that I move this SKILL
> > > > > documentation to a separate PR?
> > > > >
> > > > > [1]:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md
> > > > >
> > > > > Regards,
> > > > > Pavan
> > > > >
> > > > >
> > > > > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai <am...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Hi Pavan,
> > > > > >
> > > > > > First of all, +1 to this.
> > > > > >
> > > > > > Now, few things:
> > > > > >
> > > > > > * On naming: dataquality over dq for me honestly. Our existing
> > > provider
> > > > > > names spell things out
> > > > > > (common.sql, openlineage, not abbreviated forms) and dq is
> > genuinely
> > > > > > ambiguous outside context.
> > > > > >
> > > > > > * On scope: I also agree with Niko that #69413 is too large for
> one
> > > > pass
> > > > > &
> > > > > > I am glad to see the
> > > > > > backend/UI split already happening in #69575. Would also suggest
> > > > keeping
> > > > > > the LLM assisted rule
> > > > > > generation pieces (*schema-based generate_rules_from_schema*) out
> > of
> > > > the
> > > > > > initial provider PR entirely
> > > > > > cos as I see it, its a separable capability and bundling it will
> > slow
> > > > > > review of the core DQRule or
> > > > > > RuleSet or operator surface, which is the part that actually
> needs
> > > the
> > > > > most
> > > > > > detailed review.
> > > > > >
> > > > > > In short: go for it!
> > > > > >
> > > > > >
> > > > > > Thanks & Regards,
> > > > > > Amogh Desai
> > > > > >
> > > > > >
> > > > > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <
> > > > > > gopidesupavan@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > In the meantime, the PR is ready for review. Feel free to
> review
> > > and
> > > > > > > provide any feedback.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Pavan
> > > > > > >
> > > > > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <
> > > > > > > gopidesupavan@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Sorry, I forgot to add: the draft PR is here
> > > > > > > > https://github.com/apache/airflow/pull/69413; it's still a
> > WIP.
> > > > > > > >
> > > > > > > > some screenshots
> > > > > > > >
> > > > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468
> > > > > > :)
> > > > > > > >
> > > > > > > > Pavan
> > > > > > > >
> > > > > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <
> > > > > > > > gopidesupavan@gmail.com> wrote:
> > > > > > > >
> > > > > > > >> Hi Airflow community,
> > > > > > > >>
> > > > > > > >> I would like to start a discussion regarding a new provider:
> > > > > > > >> apache-airflow-providers-dq.
> > > > > > > >>
> > > > > > > >> While Airflow already includes SQL check operators that many
> > > users
> > > > > > rely
> > > > > > > >> on for data quality, this new provider builds on that
> > foundation
> > > > by
> > > > > > > >> introducing DQRule and RuleSet objects, stable rule
> identity,
> > > > > > persisted
> > > > > > > >> history, and direct connections to Airflow assets. This
> > approach
> > > > > makes
> > > > > > > >> quality results easier to inspect over time, allows
> downstream
> > > > > > > consumers to
> > > > > > > >> gate tasks based on recent quality results, and provides a
> > > unified
> > > > > > > schema
> > > > > > > >> for LLM-assisted workflows. Execution will continue to
> utilize
> > > > > > existing
> > > > > > > >> DbApiHook connections.
> > > > > > > >>
> > > > > > > >> The initial version of the provider is intentionally
> focused:
> > > > > > > >>
> > > > > > > >> - Declarative DQRule and RuleSet objects.
> > > > > > > >> - DQCheckOperator and @task.dq_check.
> > > > > > > >> - DbApiHook-based SQL checks, including built-in checks
> and
> > > > > > > custom_sql.
> > > > > > > >> - Persisted results for tasks, runs, and rules.
> > > > > > > >> - A minimal Airflow UI plugin for viewing results and rule
> > > > > history.
> > > > > > > >> - Experimental asset helpers such as asset_quality() and
> > > > > > > >> require_quality().
> > > > > > > >>
> > > > > > > >> Regarding scope, this first iteration uses object storage
> only
> > > to
> > > > > > > persist
> > > > > > > >> DQ results and history; checks are executed via database
> > > > > connections.
> > > > > > > >> Future iterations may include file or object-store based
> > checks
> > > > > (e.g.,
> > > > > > > S3,
> > > > > > > >> GCS) where Airflow runs quality rules against data directly.
> > > > > > > >>
> > > > > > > >> This proposal does not require changes to Airflow core.
> Asset
> > > > > support
> > > > > > is
> > > > > > > >> currently provider-owned metadata, with static configuration
> > > > stored
> > > > > on
> > > > > > > the
> > > > > > > >> asset and runtime summaries stored on asset events. If the
> > > > provider
> > > > > > > gains
> > > > > > > >> traction, we can discuss making Data Quality a first-class
> > > > component
> > > > > > of
> > > > > > > >> Airflow assets.
> > > > > > > >>
> > > > > > > >> This work also serves as a practical follow-up to the data
> > > quality
> > > > > > > >> direction mentioned in AIP-99. Persisted history is valuable
> > for
> > > > > users
> > > > > > > and
> > > > > > > >> future LLM-assisted workflows, such as those from Anthropic
> or
> > > > > > > common.ai,
> > > > > > > >> to understand rule performance and generate candidate rules
> > > based
> > > > on
> > > > > > > schema
> > > > > > > >> context.
> > > > > > > >>
> > > > > > > >> A rough pseudo-flow is provided below:
> > > > > > > >>
> > > > > > > >> seed_rules = RuleSet(
> > > > > > > >> name="orders_quality",
> > > > > > > >> rules=[
> > > > > > > >> DQRule(name="order_id_not_null", check="null_count",
> > > > > > > >> column="order_id", condition={"equal_to": 0}),
> > > > > > > >> DQRule(name="amount_valid", check="min",
> > > column="amount",
> > > > > > > >> condition={"geq_to": 0}),
> > > > > > > >> ],
> > > > > > > >> )
> > > > > > > >>
> > > > > > > >> orders_asset = asset_quality(
> > > > > > > >> Asset("orders"),
> > > > > > > >> conn_id="warehouse",
> > > > > > > >> table="orders",
> > > > > > > >> ruleset=seed_rules,
> > > > > > > >> )
> > > > > > > >>
> > > > > > > >> # Optional: common.ai / Anthropic provider can generate a
> > > RuleSet
> > > > > > from
> > > > > > > >> schema context.
> > > > > > > >> generated_rules = generate_rules_from_schema(...)
> > > > > > > >>
> > > > > > > >> @task.dq_check(asset=orders_asset)
> > > > > > > >> def check_orders(ruleset):
> > > > > > > >> return ruleset
> > > > > > > >>
> > > > > > > >> checked_orders = check_orders(generated_rules)
> > > > > > > >>
> > > > > > > >> with DAG("orders_consumer", schedule=orders_asset):
> > > > > > > >> require_quality(orders_asset, min_score=0.95) >>
> > > > > consume_orders()
> > > > > > > >>
> > > > > > > >> The UI remains deliberately minimal for this initial
> release,
> > > > > focusing
> > > > > > > on
> > > > > > > >> result and history inspection.
> > > > > > > >>
> > > > > > > >> You can view examples [1] of how it's integrated with
> > > assets/llms.
> > > > > > > >>
> > > > > > > >> currently i named it providers
> `apache-airflow-providers-dq`.
> > if
> > > > any
> > > > > > > >> other preference likely with `dataquality`. Please let me
> know
> > > if
> > > > > you
> > > > > > > have
> > > > > > > >> a preference. naming is hard :)
> > > > > > > >>
> > > > > > > >> [1]:
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> Pavan
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: New Community Provider: Data Quality
Posted by Pavankumar Gopidesu <go...@gmail.com>.
Thanks Bugra,
I have updated the naming convention now.
Pavan
On Wed, Jul 8, 2026 at 7:07 PM Buğra Öztürk <oz...@gmail.com> wrote:
> Thanks Pavan for bringing this together and starting the discussion!
> Sounds good! +1 on the idea.
>
> Harder than solving problems. Not a strong suggestion, but
> `common-dataquality` sounds more reasonable to me. It also adds the value
> of the `common` part, which provides the separation pattern Jarek
> mentioned. It gives a better understanding that it is a common offering.
>
> Best regards,
> Bugra Ozturk
>
> On Wed, Jul 8, 2026 at 7:15 PM Pavankumar Gopidesu <
> gopidesupavan@gmail.com>
> wrote:
>
> > Thanks Jarek, I agree that the separate provider approach offers much
> more
> > flexibility for iterating on features and fixes.
> >
> > Naming is always hard :)
> >
> > Option 1: apache-airflow-providers-dataquality
> > Option 2: apache-airflow-providers-common-dataquality (This goes inside
> the
> > common providers folder we already have)
> >
> > So, I am up for either option :)
> >
> > have removed first short name `apache-airflow-providers-dq`.
> >
> > Thanks,
> > Pavan
> >
> >
> > On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > +1 Good design/ idea. No objections - dataquality is a good name - but
> I
> > > would also consider `common-dataquality" - even if it's longer, it
> builds
> > > on the pattern we have already with common-ai. But not a blocker.
> > >
> > > I also think it's good to have it as a separate provider, even if it
> > gains
> > > traction for two reasons:
> > >
> > > a) ability to add features or fix issues independently from the core
> > > b) an explicit "optional" feature that is easy to promote
> > >
> > > I think what we saw with common is that people see airflow already as
> too
> > > heavy - and "too many releases" sometimes, so quite
> counter-intuitively -
> > > by having separate providers adding features that "hook in" existing
> > > functionalities of core - we do not make airflow "heavier" and we do
> not
> > > force people to migrating to future newer versions to use new features.
> > >
> > > J.
> > >
> > >
> > > On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu <
> > > gopidesupavan@gmail.com>
> > > wrote:
> > >
> > > > Hi Amogh,
> > > >
> > > > Thanks for the feedback.
> > > >
> > > > I am happy to change the provider name to dataquality.
> > > >
> > > > Regarding the LLM-assisted features, the current PR does not include
> > any
> > > > implementation. It only adds the SKILLS [1 ]and the reference schema
> > for
> > > > the DQ Rule structure. Are you suggesting that I move this SKILL
> > > > documentation to a separate PR?
> > > >
> > > > [1]:
> > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md
> > > >
> > > > Regards,
> > > > Pavan
> > > >
> > > >
> > > > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai <am...@apache.org>
> > > wrote:
> > > >
> > > > > Hi Pavan,
> > > > >
> > > > > First of all, +1 to this.
> > > > >
> > > > > Now, few things:
> > > > >
> > > > > * On naming: dataquality over dq for me honestly. Our existing
> > provider
> > > > > names spell things out
> > > > > (common.sql, openlineage, not abbreviated forms) and dq is
> genuinely
> > > > > ambiguous outside context.
> > > > >
> > > > > * On scope: I also agree with Niko that #69413 is too large for one
> > > pass
> > > > &
> > > > > I am glad to see the
> > > > > backend/UI split already happening in #69575. Would also suggest
> > > keeping
> > > > > the LLM assisted rule
> > > > > generation pieces (*schema-based generate_rules_from_schema*) out
> of
> > > the
> > > > > initial provider PR entirely
> > > > > cos as I see it, its a separable capability and bundling it will
> slow
> > > > > review of the core DQRule or
> > > > > RuleSet or operator surface, which is the part that actually needs
> > the
> > > > most
> > > > > detailed review.
> > > > >
> > > > > In short: go for it!
> > > > >
> > > > >
> > > > > Thanks & Regards,
> > > > > Amogh Desai
> > > > >
> > > > >
> > > > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <
> > > > > gopidesupavan@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > In the meantime, the PR is ready for review. Feel free to review
> > and
> > > > > > provide any feedback.
> > > > > >
> > > > > > Regards,
> > > > > > Pavan
> > > > > >
> > > > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <
> > > > > > gopidesupavan@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Sorry, I forgot to add: the draft PR is here
> > > > > > > https://github.com/apache/airflow/pull/69413; it's still a
> WIP.
> > > > > > >
> > > > > > > some screenshots
> > > > > > >
> > > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468
> > > > > :)
> > > > > > >
> > > > > > > Pavan
> > > > > > >
> > > > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <
> > > > > > > gopidesupavan@gmail.com> wrote:
> > > > > > >
> > > > > > >> Hi Airflow community,
> > > > > > >>
> > > > > > >> I would like to start a discussion regarding a new provider:
> > > > > > >> apache-airflow-providers-dq.
> > > > > > >>
> > > > > > >> While Airflow already includes SQL check operators that many
> > users
> > > > > rely
> > > > > > >> on for data quality, this new provider builds on that
> foundation
> > > by
> > > > > > >> introducing DQRule and RuleSet objects, stable rule identity,
> > > > > persisted
> > > > > > >> history, and direct connections to Airflow assets. This
> approach
> > > > makes
> > > > > > >> quality results easier to inspect over time, allows downstream
> > > > > > consumers to
> > > > > > >> gate tasks based on recent quality results, and provides a
> > unified
> > > > > > schema
> > > > > > >> for LLM-assisted workflows. Execution will continue to utilize
> > > > > existing
> > > > > > >> DbApiHook connections.
> > > > > > >>
> > > > > > >> The initial version of the provider is intentionally focused:
> > > > > > >>
> > > > > > >> - Declarative DQRule and RuleSet objects.
> > > > > > >> - DQCheckOperator and @task.dq_check.
> > > > > > >> - DbApiHook-based SQL checks, including built-in checks and
> > > > > > custom_sql.
> > > > > > >> - Persisted results for tasks, runs, and rules.
> > > > > > >> - A minimal Airflow UI plugin for viewing results and rule
> > > > history.
> > > > > > >> - Experimental asset helpers such as asset_quality() and
> > > > > > >> require_quality().
> > > > > > >>
> > > > > > >> Regarding scope, this first iteration uses object storage only
> > to
> > > > > > persist
> > > > > > >> DQ results and history; checks are executed via database
> > > > connections.
> > > > > > >> Future iterations may include file or object-store based
> checks
> > > > (e.g.,
> > > > > > S3,
> > > > > > >> GCS) where Airflow runs quality rules against data directly.
> > > > > > >>
> > > > > > >> This proposal does not require changes to Airflow core. Asset
> > > > support
> > > > > is
> > > > > > >> currently provider-owned metadata, with static configuration
> > > stored
> > > > on
> > > > > > the
> > > > > > >> asset and runtime summaries stored on asset events. If the
> > > provider
> > > > > > gains
> > > > > > >> traction, we can discuss making Data Quality a first-class
> > > component
> > > > > of
> > > > > > >> Airflow assets.
> > > > > > >>
> > > > > > >> This work also serves as a practical follow-up to the data
> > quality
> > > > > > >> direction mentioned in AIP-99. Persisted history is valuable
> for
> > > > users
> > > > > > and
> > > > > > >> future LLM-assisted workflows, such as those from Anthropic or
> > > > > > common.ai,
> > > > > > >> to understand rule performance and generate candidate rules
> > based
> > > on
> > > > > > schema
> > > > > > >> context.
> > > > > > >>
> > > > > > >> A rough pseudo-flow is provided below:
> > > > > > >>
> > > > > > >> seed_rules = RuleSet(
> > > > > > >> name="orders_quality",
> > > > > > >> rules=[
> > > > > > >> DQRule(name="order_id_not_null", check="null_count",
> > > > > > >> column="order_id", condition={"equal_to": 0}),
> > > > > > >> DQRule(name="amount_valid", check="min",
> > column="amount",
> > > > > > >> condition={"geq_to": 0}),
> > > > > > >> ],
> > > > > > >> )
> > > > > > >>
> > > > > > >> orders_asset = asset_quality(
> > > > > > >> Asset("orders"),
> > > > > > >> conn_id="warehouse",
> > > > > > >> table="orders",
> > > > > > >> ruleset=seed_rules,
> > > > > > >> )
> > > > > > >>
> > > > > > >> # Optional: common.ai / Anthropic provider can generate a
> > RuleSet
> > > > > from
> > > > > > >> schema context.
> > > > > > >> generated_rules = generate_rules_from_schema(...)
> > > > > > >>
> > > > > > >> @task.dq_check(asset=orders_asset)
> > > > > > >> def check_orders(ruleset):
> > > > > > >> return ruleset
> > > > > > >>
> > > > > > >> checked_orders = check_orders(generated_rules)
> > > > > > >>
> > > > > > >> with DAG("orders_consumer", schedule=orders_asset):
> > > > > > >> require_quality(orders_asset, min_score=0.95) >>
> > > > consume_orders()
> > > > > > >>
> > > > > > >> The UI remains deliberately minimal for this initial release,
> > > > focusing
> > > > > > on
> > > > > > >> result and history inspection.
> > > > > > >>
> > > > > > >> You can view examples [1] of how it's integrated with
> > assets/llms.
> > > > > > >>
> > > > > > >> currently i named it providers `apache-airflow-providers-dq`.
> if
> > > any
> > > > > > >> other preference likely with `dataquality`. Please let me know
> > if
> > > > you
> > > > > > have
> > > > > > >> a preference. naming is hard :)
> > > > > > >>
> > > > > > >> [1]:
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Pavan
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: New Community Provider: Data Quality
Posted by Buğra Öztürk <oz...@gmail.com>.
Thanks Pavan for bringing this together and starting the discussion!
Sounds good! +1 on the idea.
Harder than solving problems. Not a strong suggestion, but
`common-dataquality` sounds more reasonable to me. It also adds the value
of the `common` part, which provides the separation pattern Jarek
mentioned. It gives a better understanding that it is a common offering.
Best regards,
Bugra Ozturk
On Wed, Jul 8, 2026 at 7:15 PM Pavankumar Gopidesu <go...@gmail.com>
wrote:
> Thanks Jarek, I agree that the separate provider approach offers much more
> flexibility for iterating on features and fixes.
>
> Naming is always hard :)
>
> Option 1: apache-airflow-providers-dataquality
> Option 2: apache-airflow-providers-common-dataquality (This goes inside the
> common providers folder we already have)
>
> So, I am up for either option :)
>
> have removed first short name `apache-airflow-providers-dq`.
>
> Thanks,
> Pavan
>
>
> On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > +1 Good design/ idea. No objections - dataquality is a good name - but I
> > would also consider `common-dataquality" - even if it's longer, it builds
> > on the pattern we have already with common-ai. But not a blocker.
> >
> > I also think it's good to have it as a separate provider, even if it
> gains
> > traction for two reasons:
> >
> > a) ability to add features or fix issues independently from the core
> > b) an explicit "optional" feature that is easy to promote
> >
> > I think what we saw with common is that people see airflow already as too
> > heavy - and "too many releases" sometimes, so quite counter-intuitively -
> > by having separate providers adding features that "hook in" existing
> > functionalities of core - we do not make airflow "heavier" and we do not
> > force people to migrating to future newer versions to use new features.
> >
> > J.
> >
> >
> > On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu <
> > gopidesupavan@gmail.com>
> > wrote:
> >
> > > Hi Amogh,
> > >
> > > Thanks for the feedback.
> > >
> > > I am happy to change the provider name to dataquality.
> > >
> > > Regarding the LLM-assisted features, the current PR does not include
> any
> > > implementation. It only adds the SKILLS [1 ]and the reference schema
> for
> > > the DQ Rule structure. Are you suggesting that I move this SKILL
> > > documentation to a separate PR?
> > >
> > > [1]:
> > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md
> > >
> > > Regards,
> > > Pavan
> > >
> > >
> > > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai <am...@apache.org>
> > wrote:
> > >
> > > > Hi Pavan,
> > > >
> > > > First of all, +1 to this.
> > > >
> > > > Now, few things:
> > > >
> > > > * On naming: dataquality over dq for me honestly. Our existing
> provider
> > > > names spell things out
> > > > (common.sql, openlineage, not abbreviated forms) and dq is genuinely
> > > > ambiguous outside context.
> > > >
> > > > * On scope: I also agree with Niko that #69413 is too large for one
> > pass
> > > &
> > > > I am glad to see the
> > > > backend/UI split already happening in #69575. Would also suggest
> > keeping
> > > > the LLM assisted rule
> > > > generation pieces (*schema-based generate_rules_from_schema*) out of
> > the
> > > > initial provider PR entirely
> > > > cos as I see it, its a separable capability and bundling it will slow
> > > > review of the core DQRule or
> > > > RuleSet or operator surface, which is the part that actually needs
> the
> > > most
> > > > detailed review.
> > > >
> > > > In short: go for it!
> > > >
> > > >
> > > > Thanks & Regards,
> > > > Amogh Desai
> > > >
> > > >
> > > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <
> > > > gopidesupavan@gmail.com>
> > > > wrote:
> > > >
> > > > > In the meantime, the PR is ready for review. Feel free to review
> and
> > > > > provide any feedback.
> > > > >
> > > > > Regards,
> > > > > Pavan
> > > > >
> > > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <
> > > > > gopidesupavan@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Sorry, I forgot to add: the draft PR is here
> > > > > > https://github.com/apache/airflow/pull/69413; it's still a WIP.
> > > > > >
> > > > > > some screenshots
> > > > > >
> > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468
> > > > :)
> > > > > >
> > > > > > Pavan
> > > > > >
> > > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <
> > > > > > gopidesupavan@gmail.com> wrote:
> > > > > >
> > > > > >> Hi Airflow community,
> > > > > >>
> > > > > >> I would like to start a discussion regarding a new provider:
> > > > > >> apache-airflow-providers-dq.
> > > > > >>
> > > > > >> While Airflow already includes SQL check operators that many
> users
> > > > rely
> > > > > >> on for data quality, this new provider builds on that foundation
> > by
> > > > > >> introducing DQRule and RuleSet objects, stable rule identity,
> > > > persisted
> > > > > >> history, and direct connections to Airflow assets. This approach
> > > makes
> > > > > >> quality results easier to inspect over time, allows downstream
> > > > > consumers to
> > > > > >> gate tasks based on recent quality results, and provides a
> unified
> > > > > schema
> > > > > >> for LLM-assisted workflows. Execution will continue to utilize
> > > > existing
> > > > > >> DbApiHook connections.
> > > > > >>
> > > > > >> The initial version of the provider is intentionally focused:
> > > > > >>
> > > > > >> - Declarative DQRule and RuleSet objects.
> > > > > >> - DQCheckOperator and @task.dq_check.
> > > > > >> - DbApiHook-based SQL checks, including built-in checks and
> > > > > custom_sql.
> > > > > >> - Persisted results for tasks, runs, and rules.
> > > > > >> - A minimal Airflow UI plugin for viewing results and rule
> > > history.
> > > > > >> - Experimental asset helpers such as asset_quality() and
> > > > > >> require_quality().
> > > > > >>
> > > > > >> Regarding scope, this first iteration uses object storage only
> to
> > > > > persist
> > > > > >> DQ results and history; checks are executed via database
> > > connections.
> > > > > >> Future iterations may include file or object-store based checks
> > > (e.g.,
> > > > > S3,
> > > > > >> GCS) where Airflow runs quality rules against data directly.
> > > > > >>
> > > > > >> This proposal does not require changes to Airflow core. Asset
> > > support
> > > > is
> > > > > >> currently provider-owned metadata, with static configuration
> > stored
> > > on
> > > > > the
> > > > > >> asset and runtime summaries stored on asset events. If the
> > provider
> > > > > gains
> > > > > >> traction, we can discuss making Data Quality a first-class
> > component
> > > > of
> > > > > >> Airflow assets.
> > > > > >>
> > > > > >> This work also serves as a practical follow-up to the data
> quality
> > > > > >> direction mentioned in AIP-99. Persisted history is valuable for
> > > users
> > > > > and
> > > > > >> future LLM-assisted workflows, such as those from Anthropic or
> > > > > common.ai,
> > > > > >> to understand rule performance and generate candidate rules
> based
> > on
> > > > > schema
> > > > > >> context.
> > > > > >>
> > > > > >> A rough pseudo-flow is provided below:
> > > > > >>
> > > > > >> seed_rules = RuleSet(
> > > > > >> name="orders_quality",
> > > > > >> rules=[
> > > > > >> DQRule(name="order_id_not_null", check="null_count",
> > > > > >> column="order_id", condition={"equal_to": 0}),
> > > > > >> DQRule(name="amount_valid", check="min",
> column="amount",
> > > > > >> condition={"geq_to": 0}),
> > > > > >> ],
> > > > > >> )
> > > > > >>
> > > > > >> orders_asset = asset_quality(
> > > > > >> Asset("orders"),
> > > > > >> conn_id="warehouse",
> > > > > >> table="orders",
> > > > > >> ruleset=seed_rules,
> > > > > >> )
> > > > > >>
> > > > > >> # Optional: common.ai / Anthropic provider can generate a
> RuleSet
> > > > from
> > > > > >> schema context.
> > > > > >> generated_rules = generate_rules_from_schema(...)
> > > > > >>
> > > > > >> @task.dq_check(asset=orders_asset)
> > > > > >> def check_orders(ruleset):
> > > > > >> return ruleset
> > > > > >>
> > > > > >> checked_orders = check_orders(generated_rules)
> > > > > >>
> > > > > >> with DAG("orders_consumer", schedule=orders_asset):
> > > > > >> require_quality(orders_asset, min_score=0.95) >>
> > > consume_orders()
> > > > > >>
> > > > > >> The UI remains deliberately minimal for this initial release,
> > > focusing
> > > > > on
> > > > > >> result and history inspection.
> > > > > >>
> > > > > >> You can view examples [1] of how it's integrated with
> assets/llms.
> > > > > >>
> > > > > >> currently i named it providers `apache-airflow-providers-dq`. if
> > any
> > > > > >> other preference likely with `dataquality`. Please let me know
> if
> > > you
> > > > > have
> > > > > >> a preference. naming is hard :)
> > > > > >>
> > > > > >> [1]:
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Pavan
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: New Community Provider: Data Quality
Posted by Pavankumar Gopidesu <go...@gmail.com>.
Thanks Jarek, I agree that the separate provider approach offers much more
flexibility for iterating on features and fixes.
Naming is always hard :)
Option 1: apache-airflow-providers-dataquality
Option 2: apache-airflow-providers-common-dataquality (This goes inside the
common providers folder we already have)
So, I am up for either option :)
have removed first short name `apache-airflow-providers-dq`.
Thanks,
Pavan
On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> +1 Good design/ idea. No objections - dataquality is a good name - but I
> would also consider `common-dataquality" - even if it's longer, it builds
> on the pattern we have already with common-ai. But not a blocker.
>
> I also think it's good to have it as a separate provider, even if it gains
> traction for two reasons:
>
> a) ability to add features or fix issues independently from the core
> b) an explicit "optional" feature that is easy to promote
>
> I think what we saw with common is that people see airflow already as too
> heavy - and "too many releases" sometimes, so quite counter-intuitively -
> by having separate providers adding features that "hook in" existing
> functionalities of core - we do not make airflow "heavier" and we do not
> force people to migrating to future newer versions to use new features.
>
> J.
>
>
> On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu <
> gopidesupavan@gmail.com>
> wrote:
>
> > Hi Amogh,
> >
> > Thanks for the feedback.
> >
> > I am happy to change the provider name to dataquality.
> >
> > Regarding the LLM-assisted features, the current PR does not include any
> > implementation. It only adds the SKILLS [1 ]and the reference schema for
> > the DQ Rule structure. Are you suggesting that I move this SKILL
> > documentation to a separate PR?
> >
> > [1]:
> >
> >
> https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md
> >
> > Regards,
> > Pavan
> >
> >
> > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai <am...@apache.org>
> wrote:
> >
> > > Hi Pavan,
> > >
> > > First of all, +1 to this.
> > >
> > > Now, few things:
> > >
> > > * On naming: dataquality over dq for me honestly. Our existing provider
> > > names spell things out
> > > (common.sql, openlineage, not abbreviated forms) and dq is genuinely
> > > ambiguous outside context.
> > >
> > > * On scope: I also agree with Niko that #69413 is too large for one
> pass
> > &
> > > I am glad to see the
> > > backend/UI split already happening in #69575. Would also suggest
> keeping
> > > the LLM assisted rule
> > > generation pieces (*schema-based generate_rules_from_schema*) out of
> the
> > > initial provider PR entirely
> > > cos as I see it, its a separable capability and bundling it will slow
> > > review of the core DQRule or
> > > RuleSet or operator surface, which is the part that actually needs the
> > most
> > > detailed review.
> > >
> > > In short: go for it!
> > >
> > >
> > > Thanks & Regards,
> > > Amogh Desai
> > >
> > >
> > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <
> > > gopidesupavan@gmail.com>
> > > wrote:
> > >
> > > > In the meantime, the PR is ready for review. Feel free to review and
> > > > provide any feedback.
> > > >
> > > > Regards,
> > > > Pavan
> > > >
> > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <
> > > > gopidesupavan@gmail.com>
> > > > wrote:
> > > >
> > > > > Sorry, I forgot to add: the draft PR is here
> > > > > https://github.com/apache/airflow/pull/69413; it's still a WIP.
> > > > >
> > > > > some screenshots
> > > > >
> https://github.com/apache/airflow/pull/69413#issuecomment-4886311468
> > > :)
> > > > >
> > > > > Pavan
> > > > >
> > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <
> > > > > gopidesupavan@gmail.com> wrote:
> > > > >
> > > > >> Hi Airflow community,
> > > > >>
> > > > >> I would like to start a discussion regarding a new provider:
> > > > >> apache-airflow-providers-dq.
> > > > >>
> > > > >> While Airflow already includes SQL check operators that many users
> > > rely
> > > > >> on for data quality, this new provider builds on that foundation
> by
> > > > >> introducing DQRule and RuleSet objects, stable rule identity,
> > > persisted
> > > > >> history, and direct connections to Airflow assets. This approach
> > makes
> > > > >> quality results easier to inspect over time, allows downstream
> > > > consumers to
> > > > >> gate tasks based on recent quality results, and provides a unified
> > > > schema
> > > > >> for LLM-assisted workflows. Execution will continue to utilize
> > > existing
> > > > >> DbApiHook connections.
> > > > >>
> > > > >> The initial version of the provider is intentionally focused:
> > > > >>
> > > > >> - Declarative DQRule and RuleSet objects.
> > > > >> - DQCheckOperator and @task.dq_check.
> > > > >> - DbApiHook-based SQL checks, including built-in checks and
> > > > custom_sql.
> > > > >> - Persisted results for tasks, runs, and rules.
> > > > >> - A minimal Airflow UI plugin for viewing results and rule
> > history.
> > > > >> - Experimental asset helpers such as asset_quality() and
> > > > >> require_quality().
> > > > >>
> > > > >> Regarding scope, this first iteration uses object storage only to
> > > > persist
> > > > >> DQ results and history; checks are executed via database
> > connections.
> > > > >> Future iterations may include file or object-store based checks
> > (e.g.,
> > > > S3,
> > > > >> GCS) where Airflow runs quality rules against data directly.
> > > > >>
> > > > >> This proposal does not require changes to Airflow core. Asset
> > support
> > > is
> > > > >> currently provider-owned metadata, with static configuration
> stored
> > on
> > > > the
> > > > >> asset and runtime summaries stored on asset events. If the
> provider
> > > > gains
> > > > >> traction, we can discuss making Data Quality a first-class
> component
> > > of
> > > > >> Airflow assets.
> > > > >>
> > > > >> This work also serves as a practical follow-up to the data quality
> > > > >> direction mentioned in AIP-99. Persisted history is valuable for
> > users
> > > > and
> > > > >> future LLM-assisted workflows, such as those from Anthropic or
> > > > common.ai,
> > > > >> to understand rule performance and generate candidate rules based
> on
> > > > schema
> > > > >> context.
> > > > >>
> > > > >> A rough pseudo-flow is provided below:
> > > > >>
> > > > >> seed_rules = RuleSet(
> > > > >> name="orders_quality",
> > > > >> rules=[
> > > > >> DQRule(name="order_id_not_null", check="null_count",
> > > > >> column="order_id", condition={"equal_to": 0}),
> > > > >> DQRule(name="amount_valid", check="min", column="amount",
> > > > >> condition={"geq_to": 0}),
> > > > >> ],
> > > > >> )
> > > > >>
> > > > >> orders_asset = asset_quality(
> > > > >> Asset("orders"),
> > > > >> conn_id="warehouse",
> > > > >> table="orders",
> > > > >> ruleset=seed_rules,
> > > > >> )
> > > > >>
> > > > >> # Optional: common.ai / Anthropic provider can generate a RuleSet
> > > from
> > > > >> schema context.
> > > > >> generated_rules = generate_rules_from_schema(...)
> > > > >>
> > > > >> @task.dq_check(asset=orders_asset)
> > > > >> def check_orders(ruleset):
> > > > >> return ruleset
> > > > >>
> > > > >> checked_orders = check_orders(generated_rules)
> > > > >>
> > > > >> with DAG("orders_consumer", schedule=orders_asset):
> > > > >> require_quality(orders_asset, min_score=0.95) >>
> > consume_orders()
> > > > >>
> > > > >> The UI remains deliberately minimal for this initial release,
> > focusing
> > > > on
> > > > >> result and history inspection.
> > > > >>
> > > > >> You can view examples [1] of how it's integrated with assets/llms.
> > > > >>
> > > > >> currently i named it providers `apache-airflow-providers-dq`. if
> any
> > > > >> other preference likely with `dataquality`. Please let me know if
> > you
> > > > have
> > > > >> a preference. naming is hard :)
> > > > >>
> > > > >> [1]:
> > > > >>
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> > > > >>
> > > > >> Thanks,
> > > > >> Pavan
> > > > >>
> > > > >>
> > > >
> > >
> >
>
Re: New Community Provider: Data Quality
Posted by Jarek Potiuk <ja...@potiuk.com>.
+1 Good design/ idea. No objections - dataquality is a good name - but I
would also consider `common-dataquality" - even if it's longer, it builds
on the pattern we have already with common-ai. But not a blocker.
I also think it's good to have it as a separate provider, even if it gains
traction for two reasons:
a) ability to add features or fix issues independently from the core
b) an explicit "optional" feature that is easy to promote
I think what we saw with common is that people see airflow already as too
heavy - and "too many releases" sometimes, so quite counter-intuitively -
by having separate providers adding features that "hook in" existing
functionalities of core - we do not make airflow "heavier" and we do not
force people to migrating to future newer versions to use new features.
J.
On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu <go...@gmail.com>
wrote:
> Hi Amogh,
>
> Thanks for the feedback.
>
> I am happy to change the provider name to dataquality.
>
> Regarding the LLM-assisted features, the current PR does not include any
> implementation. It only adds the SKILLS [1 ]and the reference schema for
> the DQ Rule structure. Are you suggesting that I move this SKILL
> documentation to a separate PR?
>
> [1]:
>
> https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md
>
> Regards,
> Pavan
>
>
> On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai <am...@apache.org> wrote:
>
> > Hi Pavan,
> >
> > First of all, +1 to this.
> >
> > Now, few things:
> >
> > * On naming: dataquality over dq for me honestly. Our existing provider
> > names spell things out
> > (common.sql, openlineage, not abbreviated forms) and dq is genuinely
> > ambiguous outside context.
> >
> > * On scope: I also agree with Niko that #69413 is too large for one pass
> &
> > I am glad to see the
> > backend/UI split already happening in #69575. Would also suggest keeping
> > the LLM assisted rule
> > generation pieces (*schema-based generate_rules_from_schema*) out of the
> > initial provider PR entirely
> > cos as I see it, its a separable capability and bundling it will slow
> > review of the core DQRule or
> > RuleSet or operator surface, which is the part that actually needs the
> most
> > detailed review.
> >
> > In short: go for it!
> >
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <
> > gopidesupavan@gmail.com>
> > wrote:
> >
> > > In the meantime, the PR is ready for review. Feel free to review and
> > > provide any feedback.
> > >
> > > Regards,
> > > Pavan
> > >
> > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <
> > > gopidesupavan@gmail.com>
> > > wrote:
> > >
> > > > Sorry, I forgot to add: the draft PR is here
> > > > https://github.com/apache/airflow/pull/69413; it's still a WIP.
> > > >
> > > > some screenshots
> > > > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468
> > :)
> > > >
> > > > Pavan
> > > >
> > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <
> > > > gopidesupavan@gmail.com> wrote:
> > > >
> > > >> Hi Airflow community,
> > > >>
> > > >> I would like to start a discussion regarding a new provider:
> > > >> apache-airflow-providers-dq.
> > > >>
> > > >> While Airflow already includes SQL check operators that many users
> > rely
> > > >> on for data quality, this new provider builds on that foundation by
> > > >> introducing DQRule and RuleSet objects, stable rule identity,
> > persisted
> > > >> history, and direct connections to Airflow assets. This approach
> makes
> > > >> quality results easier to inspect over time, allows downstream
> > > consumers to
> > > >> gate tasks based on recent quality results, and provides a unified
> > > schema
> > > >> for LLM-assisted workflows. Execution will continue to utilize
> > existing
> > > >> DbApiHook connections.
> > > >>
> > > >> The initial version of the provider is intentionally focused:
> > > >>
> > > >> - Declarative DQRule and RuleSet objects.
> > > >> - DQCheckOperator and @task.dq_check.
> > > >> - DbApiHook-based SQL checks, including built-in checks and
> > > custom_sql.
> > > >> - Persisted results for tasks, runs, and rules.
> > > >> - A minimal Airflow UI plugin for viewing results and rule
> history.
> > > >> - Experimental asset helpers such as asset_quality() and
> > > >> require_quality().
> > > >>
> > > >> Regarding scope, this first iteration uses object storage only to
> > > persist
> > > >> DQ results and history; checks are executed via database
> connections.
> > > >> Future iterations may include file or object-store based checks
> (e.g.,
> > > S3,
> > > >> GCS) where Airflow runs quality rules against data directly.
> > > >>
> > > >> This proposal does not require changes to Airflow core. Asset
> support
> > is
> > > >> currently provider-owned metadata, with static configuration stored
> on
> > > the
> > > >> asset and runtime summaries stored on asset events. If the provider
> > > gains
> > > >> traction, we can discuss making Data Quality a first-class component
> > of
> > > >> Airflow assets.
> > > >>
> > > >> This work also serves as a practical follow-up to the data quality
> > > >> direction mentioned in AIP-99. Persisted history is valuable for
> users
> > > and
> > > >> future LLM-assisted workflows, such as those from Anthropic or
> > > common.ai,
> > > >> to understand rule performance and generate candidate rules based on
> > > schema
> > > >> context.
> > > >>
> > > >> A rough pseudo-flow is provided below:
> > > >>
> > > >> seed_rules = RuleSet(
> > > >> name="orders_quality",
> > > >> rules=[
> > > >> DQRule(name="order_id_not_null", check="null_count",
> > > >> column="order_id", condition={"equal_to": 0}),
> > > >> DQRule(name="amount_valid", check="min", column="amount",
> > > >> condition={"geq_to": 0}),
> > > >> ],
> > > >> )
> > > >>
> > > >> orders_asset = asset_quality(
> > > >> Asset("orders"),
> > > >> conn_id="warehouse",
> > > >> table="orders",
> > > >> ruleset=seed_rules,
> > > >> )
> > > >>
> > > >> # Optional: common.ai / Anthropic provider can generate a RuleSet
> > from
> > > >> schema context.
> > > >> generated_rules = generate_rules_from_schema(...)
> > > >>
> > > >> @task.dq_check(asset=orders_asset)
> > > >> def check_orders(ruleset):
> > > >> return ruleset
> > > >>
> > > >> checked_orders = check_orders(generated_rules)
> > > >>
> > > >> with DAG("orders_consumer", schedule=orders_asset):
> > > >> require_quality(orders_asset, min_score=0.95) >>
> consume_orders()
> > > >>
> > > >> The UI remains deliberately minimal for this initial release,
> focusing
> > > on
> > > >> result and history inspection.
> > > >>
> > > >> You can view examples [1] of how it's integrated with assets/llms.
> > > >>
> > > >> currently i named it providers `apache-airflow-providers-dq`. if any
> > > >> other preference likely with `dataquality`. Please let me know if
> you
> > > have
> > > >> a preference. naming is hard :)
> > > >>
> > > >> [1]:
> > > >>
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> > > >>
> > > >> Thanks,
> > > >> Pavan
> > > >>
> > > >>
> > >
> >
>
Re: New Community Provider: Data Quality
Posted by Pavankumar Gopidesu <go...@gmail.com>.
Hi Amogh,
Thanks for the feedback.
I am happy to change the provider name to dataquality.
Regarding the LLM-assisted features, the current PR does not include any
implementation. It only adds the SKILLS [1 ]and the reference schema for
the DQ Rule structure. Are you suggesting that I move this SKILL
documentation to a separate PR?
[1]:
https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md
Regards,
Pavan
On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai <am...@apache.org> wrote:
> Hi Pavan,
>
> First of all, +1 to this.
>
> Now, few things:
>
> * On naming: dataquality over dq for me honestly. Our existing provider
> names spell things out
> (common.sql, openlineage, not abbreviated forms) and dq is genuinely
> ambiguous outside context.
>
> * On scope: I also agree with Niko that #69413 is too large for one pass &
> I am glad to see the
> backend/UI split already happening in #69575. Would also suggest keeping
> the LLM assisted rule
> generation pieces (*schema-based generate_rules_from_schema*) out of the
> initial provider PR entirely
> cos as I see it, its a separable capability and bundling it will slow
> review of the core DQRule or
> RuleSet or operator surface, which is the part that actually needs the most
> detailed review.
>
> In short: go for it!
>
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <
> gopidesupavan@gmail.com>
> wrote:
>
> > In the meantime, the PR is ready for review. Feel free to review and
> > provide any feedback.
> >
> > Regards,
> > Pavan
> >
> > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <
> > gopidesupavan@gmail.com>
> > wrote:
> >
> > > Sorry, I forgot to add: the draft PR is here
> > > https://github.com/apache/airflow/pull/69413; it's still a WIP.
> > >
> > > some screenshots
> > > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468
> :)
> > >
> > > Pavan
> > >
> > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <
> > > gopidesupavan@gmail.com> wrote:
> > >
> > >> Hi Airflow community,
> > >>
> > >> I would like to start a discussion regarding a new provider:
> > >> apache-airflow-providers-dq.
> > >>
> > >> While Airflow already includes SQL check operators that many users
> rely
> > >> on for data quality, this new provider builds on that foundation by
> > >> introducing DQRule and RuleSet objects, stable rule identity,
> persisted
> > >> history, and direct connections to Airflow assets. This approach makes
> > >> quality results easier to inspect over time, allows downstream
> > consumers to
> > >> gate tasks based on recent quality results, and provides a unified
> > schema
> > >> for LLM-assisted workflows. Execution will continue to utilize
> existing
> > >> DbApiHook connections.
> > >>
> > >> The initial version of the provider is intentionally focused:
> > >>
> > >> - Declarative DQRule and RuleSet objects.
> > >> - DQCheckOperator and @task.dq_check.
> > >> - DbApiHook-based SQL checks, including built-in checks and
> > custom_sql.
> > >> - Persisted results for tasks, runs, and rules.
> > >> - A minimal Airflow UI plugin for viewing results and rule history.
> > >> - Experimental asset helpers such as asset_quality() and
> > >> require_quality().
> > >>
> > >> Regarding scope, this first iteration uses object storage only to
> > persist
> > >> DQ results and history; checks are executed via database connections.
> > >> Future iterations may include file or object-store based checks (e.g.,
> > S3,
> > >> GCS) where Airflow runs quality rules against data directly.
> > >>
> > >> This proposal does not require changes to Airflow core. Asset support
> is
> > >> currently provider-owned metadata, with static configuration stored on
> > the
> > >> asset and runtime summaries stored on asset events. If the provider
> > gains
> > >> traction, we can discuss making Data Quality a first-class component
> of
> > >> Airflow assets.
> > >>
> > >> This work also serves as a practical follow-up to the data quality
> > >> direction mentioned in AIP-99. Persisted history is valuable for users
> > and
> > >> future LLM-assisted workflows, such as those from Anthropic or
> > common.ai,
> > >> to understand rule performance and generate candidate rules based on
> > schema
> > >> context.
> > >>
> > >> A rough pseudo-flow is provided below:
> > >>
> > >> seed_rules = RuleSet(
> > >> name="orders_quality",
> > >> rules=[
> > >> DQRule(name="order_id_not_null", check="null_count",
> > >> column="order_id", condition={"equal_to": 0}),
> > >> DQRule(name="amount_valid", check="min", column="amount",
> > >> condition={"geq_to": 0}),
> > >> ],
> > >> )
> > >>
> > >> orders_asset = asset_quality(
> > >> Asset("orders"),
> > >> conn_id="warehouse",
> > >> table="orders",
> > >> ruleset=seed_rules,
> > >> )
> > >>
> > >> # Optional: common.ai / Anthropic provider can generate a RuleSet
> from
> > >> schema context.
> > >> generated_rules = generate_rules_from_schema(...)
> > >>
> > >> @task.dq_check(asset=orders_asset)
> > >> def check_orders(ruleset):
> > >> return ruleset
> > >>
> > >> checked_orders = check_orders(generated_rules)
> > >>
> > >> with DAG("orders_consumer", schedule=orders_asset):
> > >> require_quality(orders_asset, min_score=0.95) >> consume_orders()
> > >>
> > >> The UI remains deliberately minimal for this initial release, focusing
> > on
> > >> result and history inspection.
> > >>
> > >> You can view examples [1] of how it's integrated with assets/llms.
> > >>
> > >> currently i named it providers `apache-airflow-providers-dq`. if any
> > >> other preference likely with `dataquality`. Please let me know if you
> > have
> > >> a preference. naming is hard :)
> > >>
> > >> [1]:
> > >>
> >
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> > >>
> > >> Thanks,
> > >> Pavan
> > >>
> > >>
> >
>
Re: New Community Provider: Data Quality
Posted by Amogh Desai <am...@apache.org>.
Hi Pavan,
First of all, +1 to this.
Now, few things:
* On naming: dataquality over dq for me honestly. Our existing provider
names spell things out
(common.sql, openlineage, not abbreviated forms) and dq is genuinely
ambiguous outside context.
* On scope: I also agree with Niko that #69413 is too large for one pass &
I am glad to see the
backend/UI split already happening in #69575. Would also suggest keeping
the LLM assisted rule
generation pieces (*schema-based generate_rules_from_schema*) out of the
initial provider PR entirely
cos as I see it, its a separable capability and bundling it will slow
review of the core DQRule or
RuleSet or operator surface, which is the part that actually needs the most
detailed review.
In short: go for it!
Thanks & Regards,
Amogh Desai
On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <go...@gmail.com>
wrote:
> In the meantime, the PR is ready for review. Feel free to review and
> provide any feedback.
>
> Regards,
> Pavan
>
> On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <
> gopidesupavan@gmail.com>
> wrote:
>
> > Sorry, I forgot to add: the draft PR is here
> > https://github.com/apache/airflow/pull/69413; it's still a WIP.
> >
> > some screenshots
> > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468 :)
> >
> > Pavan
> >
> > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <
> > gopidesupavan@gmail.com> wrote:
> >
> >> Hi Airflow community,
> >>
> >> I would like to start a discussion regarding a new provider:
> >> apache-airflow-providers-dq.
> >>
> >> While Airflow already includes SQL check operators that many users rely
> >> on for data quality, this new provider builds on that foundation by
> >> introducing DQRule and RuleSet objects, stable rule identity, persisted
> >> history, and direct connections to Airflow assets. This approach makes
> >> quality results easier to inspect over time, allows downstream
> consumers to
> >> gate tasks based on recent quality results, and provides a unified
> schema
> >> for LLM-assisted workflows. Execution will continue to utilize existing
> >> DbApiHook connections.
> >>
> >> The initial version of the provider is intentionally focused:
> >>
> >> - Declarative DQRule and RuleSet objects.
> >> - DQCheckOperator and @task.dq_check.
> >> - DbApiHook-based SQL checks, including built-in checks and
> custom_sql.
> >> - Persisted results for tasks, runs, and rules.
> >> - A minimal Airflow UI plugin for viewing results and rule history.
> >> - Experimental asset helpers such as asset_quality() and
> >> require_quality().
> >>
> >> Regarding scope, this first iteration uses object storage only to
> persist
> >> DQ results and history; checks are executed via database connections.
> >> Future iterations may include file or object-store based checks (e.g.,
> S3,
> >> GCS) where Airflow runs quality rules against data directly.
> >>
> >> This proposal does not require changes to Airflow core. Asset support is
> >> currently provider-owned metadata, with static configuration stored on
> the
> >> asset and runtime summaries stored on asset events. If the provider
> gains
> >> traction, we can discuss making Data Quality a first-class component of
> >> Airflow assets.
> >>
> >> This work also serves as a practical follow-up to the data quality
> >> direction mentioned in AIP-99. Persisted history is valuable for users
> and
> >> future LLM-assisted workflows, such as those from Anthropic or
> common.ai,
> >> to understand rule performance and generate candidate rules based on
> schema
> >> context.
> >>
> >> A rough pseudo-flow is provided below:
> >>
> >> seed_rules = RuleSet(
> >> name="orders_quality",
> >> rules=[
> >> DQRule(name="order_id_not_null", check="null_count",
> >> column="order_id", condition={"equal_to": 0}),
> >> DQRule(name="amount_valid", check="min", column="amount",
> >> condition={"geq_to": 0}),
> >> ],
> >> )
> >>
> >> orders_asset = asset_quality(
> >> Asset("orders"),
> >> conn_id="warehouse",
> >> table="orders",
> >> ruleset=seed_rules,
> >> )
> >>
> >> # Optional: common.ai / Anthropic provider can generate a RuleSet from
> >> schema context.
> >> generated_rules = generate_rules_from_schema(...)
> >>
> >> @task.dq_check(asset=orders_asset)
> >> def check_orders(ruleset):
> >> return ruleset
> >>
> >> checked_orders = check_orders(generated_rules)
> >>
> >> with DAG("orders_consumer", schedule=orders_asset):
> >> require_quality(orders_asset, min_score=0.95) >> consume_orders()
> >>
> >> The UI remains deliberately minimal for this initial release, focusing
> on
> >> result and history inspection.
> >>
> >> You can view examples [1] of how it's integrated with assets/llms.
> >>
> >> currently i named it providers `apache-airflow-providers-dq`. if any
> >> other preference likely with `dataquality`. Please let me know if you
> have
> >> a preference. naming is hard :)
> >>
> >> [1]:
> >>
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> >>
> >> Thanks,
> >> Pavan
> >>
> >>
>
Re: New Community Provider: Data Quality
Posted by Pavankumar Gopidesu <go...@gmail.com>.
In the meantime, the PR is ready for review. Feel free to review and
provide any feedback.
Regards,
Pavan
On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <go...@gmail.com>
wrote:
> Sorry, I forgot to add: the draft PR is here
> https://github.com/apache/airflow/pull/69413; it's still a WIP.
>
> some screenshots
> https://github.com/apache/airflow/pull/69413#issuecomment-4886311468 :)
>
> Pavan
>
> On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <
> gopidesupavan@gmail.com> wrote:
>
>> Hi Airflow community,
>>
>> I would like to start a discussion regarding a new provider:
>> apache-airflow-providers-dq.
>>
>> While Airflow already includes SQL check operators that many users rely
>> on for data quality, this new provider builds on that foundation by
>> introducing DQRule and RuleSet objects, stable rule identity, persisted
>> history, and direct connections to Airflow assets. This approach makes
>> quality results easier to inspect over time, allows downstream consumers to
>> gate tasks based on recent quality results, and provides a unified schema
>> for LLM-assisted workflows. Execution will continue to utilize existing
>> DbApiHook connections.
>>
>> The initial version of the provider is intentionally focused:
>>
>> - Declarative DQRule and RuleSet objects.
>> - DQCheckOperator and @task.dq_check.
>> - DbApiHook-based SQL checks, including built-in checks and custom_sql.
>> - Persisted results for tasks, runs, and rules.
>> - A minimal Airflow UI plugin for viewing results and rule history.
>> - Experimental asset helpers such as asset_quality() and
>> require_quality().
>>
>> Regarding scope, this first iteration uses object storage only to persist
>> DQ results and history; checks are executed via database connections.
>> Future iterations may include file or object-store based checks (e.g., S3,
>> GCS) where Airflow runs quality rules against data directly.
>>
>> This proposal does not require changes to Airflow core. Asset support is
>> currently provider-owned metadata, with static configuration stored on the
>> asset and runtime summaries stored on asset events. If the provider gains
>> traction, we can discuss making Data Quality a first-class component of
>> Airflow assets.
>>
>> This work also serves as a practical follow-up to the data quality
>> direction mentioned in AIP-99. Persisted history is valuable for users and
>> future LLM-assisted workflows, such as those from Anthropic or common.ai,
>> to understand rule performance and generate candidate rules based on schema
>> context.
>>
>> A rough pseudo-flow is provided below:
>>
>> seed_rules = RuleSet(
>> name="orders_quality",
>> rules=[
>> DQRule(name="order_id_not_null", check="null_count",
>> column="order_id", condition={"equal_to": 0}),
>> DQRule(name="amount_valid", check="min", column="amount",
>> condition={"geq_to": 0}),
>> ],
>> )
>>
>> orders_asset = asset_quality(
>> Asset("orders"),
>> conn_id="warehouse",
>> table="orders",
>> ruleset=seed_rules,
>> )
>>
>> # Optional: common.ai / Anthropic provider can generate a RuleSet from
>> schema context.
>> generated_rules = generate_rules_from_schema(...)
>>
>> @task.dq_check(asset=orders_asset)
>> def check_orders(ruleset):
>> return ruleset
>>
>> checked_orders = check_orders(generated_rules)
>>
>> with DAG("orders_consumer", schedule=orders_asset):
>> require_quality(orders_asset, min_score=0.95) >> consume_orders()
>>
>> The UI remains deliberately minimal for this initial release, focusing on
>> result and history inspection.
>>
>> You can view examples [1] of how it's integrated with assets/llms.
>>
>> currently i named it providers `apache-airflow-providers-dq`. if any
>> other preference likely with `dataquality`. Please let me know if you have
>> a preference. naming is hard :)
>>
>> [1]:
>> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
>>
>> Thanks,
>> Pavan
>>
>>
Re: New Community Provider: Data Quality
Posted by Pavankumar Gopidesu <go...@gmail.com>.
Sorry, I forgot to add: the draft PR is here
https://github.com/apache/airflow/pull/69413; it's still a WIP.
some screenshots
https://github.com/apache/airflow/pull/69413#issuecomment-4886311468 :)
Pavan
On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <go...@gmail.com>
wrote:
> Hi Airflow community,
>
> I would like to start a discussion regarding a new provider:
> apache-airflow-providers-dq.
>
> While Airflow already includes SQL check operators that many users rely on
> for data quality, this new provider builds on that foundation by
> introducing DQRule and RuleSet objects, stable rule identity, persisted
> history, and direct connections to Airflow assets. This approach makes
> quality results easier to inspect over time, allows downstream consumers to
> gate tasks based on recent quality results, and provides a unified schema
> for LLM-assisted workflows. Execution will continue to utilize existing
> DbApiHook connections.
>
> The initial version of the provider is intentionally focused:
>
> - Declarative DQRule and RuleSet objects.
> - DQCheckOperator and @task.dq_check.
> - DbApiHook-based SQL checks, including built-in checks and custom_sql.
> - Persisted results for tasks, runs, and rules.
> - A minimal Airflow UI plugin for viewing results and rule history.
> - Experimental asset helpers such as asset_quality() and
> require_quality().
>
> Regarding scope, this first iteration uses object storage only to persist
> DQ results and history; checks are executed via database connections.
> Future iterations may include file or object-store based checks (e.g., S3,
> GCS) where Airflow runs quality rules against data directly.
>
> This proposal does not require changes to Airflow core. Asset support is
> currently provider-owned metadata, with static configuration stored on the
> asset and runtime summaries stored on asset events. If the provider gains
> traction, we can discuss making Data Quality a first-class component of
> Airflow assets.
>
> This work also serves as a practical follow-up to the data quality
> direction mentioned in AIP-99. Persisted history is valuable for users and
> future LLM-assisted workflows, such as those from Anthropic or common.ai,
> to understand rule performance and generate candidate rules based on schema
> context.
>
> A rough pseudo-flow is provided below:
>
> seed_rules = RuleSet(
> name="orders_quality",
> rules=[
> DQRule(name="order_id_not_null", check="null_count",
> column="order_id", condition={"equal_to": 0}),
> DQRule(name="amount_valid", check="min", column="amount",
> condition={"geq_to": 0}),
> ],
> )
>
> orders_asset = asset_quality(
> Asset("orders"),
> conn_id="warehouse",
> table="orders",
> ruleset=seed_rules,
> )
>
> # Optional: common.ai / Anthropic provider can generate a RuleSet from
> schema context.
> generated_rules = generate_rules_from_schema(...)
>
> @task.dq_check(asset=orders_asset)
> def check_orders(ruleset):
> return ruleset
>
> checked_orders = check_orders(generated_rules)
>
> with DAG("orders_consumer", schedule=orders_asset):
> require_quality(orders_asset, min_score=0.95) >> consume_orders()
>
> The UI remains deliberately minimal for this initial release, focusing on
> result and history inspection.
>
> You can view examples [1] of how it's integrated with assets/llms.
>
> currently i named it providers `apache-airflow-providers-dq`. if any
> other preference likely with `dataquality`. Please let me know if you have
> a preference. naming is hard :)
>
> [1]:
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
>
> Thanks,
> Pavan
>
>