[DISCUSS] Provider-contributed tables for 'airflow db clean'

[Tzu-ping Chung via dev <de...@airflow.apache.org>]

Hi all,

The command 'airflow db clean' currently hardcodes some provider-owned tables in Core. This causes a bug when those tables are configurable. Specifically 'celery_taskmeta' and 'celery_tasksetmeta' may not be correctly located if the user sets the '[celery] result_backend_schema' configuration since the config is not respected in core.

I have proposed #70639 to address this. This builds on #70147 (already merged), which allows the core clean command to handle a schema-qualified table name. The new PR #70639 adds a provider.yml key db-cleanup-tables key pointing at a callable that returns cleanup configs. ProvidersManager discovers them (like db-managers) and db clean merges them in; a provider resolves its own settings, so core needn't read provider config for tables it contributes.

The PR introduces the extension point and moves Celery onto it. I also identified a couple of possibilities:

* FAB. The session table is currently hard-coded in core as a carry-over from Airflow 2, when FAB was a required dependency.
* Edge3. edge_logs looks like a fit, edge_job would need a recency/terminal-state decision, and edge_worker probably isn't a cleanup target.

What do y'all think? Especially for the Edge3 tables; I'm not involved in the provider enough to make a call. The FAB one is a toss-up to me; it seems clean to move it to the provider, but also OK to be kept as-is since it's always had special status compared to most providers.

Also, feel free to raise any suggestions to the interface put out in #70639.

Thanks,
TP

[#70639]: https://github.com/apache/airflow/pull/70639
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org