[D] Command airflow db clean does not work anymore with BashOperator in Airflow 3+ [airflow]

["jlandercy (via GitHub)" <gi...@apache.org>]

GitHub user jlandercy created a discussion: Command airflow db clean does not work anymore with BashOperator in Airflow 3+

I use Airflow 3.1.0 with Docker Compose (official file).
I would like to automatize Airflow cleaning using Airflow.

If I had to make it by hand, I would open the worker container: 

`docker exec -it airflow-airflow-worker-1 bash`

And issue the following command:

`airflow db clean --clean-before-timestamp 20250930 --yes`

I can also confirm the container does know the connection string:

```
echo $AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
postgresql+psycopg2://airflow:secret@postgres/airflow
```

Anyway when I run this command using a `BashOperator`:

```
    clean_database = BashOperator(
        task_id="clean-database",
        bash_command=f'airflow db clean --clean-before-timestamp {clean_before.isoformat()} --yes',
    )
```

I get the following error:

`sqlalchemy.exc.ArgumentError: Could not parse SQLAlchemy URL from string 'airflow-db-not-allowed:///'`

Which seems to be an explicit forbidden with a volontary bad shaped connection string.
I have not found any relevant documentation about this error.

I have confirmed that the `BashOperator` does execute within the same container `airflow-airflow-worker-1`.

My question is why my `BashOperator` blocked when issuing this command and how can I configure it properly to make it work ?

Related question I opened on SO: https://stackoverflow.com/questions/79779432/how-can-i-run-airflow-db-clean-command-using-bashoperator-with-airlfow-3


GitHub link: https://github.com/apache/airflow/discussions/56281

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Command airflow db clean does not work anymore with BashOperator in Airflow 3+ [airflow]

["salimuddin07 (via GitHub)" <gi...@apache.org>]

GitHub user salimuddin07 added a comment to the discussion: Command airflow db clean does not work anymore with BashOperator in Airflow 3+

You have a few ways to handle this:

1. Use `airflow db clean` outside the DAG

The simplest approach is to keep DB cleaning as a manual or scheduled job outside Airflow:

Use a cron job or systemd timer that executes the command inside the container.

Example:

`docker exec airflow-airflow-worker-1 airflow db clean --clean-before-timestamp 20250930 --yes`

2. Use a separate script and run it as a container task

Create a small Docker container (or script) that runs the cleaning command.

Schedule it via Airflow using a KubernetesPodOperator or DockerOperator instead of BashOperator.

The container can safely have the real DB connection string, bypassing the “blocked” environment.

3. Explicitly pass the DB connection as an environment variable

If you really want to use BashOperator, you can override the environment:

```
clean_database = BashOperator(
    task_id="clean-database",
    bash_command=f'airflow db clean --clean-before-timestamp {clean_before.isoformat()} --yes',
    env={
        "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN": "postgresql+psycopg2://airflow:secret@postgres/airflow"
    }
)
```


Caution: This works, but it bypasses the security protection, so use it only if you understand the risks.

4. Use Airflow API / maintenance commands instead

If your goal is to clean old DAG runs or logs, you can use Airflow’s Python API (airflow.models.DagRun deletion) in a PythonOperator, which is safer than calling destructive CLI commands.

Recommended Approach

For production, the safest and most maintainable way is:

Keep airflow db clean outside DAGs, running either via cron or as a one-off container job.

If you need to automate via DAG, use DockerOperator or KubernetesPodOperator with explicit DB connection, not BashOperator.

Avoid passing destructive commands directly inside DAGs, because Airflow 3 intentionally prevents it for safety.

GitHub link: https://github.com/apache/airflow/discussions/56281#discussioncomment-14560197

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Command airflow db clean does not work anymore with BashOperator in Airflow 3+ [airflow]

["seancaulfield (via GitHub)" <gi...@apache.org>]

GitHub user seancaulfield added a comment to the discussion: Command airflow db clean does not work anymore with BashOperator in Airflow 3+

This should be a bug, as there's no reason why the `airflow db` commands can't use the API server like everything else does in Airflow now.

Having to use a separate scheduling system is a ridiculous suggestion as that's what Airflow _does_.

GitHub link: https://github.com/apache/airflow/discussions/56281#discussioncomment-14681120

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Command airflow db clean does not work anymore with BashOperator in Airflow 3+ [airflow]

["seancaulfield (via GitHub)" <gi...@apache.org>]

GitHub user seancaulfield deleted a comment on the discussion: Command airflow db clean does not work anymore with BashOperator in Airflow 3+

This should be a bug, as there's no reason why the `airflow db` commands can't use the API server like everything else does in Airflow now.

Having to use a separate scheduling system is a ridiculous suggestion as that's what Airflow _does_.

GitHub link: https://github.com/apache/airflow/discussions/56281#discussioncomment-14681120

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Command airflow db clean does not work anymore with BashOperator in Airflow 3+ [airflow]

["nicot29 (via GitHub)" <gi...@apache.org>]

GitHub user nicot29 added a comment to the discussion: Command airflow db clean does not work anymore with BashOperator in Airflow 3+

add `append_env=True` to your BashOperator(...) call

GitHub link: https://github.com/apache/airflow/discussions/56281#discussioncomment-15547257

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org