[PR] [v3-2-test] Add --error-on-cleanup-failure flag to airflow db clean (#65239) [airflow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] opened a new pull request, #66606:
URL: https://github.com/apache/airflow/pull/66606

   * fix(db_cleanup): add --error-on-cleanup-failure flag to airflow db clean
   
   airflow db clean suppresses all per-table cleanup errors via
   _suppress_with_logging() and exits 0 even when tables could not be
   cleaned. This makes it impossible to detect silent failures in automated
   DAG-based maintenance workflows, which can lead to unchecked table
   growth and eventual migration failures on upgrade.
   
   This commit adds an opt-in --error-on-cleanup-failure flag that causes
   run_cleanup() to raise AirflowException (and the CLI to exit 1) if any
   table cleanup encountered an error. Default behaviour is unchanged.
   
   Additionally, a warning listing all tables that were not cleaned is now
   always emitted when failures occur, even without the flag, improving
   observability without requiring any opt-in.
   
   Changes:
   - airflow/utils/db_cleanup.py: update _suppress_with_logging to track
     whether an exception was suppressed via a SimpleNamespace context
     object; collect failed table names in run_cleanup(); emit a warning
     summary and optionally raise AirflowException.
   - airflow/cli/cli_config.py: add ARG_DB_ERROR_ON_CLEANUP_FAILURE and
     wire it into the db clean ActionCommand args list.
   - airflow/cli/commands/db_command.py: forward error_on_cleanup_failure
     from CLI args to run_cleanup().
   - tests/utils/test_db_cleanup.py: add unit tests covering the new flag
     and the warning summary behaviour.
   
   Made-with: Cursor
   
   * fix(db_cleanup): improve type annotations and add docs for --error-on-cleanup-failure
   
   - Use collections.abc.Generator for the _suppress_with_logging return
     type annotation (ruff UP035 compliant) instead of typing.Generator.
   - Expand the _suppress_with_logging docstring to describe the yielded
     SimpleNamespace context object and the failure-tracking behaviour.
   - Add a new "Detecting cleanup failures" section to
     docs/howto/usage-cli.rst documenting the --error-on-cleanup-failure
     flag and the --skip-archive recommendation for large tables.
   
   Made-with: Cursor
   
   * fix(db_cleanup): address CI failures on PR #65239
   
   - Fix mypy arg-type errors: OperationalError third argument must be a
     BaseException, not None. Replace OperationalError("", {}, None) with
     OperationalError("", {}, Exception("mock db error")) in three new
     tests in test_db_cleanup.py.
   - Fix ruff ISC violation: collapse implicit string concatenation in the
     run_cleanup() warning call into a single string literal.
   - Update existing CLI tests in test_db_command.py to include the new
     error_on_cleanup_failure=False kwarg in all ten
     assert_called_once_with assertions.
   - Add test_error_on_cleanup_failure to test_db_command.py to verify the
     --error-on-cleanup-failure flag is correctly forwarded to run_cleanup.
   
   Made-with: Cursor
   
   * fix(db_cleanup): add --error-on-cleanup-failure flag documentation and tests
   
   - Introduce a new news fragment detailing the addition of the ``--error-on-cleanup-failure`` flag to the ``airflow db clean`` command, allowing for better error handling during table cleanup.
   - Update unit tests in `test_db_cleanup.py` to ensure proper functionality of the new flag, including checks for raised exceptions and warning messages for failed tables.
   - Adjust the known exceptions list to reflect changes in `db_cleanup.py`.
   
   * fix(db_cleanup): update error handling for table cleanup failures
   
   - Change the behavior of the `--error-on-cleanup-failure` flag to raise a RuntimeError instead of an AirflowException when table cleanup encounters errors.
   - Update the documentation and help text for the flag to clarify its functionality.
   - Ensure that warning messages for failed tables are always emitted, regardless of the flag's state.
   - Modify unit tests in `test_db_cleanup.py` to reflect the new error handling and verify the correct logging behavior.
   
   This update improves error visibility during automated workflows by ensuring that cleanup failures are properly reported.
   
   * fix(db_cleanup): remove redundant cleanup commit
   (cherry picked from commit 4364908da2c0e798800509f2427fce90777af3ec)
   
   Co-authored-by: Hemkumar Chheda <95...@users.noreply.github.com>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [v3-2-test] Add --error-on-cleanup-failure flag to airflow db clean (#65239) [airflow]

["jscheffl (via GitHub)" <gi...@apache.org>]

jscheffl merged PR #66606:
URL: https://github.com/apache/airflow/pull/66606


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org