You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Chris Riccomini (JIRA)" <ji...@apache.org> on 2016/05/12 15:42:12 UTC
[jira] [Created] (AIRFLOW-108) Add data retention to Airflow
Chris Riccomini created AIRFLOW-108:
---------------------------------------
Summary: Add data retention to Airflow
Key: AIRFLOW-108
URL: https://issues.apache.org/jira/browse/AIRFLOW-108
Project: Apache Airflow
Issue Type: New Feature
Components: db, scheduler
Reporter: Chris Riccomini
Airflow's DB currently holds the entire history of all executions for all time. This is problematic as the DB grows. The UI starts to get slower, and the DB's disk usage grows. There is no bound to how large the DB will grow.
It would be useful to add a feature in Airflow to do two things:
# Delete old data from the DB
# Mark some lower watermark, past which DAG executions are ignored
For example, (2) would allow you to tell the scheduler "ignore all data prior to a year ago". And (1) would allow Airflow to delete all data prior to January 1, 2015.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)