You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by po...@apache.org on 2020/10/18 14:40:50 UTC
[airflow] branch master updated: Name and optionally preserve data
volumes in Breeze (#11628)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new 66ced72 Name and optionally preserve data volumes in Breeze (#11628)
66ced72 is described below
commit 66ced72fcac9fc346651a66a9d8cb9639ccc43f1
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Sun Oct 18 16:39:44 2020 +0200
Name and optionally preserve data volumes in Breeze (#11628)
So far breeze used in-container data for persisting it (mysql redis,
postgres). This means that the data was kept as long, as long the
containers were running. If you stopped Breeze via `stop` command
the data was always deleted.
This changes the behaviour - each of the Breeze containers has
a named volume where data is kept. Those volumes are also deleted
by default when Breeze is stopped, but you can choose to preserve
them by adding ``--preserve-volumes`` when you run ``stop`` or
``restart`` command.
Fixes: #11625
---
BREEZE.rst | 42 +++++++++++++++++---
breeze | 45 ++++++++++++++++++++--
breeze-complete | 1 +
scripts/ci/docker-compose/backend-mysql.yml | 3 ++
scripts/ci/docker-compose/backend-postgres.yml | 3 ++
scripts/ci/docker-compose/backend-sqlite.yml | 5 +++
.../ci/docker-compose/integration-cassandra.yml | 3 ++
scripts/ci/docker-compose/integration-mongo.yml | 3 ++
scripts/ci/docker-compose/integration-openldap.yml | 3 ++
scripts/ci/docker-compose/integration-presto.yml | 3 ++
scripts/ci/docker-compose/integration-rabbitmq.yml | 3 ++
scripts/ci/docker-compose/integration-redis.yml | 3 ++
scripts/ci/libraries/_initialization.sh | 4 ++
scripts/ci/mysql/conf.d/airflow.cnf | 2 +-
tests/providers/mysql/hooks/test_mysql.py | 9 ++++-
15 files changed, 119 insertions(+), 13 deletions(-)
diff --git a/BREEZE.rst b/BREEZE.rst
index 68000f4..6827521 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -376,6 +376,16 @@ Set up local development environment:
* Setup local virtualenv with ``breeze setup-virtualenv`` command
* Setup autocomplete for itself with ``breeze setup-autocomplete`` command
+Database volumes in Breeze
+--------------------------
+
+Breeze keeps data for all it's integration in named docker volumes. Each backend and integration
+keeps data in their own volume. Those volumes are persisted until ``./breeze stop`` command or
+``./breeze restart`` command is run. You can also preserve the volumes by adding flag
+``--preserve-volumes`` when you run either of those commands. Then, next time when you start
+``Breeze``, it will have the data pre-populated. You can always delete the volumes by
+runnint ``./breeze stop`` without the ``--preserve-volumes`` flag.
+
Launching multiple terminals
----------------------------
@@ -1141,10 +1151,6 @@ This is the current syntax for `./breeze <./breeze>`_:
'breeze \
--github-image-id 209845560' - pull/use image with RUN_ID
- Flags:
-
- Run 'breeze flags' to see all applicable flags.
-
####################################################################################################
@@ -1620,6 +1626,15 @@ This is the current syntax for `./breeze <./breeze>`_:
containers will continue running so that startup time is shorter. But they take quite a lot of
memory and CPU. This command stops all running containers from the environment.
+ Flags:
+
+ --preserve-volumes
+ Use this flag if you would like to preserve data volumes from the databases used
+ by the integrations. By default, those volumes are deleted, so when you run 'stop'
+ or 'restart' commands you start from scratch, but by using this flag you can
+ preserve them. If you want to delete those volumes after stopping Breeze, just
+ run the 'breeze stop' again without this flag.
+
####################################################################################################
@@ -1635,7 +1650,12 @@ This is the current syntax for `./breeze <./breeze>`_:
Flags:
- Run 'breeze flags' to see all applicable flags.
+ --preserve-volumes
+ Use this flag if you would like to preserve data volumes from the databases used
+ by the integrations. By default, those volumes are deleted, so when you run 'stop'
+ or 'restart' commands you start from scratch, but by using this flag you can
+ preserve them. If you want to delete those volumes after stopping Breeze, just
+ run the 'breeze stop' again without this flag.
####################################################################################################
@@ -2125,7 +2145,7 @@ This is the current syntax for `./breeze <./breeze>`_:
init.sh. It will be executed after the environment is configured and started.
****************************************************************************************************
- Additional actions executed starting Airflow
+ Additional actions executed while starting Airflow
--load-example-dags
Include Airflow example dags.
@@ -2134,6 +2154,16 @@ This is the current syntax for `./breeze <./breeze>`_:
Include Airflow Default Connections.
****************************************************************************************************
+ Cleanup options when stopping Airflow
+
+ --preserve-volumes
+ Use this flag if you would like to preserve data volumes from the databases used
+ by the integrations. By default, those volumes are deleted, so when you run 'stop'
+ or 'restart' commands you start from scratch, but by using this flag you can
+ preserve them. If you want to delete those volumes after stopping Breeze, just
+ run the 'breeze stop' again without this flag.
+
+ ****************************************************************************************************
Kind kubernetes and Kubernetes tests configuration(optional)
Configuration for the KinD Kubernetes cluster and tests:
diff --git a/breeze b/breeze
index 321b46c..7a9cea8 100755
--- a/breeze
+++ b/breeze
@@ -123,6 +123,10 @@ function breeze::setup_default_breeze_constants() {
# If set to true, the sample dags will be used
export LOAD_EXAMPLES="false"
+ # If set to true, Breeze db volumes will be preserved when breeze is stopped and reused next time
+ # Which means that you do not have to start from scratch
+ export PRESERVE_VOLUMES="false"
+
# If set to true, Backport packages are prepared not the Regular ones
export BACKPORT_PACKAGES="false"
@@ -1136,6 +1140,12 @@ function breeze::parse_arguments() {
echo
shift
;;
+ --preserve-volumes)
+ export PRESERVE_VOLUMES="true"
+ echo "Preserves data volumes when stopping airflow"
+ echo
+ shift
+ ;;
--no-rbac-ui)
export DISABLE_RBAC="true"
echo "When installing Airflow 1.10, RBAC UI will be disabled."
@@ -1376,6 +1386,9 @@ function breeze::parse_arguments() {
exit 0
fi
+ if [[ ${PRESERVE_VOLUMES} != "true" ]]; then
+ EXTRA_DC_OPTIONS+=("--volumes")
+ fi
# EXTRA_DC_OPTIONS is only used by Breeze. It's value is set here as well.
readonly EXTRA_DC_OPTIONS
@@ -1567,8 +1580,6 @@ ${CMDNAME} shell [FLAGS] [-- <EXTRA_ARGS>]
'${CMDNAME} \\
--github-image-id 209845560' - pull/use image with RUN_ID
-Flags:
-$(breeze::flag_footer)
"
readonly DETAILED_USAGE_SHELL
export DETAILED_USAGE_EXEC="
@@ -1817,6 +1828,9 @@ ${CMDNAME} stop
Brings down running docker compose environment. When you start the environment, the docker
containers will continue running so that startup time is shorter. But they take quite a lot of
memory and CPU. This command stops all running containers from the environment.
+
+Flags:
+$(breeze::flag_stop_airflow)
"
readonly DETAILED_USAGE_STOP
export DETAILED_USAGE_RESTART="
@@ -1827,7 +1841,7 @@ ${CMDNAME} restart [FLAGS]
especially useful if you switch between different versions of Airflow.
Flags:
-$(breeze::flag_footer)
+$(breeze::flag_stop_airflow)
"
readonly DETAILED_USAGE_RESTART
export DETAILED_USAGE_STATIC_CHECK="
@@ -2518,6 +2532,25 @@ function breeze::flag_start_airflow() {
#####################################################################################################
#
+# Prints flags that control how Airflow should be populated with the command stop-airflow
+#
+# Outputs:
+# Flag information.
+#######################################################################################################
+function breeze::flag_stop_airflow() {
+ echo "
+--preserve-volumes
+ Use this flag if you would like to preserve data volumes from the databases used
+ by the integrations. By default, those volumes are deleted, so when you run 'stop'
+ or 'restart' commands you start from scratch, but by using this flag you can
+ preserve them. If you want to delete those volumes after stopping Breeze, just
+ run the 'breeze stop' again without this flag.
+"
+}
+
+
+#####################################################################################################
+#
# Prints flags that control tests
#
# Outputs:
@@ -2565,10 +2598,14 @@ $(breeze::print_star_line)
$(breeze::flag_breeze_actions)
$(breeze::print_star_line)
- Additional actions executed starting Airflow
+ Additional actions executed while starting Airflow
$(breeze::flag_start_airflow)
$(breeze::print_star_line)
+ Cleanup options when stopping Airflow
+$(breeze::flag_stop_airflow)
+
+$(breeze::print_star_line)
Kind kubernetes and Kubernetes tests configuration(optional)
$(breeze::flag_kubernetes_configuration)
diff --git a/breeze-complete b/breeze-complete
index 2616bad..d0bafab 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -157,6 +157,7 @@ runtime-apt-deps: additional-runtime-apt-deps: runtime-apt-command: additional-r
load-default-connections load-example-dags
install-wheels no-rbac-ui
test-type:
+preserve-volumes
"
_breeze_commands="
diff --git a/scripts/ci/docker-compose/backend-mysql.yml b/scripts/ci/docker-compose/backend-mysql.yml
index de69c22..6ec8d29 100644
--- a/scripts/ci/docker-compose/backend-mysql.yml
+++ b/scripts/ci/docker-compose/backend-mysql.yml
@@ -34,5 +34,8 @@ services:
volumes:
- ../mysql/conf.d:/etc/mysql/conf.d:ro
- /dev/urandom:/dev/random # Required to get non-blocking entropy source
+ - mysql-db-volume:/var/lib/mysql
ports:
- "${MYSQL_HOST_PORT}:3306"
+volumes:
+ mysql-db-volume:
diff --git a/scripts/ci/docker-compose/backend-postgres.yml b/scripts/ci/docker-compose/backend-postgres.yml
index 74b4f25..3e4da87 100644
--- a/scripts/ci/docker-compose/backend-postgres.yml
+++ b/scripts/ci/docker-compose/backend-postgres.yml
@@ -32,5 +32,8 @@ services:
- POSTGRES_DB=airflow
volumes:
- /dev/urandom:/dev/random # Required to get non-blocking entropy source
+ - postgres-db-volume:/var/lib/postgresql/data
ports:
- "${POSTGRES_HOST_PORT}:5432"
+volumes:
+ postgres-db-volume:
diff --git a/scripts/ci/docker-compose/backend-sqlite.yml b/scripts/ci/docker-compose/backend-sqlite.yml
index 50972b6..1d313d0 100644
--- a/scripts/ci/docker-compose/backend-sqlite.yml
+++ b/scripts/ci/docker-compose/backend-sqlite.yml
@@ -22,3 +22,8 @@ services:
- BACKEND=sqlite
- AIRFLOW__CORE__SQL_ALCHEMY_CONN=${SQLITE_URL}
- AIRFLOW__CORE__EXECUTOR=SequentialExecutor
+ volumes:
+ - /dev/urandom:/dev/random # Required to get non-blocking entropy source
+ - sqlite-db-volume:/root/airflow
+volumes:
+ sqlite-db-volume:
diff --git a/scripts/ci/docker-compose/integration-cassandra.yml b/scripts/ci/docker-compose/integration-cassandra.yml
index 068134b..6a6f9f2 100644
--- a/scripts/ci/docker-compose/integration-cassandra.yml
+++ b/scripts/ci/docker-compose/integration-cassandra.yml
@@ -21,8 +21,11 @@ services:
image: cassandra:3.0
volumes:
- /dev/urandom:/dev/random # Required to get non-blocking entropy source
+ - cassandra-db-volume:/var/lib/cassandra
airflow:
environment:
- INTEGRATION_CASSANDRA=true
depends_on:
- cassandra
+volumes:
+ cassandra-db-volume:
diff --git a/scripts/ci/docker-compose/integration-mongo.yml b/scripts/ci/docker-compose/integration-mongo.yml
index f2dad66..7feeda0 100644
--- a/scripts/ci/docker-compose/integration-mongo.yml
+++ b/scripts/ci/docker-compose/integration-mongo.yml
@@ -21,8 +21,11 @@ services:
image: mongo:3
volumes:
- /dev/urandom:/dev/random # Required to get non-blocking entropy source
+ - mongo-db-volume:/data/db
airflow:
environment:
- INTEGRATION_MONGO=true
depends_on:
- mongo
+volumes:
+ mongo-db-volume:
diff --git a/scripts/ci/docker-compose/integration-openldap.yml b/scripts/ci/docker-compose/integration-openldap.yml
index dedbb0b..a100f2f 100644
--- a/scripts/ci/docker-compose/integration-openldap.yml
+++ b/scripts/ci/docker-compose/integration-openldap.yml
@@ -27,8 +27,11 @@ services:
volumes:
- ../openldap/ldif:/container/service/slapd/assets/config/bootstrap/ldif/custom:ro
- /dev/urandom:/dev/random # Required to get non-blocking entropy source
+ - openldap-db-volume:/var/lib/ldap
airflow:
environment:
- INTEGRATION_OPENLDAP=true
depends_on:
- openldap
+volumes:
+ openldap-db-volume:
diff --git a/scripts/ci/docker-compose/integration-presto.yml b/scripts/ci/docker-compose/integration-presto.yml
index 9c139b5..3cbc077 100644
--- a/scripts/ci/docker-compose/integration-presto.yml
+++ b/scripts/ci/docker-compose/integration-presto.yml
@@ -21,8 +21,11 @@ services:
image: prestosql/presto:330
volumes:
- /dev/urandom:/dev/random # Required to get non-blocking entropy source
+ - presto-db-volume:/data/presto
airflow:
environment:
- INTEGRATION_PRESTO=true
depends_on:
- presto
+volumes:
+ presto-db-volume:
diff --git a/scripts/ci/docker-compose/integration-rabbitmq.yml b/scripts/ci/docker-compose/integration-rabbitmq.yml
index c9d750d..dc2eaed 100644
--- a/scripts/ci/docker-compose/integration-rabbitmq.yml
+++ b/scripts/ci/docker-compose/integration-rabbitmq.yml
@@ -21,8 +21,11 @@ services:
image: rabbitmq:3.7
volumes:
- /dev/urandom:/dev/random # Required to get non-blocking entropy source
+ - rabbitmq-db-volume:/var/lib/rabbitmq
airflow:
environment:
- INTEGRATION_RABBITMQ=true
depends_on:
- rabbitmq
+volumes:
+ rabbitmq-db-volume:
diff --git a/scripts/ci/docker-compose/integration-redis.yml b/scripts/ci/docker-compose/integration-redis.yml
index bf754f4..99e286d 100644
--- a/scripts/ci/docker-compose/integration-redis.yml
+++ b/scripts/ci/docker-compose/integration-redis.yml
@@ -21,6 +21,7 @@ services:
image: redis:5.0.1
volumes:
- /dev/urandom:/dev/random # Required to get non-blocking entropy source
+ - redis-db-volume:/data/presto
ports:
- "${REDIS_HOST_PORT}:6379"
airflow:
@@ -28,3 +29,5 @@ services:
- INTEGRATION_REDIS=true
depends_on:
- redis
+volumes:
+ redis-db-volume:
diff --git a/scripts/ci/libraries/_initialization.sh b/scripts/ci/libraries/_initialization.sh
index dcbfa19..ddc15a7 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -125,6 +125,10 @@ function initialization::initialize_base_variables() {
# If set to true, the test connections will be created
export LOAD_DEFAULT_CONNECTIONS=${LOAD_DEFAULT_CONNECTIONS:="false"}
+ # If set to true, Breeze db volumes will be persisted when breeze is stopped and reused next time
+ # Which means that you do not have to start from scratch
+ export PRESERVE_VOLUMES="false"
+
# If set to true, RBAC UI will not be used for 1.10 version
export DISABLE_RBAC=${DISABLE_RBAC:="false"}
diff --git a/scripts/ci/mysql/conf.d/airflow.cnf b/scripts/ci/mysql/conf.d/airflow.cnf
index 58f2b37..83b21cb 100644
--- a/scripts/ci/mysql/conf.d/airflow.cnf
+++ b/scripts/ci/mysql/conf.d/airflow.cnf
@@ -19,5 +19,5 @@
[mysqld]
explicit_defaults_for_timestamp = 1
-secure_file_priv = ""
+secure_file_priv = "/var/lib/mysql"
local_infile = 1
diff --git a/tests/providers/mysql/hooks/test_mysql.py b/tests/providers/mysql/hooks/test_mysql.py
index 250e869..86b3107 100644
--- a/tests/providers/mysql/hooks/test_mysql.py
+++ b/tests/providers/mysql/hooks/test_mysql.py
@@ -20,6 +20,7 @@
import json
import os
import unittest
+import uuid
from unittest import mock
import MySQLdb.cursors
@@ -398,11 +399,15 @@ class TestMySql(unittest.TestCase):
with MySqlContext(client):
hook = MySqlHook('airflow_db')
priv = hook.get_first("SELECT @@global.secure_file_priv")
+ # Use random names to alllow re-running
if priv and priv[0]:
# Confirm that no error occurs
- hook.bulk_dump("INFORMATION_SCHEMA.TABLES", os.path.join(priv[0], "TABLES_{}".format(client)))
+ hook.bulk_dump(
+ "INFORMATION_SCHEMA.TABLES",
+ os.path.join(priv[0], "TABLES_{}-{}".format(client, uuid.uuid1())),
+ )
elif priv == ("",):
- hook.bulk_dump("INFORMATION_SCHEMA.TABLES", "TABLES_{}".format(client))
+ hook.bulk_dump("INFORMATION_SCHEMA.TABLES", "TABLES_{}_{}".format(client, uuid.uuid1()))
else:
self.skipTest("Skip test_mysql_hook_test_bulk_load " "since file output is not permitted")