You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2021/01/05 21:42:48 UTC
[airflow] branch master updated: Check for minimum version of
Sqlite (#13496)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new b213dbe Check for minimum version of Sqlite (#13496)
b213dbe is described below
commit b213dbef0a171671994511fc826358bb79278770
Author: Jarek Potiuk <ja...@polidea.com>
AuthorDate: Tue Jan 5 22:42:37 2021 +0100
Check for minimum version of Sqlite (#13496)
Some users testing Airlfow 2.0 with sqlite noticed that for
old versions of sqlite, Airflow does not run tasks and fails with
'sqlite3.OperationalError: near ",": syntax error' when running
tasks. More details about it in #13397.
Bisecting had shown that minimum supported version of sqlite is
3.15.0, therefore this PR adds checking if sqlite version is
higher than that and fails hard if it is not.
Documentation has been updated with minimum requirements, some
inconsisttencies have been removed, also the minimum requirements
for stable 2.0 version were moved to installation.rst because
the requirements were never explicitely stated in the user-facing
documentation.
---
README.md | 9 ++++-----
airflow/configuration.py | 14 ++++++++++----
docs/apache-airflow/installation.rst | 22 ++++++++++++++++++++++
3 files changed, 36 insertions(+), 9 deletions(-)
diff --git a/README.md b/README.md
index b969894..141e8b5 100644
--- a/README.md
+++ b/README.md
@@ -83,15 +83,14 @@ Apache Airflow is tested with:
| Python | 3.6, 3.7, 3.8 | 3.6, 3.7, 3.8 | 2.7, 3.5, 3.6, 3.7, 3.8 |
| PostgreSQL | 9.6, 10, 11, 12, 13 | 9.6, 10, 11, 12, 13 | 9.6, 10, 11, 12, 13 |
| MySQL | 5.7, 8 | 5.7, 8 | 5.6, 5.7 |
-| SQLite | latest stable | latest stable | latest stable |
+| SQLite | 3.15.0+ | 3.15.0+ | 3.15.0+ |
| Kubernetes | 1.16.9, 1.17.5, 1.18.6 | 1.16.9, 1.17.5, 1.18.6 | 1.16.9, 1.17.5, 1.18.6 |
-**Note:** MariaDB and MySQL 5.x are unable to or have limitations with
-running multiple schedulers -- please see the "Scheduler" docs.
+**Note:** MySQL 5.x versions are unable to or have limitations with
+running multiple schedulers -- please see the "Scheduler" docs. MariaDB is not tested/recommended.
**Note:** SQLite is used in Airflow tests. Do not use it in production. We recommend
-using the latest stable version of SQLite for local development. Some older versions
-of SQLite might not work well, however anything at or above 3.27.2 should work fine.
+using the latest stable version of SQLite for local development.
## Support for Python versions
diff --git a/airflow/configuration.py b/airflow/configuration.py
index 2b6d363..14b731e 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -30,6 +30,7 @@ from collections import OrderedDict
# Ignored Mypy on configparser because it thinks the configparser module has no _UNSET attribute
from configparser import _UNSET, ConfigParser, NoOptionError, NoSectionError # type: ignore
+from distutils.version import StrictVersion
from json.decoder import JSONDecodeError
from typing import Dict, List, Optional, Tuple, Union
@@ -230,10 +231,15 @@ class AirflowConfigParser(ConfigParser): # pylint: disable=too-many-ancestors
'SequentialExecutor',
)
is_sqlite = "sqlite" in self.get('core', 'sql_alchemy_conn')
- if is_executor_without_sqlite_support and is_sqlite:
- raise AirflowConfigException(
- "error: cannot use sqlite with the {}".format(self.get('core', 'executor'))
- )
+ if is_sqlite and is_executor_without_sqlite_support:
+ raise AirflowConfigException(f"error: cannot use sqlite with the {self.get('core', 'executor')}")
+ if is_sqlite:
+ import sqlite3
+
+ # Some of the features in storing rendered fields require sqlite version >= 3.15.0
+ min_sqlite_version = '3.15.0'
+ if StrictVersion(sqlite3.sqlite_version) < StrictVersion(min_sqlite_version):
+ raise AirflowConfigException(f"error: cannot use sqlite version < {min_sqlite_version}")
if self.has_option('core', 'mp_start_method'):
mp_start_method = self.get('core', 'mp_start_method')
diff --git a/docs/apache-airflow/installation.rst b/docs/apache-airflow/installation.rst
index 590d689..9bcb0e7 100644
--- a/docs/apache-airflow/installation.rst
+++ b/docs/apache-airflow/installation.rst
@@ -21,6 +21,28 @@ Installation
.. contents:: :local:
+
+Prerequisites
+-------------
+
+Airflow is tested with:
+
+* Python: 3.6, 3.7, 3.8
+
+* Databases:
+
+ * PostgreSQL: 9.6, 10, 11, 12, 13
+ * MySQL: 5.7, 8
+ * SQLite: 3.15.0+
+
+* Kubernetes: 1.16.9, 1.17.5, 1.18.6
+
+**Note:** MySQL 5.x versions are unable to or have limitations with
+running multiple schedulers -- please see the "Scheduler" docs. MariaDB is not tested/recommended.
+
+**Note:** SQLite is used in Airflow tests. Do not use it in production. We recommend
+using the latest stable version of SQLite for local development.
+
Getting Airflow
'''''''''''''''