You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2020/06/30 19:17:51 UTC
[airflow] branch v1-10-test updated (834182e -> b7a7508)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from 834182e Fix pickling failure when spawning processes (#8671)
new 5bb80f8 Use fork when test relies on mock.patch in parent process. (#8794)
new e94ddc5 Access function to be pickled as attribute, not method, to avoid error. (#8823)
new b7a7508 Add 3.8 to the test matrices (#8836)
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/ci.yml | 14 ++++++------
BREEZE.rst | 14 ++++++------
Dockerfile.ci | 9 +++++---
README.md | 6 ++---
airflow/utils/dag_processing.py | 3 ++-
breeze | 4 ++--
breeze-complete | 2 +-
...ts-python3.7.txt => requirements-python3.8.txt} | 1 -
requirements/setup-3.6.md5 | 2 +-
requirements/setup-3.7.md5 | 2 +-
requirements/setup-3.8.md5 | 1 +
setup.py | 10 ++++++---
.../operators/test_mssql_to_gcs_operator.py | 7 ++++--
tests/jobs/test_scheduler_job.py | 26 ++++++++++++----------
14 files changed, 57 insertions(+), 44 deletions(-)
copy requirements/{requirements-python3.7.txt => requirements-python3.8.txt} (99%)
create mode 100644 requirements/setup-3.8.md5
[airflow] 02/03: Access function to be pickled as attribute,
not method, to avoid error. (#8823)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e94ddc501794e6b7df9dc38d260328a53320c30d
Author: James Timmins <ja...@astronomer.io>
AuthorDate: Tue May 12 00:55:39 2020 -0700
Access function to be pickled as attribute, not method, to avoid error. (#8823)
* Access function to be pickled as attribute, not method, to avoid error.
* Access type attribute to allow pickling.
* Use getattr instead of type(self) to fix linting error.
(cherry picked from commit 7533378df99a172ad681960243452a84280aa8b2)
---
airflow/utils/dag_processing.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/airflow/utils/dag_processing.py b/airflow/utils/dag_processing.py
index d13b192..3a390e7 100644
--- a/airflow/utils/dag_processing.py
+++ b/airflow/utils/dag_processing.py
@@ -547,7 +547,8 @@ class DagFileProcessorAgent(LoggingMixin, MultiprocessingStartMethodMixin):
self._dag_directory,
self._file_paths,
self._max_runs,
- self._processor_factory,
+ # getattr prevents error while pickling an instance method.
+ getattr(self, "_processor_factory"),
self._processor_timeout,
child_signal_conn,
self._dag_ids,
[airflow] 01/03: Use fork when test relies on mock.patch in parent
process. (#8794)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5bb80f81a807920893843156c5432f3d36a719f5
Author: James Timmins <ja...@astronomer.io>
AuthorDate: Mon May 11 13:42:38 2020 -0700
Use fork when test relies on mock.patch in parent process. (#8794)
* Use 'fork' in test bc 'spawn' breaks mocks.
* Use fork when making process w test_scheduler_executor_overflow.
(cherry picked from commit f410d64de56e3a2c9f3f0bec1f25aa4ad8fc815d)
---
tests/jobs/test_scheduler_job.py | 26 ++++++++++++++------------
1 file changed, 14 insertions(+), 12 deletions(-)
diff --git a/tests/jobs/test_scheduler_job.py b/tests/jobs/test_scheduler_job.py
index d636fd0..48f70a9 100644
--- a/tests/jobs/test_scheduler_job.py
+++ b/tests/jobs/test_scheduler_job.py
@@ -2092,12 +2092,13 @@ class SchedulerJobTest(unittest.TestCase):
# Use a empty file since the above mock will return the
# expected DAGs. Also specify only a single file so that it doesn't
# try to schedule the above DAG repeatedly.
- scheduler = SchedulerJob(num_runs=1,
- executor=executor,
- subdir=os.path.join(settings.DAGS_FOLDER,
- "no_dags.py"))
- scheduler.heartrate = 0
- scheduler.run()
+ with conf_vars({('core', 'mp_start_method'): 'fork'}):
+ scheduler = SchedulerJob(num_runs=1,
+ executor=executor,
+ subdir=os.path.join(settings.DAGS_FOLDER,
+ "no_dags.py"))
+ scheduler.heartrate = 0
+ scheduler.run()
do_schedule()
for ti in tis:
@@ -2309,12 +2310,13 @@ class SchedulerJobTest(unittest.TestCase):
# Use a empty file since the above mock will return the
# expected DAGs. Also specify only a single file so that it doesn't
# try to schedule the above DAG repeatedly.
- scheduler = SchedulerJob(num_runs=1,
- executor=executor,
- subdir=os.path.join(settings.DAGS_FOLDER,
- "no_dags.py"))
- scheduler.heartrate = 0
- scheduler.run()
+ with conf_vars({('core', 'mp_start_method'): 'fork'}):
+ scheduler = SchedulerJob(num_runs=1,
+ executor=executor,
+ subdir=os.path.join(settings.DAGS_FOLDER,
+ "no_dags.py"))
+ scheduler.heartrate = 0
+ scheduler.run()
do_schedule()
with create_session() as session:
[airflow] 03/03: Add 3.8 to the test matrices (#8836)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit b7a75089d679b14742c1a5a8939c6d478708995b
Author: James Timmins <ja...@astronomer.io>
AuthorDate: Fri Jun 5 10:39:28 2020 -0700
Add 3.8 to the test matrices (#8836)
(cherry picked from commit 5918efc86a2217caa641a6ada289eee1c21407f8)
---
.github/workflows/ci.yml | 14 +-
BREEZE.rst | 14 +-
Dockerfile.ci | 9 +-
README.md | 6 +-
breeze | 4 +-
breeze-complete | 2 +-
requirements/requirements-python3.8.txt | 327 +++++++++++++++++++++
requirements/setup-3.6.md5 | 2 +-
requirements/setup-3.7.md5 | 2 +-
requirements/setup-3.8.md5 | 1 +
setup.py | 10 +-
.../operators/test_mssql_to_gcs_operator.py | 7 +-
12 files changed, 368 insertions(+), 30 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index bc86961..d091d2e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -93,7 +93,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
- python-version: [2.7, 3.5, 3.6, 3.7]
+ python-version: [2.7, 3.5, 3.6, 3.7, 3.8]
env:
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
CI_JOB_TYPE: "Prod image"
@@ -179,7 +179,7 @@ ${{ hashFiles('requirements/requirements-python${{matrix.python-version}}.txt')
needs: [static-checks, pyfiles]
strategy:
matrix:
- python-version: [2.7, 3.5, 3.6, 3.7]
+ python-version: [2.7, 3.5, 3.6, 3.7, 3.8]
postgres-version: [9.6, 10]
test-type: [Core, Integration]
fail-fast: false
@@ -211,7 +211,7 @@ ${{ hashFiles('requirements/requirements-python${{matrix.python-version}}.txt')
needs: [static-checks, pyfiles]
strategy:
matrix:
- python-version: [2.7, 3.5, 3.6, 3.7]
+ python-version: [2.7, 3.5, 3.6, 3.7, 3.8]
mysql-version: [5.6, 5.7]
test-type: [Core, Integration]
fail-fast: false
@@ -243,7 +243,7 @@ ${{ hashFiles('requirements/requirements-python${{matrix.python-version}}.txt')
needs: [static-checks, pyfiles]
strategy:
matrix:
- python-version: [2.7, 3.5, 3.6, 3.7]
+ python-version: [2.7, 3.5, 3.6, 3.7, 3.8]
test-type: [Core, Integration]
fail-fast: false
env:
@@ -303,7 +303,7 @@ ${{ hashFiles('requirements/requirements-python${{matrix.python-version}}.txt')
runs-on: ubuntu-latest
strategy:
matrix:
- python-version: [2.7, 3.5, 3.6, 3.7]
+ python-version: [2.7, 3.5, 3.6, 3.7, 3.8]
fail-fast: false
env:
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
@@ -334,7 +334,7 @@ ${{ hashFiles('requirements/requirements-python${{matrix.python-version}}.txt')
if: github.ref == 'refs/heads/master' && github.event_name != 'schedule'
strategy:
matrix:
- python-version: [2.7, 3.5, 3.6, 3.7]
+ python-version: [2.7, 3.5, 3.6, 3.7, 3.8]
env:
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
CI_JOB_TYPE: "Prod image"
@@ -364,7 +364,7 @@ ${{ hashFiles('requirements/requirements-python${{matrix.python-version}}.txt')
github.event_name != 'schedule'
strategy:
matrix:
- python-version: [2.7, 3.5, 3.6, 3.7]
+ python-version: [2.7, 3.5, 3.6, 3.7, 3.8]
env:
PULL_PYTHON_BASE_IMAGES_FROM_CACHE: "false"
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
diff --git a/BREEZE.rst b/BREEZE.rst
index ee0ff01..9d2d719 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -802,7 +802,7 @@ This is the current syntax for `./breeze <./breeze>`_:
One of:
- 2.7 3.5 3.6 3.7
+ 2.7 3.5 3.6 3.7 3.8
-a, --install-airflow-version <INSTALL_AIRFLOW_VERSION>
If specified, installs Airflow directly from PIP released version. This happens at
@@ -922,7 +922,7 @@ This is the current syntax for `./breeze <./breeze>`_:
One of:
- 2.7 3.5 3.6 3.7
+ 2.7 3.5 3.6 3.7 3.8
-I, --production-image
Use production image for entering the environment and builds (not for tests).
@@ -968,7 +968,7 @@ This is the current syntax for `./breeze <./breeze>`_:
One of:
- 2.7 3.5 3.6 3.7
+ 2.7 3.5 3.6 3.7 3.8
-v, --verbose
Show verbose information about executed commands (enabled by default for running test).
@@ -1047,7 +1047,7 @@ This is the current syntax for `./breeze <./breeze>`_:
One of:
- 2.7 3.5 3.6 3.7
+ 2.7 3.5 3.6 3.7 3.8
####################################################################################################
@@ -1136,7 +1136,7 @@ This is the current syntax for `./breeze <./breeze>`_:
One of:
- 2.7 3.5 3.6 3.7
+ 2.7 3.5 3.6 3.7 3.8
-b, --backend <BACKEND>
Backend to use for tests - it determines which database is used.
@@ -1186,7 +1186,7 @@ This is the current syntax for `./breeze <./breeze>`_:
One of:
- 2.7 3.5 3.6 3.7
+ 2.7 3.5 3.6 3.7 3.8
-F, --force-build-images
Forces building of the local docker images. The images are rebuilt
@@ -1351,7 +1351,7 @@ This is the current syntax for `./breeze <./breeze>`_:
One of:
- 2.7 3.5 3.6 3.7
+ 2.7 3.5 3.6 3.7 3.8
****************************************************************************************************
Choose backend to run for Airflow
diff --git a/Dockerfile.ci b/Dockerfile.ci
index f4ba142..c4da431 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -223,11 +223,14 @@ ENV AIRFLOW_CI_BUILD_EPOCH=${AIRFLOW_CI_BUILD_EPOCH}
# We do not have to always reinstall it from the scratch.
# This can be reinstalled from latest master by increasing PIP_DEPENDENCIES_EPOCH_NUMBER.
# And is automatically reinstalled from the scratch every time patch release of python gets released
-RUN pip install \
- "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
- --constraint "https://raw.githubusercontent.com/${AIRFLOW_REPO}/${AIRFLOW_BRANCH}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt" \
+ARG AIRFLOW_PY_VERSION_REQ_FILE="https://raw.githubusercontent.com/${AIRFLOW_REPO}/${AIRFLOW_BRANCH}/requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
+ENV AIRFLOW_PY_VERSION_REQ_FILE_STATUS="$(curl -s --head -w %{http_code} ${AIRFLOW_PY_VERSION_REQ_FILE} -o /dev/null)"
+RUN echo $AIRFLOW_PY_VERSION_REQ_FILE_STATUS;
+
+RUN pip install "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
&& pip uninstall --yes apache-airflow;
+
# Link dumb-init for backwards compatibility (so that older images also work)
RUN ln -sf /usr/bin/dumb-init /usr/local/bin/dumb-init
diff --git a/README.md b/README.md
index 0c0aa7c..d5883bf 100644
--- a/README.md
+++ b/README.md
@@ -61,7 +61,7 @@ Apache Airflow is tested with:
### Master version (2.0.0dev)
-* Python versions: 3.6, 3.7
+* Python versions: 3.6, 3.7, 3.8
* Postgres DB: 9.6, 10
* MySQL DB: 5.7
* Sqlite - latest stable (it is used mainly for development purpose)
@@ -69,7 +69,7 @@ Apache Airflow is tested with:
### Stable version (1.10.10)
-* Python versions: 2.7, 3.5, 3.6, 3.7
+* Python versions: 2.7, 3.5, 3.6, 3.7, 3.8
* Postgres DB: 9.6, 10
* MySQL DB: 5.6, 5.7
* Sqlite - latest stable (it is used mainly for development purpose)
@@ -100,7 +100,7 @@ produce unusable Airflow installation.
In order to have repeatable installation, however, starting from **Airflow 1.10.10** we also keep a set of
"known-to-be-working" requirement files in the `requirements` folder. Those "known-to-be-working"
-requirements are per major/minor python version (3.6/3.7). You can use them as constraint files
+requirements are per major/minor python version (3.6/3.7/3.8). You can use them as constraint files
when installing Airflow from PyPI. Note that you have to specify correct Airflow version and python versions
in the URL.
diff --git a/breeze b/breeze
index 16826bc..4321c49 100755
--- a/breeze
+++ b/breeze
@@ -68,7 +68,7 @@ function setup_default_breeze_variables() {
# We have different versions of images depending on the python version used. We keep up with the
# Latest patch-level changes in Python (this is done automatically during CI builds) so we have
# To only take into account MAJOR and MINOR version of python. This variable keeps the major/minor
- # version of python in X.Y format (2.7, 3.5, 3.6, 3.7 etc).
+ # version of python in X.Y format (2.7, 3.5, 3.6, 3.7, 3.8 etc).
export PYTHON_MAJOR_MINOR_VERSION="${PYTHON_MAJOR_MINOR_VERSION:=$(read_from_file PYTHON_MAJOR_MINOR_VERSION)}"
# When we generate documentation for README files, we want to force the width of terminal so that
@@ -197,7 +197,7 @@ function initialize_virtualenv() {
if [[ ${SYSTEM} == "Darwin" ]]; then
echo " brew install sqlite mysql postgresql openssl"
else
- echo " sudo apt install build-essentials python3.6-dev python3.7-dev python-dev openssl \\"
+ echo " sudo apt install build-essentials python3.6-dev python3.7-dev python3.8-dev python-dev openssl \\"
echo " sqlite sqlite-dev default-libmysqlclient-dev libmysqld-dev postgresql"
fi
echo
diff --git a/breeze-complete b/breeze-complete
index a3cc1eb..d9227a5 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -17,7 +17,7 @@
# specific language governing permissions and limitations
# under the License.
-_BREEZE_ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS="2.7 3.5 3.6 3.7"
+_BREEZE_ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS="2.7 3.5 3.6 3.7 3.8"
_BREEZE_ALLOWED_BACKENDS="sqlite mysql postgres"
_BREEZE_ALLOWED_INTEGRATIONS="cassandra kerberos mongo openldap rabbitmq redis"
_BREEZE_ALLOWED_KUBERNETES_MODES="image git"
diff --git a/requirements/requirements-python3.8.txt b/requirements/requirements-python3.8.txt
new file mode 100644
index 0000000..4e24246
--- /dev/null
+++ b/requirements/requirements-python3.8.txt
@@ -0,0 +1,327 @@
+# Editable install with no version control (apache-airflow==1.10.10)
+Babel==2.8.0
+Flask-Admin==1.5.4
+Flask-AppBuilder==2.3.4
+Flask-Babel==1.0.0
+Flask-Bcrypt==0.7.1
+Flask-Caching==1.3.3
+Flask-JWT-Extended==3.24.1
+Flask-Login==0.4.1
+Flask-OpenID==1.2.5
+Flask-SQLAlchemy==2.4.3
+Flask-WTF==0.14.3
+Flask==1.1.2
+JPype1==0.7.1
+JayDeBeApi==1.2.3
+Jinja2==2.10.3
+Mako==1.1.3
+Markdown==2.6.11
+MarkupSafe==1.1.1
+PyHive==0.6.2
+PyJWT==1.7.1
+PyNaCl==1.4.0
+PySmbClient==0.1.5
+PyYAML==5.3.1
+Pygments==2.6.1
+SQLAlchemy-JSONField==0.9.0
+SQLAlchemy-Utils==0.36.6
+SQLAlchemy==1.3.18
+Sphinx==3.1.1
+Unidecode==1.1.1
+WTForms==2.3.1
+Werkzeug==0.16.1
+adal==1.2.4
+alabaster==0.7.12
+alembic==1.4.2
+amqp==2.6.0
+analytics-python==1.2.9
+ansiwrap==0.8.4
+apipkg==1.5
+apispec==1.3.3
+appdirs==1.4.4
+argcomplete==1.11.1
+asn1crypto==1.3.0
+astroid==2.4.2
+async-generator==1.10
+atlasclient==1.0.0
+attrs==19.3.0
+aws-sam-translator==1.25.0
+aws-xray-sdk==2.6.0
+azure-common==1.1.25
+azure-cosmos==3.1.2
+azure-datalake-store==0.0.48
+azure-mgmt-containerinstance==1.5.0
+azure-mgmt-resource==10.0.0
+azure-nspkg==3.0.2
+azure-storage-blob==2.1.0
+azure-storage-common==2.1.0
+azure-storage==0.36.0
+backcall==0.2.0
+bcrypt==3.1.7
+beautifulsoup4==4.7.1
+billiard==3.6.3.0
+black==19.10b0
+blinker==1.4
+boto3==1.14.13
+boto==2.49.0
+botocore==1.17.13
+cached-property==1.5.1
+cachetools==4.1.1
+cassandra-driver==3.20.2
+cattrs==1.0.0
+celery==4.4.6
+certifi==2020.6.20
+cffi==1.14.0
+cfgv==3.1.0
+cfn-lint==0.33.2
+cgroupspy==0.1.6
+chardet==3.0.4
+click==6.7
+cloudant==0.5.10
+colorama==0.4.3
+colorlog==4.0.2
+configparser==3.5.3
+coverage==5.1
+croniter==0.3.34
+cryptography==2.9.2
+cx-Oracle==8.0.0
+datadog==0.37.1
+decorator==4.4.2
+defusedxml==0.6.0
+dill==0.3.2
+distlib==0.3.1
+dnspython==1.16.0
+docker-pycreds==0.4.0
+docker==3.7.3
+docopt==0.6.2
+docutils==0.16
+ecdsa==0.15
+elasticsearch-dsl==5.4.0
+elasticsearch==5.5.3
+email-validator==1.1.1
+entrypoints==0.3
+execnet==1.7.1
+fastavro==0.23.5
+filelock==3.0.12
+flake8-colors==0.1.6
+flake8==3.8.3
+flaky==3.6.1
+flask-swagger==0.2.13
+flower==0.9.4
+freezegun==0.3.15
+fsspec==0.7.4
+funcsigs==1.0.2
+future-fstrings==1.2.0
+future==0.18.2
+gcsfs==0.6.2
+google-api-core==1.21.0
+google-api-python-client==1.9.3
+google-auth-httplib2==0.0.3
+google-auth-oauthlib==0.4.1
+google-auth==1.18.0
+google-cloud-bigquery==1.25.0
+google-cloud-bigtable==1.2.1
+google-cloud-container==1.0.1
+google-cloud-core==1.3.0
+google-cloud-dlp==1.0.0
+google-cloud-language==1.3.0
+google-cloud-secret-manager==1.0.0
+google-cloud-spanner==1.17.1
+google-cloud-speech==1.3.2
+google-cloud-storage==1.29.0
+google-cloud-texttospeech==1.0.1
+google-cloud-translate==2.0.1
+google-cloud-videointelligence==1.15.0
+google-cloud-vision==1.0.0
+google-resumable-media==0.5.1
+googleapis-common-protos==1.52.0
+graphviz==0.14
+grpc-google-iam-v1==0.12.3
+grpcio-gcp==0.2.2
+grpcio==1.30.0
+gunicorn==19.10.0
+hdfs==2.5.8
+hmsclient==0.1.1
+httplib2==0.18.1
+humanize==0.5.1
+hvac==0.10.4
+identify==1.4.20
+idna==2.10
+ijson==2.6.1
+imagesize==1.2.0
+importlib-metadata==1.7.0
+inflection==0.5.0
+ipdb==0.13.3
+ipython-genutils==0.2.0
+ipython==7.16.1
+iso8601==0.1.12
+isodate==0.6.0
+itsdangerous==1.1.0
+jedi==0.17.1
+jira==2.0.0
+jmespath==0.10.0
+json-merge-patch==0.2
+jsondiff==1.1.2
+jsonpatch==1.26
+jsonpickle==1.4.1
+jsonpointer==2.0
+jsonschema==3.2.0
+junit-xml==1.9
+jupyter-client==6.1.5
+jupyter-core==4.6.3
+kombu==4.6.11
+kubernetes==11.0.0
+lazy-object-proxy==1.5.0
+ldap3==2.7
+lockfile==0.12.2
+marshmallow-enum==1.5.1
+marshmallow-sqlalchemy==0.23.1
+marshmallow==2.21.0
+mccabe==0.6.1
+mock==4.0.2
+mongomock==3.19.0
+more-itertools==8.4.0
+moto==1.3.14
+msrest==0.6.17
+msrestazure==0.6.4
+multi-key-dict==2.0.3
+mypy-extensions==0.4.3
+mypy==0.720
+mysqlclient==1.3.14
+natsort==7.0.1
+nbclient==0.4.0
+nbformat==5.0.7
+nest-asyncio==1.3.3
+networkx==2.4
+nodeenv==1.4.0
+nteract-scrapbook==0.4.1
+ntlm-auth==1.5.0
+numpy==1.19.0
+oauthlib==3.1.0
+oscrypto==1.2.0
+packaging==20.4
+pandas-gbq==0.13.2
+pandas==1.0.5
+papermill==2.1.2
+parameterized==0.7.4
+paramiko==2.7.1
+parso==0.7.0
+pathspec==0.8.0
+pbr==5.4.5
+pendulum==1.4.4
+pexpect==4.8.0
+pickleshare==0.7.5
+pinotdb==0.1.1
+pluggy==0.13.1
+pre-commit==2.5.1
+presto-python-client==0.7.0
+prison==0.1.3
+prompt-toolkit==3.0.5
+protobuf==3.12.2
+psutil==5.7.0
+psycopg2-binary==2.8.5
+ptyprocess==0.6.0
+py==1.9.0
+pyOpenSSL==19.1.0
+pyarrow==0.17.1
+pyasn1-modules==0.2.8
+pyasn1==0.4.8
+pycodestyle==2.6.0
+pycparser==2.20
+pycryptodomex==3.9.8
+pydata-google-auth==1.1.0
+pydruid==0.5.8
+pyflakes==2.2.0
+pykerberos==1.2.1
+pymongo==3.10.1
+pyparsing==2.4.7
+pyrsistent==0.16.0
+pysftp==0.2.9
+pytest-cov==2.10.0
+pytest-forked==1.2.0
+pytest-instafail==0.4.2
+pytest-rerunfailures==9.0
+pytest-timeout==1.4.1
+pytest-xdist==1.32.0
+pytest==5.4.3
+python-daemon==2.1.2
+python-dateutil==2.8.1
+python-editor==1.0.4
+python-http-client==3.2.7
+python-jenkins==1.7.0
+python-jose==3.1.0
+python-nvd3==0.15.0
+python-slugify==4.0.1
+python3-openid==3.2.0
+pytz==2020.1
+pytzdata==2019.3
+pywinrm==0.4.1
+pyzmq==19.0.1
+qds-sdk==1.16.0
+redis==3.5.3
+regex==2020.6.8
+requests-futures==0.9.4
+requests-kerberos==0.12.0
+requests-mock==1.8.0
+requests-ntlm==1.1.0
+requests-oauthlib==1.3.0
+requests-toolbelt==0.9.1
+requests==2.24.0
+responses==0.10.15
+rsa==4.6
+s3transfer==0.3.3
+sasl==0.2.1
+sendgrid==5.6.0
+sentinels==1.0.0
+sentry-sdk==0.15.1
+setproctitle==1.1.10
+six==1.15.0
+slackclient==1.3.2
+snowballstemmer==2.0.0
+snowflake-connector-python==2.2.8
+snowflake-sqlalchemy==1.2.3
+soupsieve==2.0.1
+sphinx-argparse==0.2.5
+sphinx-autoapi==1.0.0
+sphinx-copybutton==0.2.12
+sphinx-jinja==1.1.1
+sphinx-rtd-theme==0.5.0
+sphinxcontrib-applehelp==1.0.2
+sphinxcontrib-devhelp==1.0.2
+sphinxcontrib-dotnetdomain==0.4
+sphinxcontrib-golangdomain==0.2.0.dev0
+sphinxcontrib-htmlhelp==1.0.3
+sphinxcontrib-httpdomain==1.7.0
+sphinxcontrib-jsmath==1.0.1
+sphinxcontrib-qthelp==1.0.3
+sphinxcontrib-serializinghtml==1.1.4
+sshpubkeys==3.1.0
+sshtunnel==0.1.5
+tabulate==0.8.7
+tenacity==4.12.0
+termcolor==1.1.0
+text-unidecode==1.3
+textwrap3==0.9.2
+thrift-sasl==0.4.2
+thrift==0.13.0
+toml==0.10.1
+tornado==5.1.1
+tqdm==4.47.0
+traitlets==4.3.3
+typed-ast==1.4.1
+typing-extensions==3.7.4.2
+tzlocal==1.5.1
+unicodecsv==0.14.1
+uritemplate==3.0.1
+urllib3==1.25.9
+vertica-python==0.10.4
+vine==1.3.0
+virtualenv==20.0.25
+wcwidth==0.2.5
+websocket-client==0.57.0
+wrapt==1.12.1
+xmltodict==0.12.0
+yamllint==1.23.0
+zdesk==2.7.1
+zipp==3.1.0
+zope.deprecation==4.4.0
diff --git a/requirements/setup-3.6.md5 b/requirements/setup-3.6.md5
index 02fe09a..7cb2b1c 100644
--- a/requirements/setup-3.6.md5
+++ b/requirements/setup-3.6.md5
@@ -1 +1 @@
-f86f24b093d55ecc490bb77e74b332eb /opt/airflow/setup.py
+9688bea64971365f2ab7771658b1fc80 /opt/airflow/setup.py
diff --git a/requirements/setup-3.7.md5 b/requirements/setup-3.7.md5
index 02fe09a..7cb2b1c 100644
--- a/requirements/setup-3.7.md5
+++ b/requirements/setup-3.7.md5
@@ -1 +1 @@
-f86f24b093d55ecc490bb77e74b332eb /opt/airflow/setup.py
+9688bea64971365f2ab7771658b1fc80 /opt/airflow/setup.py
diff --git a/requirements/setup-3.8.md5 b/requirements/setup-3.8.md5
new file mode 100644
index 0000000..7cb2b1c
--- /dev/null
+++ b/requirements/setup-3.8.md5
@@ -0,0 +1 @@
+9688bea64971365f2ab7771658b1fc80 /opt/airflow/setup.py
diff --git a/setup.py b/setup.py
index 3c6defc..fc14c3c 100644
--- a/setup.py
+++ b/setup.py
@@ -36,6 +36,7 @@ logger = logging.getLogger(__name__)
version = imp.load_source('airflow.version', os.path.join('airflow', 'version.py')).version # type: ignore
PY3 = sys.version_info[0] == 3
+PY38 = PY3 and sys.version_info[1] >= 8
my_dir = dirname(__file__)
@@ -184,7 +185,7 @@ aws = [
'boto3~=1.10',
]
azure_blob_storage = [
- 'azure-storage>=0.34.0,<0.37.0',
+ 'azure-storage>=0.34.0, <0.37.0',
'azure-storage-blob<12.0',
]
azure_container_instances = [
@@ -456,8 +457,10 @@ devel_all = (all_dbs + atlas + aws +
# Snakebite is not Python 3 compatible :'(
if PY3:
- devel_all = [package for package in devel_all if package not in
- ['snakebite>=2.7.8', 'snakebite[kerberos]>=2.7.8']]
+ package_to_excludes = ['snakebite>=2.7.8', 'snakebite[kerberos]>=2.7.8']
+ if PY38:
+ package_to_excludes.extend(['pymssql~=2.1.1'])
+ devel_all = [package for package in devel_all if package not in package_to_excludes]
devel_ci = devel_all
else:
devel_ci = devel_all + ['unittest2']
@@ -642,6 +645,7 @@ def do_setup():
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
+ 'Programming Language :: Python :: 3.8',
'Topic :: System :: Monitoring',
],
author='Apache Software Foundation',
diff --git a/tests/contrib/operators/test_mssql_to_gcs_operator.py b/tests/contrib/operators/test_mssql_to_gcs_operator.py
index 19ef774..27d7bad 100644
--- a/tests/contrib/operators/test_mssql_to_gcs_operator.py
+++ b/tests/contrib/operators/test_mssql_to_gcs_operator.py
@@ -20,11 +20,13 @@
import sys
import unittest
-from airflow.contrib.operators.mssql_to_gcs import \
- MsSqlToGoogleCloudStorageOperator
from tests.compat import mock
PY3 = sys.version_info[0] == 3
+PY38 = PY3 and sys.version_info[1] >= 8
+
+if not PY38:
+ from airflow.contrib.operators.mssql_to_gcs import MsSqlToGoogleCloudStorageOperator
TASK_ID = 'test-mssql-to-gcs'
MSSQL_CONN_ID = 'mssql_conn_test'
@@ -54,6 +56,7 @@ SCHEMA_JSON = [
]
+@unittest.skipIf(PY38, "Mssql package not available when Python >= 3.8.")
class MsSqlToGoogleCloudStorageOperatorTest(unittest.TestCase):
def test_init(self):