You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/04 13:35:30 UTC
[GitHub] [airflow] uranusjr opened a new issue #18705: Do not fetch providers from PyPI in CI runs
uranusjr opened a new issue #18705:
URL: https://github.com/apache/airflow/issues/18705
I think this is the problem hit by #18684. I am not entirely familiar with the CI script, but this is my best guess reading the CI logs. https://github.com/apache/airflow/pull/18684/checks?check_run_id=3788521722
cc @eladkal @potiuk
## The problem
Currently the *Installing all packages with eager upgrade* job installs `airflow[devel_ci]` with `--upgrade-strategy=eager`, which installs the latest versions of all Airflow providers. However, those providers are specified by name, so the "latest version" is defined as the version on PyPI, with whatever the dependencies defined in those released versions. This is usually not a problem, but if we change any of the providers’ dependencies to a new range mutually exclusive with the released provider version, this eager-upgrade job would fail because it’s impossible to both satisfy the new version range included as a part of the `devel_ci` extra, and the old version range included by the provider release from PyPI.
This becomes an issue in e.g. #18684, where a version range (`sshtunnel>=0.1.4,<0.2`) is changed to an entirely non-overlapping one (`sshtunnel>=0.3.2,<0.5`).
## Proposed solution
Before running the *eager upgrade* job, CI should first build all local providers into installable distributions (perferrably wheels), and constrain the `pip install` command like this:
```
# provider-constraints.txt
apache-airflow-providers-airbyte @ ./provider-wheels/apache_airflow_providers_airbyte-...whl
# ... all the wheel paths
```
```shell
pip install \
--upgrade \
--upgrade-strategy eager \
'.[devel_ci]' \
'lazy-object-proxy<1.5.0' \
'pyjwt<2.0.0' \
'dill<0.3.3' \
'certifi<2021.0.0' \
-c provider-constraints.txt # This line is new.
```
So providers are always fetched locally instead of from PyPI, thus always include the most up-to-date dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933653013
Actually I believe ` --find-links="file:///docker-context-files,https://pypi.python.org/simple" should be the right one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933651000
I think it's ALMOST good advise. I believe `--constraints` and `--upgrade` are conflicting and you cannot have them both. But this should be solvable by using `` --find-links="file:///docker-context-files"` which I am already using in "install_from_docker_context_files" to do exactly this - install providers prepared locally from sources (I am using it to build production docker image).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933701818
This is the diff of deps updated in 3.6 at the time of the failed build. It is likely one of those is the reason:
```
> Rx==3.2.0
103c94
< boto3==1.18.48
---
> boto3==1.18.50
105c96
< botocore==1.21.48
---
> botocore==1.21.50
118c109
< click-didyoumean==0.0.3
---
> click-didyoumean==0.3.0
177c168
< google-cloud-appengine-logging==0.1.4
---
> google-cloud-appengine-logging==0.1.5
180,182c171,173
< google-cloud-bigquery-datatransfer==3.3.2
< google-cloud-bigquery-storage==2.8.0
< google-cloud-bigquery==2.27.0
---
> google-cloud-bigquery-datatransfer==3.3.3
> google-cloud-bigquery-storage==2.9.0
> google-cloud-bigquery==2.27.1
187c178
< google-cloud-datacatalog==3.4.1
---
> google-cloud-datacatalog==3.4.2
190c181
< google-cloud-kms==2.6.0
---
> google-cloud-kms==2.6.1
194,195c185,186
< google-cloud-monitoring==2.5.0
< google-cloud-os-login==2.3.1
---
> google-cloud-monitoring==2.5.1
> google-cloud-os-login==2.3.2
197c188
< google-cloud-redis==2.2.2
---
> google-cloud-redis==2.2.3
207c198
< google-cloud-workflows==1.2.1
---
> google-cloud-workflows==1.2.2
212c203
< greenlet==1.1.1
---
> greenlet==1.1.2
215c206
< grpcio==1.40.0
---
> grpcio==1.41.0
233a225
> influxdb-client==1.21.0
249c241
< jupyter-client==7.0.3
---
> jupyter-client==7.0.5
322c314
< proto-plus==1.19.0
---
> proto-plus==1.19.1
343c335
< pyexasol==0.20.0
---
> pyexasol==0.21.1
397c389
< sentry-sdk==1.4.1
---
> sentry-sdk==1.4.3
406c398
< snowflake-connector-python==2.6.1
---
> snowflake-connector-python==2.6.2
468c460
< yandexcloud==0.106.1
---
> yandexcloud==0.107.0
472c464
< zipp==3.5.0
---
> zipp==3.6.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933693650
OK. It was not the InfluxDB Hook. False trail.
It broke "suddenly" 5 days ago:
* This one was ok: https://github.com/apache/airflow/actions/runs/1287338265, Wed, 29 Sep 2021 14:53:44 GMT. It was already after HookDB was merged.
* This one failed 2hrs later: https://github.com/apache/airflow/actions/runs/1287708682, Wed, 29 Sep 2021 16:36:23 GMT
Something was released between the two that broke the eager upgrade. There was no change in either setup.py or setup.cfg between the two builds (I looked at all commits) so this rules out any of our changes to cause it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933746150
good news @eladkal @uranusjr . I found the culprit:
google-ads released 14.0.1 version Sep 29: https://pypi.org/project/google-ads/14.0.1/
Guess what between the version 14.0.0 they bumped minimum google-api-core from 1.31.2 to 2.0.0
However other deps of ours have < 2.0.0
Python 3.6 is not affected becasue there we have version 12.0.0 of google-ads librarty (version 14 is Python 3.7+)
```
google-api-python-client==1.12.8
- google-api-core [required: >=1.21.0,<2dev, installed: 1.31.3]
```
14.0.0: install_requires:
```
install_requires = [
"google-auth-oauthlib >= 0.3.0, < 1.0.0",
# NOTE: Require google-api-core > 1.31.2 until
# https://github.com/googleapis/google-cloud-python/issues/10566
# is resolved. Once resolved, require google-api-core >=2.x.x
"google-api-core >= 1.31.2, < 3.0.0", <- !!!!!!!!!!!!!!!!!!!!!!!!!! LOOK HERE !!!!!!!!!!!!!!!!!!!!!!!!!
"googleapis-common-protos >= 1.5.8, < 2.0.0",
"grpcio >= 1.38.1, < 2.0.0",
"proto-plus == 1.18.1",
"PyYAML >= 5.1, < 6.0",
"setuptools >= 40.3.0",
"nox == 2020.12.31",
]
````
14.0.1: install requires:
```
install_requires = [
"google-auth-oauthlib >= 0.3.0, < 1.0.0",
"google-api-core >= 2.0.1, < 3.0.0", <- !!!!!!!!!!!!!!!!!!!!!! LOOOK HERE !!!!!!!!!!!!!!!!!!!!!!!!!
"googleapis-common-protos >= 1.5.8, < 2.0.0",
"grpcio >= 1.38.1, < 2.0.0",
"proto-plus == 1.18.1",
"PyYAML >= 5.1, < 6.0",
"setuptools >= 40.3.0",
"nox >= 2020.12.31, < 2022.6",
"protobuf >= 3.12.0, < 3.18.0"
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933651000
I think it's ALMOST good advise. I believe `--constraints` and `--upgrade` are conflicting and you cannot have them both. But this should be solvable by using ` --find-links="file:///docker-context-files"` which I am already using in "install_from_docker_context_files" to do exactly this - install providers prepared locally from sources (I am using it to build production docker image).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933655141
Ah no - seems single find-links is enough :), index is also searched by default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933653013
Actually I believe ` --find-links="file:///docker-context-files,https://pypi.python.org/simple)" should be the right one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933665521
Actually after closer look, this is not it. The current script is ok for that. if you look closer it uses "INSTALL_PROVIDERS_FROM_SOURCES: true" - which means that it does not even attempt to look at providers from PyPI. It takes the dependencies from local `setup.py`, so it **should** work in general, regardless what providers are released in PyPI:
```
include_provider_namespace_packages_when_installing_from_sources()
if os.getenv(INSTALL_PROVIDERS_FROM_SOURCES) == 'true':
print("Installing providers from sources. Skip adding providers as dependencies")
else:
add_all_provider_packages()
```
So I continue looking for a reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933653013
~Actually I believe ` --find-links="file:///docker-context-files,https://pypi.python.org/simple" should be the right one.~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933701818
This is the diff of deps updated in 3.6 at the time of the failed build. It is likely one of those is the reason:
```
< # This constraints file was automatically generated on 2021-09-27T11:33:12Z
---
> # This constraints file was automatically generated on 2021-09-18T22:41:56Z
57c57
< anyio==3.3.2
---
> anyio==3.3.1
103c103
< boto3==1.18.48
---
> boto3==1.17.112
105c105
< botocore==1.21.48
---
> botocore==1.20.112
141c141
< distlib==0.3.3
---
> distlib==0.3.2
150c150
< elasticsearch==7.15.0
---
> elasticsearch==7.14.1
156,157c156,157
< fastavro==1.4.5
< filelock==3.1.0
---
> fastavro==1.4.4
> filelock==3.0.12
164c164
< fsspec==2021.9.0
---
> fsspec==2021.8.1
166c166
< gcsfs==2021.9.0
---
> gcsfs==2021.8.1
172c172
< google-api-core==1.31.3
---
> google-api-core==1.31.2
182c182
< google-cloud-bigquery==2.27.0
---
> google-cloud-bigquery==2.26.0
184c184
< google-cloud-build==3.5.1
---
> google-cloud-build==3.5.0
202c202
< google-cloud-tasks==2.5.2
---
> google-cloud-tasks==2.5.1
208,209c208,209
< google-crc32c==1.2.0
< google-resumable-media==2.0.3
---
> google-crc32c==1.1.2
> google-resumable-media==2.0.2
224,225c224,225
< hvac==0.11.2
< identify==2.2.15
---
> hvac==0.11.0
> identify==2.2.14
258c258
< libcst==0.3.21
---
> libcst==0.3.20
269c269
< moto==2.2.8
---
> moto==2.2.7
283c283
< neo4j==4.3.5
---
> neo4j==4.3.4
312c312
< platformdirs==2.4.0
---
> platformdirs==2.3.0
323c323
< protobuf==3.17.3
---
> protobuf==3.18.0
337,338c337,338
< pycryptodome==3.10.4
< pycryptodomex==3.10.4
---
> pycryptodome==3.10.1
> pycryptodomex==3.10.1
354c354
< pyspnego==0.2.0
---
> pyspnego==0.1.6
362c362
< pytest-xdist==2.4.0
---
> pytest-xdist==2.3.0
366c366
< python-http-client==3.3.3
---
> python-http-client==3.3.2
380c380
< regex==2021.9.24
---
> regex==2021.8.28
389c389
< rich==10.11.0
---
> rich==10.10.0
391c391
< s3transfer==0.5.0
---
> s3transfer==0.4.2
395c395
< sendgrid==6.8.2
---
> sendgrid==6.8.1
397c397
< sentry-sdk==1.4.1
---
> sentry-sdk==1.3.1
401,402c401,402
< slack-sdk==3.11.2
< smbprotocol==1.7.0
---
> slack-sdk==3.11.1
> smbprotocol==1.6.2
421c421
< sphinxcontrib-httpdomain==1.8.0
---
> sphinxcontrib-httpdomain==1.7.0
442c442
< thrift==0.15.0
---
> thrift==0.13.0
447c447
< tqdm==4.62.3
---
> tqdm==4.62.2
457c457
< urllib3==1.26.7
---
> urllib3==1.26.6
460c460
< virtualenv==20.8.1
---
> virtualenv==20.8.0
468c468
< yandexcloud==0.106.1
---
> yandexcloud==0.105.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933731507
All right. I narrowed it down:
I generated diff of deps in 3.6 in the "ok" build vs. "NOK" one (note both of them succeded in python 3.6 so there is no guarantee those were the culprit ones
```
< proto-plus==1.19.0
---
> proto-plus==1.19.1
464c464
< zipp==3.5.1
---
> zipp==3.6.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933752716
Confirmed. I just built `eager-upgraded` 3.7 image by limiting google-ads to < 14.0.1. PR is coming.
@uranusjr - we REALLY have to find a way for PIP to help us with those kind of problems. I have some ideas (based on my investigation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933734104
OK. I am a bit closer. For the "good build" "3.6" vs. "3.7" is below - so it's either one of the two above or below I think:
```
16c16
< GitPython==3.1.18
---
> GitPython==3.1.24
57d56
< async-generator==1.10
101c100
< cattrs==1.0.0
---
> cattrs==1.5.0
119d117
< contextvars==2.4
126,127c124
< dask==2021.3.0
< dataclasses==0.8
---
> dask==2021.6.0
162c159
< google-ads==12.0.0
---
> google-ads==14.0.0
217d213
< idna-ssl==1.1.0
221d216
< immutables==0.16
229c224
< ipython==7.16.1
---
> ipython==7.28.0
250a246
> locket==0.2.1
255a252
> matplotlib-inline==0.1.3
280c277
< numpy==1.19.5
---
> numpy==1.20.3
285c282
< orjson==3.6.1
---
> orjson==3.6.3
290c287
< pandas==1.1.5
---
> pandas==1.3.3
294a292
> partd==1.2.0
299d296
< pep562==1.1
314c311
< proto-plus==1.19.0
---
> proto-plus==1.18.1
440c437
< traitlets==4.3.3
---
> traitlets==5.1.0
445d441
< typing==3.7.4.3
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933727612
Those packages were released 29th September:
* greenlet
* jupyter-clent
* proto-plus
* sentry-sdk
* zipp
I looked at the released versions - so far this is a dead trail - (but I could have missed something ). .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933674992
Though, it's strange as we alredy had same pandas there. BTW. THe problem is exactly the same in regular `main` build so we do not need to look to far when exactly it broke
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933673011
Quite likely it's pandas: `influxdb = ['pandas>=0.17.1, <2.0', 'influxdb-client>=1.19.0']`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18705:
URL: https://github.com/apache/airflow/issues/18705#issuecomment-933672104
Luckily we can trace quite easily WHEN things broke:
Currently the real problem we have (and you can actually see that in #18684 that while Python 3.6 builds just fine with eager-upgrade, but 3.7, 3.8 and 3.9 are not (this is what I also noticed yesterday) and I am pretty sure this is some "3rd-party" dependency changed after this one: https://github.com/apache/airflow/commit/45e2970c0ac0f509d2ea56c6f08546d3d65d2935
At the time this commit was pushed automatically (7 days ago) the eager upgrade worked fine for all images (otherwise it would not have been pushed). SOMETHING broke after this commit was merged: https://github.com/apache/airflow/commit/fa1a088d2c02ac85a8e01f91ff2ae80ad0def212
It's either a 3rd-party change ... or https://github.com/apache/airflow/pull/17068 (the latter is most probable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #18705: Do not fetch providers from PyPI in CI runs
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #18705:
URL: https://github.com/apache/airflow/issues/18705
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org