You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/24 10:07:15 UTC
[GitHub] [airflow] seongwon77 opened a new issue, #25929: Google provider package should have `google-ads<18.1.0` constraint
seongwon77 opened a new issue, #25929:
URL: https://github.com/apache/airflow/issues/25929
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
alembic==1.8.1
anyio==3.6.1
apache-airflow==2.3.4
apache-airflow-providers-common-sql==1.1.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.3.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-sqlite==3.2.0
apispec==3.3.2
argcomplete==2.0.0
attrs==22.1.0
Babel==2.10.3
blinker==1.5
cachelib==0.9.0
cachetools==5.2.0
cattrs==22.1.0
certifi==2022.6.15
cffi==1.15.1
charset-normalizer==2.1.1
click==8.1.3
clickclick==20.10.2
colorama==0.4.5
colorlog==4.8.0
commonmark==0.9.1
connexion==2.14.1
cron-descriptor==1.2.31
croniter==1.3.5
cryptography==37.0.4
db-dtypes==1.0.3
Deprecated==1.2.13
dill==0.3.5.1
dnspython==2.2.1
docutils==0.19
email-validator==1.2.1
exceptiongroup==1.0.0rc8
Flask==2.2.2
Flask-AppBuilder==4.1.3
Flask-Babel==2.0.0
Flask-Caching==2.0.1
Flask-JWT-Extended==4.4.4
Flask-Login==0.6.2
Flask-Session==0.4.0
Flask-SQLAlchemy==2.5.1
Flask-WTF==0.15.1
future==0.18.2
google-ads==18.1.0
google-api-core==2.8.2
google-api-python-client==1.12.11
google-auth==2.11.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.5.2
google-cloud-aiplatform==1.13.1
google-cloud-appengine-logging==1.1.4
google-cloud-audit-log==0.2.4
google-cloud-automl==2.8.1
google-cloud-bigquery==2.34.3
google-cloud-bigquery-datatransfer==3.7.1
google-cloud-bigquery-storage==2.14.2
google-cloud-bigtable==1.7.1
google-cloud-build==3.9.1
google-cloud-container==2.11.2
google-cloud-core==2.3.2
google-cloud-datacatalog==3.9.1
google-cloud-dataform==0.2.1
google-cloud-dataplex==1.1.1
google-cloud-dataproc==5.0.1
google-cloud-dataproc-metastore==1.6.1
google-cloud-dlp==1.0.1
google-cloud-kms==2.12.1
google-cloud-language==1.3.1
google-cloud-logging==3.2.2
google-cloud-memcache==1.4.2
google-cloud-monitoring==2.11.1
google-cloud-orchestration-airflow==1.4.2
google-cloud-os-login==2.7.2
google-cloud-pubsub==2.13.6
google-cloud-redis==2.9.1
google-cloud-resource-manager==1.6.1
google-cloud-secret-manager==1.0.1
google-cloud-spanner==1.19.2
google-cloud-speech==1.3.3
google-cloud-storage==1.44.0
google-cloud-tasks==2.10.2
google-cloud-texttospeech==1.0.2
google-cloud-translate==1.7.1
google-cloud-videointelligence==1.16.2
google-cloud-vision==1.0.1
google-cloud-workflows==1.7.2
google-crc32c==1.3.0
google-resumable-media==2.3.3
googleapis-common-protos==1.56.4
graphviz==0.20.1
grpc-google-iam-v1==0.12.4
grpcio==1.47.0
grpcio-gcp==0.2.2
grpcio-status==1.47.0
gunicorn==20.1.0
h11==0.12.0
httpcore==0.15.0
httplib2==0.20.4
httpx==0.23.0
idna==3.3
importlib-metadata==4.12.0
importlib-resources==5.9.0
inflection==0.5.1
itsdangerous==2.1.2
Jinja2==3.1.2
json-merge-patch==0.2
jsonschema==4.14.0
lazy-object-proxy==1.7.1
linkify-it-py==2.0.0
lockfile==0.12.2
looker-sdk==22.10.0
Mako==1.2.1
Markdown==3.4.1
markdown-it-py==2.1.0
MarkupSafe==2.1.1
marshmallow==3.17.1
marshmallow-enum==1.5.1
marshmallow-oneofschema==3.0.1
marshmallow-sqlalchemy==0.26.1
mdit-py-plugins==0.3.0
mdurl==0.1.2
numpy==1.23.2
oauthlib==3.2.0
packaging==21.3
pandas==1.4.3
pandas-gbq==0.17.8
pathspec==0.9.0
pendulum==2.1.2
pkgutil_resolve_name==1.3.10
pluggy==1.0.0
prison==0.2.1
proto-plus==1.22.0
protobuf==4.21.5
psutil==5.9.1
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
pydata-google-auth==1.4.0
Pygments==2.13.0
PyJWT==2.4.0
pyOpenSSL==22.0.0
pyparsing==3.0.9
pyrsistent==0.18.1
python-daemon==2.3.1
python-dateutil==2.8.2
python-nvd3==0.15.0
python-slugify==6.1.2
pytz==2022.2.1
pytzdata==2020.1
PyYAML==6.0
requests==2.28.1
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
rfc3986==1.5.0
rich==12.5.1
rsa==4.9
setproctitle==1.3.2
six==1.16.0
sniffio==1.2.0
SQLAlchemy==1.4.27
sqlalchemy-bigquery==1.4.4
SQLAlchemy-JSONField==1.0.0
SQLAlchemy-Utils==0.38.3
sqlparse==0.4.2
swagger-ui-bundle==0.0.9
tabulate==0.8.10
tenacity==8.0.1
termcolor==1.1.0
text-unidecode==1.3
typing_extensions==4.3.0
uc-micro-py==1.0.1
unicodecsv==0.14.1
uritemplate==3.0.1
urllib3==1.26.12
Werkzeug==2.2.2
wrapt==1.14.1
WTForms==2.3.3
zipp==3.8.1
### Apache Airflow version
2.3.4
### Operating System
Debian 11 bullseye
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
Starting from 18.1.0 of the package `google-ads`, one of its constraint has change, which is `protobuf >= 4.21.5`.
- Before: https://github.com/googleads/google-ads-python/blob/18.0.0/setup.py#L34
- After: https://github.com/googleads/google-ads-python/blob/18.1.0/setup.py#L31
With 18.1.0 of `google-ads`, it tries to install `protobuf==4.21.5`. Then, the Airflow DB migration job fails when initiating an Airflow instance with the following log. (I deployed the instance to a k8s cluster using the official Helm chart)
```
Info
2022-08-24 02:33:47.645 PDTTraceback (most recent call last): File "/home/airflow/.local/bin/airflow", line 5, in <module> from airflow.__main__ import main File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__init__.py", line 35, in <module> from airflow import settings File "/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py", line 35, in <module> from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf # NOQA F401 File "/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py", line 1618, in <module> secrets_backend_list = initialize_secrets_backends() File "/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py", line 1540, in initialize_secrets_backends custom_secret_backend = get_custom_secret_backend() File "/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py", line 1515, in get_custom_secret_backend secrets_backend_cls = conf.getimport(section='secrets', key='backend') Fil
e "/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py", line 684, in getimport return import_string(full_qualified_path) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string module = import_module(module_path) File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/secrets/secret_manager.py", line 27, in <module> from airflow.providers.google.cloud._internal_client.secret_manager_client import _SecretManagerClient File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/_internal_client/secret_manager_client.py", line 23, in <module> from google.cloud.secretmanager_v1 import SecretManagerServiceClient File "/home/airflow/.local/lib/python3.8/site-packages/google/cloud/secretmanager_v1/__init__.py", lin
e 22, in <module> from google.cloud.secretmanager_v1 import types File "/home/airflow/.local/lib/python3.8/site-packages/google/cloud/secretmanager_v1/types.py", line 23, in <module> from google.cloud.secretmanager_v1.proto import resources_pb2 File "/home/airflow/.local/lib/python3.8/site-packages/google/cloud/secretmanager_v1/proto/resources_pb2.py", line 57, in <module> _descriptor.EnumValueDescriptor( File "/home/airflow/.local/lib/python3.8/site-packages/google/protobuf/descriptor.py", line 755, in __new__ _message.Message._CheckCalledFromGeneratedFile() TypeError: Descriptors cannot not be created directly.
Info
2022-08-24 02:33:47.645 PDTIf this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
Info
2022-08-24 02:33:47.645 PDTIf you cannot immediately regenerate your protos, some other possible workarounds are:
Info
2022-08-24 02:33:47.645 PDT 1. Downgrade the protobuf package to 3.20.x or lower.
Info
2022-08-24 02:33:47.645 PDT 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
Info
2022-08-24 02:33:47.645 PDT{}
Info
2022-08-24 02:33:47.645 PDTMore information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
Info
2022-08-24 02:33:47.645 PDT{}
```
### What you think should happen instead
Looks like the version of the package must be equal to or lower than `3.20.x`.
### How to reproduce
1. Build a custom Airflow image with the following Dockerfile.
```
FROM apache/airflow:slim-2.3.4-python3.8
RUN pip install apache-airflow[google]
```
2. Deploy a new Airflow instance to a k8s cluster using the official Helm chart.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25929: Google provider package should have `google-ads<18.1.0` constraint
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25929:
URL: https://github.com/apache/airflow/issues/25929#issuecomment-1225515942
Thanks for notifying us.
It's already been handled few days ago (our CI tests caught the problem). We handled that by limiting the protobuf rather than ads. https://github.com/apache/airflow/pull/25886. It will be released in the next version.
BTW, It's not 'constraint' - it's `requirement` :). Those two have completely different meaning in `pip` (and we are using constraints heavily so it makes a difference)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #25929: Google provider package should have `google-ads<18.1.0` constraint
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #25929: Google provider package should have `google-ads<18.1.0` constraint
URL: https://github.com/apache/airflow/issues/25929
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25929: Google provider package should have `google-ads<18.1.0` constraint
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25929:
URL: https://github.com/apache/airflow/issues/25929#issuecomment-1225534731
BTW. I am also talking to Google team - they need to upgrade the integraions soon for some old clients as otherwise things like that will start happening more and more often
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] seongwon77 commented on issue #25929: Google provider package should have `google-ads<18.1.0` constraint
Posted by GitBox <gi...@apache.org>.
seongwon77 commented on issue #25929:
URL: https://github.com/apache/airflow/issues/25929#issuecomment-1225518865
Oh, thanks for letting me know. Glad to see the next release! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org