You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Xiroo (via GitHub)" <gi...@apache.org> on 2023/07/19 06:54:31 UTC
[GitHub] [airflow] Xiroo opened a new issue, #32686: Google Cloud client libraries broke with protobuf 4.23.x version.
Xiroo opened a new issue, #32686:
URL: https://github.com/apache/airflow/issues/32686
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
In airflow version 2.6.2 and 2.6.3, there's a constrain like following.
In 2.6.2
protobuf==4.23.2
In 2.6.3
protobuf==4.23.4
But protobuf with version above 3.20.x, it return failure log like following.
I think it is known issue also in the [protobuf project](https://github.com/protocolbuffers/protobuf/issues/10051)
```Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.10/site-packages/google/cloud/logging/__init__.py", line 18, in <module>
from google.cloud.logging_v2 import __version__
File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/__init__.py", line 25, in <module>
from google.cloud.logging_v2.client import Client
File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/client.py", line 25, in <module>
from google.cloud.logging_v2._helpers import _add_defaults_to_filter
File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/_helpers.py", line 25, in <module>
from google.cloud.logging_v2.entries import LogEntry
File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/entries.py", line 31, in <module>
import google.cloud.audit.audit_log_pb2 # noqa: F401
File "/usr/local/lib/python3.10/site-packages/google/cloud/audit/audit_log_pb2.py", line 62, in <module>
_descriptor.FieldDescriptor(
File "/usr/local/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 560, in __new__
_message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates```
### What you think should happen instead
_No response_
### How to reproduce
using 'from airflow.providers.google.common.hooks.base_google import GoogleBaseHook' with protobuf above 3.20.x
### Operating System
macOS 13.0
### Versions of Apache Airflow Providers
apache-airflow-providers-google==10.2.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #32686: Google Cloud client libraries broke with protobuf 4.23.x version.
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #32686:
URL: https://github.com/apache/airflow/issues/32686#issuecomment-1656891366
No. Google client libraries deliberately moved to Protobuf 4. Protobuf 3 is end of life already so there is no way we are going to set it in constraints. If this is a problem, then this is a problem with cloud logging library compiled with protobuf that are already end of life.
We have no control over 3rd-party libraries like google logging. It the version that has been fixed in our constraints had protobuf3 -compiled code, then it's the problem of theirs not ours.
The good news is that Airflow mechanism of constraint is prepared and gives yout ways to fix it.
You are free to upgrade those dependncies that have some features that might be limiting you. You can definitely try to upgrade to latest version of google provider and especially to the latest google logging library 3..6.0 that has been released 14th of July in the hopes it will work. In fact it is already tested and runs in `main` so it should be quite safe to upgrade it (though of course you should test if it works in your case, ideally on staging environment of yours). Just follow the docs of ours - where we explain how to install and upgrade airflow depenendencies after installing airflow "with constraints" https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#managing-providers-separately-from-airflow-core
Or you can also upgrade to 2.7.0 when it is out (likely in the next few weeks) if you are willing to wait - we are automatically upgrading the dependencies of ours to latest "compatible versions" and every new relase of airflow will have upgraded dependencies - so if you prefer to base on what community provides, wait for 2.7.0.
Those are basicaly two choices you have:
* if you want to rely on the set of dependencies that community releases - you just upgrade to latest version of Airflow with all the goodies /provider version it comes with - latest and gratest
* or if you need ot update those dependencies out-of-band - follow the instructions we gave to our users so that they can follow them on their own and selectively update what they need.
converting to discussion if more is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #32686: Google Cloud client libraries broke with protobuf 4.23.x version.
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #32686: Google Cloud client libraries broke with protobuf 4.23.x version.
URL: https://github.com/apache/airflow/issues/32686
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org