You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2023/01/02 03:57:57 UTC
[GitHub] [airflow] sudohainguyen opened a new issue, #28668: GCP Dataproc suddently not accepting current config default values
sudohainguyen opened a new issue, #28668:
URL: https://github.com/apache/airflow/issues/28668
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
```shell
apache-airflow-providers-amazon 6.2.0
apache-airflow-providers-apache-beam 4.1.0
apache-airflow-providers-atlassian-jira 1.1.0
apache-airflow-providers-celery 3.1.0
apache-airflow-providers-cncf-kubernetes 5.0.0
apache-airflow-providers-common-sql 1.3.1
apache-airflow-providers-docker 3.1.0
apache-airflow-providers-elasticsearch 4.3.1
apache-airflow-providers-ftp 3.2.0
apache-airflow-providers-google 8.6.0
apache-airflow-providers-grpc 3.1.0
apache-airflow-providers-hashicorp 3.2.0
apache-airflow-providers-http 4.1.0
apache-airflow-providers-imap 3.1.0
apache-airflow-providers-jira 3.1.0
apache-airflow-providers-microsoft-azure 5.0.0
apache-airflow-providers-mysql 3.4.0
apache-airflow-providers-odbc 3.2.1
apache-airflow-providers-oracle 3.5.1
apache-airflow-providers-postgres 5.3.1
apache-airflow-providers-redis 3.1.0
apache-airflow-providers-sendgrid 3.1.0
apache-airflow-providers-sftp 4.2.0
apache-airflow-providers-slack 7.1.0
apache-airflow-providers-sqlite 3.3.1
apache-airflow-providers-ssh 3.3.0
apache-airflow-providers-vertica 3.3.1
```
### Apache Airflow version
2.3.2
### Operating System
ubuntu 20.04
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
The incident occurred 2 days ago, all of DataprocCreateCluster thrown this error
``` bash
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "Network 'default' requires explicit subnetworks for instance creation. Specify a 'legacy' or 'auto' network, or specify subnetwork explicitly."
debug_error_string = "UNKNOWN:Error received from peer ipv4:172.217.194.95:443 {grpc_message:"Network \'default\' requires explicit subnetworks for instance creation. Specify a \'legacy\' or \'auto\' network, or specify subnetwork explicitly.", grpc_status:3, created_time:"2022-12-30T09:48:37.000807792+00:00"}"
```
### What you think should happen instead
After a few tries with gcloud and pure python client, I found out it was not because code implementation, maybe on server side, GCP had a few changes.
### How to reproduce
1. Create ClusterGenerator object with a few required params, left the rest as default
2. Create a DAG containing DataprocCreateClusterOperator with the config
3. Execute the DAG
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28668: GCP Dataproc suddently not accepting current config default values
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28668:
URL: https://github.com/apache/airflow/issues/28668#issuecomment-1368656603
I believe so, but I think this is more question to Dataproc team - seem that they have changed the API. cc: @bhirsz @bjankie1 @VladaZakharova - and whether there is somethign to be done in the google provider about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudohainguyen commented on issue #28668: GCP Dataproc suddently not accepting current config default values
Posted by GitBox <gi...@apache.org>.
sudohainguyen commented on issue #28668:
URL: https://github.com/apache/airflow/issues/28668#issuecomment-1368636340
> my team resolved this issue by assigning specific value to `subnetwork_uri` attributes like this
>
> ```
> cluster = {
> "project_id": project_id,
> "cluster_name": cluster_name,
> "config": {
> ...
> "gce_cluster_config": {
> "subnetwork_uri": f"projects/{project_id}/regions/{region}/subnetworks/default",
> },
> ...
> },
> }
> ```
so my question is, should we put it as default value of ClusterGenerator on behalf of airflow users?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudohainguyen commented on issue #28668: GCP Dataproc suddently not accepting current config default values
Posted by GitBox <gi...@apache.org>.
sudohainguyen commented on issue #28668:
URL: https://github.com/apache/airflow/issues/28668#issuecomment-1368635929
my team resolved this issue by assigning specific value to `subnetwork_uri` attributes like this
```
cluster = {
"project_id": project_id,
"cluster_name": cluster_name,
"config": {
...
"gce_cluster_config": {
"subnetwork_uri": f"projects/{project_id}/regions/{region}/subnetworks/default",
},
...
},
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #28668: GCP Dataproc suddently not accepting current config default values
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #28668: GCP Dataproc suddently not accepting current config default values
URL: https://github.com/apache/airflow/issues/28668
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudohainguyen commented on issue #28668: GCP Dataproc suddently not accepting current config default values
Posted by GitBox <gi...@apache.org>.
sudohainguyen commented on issue #28668:
URL: https://github.com/apache/airflow/issues/28668#issuecomment-1368636163
should we put it as default value of ClusterGenerator on behalf of airflow users?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org