You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/30 23:46:41 UTC
[GitHub] [airflow] leahecole opened a new issue #15622: Inconsistencies with Dataproc Operator parameters
leahecole opened a new issue #15622:
URL: https://github.com/apache/airflow/issues/15622
I'm looking at the GCP Dataproc operator and noticed that the `DataprocCreateClusterOperator` requires a `region` parameter, but other operators, like the `DataprocSubmitJobOperator` and the `DataprocDeleteClusterOperator` require a `location` parameter instead. I think it would be best to consistently enforce the parameter as `region`, because that's what is required in the protos for all of the [cluster CRUD operations](https://github.com/googleapis/python-dataproc/blob/master/google/cloud/dataproc_v1/proto/clusters.proto) and for [job submission](https://github.com/googleapis/python-dataproc/blob/d4b299216ad833f68ad63866dbdb2c8f2755c6b4/google/cloud/dataproc_v1/proto/jobs.proto#L741). `location` also feels too ambiguous imho, because it implies we could also pass a GCE zone, which in this case is either unnecessary or not supported (I can't remember which and it's too late in my Friday for me to double check 🙃 )
This might be similar to #13454 but I'm not 100% sure. If we think this is worth working on, I could maybe take this on as a PR, but it would be low priority for me, and if someone else wants to take it on, they should feel free to 😄
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**:
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
**Environment**:
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
<!-- (please include exact error messages if you can) -->
**What you expected to happen**:
<!-- What do you think went wrong? -->
**How to reproduce it**:
<!---
As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
## Install minikube/kind
- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/
If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
You can include images using the .md style of
To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
--->
**Anything else we need to know**:
<!--
How often does this problem occur? Once? Every time etc?
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Daniel-Han-Yang commented on issue #15622: Inconsistencies with Dataproc Operator parameters
Posted by GitBox <gi...@apache.org>.
Daniel-Han-Yang commented on issue #15622:
URL: https://github.com/apache/airflow/issues/15622#issuecomment-841887256
Can I work on this @leahecole @mik-laj ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #15622: Inconsistencies with Dataproc Operator parameters
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #15622:
URL: https://github.com/apache/airflow/issues/15622#issuecomment-830463037
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #15622: Inconsistencies with Dataproc Operator parameters
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #15622:
URL: https://github.com/apache/airflow/issues/15622#issuecomment-841887958
@Daniel-Han-Yang feel free. Assigned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Daniel-Han-Yang commented on issue #15622: Inconsistencies with Dataproc Operator parameters
Posted by GitBox <gi...@apache.org>.
Daniel-Han-Yang commented on issue #15622:
URL: https://github.com/apache/airflow/issues/15622#issuecomment-850995772
@mik-laj, I've put in a PR for this change, appreciate it if you let me know what needs to be added/changed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Daniel-Han-Yang commented on issue #15622: Inconsistencies with Dataproc Operator parameters
Posted by GitBox <gi...@apache.org>.
Daniel-Han-Yang commented on issue #15622:
URL: https://github.com/apache/airflow/issues/15622#issuecomment-841887256
Can I work on this @leahecole @mik-laj ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Daniel-Han-Yang edited a comment on issue #15622: Inconsistencies with Dataproc Operator parameters
Posted by GitBox <gi...@apache.org>.
Daniel-Han-Yang edited a comment on issue #15622:
URL: https://github.com/apache/airflow/issues/15622#issuecomment-850995772
@mik-laj, I've put in a PR for this change, appreciate it if you let me know what needs to be added/changed. Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #15622: Inconsistencies with Dataproc Operator parameters
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #15622:
URL: https://github.com/apache/airflow/issues/15622#issuecomment-830729125
Hi @leahecole
Are you willing to submit a PR? I'm happy to help with the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #15622: Inconsistencies with Dataproc Operator parameters
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #15622:
URL: https://github.com/apache/airflow/issues/15622#issuecomment-841887958
@Daniel-Han-Yang feel free. Assigned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] leahecole closed issue #15622: Inconsistencies with Dataproc Operator parameters
Posted by GitBox <gi...@apache.org>.
leahecole closed issue #15622:
URL: https://github.com/apache/airflow/issues/15622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org