You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/19 19:16:10 UTC
[GitHub] [airflow] iostreamdoth opened a new issue #17089: cluster config generation using cluster config YAML file.
iostreamdoth opened a new issue #17089:
URL: https://github.com/apache/airflow/issues/17089
**Description**
Add cluster config generation from yaml file which is generated using `gcloud dataproc clusters export` command.
**Use case / motivation**
We are generating quite a lot of clusters using a cluster config which keeps on changing depending upon project to project. I wrote custom operator which reads yaml config and converts it to dict and then uses the same config to genreate cluster. Wondering if it will be useful if cluster config generator operator can accept a yaml config or can create a new operator.
**Are you willing to submit a PR?**
I do have a working code for converting yaml stored on gcs to dict.
**Related Issues**
There is no issue as such, it is a functionality which allows users to specify cluster config as yaml file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882871939
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882794238
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882871939
> config generator
DataprocCreateClusterOperator class is a deprecated and we are not adding new features to it.
I think we can extend `DataprocCreateClusterOperator` to accept the file path in `cluster_config` parameter like Cloud Build operator.
https://github.com/apache/airflow/blob/d268016a7a6ff4b65079f1dea080ead02aea99bb/airflow/providers/google/cloud/operators/cloud_build.py#L172-L175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] iostreamdoth commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
iostreamdoth commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882914685
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882871939
> config generator
<s>DataprocCreateClusterOperator</s>ClusterGenerator class is a deprecated and we are not adding new features to it.
I think we can extend `DataprocCreateClusterOperator` to accept the file path in `cluster_config` parameter like Cloud Build operator.
https://github.com/apache/airflow/blob/d268016a7a6ff4b65079f1dea080ead02aea99bb/airflow/providers/google/cloud/operators/cloud_build.py#L172-L175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-981104018
@iostreamdoth do you plan to finish the stale PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] iostreamdoth edited a comment on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
iostreamdoth edited a comment on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882914685
you mean `airflow.providers.google.cloud.operators.dataproc.DataprocCreateClusterOperator` or you mean `airflow.contrib.operators.dataproc_operator.DataprocClusterCreateOperator` ? I thought this is going to be in providers package.
@mik-laj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882795290
Feel free to make a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882795290
Feel free to make a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882919242
@iostreamdoth ClusterGenerator from providers is legacy/deprecated method to generate dataproc cluster configuration. You should use `cluster_config` parameter. For details, see: https://github.com/apache/airflow/blob/026ffe65d4738674512f691a56b922e82d0a2309/airflow/providers/google/cloud/operators/dataproc.py#L553-L559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882871939
> config generator
<s>DataprocCreateClusterOperator</s>ClusterGenerator class is a deprecated and we are not adding new features to it.
I think we can extend `DataprocCreateClusterOperator` to accept the file path in `cluster_config` parameter like Cloud Build operator.
https://github.com/apache/airflow/blob/d268016a7a6ff4b65079f1dea080ead02aea99bb/airflow/providers/google/cloud/operators/cloud_build.py#L172-L175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] iostreamdoth commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
iostreamdoth commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882914685
you mean `airflow.providers.google.cloud.operators.dataproc.DataprocCreateClusterOperator` or you mean `airflow.contrib.operators.dataproc_operator.DataprocClusterCreateOperator` ? I thought everything is going to be in providers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882871939
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] iostreamdoth edited a comment on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
iostreamdoth edited a comment on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882914685
you mean `airflow.providers.google.cloud.operators.dataproc.DataprocCreateClusterOperator` or you mean `airflow.contrib.operators.dataproc_operator.DataprocClusterCreateOperator` ? I thought everything is going to be in providers.
@mik-laj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882871939
> config generator
<s>DataprocCreateClusterOperator</s>ClusterGenerator class is a deprecated and we are not adding new features to it.
I think we can extend `DataprocCreateClusterOperator` to accept the file path in `cluster_config` parameter like Cloud Build operator.
https://github.com/apache/airflow/blob/d268016a7a6ff4b65079f1dea080ead02aea99bb/airflow/providers/google/cloud/operators/cloud_build.py#L172-L175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882794238
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] iostreamdoth edited a comment on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
iostreamdoth edited a comment on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882914685
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] iostreamdoth edited a comment on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
iostreamdoth edited a comment on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882914685
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] iostreamdoth commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
iostreamdoth commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882914685
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] iostreamdoth commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
iostreamdoth commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882928050
Got it, I got confused with the Operator name earlier.
Thanks @mik-laj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882794238
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17089: cluster config generation using cluster config YAML file.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17089:
URL: https://github.com/apache/airflow/issues/17089#issuecomment-882795290
Feel free to make a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org