You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/07 10:38:12 UTC
[GitHub] [airflow] wsmolak opened a new issue #20100: google providers > dataproc > ClusterGenerator does not implement all parameters of Dataproc ClusterConfig object
wsmolak opened a new issue #20100:
URL: https://github.com/apache/airflow/issues/20100
### Description
ClusterGenerator class should support all parameters of [ClusterConfig](https://cloud.google.com/dataproc/docs/reference/rest/v1/ClusterConfig) object.
### Use case/motivation
Using current version of ClusterGenerator user can't set all DataProc's parameters. It is possible only by manual update of result of `ClusterGenerator.make()`.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20100: google providers > dataproc > ClusterGenerator does not implement all parameters of Dataproc ClusterConfig object
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20100:
URL: https://github.com/apache/airflow/issues/20100#issuecomment-988652407
@mik-laj @wsmolak - In principle - I do agree with Kamil.
However I have a thought.
I would not dismiss the option of leaving the ClusterGenerator stratight away.
The API Guide was written for Airflow 1.10, and now when we are releasing new Google Provider every month, regularly like a clock. So as long as someone (your team @wsmolak :) ? ) woudl regularly (even automatically - we used to have such a simple tool) monitor for changes and update the ClusterGenerator, that would pretty much work - there would be at most weeks delays between the release of updated API and release of Google Provider, and likely you could even add more parameters when they are in Beta or in-development.
Another option (not sure if feasible) - maybe there is (even a convoluted and complex) way of adding new "unknown" fields to cluster config generated by the generator? In this case we could simply describe the user how they can add new field while they are waiting for a new provider.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #20100: google providers > dataproc > ClusterGenerator does not implement all parameters of Dataproc ClusterConfig object
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #20100:
URL: https://github.com/apache/airflow/issues/20100#issuecomment-987941388
ClusterGenerator is a deprecated approach because it does not follow [the GCP Service Airflow Integration Guide](https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit#heading=h.3sxddzjr5f6q). More precisely, this class needs to be updated with almost every API change. The Airflow operators should be a thin layer that passes the parameters unchanged to the API client. I think we should update the documentation to make it clear that this class will not be further developed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] wsmolak commented on issue #20100: google providers > dataproc > ClusterGenerator does not implement all parameters of Dataproc ClusterConfig object
Posted by GitBox <gi...@apache.org>.
wsmolak commented on issue #20100:
URL: https://github.com/apache/airflow/issues/20100#issuecomment-988563395
Thanks @mik-laj!
What is the recommended way to build a cluster config? Usage of ClusterGenerator simplifies running a DataProc pipelines form Composer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #20100: google providers > dataproc > ClusterGenerator does not implement all parameters of Dataproc ClusterConfig object
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20100:
URL: https://github.com/apache/airflow/issues/20100#issuecomment-988652407
@mik-laj @wsmolak - In principle - I do agree with Kamil.
However I have a thought.
I would not dismiss the option of leaving the ClusterGenerator stratight away.
The API Guide was written for Airflow 1.10, and now when we are releasing new Google Provider every month, regularly like a clock things changed a bit.
As long as someone (your team @wsmolak :) ? ) woudl regularly (even automatically - we used to have such a simple tool) monitor for changes and update the ClusterGenerator, that would pretty much work - there would be at most weeks delays between the release of updated API and release of Google Provider, and likely you could even add more parameters when they are in Beta or in-development.
Another option (not sure if feasible) - maybe there is (even a convoluted and complex) way of adding new "unknown" fields to cluster config generated by the generator? In this case we could simply describe the user how they can add new field while they are waiting for a new provider.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #20100: google providers > dataproc > ClusterGenerator does not implement all parameters of Dataproc ClusterConfig object
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #20100:
URL: https://github.com/apache/airflow/issues/20100#issuecomment-987792060
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #20100: google providers > dataproc > ClusterGenerator does not implement all parameters of Dataproc ClusterConfig object
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #20100:
URL: https://github.com/apache/airflow/issues/20100#issuecomment-988672027
The ClusterGenerator is 350 lines of code that does not look like it can be generated.
https://github.com/apache/airflow/blob/bf68b9a8461eda634a7d91aa56575fb950960eaa/airflow/providers/google/cloud/operators/dataproc.py#L90-L444
I think that if we make an exception, similar problems arise with other operators such as BigQuery, which will cause a heavy burden of maintenance with limited profit. On the other hand, Google has a very helpful generator on its website, which slightly improves could fulfill a similar role.
<img width="388" alt="Screenshot 2021-12-08 at 11 04 44" src="https://user-images.githubusercontent.com/12058428/145189463-de161c6a-0aaa-40dd-bcd0-7c2715d48df5.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #20100: google providers > dataproc > ClusterGenerator does not implement all parameters of Dataproc ClusterConfig object
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #20100:
URL: https://github.com/apache/airflow/issues/20100#issuecomment-988643273
@wsmolak I would recommend looking at the API reference documentation and building based on this code. That might mean a bit more code, but it's up to Google that they built an API that is quite verbose.
https://cloud.google.com/dataproc/docs/reference/rest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org