You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/16 07:41:21 UTC
[GitHub] [airflow] sudohainguyen opened a new issue, #28399: Dataproc Generator should allow creating spot instance
sudohainguyen opened a new issue, #28399:
URL: https://github.com/apache/airflow/issues/28399
### Description
GCP has released their new generation of Preemptible instance called Spot
background: https://cloud.google.com/dataproc/docs/reference/rest/v1/ClusterConfig#preemptibility
`DataprocCreateClusterOperator` should allow passing options for secondary workers to be created as spot instances
### Use case/motivation
as the background: https://cloud.google.com/compute/docs/instances/spot
Spot instances have a some advantages comparing to preemptible ones. And most important is, it can save cost charged by GCP.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #28399: Dataproc Generator should allow creating spot instances
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #28399: Dataproc Generator should allow creating spot instances
URL: https://github.com/apache/airflow/issues/28399
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudohainguyen commented on issue #28399: Dataproc Generator should allow creating spot instances
Posted by GitBox <gi...@apache.org>.
sudohainguyen commented on issue #28399:
URL: https://github.com/apache/airflow/issues/28399#issuecomment-1356167335
hey @Taragolis, yes I tried, but behind of the picture, apparently python dataproc client have not supported SPOT yet, you can check it out here
https://github.com/googleapis/python-dataproc/issues/506
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudohainguyen commented on issue #28399: Dataproc Generator should allow creating spot instances
Posted by GitBox <gi...@apache.org>.
sudohainguyen commented on issue #28399:
URL: https://github.com/apache/airflow/issues/28399#issuecomment-1356167522
let me follow closely with the issue from google, will be back when they have any progress regarding this feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudohainguyen commented on issue #28399: Dataproc Generator should allow creating spot instances
Posted by "sudohainguyen (via GitHub)" <gi...@apache.org>.
sudohainguyen commented on issue #28399:
URL: https://github.com/apache/airflow/issues/28399#issuecomment-1400637319
to fulfill the need to create secondary workers as SPOT type, `google-cloud-dataproc` version must be `5.3.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #28399: Dataproc Generator should allow creating spot instances
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #28399:
URL: https://github.com/apache/airflow/issues/28399#issuecomment-1384687968
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #28399: Dataproc Generator should allow creating spot instances
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28399:
URL: https://github.com/apache/airflow/issues/28399#issuecomment-1451076558
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #28399: Dataproc Generator should allow creating spot instances
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28399:
URL: https://github.com/apache/airflow/issues/28399#issuecomment-1441031459
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudohainguyen commented on issue #28399: Dataproc Generator should allow creating spot instances
Posted by "sudohainguyen (via GitHub)" <gi...@apache.org>.
sudohainguyen commented on issue #28399:
URL: https://github.com/apache/airflow/issues/28399#issuecomment-1400553157
due to the latest change from google (https://github.com/googleapis/python-dataproc/pull/512)
I'm resuming the work from now on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #28399: Dataproc Generator should allow creating spot instances
Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #28399:
URL: https://github.com/apache/airflow/issues/28399#issuecomment-1354540413
Hey @sudohainguyen, did you tried to provide required config into `cluster_config` of [DataprocCreateClusterOperator](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/cloud/operators/dataproc/index.html#airflow.providers.google.cloud.operators.dataproc.DataprocCreateClusterOperator)?
See also usage of [Dataproc Operators in Airflow](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/dataproc.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org