You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/05 18:37:02 UTC
[GitHub] [airflow] SamWheating opened a new pull request #9159: Adding Google Deployment Manager Hook
SamWheating opened a new pull request #9159:
URL: https://github.com/apache/airflow/pull/9159
Contributing a Hook I made for interacting with [Google Deployment Manager](https://cloud.google.com/deployment-manager) - I use it in a DAG which automatically cleans up test deployments on a regular scheduled interval.
In its current state, this hook only supports the deployments.list and deployments.delete endpoints in but support for [other available endpoints] can be added later.
---
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Target Github ISSUE in description if exists
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#issuecomment-658698216
Awesome work, congrats on your first merged pull request!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436141004
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ def list_deployments(self, project, deployment_filter=None,
+ max_results=None, order_by=None,
+ page_token=None) -> List[Dict[str, Any]]: # pylint: disable=too-many-arguments
+ """
+ Lists deployments in a google cloud project.
+
+ :param project: The project ID for this request.
+ :type key_name: str
+ :param filter: A filter expression which limits resources returned in the response.
+ :type filter: string
+ :param max_results: The maximum number of results to return
+ :type max_results: int
+ :param order_by: A filed name to order by, ex: "creationTimestamp desc"
+ :type order_by: string
+ :param page_token: specifies a page_token to use
+ :type page_token: string
+
+ :rtype: list
+ """
+ client = self.get_conn()
+ request = client.deployments().list(project=project, # pylint: disable=no-member
+ filter=deployment_filter,
+ maxResults=max_results,
+ orderBy=order_by,
+ pageToken=page_token)
+ try:
+ deployments = request.execute()['deployments']
+ except KeyError:
+ return []
+ return deployments
+
+ def delete_deployment(self, project: str, deployment: str, delete_policy: str = "DELETE"):
+ """
+ Deletes a deployment and all associated resources in a google cloud project.
+
+ :param project: The project ID for this request.
+ :type project: str
+ :param deployment: The name of the deployment for this request.
+ :type deployment: string
Review comment:
```suggestion
:type deployment: str
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436343182
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,103 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List, Optional
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def list_deployments(self, project_id: Optional[str] = None, # pylint: disable=too-many-arguments
+ deployment_filter: Optional[str] = None,
+ max_results: Optional[int] = None,
+ order_by: Optional[str] = None,
+ page_token: Optional[str] = None) -> List[Dict[str, Any]]:
+ """
+ Lists deployments in a google cloud project.
+
+ :param project_id: The project ID for this request.
+ :type project_id: str
+ :param deployment_filter: A filter expression which limits resources returned in the response.
+ :type filter: str
Review comment:
```suggestion
:type deployment_filter: str
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436140030
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ def list_deployments(self, project, deployment_filter=None,
+ max_results=None, order_by=None,
+ page_token=None) -> List[Dict[str, Any]]: # pylint: disable=too-many-arguments
+ """
+ Lists deployments in a google cloud project.
+
+ :param project: The project ID for this request.
+ :type key_name: str
+ :param filter: A filter expression which limits resources returned in the response.
+ :type filter: string
+ :param max_results: The maximum number of results to return
+ :type max_results: int
+ :param order_by: A filed name to order by, ex: "creationTimestamp desc"
+ :type order_by: string
+ :param page_token: specifies a page_token to use
+ :type page_token: string
Review comment:
```suggestion
:type page_token: str
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436141606
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ def list_deployments(self, project, deployment_filter=None,
+ max_results=None, order_by=None,
+ page_token=None) -> List[Dict[str, Any]]: # pylint: disable=too-many-arguments
+ """
+ Lists deployments in a google cloud project.
+
+ :param project: The project ID for this request.
+ :type key_name: str
+ :param filter: A filter expression which limits resources returned in the response.
+ :type filter: string
Review comment:
```suggestion
:param deployment_filter: A filter expression which limits resources returned in the response.
:type filter: str
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj merged pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
mik-laj merged pull request #9159:
URL: https://github.com/apache/airflow/pull/9159
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r437216246
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,103 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List, Optional
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def list_deployments(self, project_id: Optional[str] = None, # pylint: disable=too-many-arguments
+ deployment_filter: Optional[str] = None,
+ max_results: Optional[int] = None,
+ order_by: Optional[str] = None,
+ page_token: Optional[str] = None) -> List[Dict[str, Any]]:
+ """
+ Lists deployments in a google cloud project.
+
+ :param project_id: The project ID for this request.
+ :type project_id: str
+ :param deployment_filter: A filter expression which limits resources returned in the response.
+ :type filter: str
+ :param max_results: The maximum number of results to return
+ :type max_results: Optional[int]
+ :param order_by: A field name to order by, ex: "creationTimestamp desc"
+ :type order_by: Optional[str]
+ :param page_token: specifies a page_token to use
+ :type page_token: str
Review comment:
If the user does not want to fetch all the elements, user can use filters to limit the range of fetched elements. We should fetch all items by default.
Here is an example of how we handle pagination in other integration.
https://github.com/apache/airflow/blob/de9d340/airflow/providers/google/marketing_platform/hooks/campaign_manager.py#L125-L132
https://github.com/apache/airflow/blob/de9d340/airflow/providers/google/cloud/hooks/dataflow.py#L196-L210
https://github.com/apache/airflow/blob/de9d340/airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py#L246-L253
https://github.com/apache/airflow/blob/de9d340/airflow/providers/google/cloud/hooks/mlengine.py#L345-L353
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#issuecomment-658705598
Thank you for this contribution. What are your plans for the next contribution? :-D
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#issuecomment-639695770
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SamWheating commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
SamWheating commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436680469
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,103 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List, Optional
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def list_deployments(self, project_id: Optional[str] = None, # pylint: disable=too-many-arguments
+ deployment_filter: Optional[str] = None,
+ max_results: Optional[int] = None,
+ order_by: Optional[str] = None,
+ page_token: Optional[str] = None) -> List[Dict[str, Any]]:
+ """
+ Lists deployments in a google cloud project.
+
+ :param project_id: The project ID for this request.
+ :type project_id: str
+ :param deployment_filter: A filter expression which limits resources returned in the response.
+ :type filter: str
+ :param max_results: The maximum number of results to return
+ :type max_results: Optional[int]
+ :param order_by: A field name to order by, ex: "creationTimestamp desc"
+ :type order_by: Optional[str]
+ :param page_token: specifies a page_token to use
+ :type page_token: str
Review comment:
This option comes straight from the [API](https://cloud.google.com/deployment-manager/docs/reference/latest/deployments/list#parameters):
_maxResults, unsigned integer:_
_The maximum number of results per page that should be returned. If the number of available results is larger than maxResults, Compute Engine returns a nextPageToken that can be used to get the next page of results in subsequent list requests. Acceptable values are 0 to 500, inclusive. (Default: 500)_
So there's no way to get all of the entries, though I would assume its super rare to have >500 deployments in the same project. Can we leave this param in just so the hook doesn't become unusable past 500 deployments?
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,103 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List, Optional
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def list_deployments(self, project_id: Optional[str] = None, # pylint: disable=too-many-arguments
+ deployment_filter: Optional[str] = None,
+ max_results: Optional[int] = None,
+ order_by: Optional[str] = None,
+ page_token: Optional[str] = None) -> List[Dict[str, Any]]:
+ """
+ Lists deployments in a google cloud project.
+
+ :param project_id: The project ID for this request.
+ :type project_id: str
+ :param deployment_filter: A filter expression which limits resources returned in the response.
+ :type filter: str
+ :param max_results: The maximum number of results to return
+ :type max_results: Optional[int]
+ :param order_by: A field name to order by, ex: "creationTimestamp desc"
+ :type order_by: Optional[str]
+ :param page_token: specifies a page_token to use
+ :type page_token: str
Review comment:
Oh now I see what you mean - thanks for the examples, that really helps. Will update and push changes now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436138422
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ def list_deployments(self, project, deployment_filter=None,
+ max_results=None, order_by=None,
Review comment:
```suggestion
@GoogleBaseHook.fallback_to_default_project_id
def list_deployments(self, project_id: str, deployment_filter: Optional[str]=None,
max_results: Optional[int]=None, order_by: Optional[str]=None,
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436139802
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ def list_deployments(self, project, deployment_filter=None,
+ max_results=None, order_by=None,
+ page_token=None) -> List[Dict[str, Any]]: # pylint: disable=too-many-arguments
+ """
+ Lists deployments in a google cloud project.
+
+ :param project: The project ID for this request.
+ :type key_name: str
+ :param filter: A filter expression which limits resources returned in the response.
+ :type filter: string
+ :param max_results: The maximum number of results to return
+ :type max_results: int
Review comment:
```suggestion
:type max_results: Optional[int]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436140175
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ def list_deployments(self, project, deployment_filter=None,
+ max_results=None, order_by=None,
+ page_token=None) -> List[Dict[str, Any]]: # pylint: disable=too-many-arguments
+ """
+ Lists deployments in a google cloud project.
+
+ :param project: The project ID for this request.
+ :type key_name: str
+ :param filter: A filter expression which limits resources returned in the response.
+ :type filter: string
+ :param max_results: The maximum number of results to return
+ :type max_results: int
+ :param order_by: A filed name to order by, ex: "creationTimestamp desc"
+ :type order_by: string
+ :param page_token: specifies a page_token to use
+ :type page_token: string
+
+ :rtype: list
+ """
+ client = self.get_conn()
+ request = client.deployments().list(project=project, # pylint: disable=no-member
Review comment:
```suggestion
request = client.deployments().list(project=project_id, # pylint: disable=no-member
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436139258
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ def list_deployments(self, project, deployment_filter=None,
+ max_results=None, order_by=None,
+ page_token=None) -> List[Dict[str, Any]]: # pylint: disable=too-many-arguments
+ """
+ Lists deployments in a google cloud project.
+
+ :param project: The project ID for this request.
+ :type key_name: str
Review comment:
```suggestion
:param project_id: The project ID for this request.
:type project_id: str
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436140863
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ def list_deployments(self, project, deployment_filter=None,
+ max_results=None, order_by=None,
+ page_token=None) -> List[Dict[str, Any]]: # pylint: disable=too-many-arguments
+ """
+ Lists deployments in a google cloud project.
+
+ :param project: The project ID for this request.
+ :type key_name: str
+ :param filter: A filter expression which limits resources returned in the response.
+ :type filter: string
+ :param max_results: The maximum number of results to return
+ :type max_results: int
+ :param order_by: A filed name to order by, ex: "creationTimestamp desc"
+ :type order_by: string
+ :param page_token: specifies a page_token to use
+ :type page_token: string
+
+ :rtype: list
+ """
+ client = self.get_conn()
+ request = client.deployments().list(project=project, # pylint: disable=no-member
+ filter=deployment_filter,
+ maxResults=max_results,
+ orderBy=order_by,
+ pageToken=page_token)
+ try:
+ deployments = request.execute()['deployments']
+ except KeyError:
+ return []
+ return deployments
+
+ def delete_deployment(self, project: str, deployment: str, delete_policy: str = "DELETE"):
+ """
+ Deletes a deployment and all associated resources in a google cloud project.
+
+ :param project: The project ID for this request.
+ :type project: str
Review comment:
```suggestion
:param project_id: The project ID for this request.
:type project_id: str
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SamWheating commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
SamWheating commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r437614992
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,103 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List, Optional
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def list_deployments(self, project_id: Optional[str] = None, # pylint: disable=too-many-arguments
+ deployment_filter: Optional[str] = None,
+ max_results: Optional[int] = None,
+ order_by: Optional[str] = None,
+ page_token: Optional[str] = None) -> List[Dict[str, Any]]:
+ """
+ Lists deployments in a google cloud project.
+
+ :param project_id: The project ID for this request.
+ :type project_id: str
+ :param deployment_filter: A filter expression which limits resources returned in the response.
+ :type filter: str
+ :param max_results: The maximum number of results to return
+ :type max_results: Optional[int]
+ :param order_by: A field name to order by, ex: "creationTimestamp desc"
+ :type order_by: Optional[str]
+ :param page_token: specifies a page_token to use
+ :type page_token: str
Review comment:
Ok - I've updated the hook to fetch all deployments. I also removed the ability to specify maxResults, since it actually specifies max results _per page_ which is irrelevant if we're fetching all pages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436342715
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,103 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List, Optional
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def list_deployments(self, project_id: Optional[str] = None, # pylint: disable=too-many-arguments
+ deployment_filter: Optional[str] = None,
+ max_results: Optional[int] = None,
+ order_by: Optional[str] = None,
+ page_token: Optional[str] = None) -> List[Dict[str, Any]]:
+ """
+ Lists deployments in a google cloud project.
+
+ :param project_id: The project ID for this request.
+ :type project_id: str
+ :param deployment_filter: A filter expression which limits resources returned in the response.
+ :type filter: str
+ :param max_results: The maximum number of results to return
+ :type max_results: Optional[int]
+ :param order_by: A field name to order by, ex: "creationTimestamp desc"
+ :type order_by: Optional[str]
+ :param page_token: specifies a page_token to use
+ :type page_token: str
Review comment:
In other handles, we always return all pages. I don't think the user should worry about pagination. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436139608
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ def list_deployments(self, project, deployment_filter=None,
+ max_results=None, order_by=None,
+ page_token=None) -> List[Dict[str, Any]]: # pylint: disable=too-many-arguments
+ """
+ Lists deployments in a google cloud project.
+
+ :param project: The project ID for this request.
+ :type key_name: str
+ :param filter: A filter expression which limits resources returned in the response.
+ :type filter: string
+ :param max_results: The maximum number of results to return
+ :type max_results: int
+ :param order_by: A filed name to order by, ex: "creationTimestamp desc"
+ :type order_by: string
Review comment:
```suggestion
:type order_by: Optional[str]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436141174
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ def list_deployments(self, project, deployment_filter=None,
+ max_results=None, order_by=None,
+ page_token=None) -> List[Dict[str, Any]]: # pylint: disable=too-many-arguments
+ """
+ Lists deployments in a google cloud project.
+
+ :param project: The project ID for this request.
+ :type key_name: str
+ :param filter: A filter expression which limits resources returned in the response.
+ :type filter: string
+ :param max_results: The maximum number of results to return
+ :type max_results: int
+ :param order_by: A filed name to order by, ex: "creationTimestamp desc"
+ :type order_by: string
+ :param page_token: specifies a page_token to use
+ :type page_token: string
+
+ :rtype: list
+ """
+ client = self.get_conn()
+ request = client.deployments().list(project=project, # pylint: disable=no-member
+ filter=deployment_filter,
+ maxResults=max_results,
+ orderBy=order_by,
+ pageToken=page_token)
+ try:
+ deployments = request.execute()['deployments']
+ except KeyError:
+ return []
+ return deployments
+
+ def delete_deployment(self, project: str, deployment: str, delete_policy: str = "DELETE"):
+ """
+ Deletes a deployment and all associated resources in a google cloud project.
+
+ :param project: The project ID for this request.
+ :type project: str
+ :param deployment: The name of the deployment for this request.
+ :type deployment: string
+ :param delete_policy: Sets the policy to use for deleting resources. (ABANDON | DELETE)
+ :type delete_policy: string
+
+ :rtype: None
+ """
+ client = self.get_conn()
+ request = client.deployments().delete(project=project, # pylint: disable=no-member
Review comment:
```suggestion
request = client.deployments().delete(project=project_id, # pylint: disable=no-member
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] SamWheating commented on pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
SamWheating commented on pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#issuecomment-656932385
@mik-laj I've made the requested changes, does it look acceptable to you?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r453176927
##########
File path: tests/providers/google/cloud/hooks/test_gdm.py
##########
@@ -0,0 +1,82 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+from unittest import mock
+
+from airflow.providers.google.cloud.hooks.gdm import GoogleDeploymentManagerHook
+
+
+def mock_init(self, gcp_conn_id, delegate_to=None): # pylint: disable=unused-argument
+ pass
+
+
+TEST_PROJECT = 'my-project'
+TEST_DEPLOYMENT = 'my-deployment'
+
+
+class TestDeploymentManagerHook(unittest.TestCase):
+
+ def setUp(self):
+ with mock.patch(
+ "airflow.providers.google.common.hooks.base_google.GoogleBaseHook.__init__",
+ new=mock_init,
+ ):
+ self.gdm_hook = GoogleDeploymentManagerHook(gcp_conn_id="test")
+
+ @mock.patch("airflow.providers.google.cloud.hooks.gdm.GoogleDeploymentManagerHook.get_conn")
+ def test_list_deployments(self, mock_get_conn):
+
+ response1 = {'deployments': [{'id': 'deployment1', 'name': 'test-deploy1'}], 'pageToken': None}
+ response2 = {'deployments': [{'id': 'deployment2', 'name': 'test-deploy2'}], 'pageToken': None}
+
+ mock_get_conn.return_value.deployments.return_value.list.return_value.execute.return_value = response1
+
+ request_mock = mock.MagicMock()
+ request_mock.execute.return_value = response2
+ mock_get_conn.return_value.deployments.return_value.list_next.side_effect = [
+ request_mock,
+ None,
+ ]
+
+ deployments = self.gdm_hook.list_deployments(project_id=TEST_PROJECT,
+ deployment_filter='filter',
+ order_by='name')
+
+ mock_get_conn.assert_called_once_with()
+
+ mock_get_conn.return_value.deployments.return_value.list.assert_called_once_with(
+ project=TEST_PROJECT,
+ filter='filter',
+ orderBy='name',
+ )
+
+ self.assertEqual(mock_get_conn.return_value.deployments.return_value.list_next.call_count, 2)
+
+ self.assertEqual(deployments, [{'id': 'deployment1', 'name': 'test-deploy1'},
+ {'id': 'deployment2', 'name': 'test-deploy2'}])
+
+ @mock.patch("airflow.providers.google.cloud.hooks.gdm.GoogleDeploymentManagerHook.get_conn")
+ def test_delete_deployment(self, mock_get_conn):
Review comment:
Can you add a test when this operation fails?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436138679
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ def list_deployments(self, project, deployment_filter=None,
+ max_results=None, order_by=None,
+ page_token=None) -> List[Dict[str, Any]]: # pylint: disable=too-many-arguments
Review comment:
```suggestion
page_token: Optional[str]=None) -> List[Dict[str, Any]]: # pylint: disable=too-many-arguments
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436342775
##########
File path: tests/providers/google/cloud/hooks/test_gdm.py
##########
@@ -0,0 +1,78 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+from unittest import mock
+
+from airflow.providers.google.cloud.hooks.gdm import GoogleDeploymentManagerHook
+
+
+def mock_init(self, gcp_conn_id, delegate_to=None): # pylint: disable=unused-argument
+ pass
+
+
+TEST_PROJECT = 'my-project'
+TEST_DEPLOYMENT = 'my-deployment'
+FILTER = 'staging-'
+MAX_RESULTS = 10
+ORDER_BY = 'name'
+
+
+class TestDeploymentManagerHook(unittest.TestCase):
+
+ def setUp(self):
+ with mock.patch(
+ "airflow.providers.google.common.hooks.base_google.GoogleBaseHook.__init__",
+ new=mock_init,
+ ):
+ self.gdm_hook = GoogleDeploymentManagerHook(gcp_conn_id="test")
+
+ @mock.patch("airflow.providers.google.cloud.hooks.gdm.GoogleDeploymentManagerHook.get_conn")
+ def test_list_deployments(self, mock_get_conn):
+ _ = self.gdm_hook.list_deployments(project_id=TEST_PROJECT)
+ mock_get_conn.assert_called_once_with()
+ mock_get_conn.return_value.deployments().list.assert_called_once_with(
+ project=TEST_PROJECT,
+ filter=None,
+ maxResults=None,
+ orderBy=None,
+ pageToken=None,
+ )
+
+ @mock.patch("airflow.providers.google.cloud.hooks.gdm.GoogleDeploymentManagerHook.get_conn")
+ def test_list_deployments_with_params(self, mock_get_conn):
+ _ = self.gdm_hook.list_deployments(project_id=TEST_PROJECT, deployment_filter=FILTER,
Review comment:
Can you add assertion for return value?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#discussion_r436140642
##########
File path: airflow/providers/google/cloud/hooks/gdm.py
##########
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import build
+
+from airflow.exceptions import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class GoogleDeploymentManagerHook(GoogleBaseHook): # pylint: disable=abstract-method
+ """
+ Interact with Google Cloud Deployment Manager using the Google Cloud Platform connection.
+ This allows for scheduled and programatic inspection and deletion fo resources managed by GDM.
+ """
+
+ def __init__(self, gcp_conn_id='google_cloud_default', delegate_to=None):
+ super(GoogleDeploymentManagerHook, self).__init__(gcp_conn_id, delegate_to=delegate_to)
+
+ def get_conn(self):
+ """
+ Returns a Google Deployment Manager service object.
+
+ :rtype: googleapiclient.discovery.Resource
+ """
+ http_authorized = self._authorize()
+ return build('deploymentmanager', 'v2', http=http_authorized, cache_discovery=False)
+
+ def list_deployments(self, project, deployment_filter=None,
+ max_results=None, order_by=None,
+ page_token=None) -> List[Dict[str, Any]]: # pylint: disable=too-many-arguments
+ """
+ Lists deployments in a google cloud project.
+
+ :param project: The project ID for this request.
+ :type key_name: str
+ :param filter: A filter expression which limits resources returned in the response.
+ :type filter: string
+ :param max_results: The maximum number of results to return
+ :type max_results: int
+ :param order_by: A filed name to order by, ex: "creationTimestamp desc"
+ :type order_by: string
+ :param page_token: specifies a page_token to use
+ :type page_token: string
+
+ :rtype: list
+ """
+ client = self.get_conn()
+ request = client.deployments().list(project=project, # pylint: disable=no-member
+ filter=deployment_filter,
+ maxResults=max_results,
+ orderBy=order_by,
+ pageToken=page_token)
+ try:
+ deployments = request.execute()['deployments']
+ except KeyError:
+ return []
+ return deployments
+
+ def delete_deployment(self, project: str, deployment: str, delete_policy: str = "DELETE"):
Review comment:
```suggestion
@GoogleBaseHook.fallback_to_default_project_id
def delete_deployment(self, project_id: str, deployment: str, delete_policy: str = "DELETE"):
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #9159: Adding Google Deployment Manager Hook
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #9159:
URL: https://github.com/apache/airflow/pull/9159#issuecomment-656919096
I wonder what the status of this change is. Do you need help?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org