You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/03/05 16:34:56 UTC
[GitHub] [airflow] michalslowikowski00 opened a new pull request #7630:
[AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
michalslowikowski00 opened a new pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
<sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r389245843
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
Review comment:
Can you explain why we need this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r391755573
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
Review comment:
I looked into others hooks, none of them have a link to the API doc. I will leave it as it is. The operator have link to API doc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7630: [AIRFLOW-6724] Add
Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#issuecomment-596135065
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=h1) Report
> Merging [#7630](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/f3abd340826289dec23e96b79a6ed9b6a1955027?src=pr&el=desc) will **decrease** coverage by `0.26%`.
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7630 +/- ##
==========================================
- Coverage 86.83% 86.56% -0.27%
==========================================
Files 897 900 +3
Lines 42805 42851 +46
==========================================
- Hits 37170 37096 -74
- Misses 5635 5755 +120
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...iders/google/marketing\_platform/hooks/analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9ob29rcy9hbmFseXRpY3MucHk=) | `100% <100%> (ø)` | |
| [...rketing\_platform/example\_dags/example\_analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9leGFtcGxlX2RhZ3MvZXhhbXBsZV9hbmFseXRpY3MucHk=) | `100% <100%> (ø)` | |
| [...s/google/marketing\_platform/operators/analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9vcGVyYXRvcnMvYW5hbHl0aWNzLnB5) | `100% <100%> (ø)` | |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0%> (-25.26%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.87% <0%> (-0.29%)` | :arrow_down: |
| [airflow/providers/google/cloud/hooks/gcs.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2djcy5weQ==) | `85.46% <0%> (-0.06%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=footer). Last update [f3abd34...ccfa0c3](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r389258274
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
+ def list_accounts(self, max_results: int, start_index: int) -> list:
Review comment:
Why this way?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r390969317
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ def list_accounts(self) -> List[Dict[str, Any]]:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ More information about API:
+ http://googleapis.github.io/google-api-python-client/docs/dyn/analytics_v3.management.accounts.html
+
+ """
+
+ self.log.info("Retrieving accounts list...")
+
+ # pylint: disable=no-member
Review comment:
Since then little refactor has been made and few things are different right now, sorry. :/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r389266880
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
+ def list_accounts(self, max_results: int, start_index: int) -> list:
Review comment:
Yes, it's more explicit and mypy can perform better checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r390967147
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ def list_accounts(self) -> List[Dict[str, Any]]:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ More information about API:
+ http://googleapis.github.io/google-api-python-client/docs/dyn/analytics_v3.management.accounts.html
+
+ """
+
+ self.log.info("Retrieving accounts list...")
+
+ # pylint: disable=no-member
+ result = [] # type: List[Dict]
+ conn = self.get_conn()
+ # return self._list_emails_from_response(accounts)
+ while True:
+ request = conn.management().accounts().list(start_index=len(result) + 1)
+ response = request.execute(num_retries=self.num_retries)
+ result.extend(response.get('items', []))
+ if response["totalResults"] >= len(result):
Review comment:
Yes, below this is a `break`.
I changed it to `<=` in the latest commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r390834307
##########
File path: airflow/providers/google/marketing_platform/operators/analytics.py
##########
@@ -0,0 +1,67 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Google Analytics 360 operators.
+"""
+
+from airflow.models import BaseOperator
+from airflow.providers.google.marketing_platform.hooks.analytics import GoogleAnalyticsHook
+from airflow.utils.decorators import apply_defaults
+
+
+class GoogleAnalyticsListAccountsOperator(BaseOperator):
+ """
+ Lists all accounts to which the user has access.
+
+ .. seealso::
+ Check official API docs:
+ https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/accounts/list
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:GoogleAnalyticsListAccountsOperator`
+
+ :param max_results: The maximum number of accounts to include in this response.
+ :type: max_results: int
+ :param: start_index: An index of the first account to retrieve
+ Use this parameter as a pagination mechanism along with the max-results parameter.
+ :type: start_index: int
Review comment:
Those arguments are not present in constructor
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r389258263
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
Review comment:
I had to add this option because of pylint.
This methods `def list_accounts(self, max_results: int, start_index: int)` has two args, but none of them is used in method itself. Args are used in operator.
AFAIK Analytics API method supports args: max_results and start_index but I cannot pass them into this request `analytics.management().accounts().list().execute()`. I have tried but it failed.
I am open for the suggestions if you have any.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r389258274
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
+ def list_accounts(self, max_results: int, start_index: int) -> list:
Review comment:
Nice :) I will do it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7630:
[AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#issuecomment-596135065
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=h1) Report
> Merging [#7630](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/137896f326cd29b59902a887e4c4e58f940ff62b?src=pr&el=desc) will **decrease** coverage by `1.02%`.
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7630 +/- ##
==========================================
- Coverage 87% 85.98% -1.03%
==========================================
Files 904 909 +5
Lines 43728 43954 +226
==========================================
- Hits 38046 37794 -252
- Misses 5682 6160 +478
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...iders/google/marketing\_platform/hooks/analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9ob29rcy9hbmFseXRpY3MucHk=) | `100% <100%> (ø)` | |
| [...rketing\_platform/example\_dags/example\_analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9leGFtcGxlX2RhZ3MvZXhhbXBsZV9hbmFseXRpY3MucHk=) | `100% <100%> (ø)` | |
| [...s/google/marketing\_platform/operators/analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9vcGVyYXRvcnMvYW5hbHl0aWNzLnB5) | `100% <100%> (ø)` | |
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0%> (-54.93%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `50% <0%> (-50%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| ... and [33 more](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=footer). Last update [137896f...d7eb2ee](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r391459941
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
Review comment:
Updated with the latest commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r390964634
##########
File path: airflow/providers/google/marketing_platform/operators/analytics.py
##########
@@ -0,0 +1,67 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Google Analytics 360 operators.
+"""
+
+from airflow.models import BaseOperator
+from airflow.providers.google.marketing_platform.hooks.analytics import GoogleAnalyticsHook
+from airflow.utils.decorators import apply_defaults
+
+
+class GoogleAnalyticsListAccountsOperator(BaseOperator):
+ """
+ Lists all accounts to which the user has access.
+
+ .. seealso::
+ Check official API docs:
+ https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/accounts/list
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:GoogleAnalyticsListAccountsOperator`
+
+ :param max_results: The maximum number of accounts to include in this response.
+ :type: max_results: int
+ :param: start_index: An index of the first account to retrieve
+ Use this parameter as a pagination mechanism along with the max-results parameter.
+ :type: start_index: int
+ :param api_version: The version of the api that will be requested for example 'v3'.
+ :type api_version: str
+ :param gcp_conn_id: The connection ID to use when fetching connection info.
+ :type gcp_conn_id: str
+ """
+
+ template_fields = ("api_version", "gcp_connection_id",)
+
+ @apply_defaults
+ def __init__(self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs):
+ super().__init__(*args, **kwargs)
+
+ self.api_version = api_version
+ self.gcp_connection_id = gcp_connection_id
+
+ def execute(self, context):
+ hook = GoogleAnalyticsHook(api_version=self.api_version,
+ gcp_connection_id=self.gcp_connection_id)
+ result = hook.list_accounts()
Review comment:
I put logging into hook. `self.log.info("Retrieving accounts list...")`
Should it be in the operator? What's the best practice?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r389258274
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
+ def list_accounts(self, max_results: int, start_index: int) -> list:
Review comment:
Why this way? I see, it is more explicit. What other benefits are?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r389305670
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
+ def list_accounts(self, max_results: int, start_index: int) -> list:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ :param max_results: The maximum number of accounts to include in this response.
+ :type: max_result: int
+ :param start_index: An index of the first account to retrieve
+ Use this parameter as a pagination mechanism along with the max-results parameter.
+ :type: start_index: int
+ """
+
+ self.log.info("Retrieving accounts list")
+
+ # pylint: disable=no-member
+ accounts = (
+ self.get_conn()
+ .management()
+ .accounts()
+ .list()
+ .execute(num_retries=self.num_retries)
+ )
+ return self._list_emails_from_response(accounts)
+
+ @staticmethod
+ def _list_emails_from_response(accounts) -> list:
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] rebeccasg commented on a change in pull request #7630:
[AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
rebeccasg commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r391261027
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
Review comment:
Update the comment here to note that this is Management API
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r389245872
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
+ def list_accounts(self, max_results: int, start_index: int) -> list:
Review comment:
```suggestion
def list_accounts(self, max_results: int, start_index: int) -> List[Dict[str, Any]]:
```
Let's try to avoid generic types ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r390965342
##########
File path: airflow/providers/google/marketing_platform/operators/analytics.py
##########
@@ -0,0 +1,67 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Google Analytics 360 operators.
+"""
+
+from airflow.models import BaseOperator
+from airflow.providers.google.marketing_platform.hooks.analytics import GoogleAnalyticsHook
+from airflow.utils.decorators import apply_defaults
+
+
+class GoogleAnalyticsListAccountsOperator(BaseOperator):
+ """
+ Lists all accounts to which the user has access.
+
+ .. seealso::
+ Check official API docs:
+ https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/accounts/list
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:GoogleAnalyticsListAccountsOperator`
+
+ :param max_results: The maximum number of accounts to include in this response.
+ :type: max_results: int
+ :param: start_index: An index of the first account to retrieve
+ Use this parameter as a pagination mechanism along with the max-results parameter.
+ :type: start_index: int
Review comment:
It already fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r391460324
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ def list_accounts(self) -> List[Dict[str, Any]]:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ More information about API:
+ http://googleapis.github.io/google-api-python-client/docs/dyn/analytics_v3.management.accounts.html
+
+ """
+
+ self.log.info("Retrieving accounts list...")
+
+ # pylint: disable=no-member
+ result = [] # type: List[Dict]
+ conn = self.get_conn()
+ # return self._list_emails_from_response(accounts)
+ while True:
+ request = conn.management().accounts().list(start_index=len(result) + 1)
+ response = request.execute(num_retries=self.num_retries)
+ result.extend(response.get('items', []))
+ if response["totalResults"] >= len(result):
Review comment:
One more fix no is `==`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r391685552
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
Review comment:
I don't think it's necessary. The `get_conn` method return resource for the whole GA360 so all methods can be implemented not only those for management API.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7630:
[AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#issuecomment-596135065
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=h1) Report
> Merging [#7630](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/137896f326cd29b59902a887e4c4e58f940ff62b&el=desc) will **decrease** coverage by `1.02%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7630 +/- ##
==========================================
- Coverage 87.00% 85.98% -1.03%
==========================================
Files 904 909 +5
Lines 43728 43954 +226
==========================================
- Hits 38046 37794 -252
- Misses 5682 6160 +478
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...rketing\_platform/example\_dags/example\_analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9leGFtcGxlX2RhZ3MvZXhhbXBsZV9hbmFseXRpY3MucHk=) | `100.00% <100.00%> (ø)` | |
| [...iders/google/marketing\_platform/hooks/analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9ob29rcy9hbmFseXRpY3MucHk=) | `100.00% <100.00%> (ø)` | |
| [...s/google/marketing\_platform/operators/analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9vcGVyYXRvcnMvYW5hbHl0aWNzLnB5) | `100.00% <100.00%> (ø)` | |
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0.00%> (-72.16%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0.00%> (-54.93%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| ... and [33 more](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=footer). Last update [137896f...d7eb2ee](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7630:
[AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r388867299
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,87 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
+ def list_accounts(self, max_results: int, start_index: int) -> list:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ :param max_results: The maximum number of accounts to include in this response.
+ :type: max_result: int
+ :param start_index: An index of the first account to retrieve
+ Use this parameter as a pagination mechanism along with the max-results parameter.
Review comment:
```suggestion
Use this parameter as a pagination mechanism along with the max-results parameter.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r391687662
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,80 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ def list_accounts(self) -> List[Dict[str, Any]]:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ More information about API for python client:
+ http://googleapis.github.io/google-api-python-client/docs/dyn/analytics_v3.management.accounts.html
+
+ """
+
+ self.log.info("Retrieving accounts list...")
+ result = [] # type: List[Dict]
+ conn = self.get_conn()
+ # pylint: disable=no-member
+ accounts = conn.management().accounts()
+ while True:
+ # start index has value 1
+ request = accounts.list(start_index=len(result) + 1)
+ response = request.execute(num_retries=self.num_retries)
+ result.extend(response.get('items', []))
+ # result is the number of fetched accounts from Analytics
+ # when all accounts will be add to the result
+ # the loop will be break
+ if response["totalResults"] == len(result):
Review comment:
Why? Don't we want to quite once all result has been collected? @michalslowikowski00 does this `totalResults` changes with subsequent request?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r390833460
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ def list_accounts(self) -> List[Dict[str, Any]]:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ More information about API:
+ http://googleapis.github.io/google-api-python-client/docs/dyn/analytics_v3.management.accounts.html
+
+ """
+
+ self.log.info("Retrieving accounts list...")
+
+ # pylint: disable=no-member
Review comment:
I think this should be in L72
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r390834950
##########
File path: airflow/providers/google/marketing_platform/operators/analytics.py
##########
@@ -0,0 +1,67 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Google Analytics 360 operators.
+"""
+
+from airflow.models import BaseOperator
+from airflow.providers.google.marketing_platform.hooks.analytics import GoogleAnalyticsHook
+from airflow.utils.decorators import apply_defaults
+
+
+class GoogleAnalyticsListAccountsOperator(BaseOperator):
+ """
+ Lists all accounts to which the user has access.
+
+ .. seealso::
+ Check official API docs:
+ https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/accounts/list
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:GoogleAnalyticsListAccountsOperator`
+
+ :param max_results: The maximum number of accounts to include in this response.
+ :type: max_results: int
+ :param: start_index: An index of the first account to retrieve
+ Use this parameter as a pagination mechanism along with the max-results parameter.
+ :type: start_index: int
+ :param api_version: The version of the api that will be requested for example 'v3'.
+ :type api_version: str
+ :param gcp_conn_id: The connection ID to use when fetching connection info.
+ :type gcp_conn_id: str
+ """
+
+ template_fields = ("api_version", "gcp_connection_id",)
+
+ @apply_defaults
+ def __init__(self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs):
+ super().__init__(*args, **kwargs)
+
+ self.api_version = api_version
+ self.gcp_connection_id = gcp_connection_id
+
+ def execute(self, context):
+ hook = GoogleAnalyticsHook(api_version=self.api_version,
+ gcp_connection_id=self.gcp_connection_id)
+ result = hook.list_accounts()
Review comment:
Should we add some info logging like `Requesting accounts...`, `Retrieved {no} accounts`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r391752126
##########
File path: airflow/providers/google/marketing_platform/operators/analytics.py
##########
@@ -0,0 +1,67 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Google Analytics 360 operators.
+"""
+
+from airflow.models import BaseOperator
+from airflow.providers.google.marketing_platform.hooks.analytics import GoogleAnalyticsHook
+from airflow.utils.decorators import apply_defaults
+
+
+class GoogleAnalyticsListAccountsOperator(BaseOperator):
+ """
+ Lists all accounts to which the user has access.
+
+ .. seealso::
+ Check official API docs:
+ https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/accounts/list
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:GoogleAnalyticsListAccountsOperator`
+
+ :param max_results: The maximum number of accounts to include in this response.
+ :type: max_results: int
+ :param: start_index: An index of the first account to retrieve
+ Use this parameter as a pagination mechanism along with the max-results parameter.
+ :type: start_index: int
+ :param api_version: The version of the api that will be requested for example 'v3'.
+ :type api_version: str
+ :param gcp_conn_id: The connection ID to use when fetching connection info.
+ :type gcp_conn_id: str
+ """
+
+ template_fields = ("api_version", "gcp_connection_id",)
+
+ @apply_defaults
+ def __init__(self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs):
+ super().__init__(*args, **kwargs)
+
+ self.api_version = api_version
+ self.gcp_connection_id = gcp_connection_id
+
+ def execute(self, context):
+ hook = GoogleAnalyticsHook(api_version=self.api_version,
+ gcp_connection_id=self.gcp_connection_id)
+ result = hook.list_accounts()
Review comment:
Perfect.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r391792290
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,80 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ def list_accounts(self) -> List[Dict[str, Any]]:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ More information about API for python client:
+ http://googleapis.github.io/google-api-python-client/docs/dyn/analytics_v3.management.accounts.html
+
+ """
+
+ self.log.info("Retrieving accounts list...")
+ result = [] # type: List[Dict]
+ conn = self.get_conn()
+ # pylint: disable=no-member
+ accounts = conn.management().accounts()
+ while True:
+ # start index has value 1
+ request = accounts.list(start_index=len(result) + 1)
+ response = request.execute(num_retries=self.num_retries)
+ result.extend(response.get('items', []))
+ # result is the number of fetched accounts from Analytics
+ # when all accounts will be add to the result
+ # the loop will be break
+ if response["totalResults"] == len(result):
Review comment:
We discussed it offline. `<=` condition will be safer then `==`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r389258760
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
+ def list_accounts(self, max_results: int, start_index: int) -> list:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ :param max_results: The maximum number of accounts to include in this response.
+ :type: max_result: int
+ :param start_index: An index of the first account to retrieve
+ Use this parameter as a pagination mechanism along with the max-results parameter.
+ :type: start_index: int
+ """
+
+ self.log.info("Retrieving accounts list")
+
+ # pylint: disable=no-member
+ accounts = (
+ self.get_conn()
+ .management()
+ .accounts()
+ .list()
+ .execute(num_retries=self.num_retries)
+ )
+ return self._list_emails_from_response(accounts)
+
+ @staticmethod
+ def _list_emails_from_response(accounts) -> list:
+ """
+ :param accounts: GET request to Analytics service
+ :type accounts: list
+ """
+
+ result = [value for key, value in accounts.items() if key == "username"]
+ return result
Review comment:
That's nit, but I like it. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r391768447
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,80 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ def list_accounts(self) -> List[Dict[str, Any]]:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ More information about API for python client:
+ http://googleapis.github.io/google-api-python-client/docs/dyn/analytics_v3.management.accounts.html
+
+ """
+
+ self.log.info("Retrieving accounts list...")
+ result = [] # type: List[Dict]
+ conn = self.get_conn()
+ # pylint: disable=no-member
+ accounts = conn.management().accounts()
Review comment:
It works when the `# pylint: disable=no-member` is above the code. But sometimes it doesn't.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk merged pull request #7630: [AIRFLOW-6724] Add
Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r389305653
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
+ def list_accounts(self, max_results: int, start_index: int) -> list:
Review comment:
Got it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r388868149
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,87 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
+ def list_accounts(self, max_results: int, start_index: int) -> list:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ :param max_results: The maximum number of accounts to include in this response.
+ :type: max_result: int
+ :param start_index: An index of the first account to retrieve
+ Use this parameter as a pagination mechanism along with the max-results parameter.
Review comment:
Thank you :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r390833963
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ def list_accounts(self) -> List[Dict[str, Any]]:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ More information about API:
+ http://googleapis.github.io/google-api-python-client/docs/dyn/analytics_v3.management.accounts.html
+
+ """
+
+ self.log.info("Retrieving accounts list...")
+
+ # pylint: disable=no-member
+ result = [] # type: List[Dict]
+ conn = self.get_conn()
+ # return self._list_emails_from_response(accounts)
+ while True:
+ request = conn.management().accounts().list(start_index=len(result) + 1)
+ response = request.execute(num_retries=self.num_retries)
+ result.extend(response.get('items', []))
+ if response["totalResults"] >= len(result):
Review comment:
```suggestion
if response["totalResults"] == len(result):
```
We want to break the loop once we get all results, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r391685868
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,80 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ def list_accounts(self) -> List[Dict[str, Any]]:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ More information about API for python client:
+ http://googleapis.github.io/google-api-python-client/docs/dyn/analytics_v3.management.accounts.html
+
+ """
+
+ self.log.info("Retrieving accounts list...")
+ result = [] # type: List[Dict]
+ conn = self.get_conn()
+ # pylint: disable=no-member
+ accounts = conn.management().accounts()
Review comment:
```suggestion
accounts = conn.management().accounts() # pylint: disable=no-member
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r391688028
##########
File path: airflow/providers/google/marketing_platform/operators/analytics.py
##########
@@ -0,0 +1,67 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Google Analytics 360 operators.
+"""
+
+from airflow.models import BaseOperator
+from airflow.providers.google.marketing_platform.hooks.analytics import GoogleAnalyticsHook
+from airflow.utils.decorators import apply_defaults
+
+
+class GoogleAnalyticsListAccountsOperator(BaseOperator):
+ """
+ Lists all accounts to which the user has access.
+
+ .. seealso::
+ Check official API docs:
+ https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/accounts/list
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:GoogleAnalyticsListAccountsOperator`
+
+ :param max_results: The maximum number of accounts to include in this response.
+ :type: max_results: int
+ :param: start_index: An index of the first account to retrieve
+ Use this parameter as a pagination mechanism along with the max-results parameter.
+ :type: start_index: int
+ :param api_version: The version of the api that will be requested for example 'v3'.
+ :type api_version: str
+ :param gcp_conn_id: The connection ID to use when fetching connection info.
+ :type gcp_conn_id: str
+ """
+
+ template_fields = ("api_version", "gcp_connection_id",)
+
+ @apply_defaults
+ def __init__(self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs):
+ super().__init__(*args, **kwargs)
+
+ self.api_version = api_version
+ self.gcp_connection_id = gcp_connection_id
+
+ def execute(self, context):
+ hook = GoogleAnalyticsHook(api_version=self.api_version,
+ gcp_connection_id=self.gcp_connection_id)
+ result = hook.list_accounts()
Review comment:
It's enough to have it in hook
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7630:
[AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#issuecomment-596135065
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=h1) Report
> Merging [#7630](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/137896f326cd29b59902a887e4c4e58f940ff62b&el=desc) will **decrease** coverage by `0.27%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7630 +/- ##
==========================================
- Coverage 87.00% 86.72% -0.28%
==========================================
Files 904 909 +5
Lines 43728 43899 +171
==========================================
+ Hits 38046 38072 +26
- Misses 5682 5827 +145
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...rketing\_platform/example\_dags/example\_analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9leGFtcGxlX2RhZ3MvZXhhbXBsZV9hbmFseXRpY3MucHk=) | `100.00% <100.00%> (ø)` | |
| [...iders/google/marketing\_platform/hooks/analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9ob29rcy9hbmFseXRpY3MucHk=) | `100.00% <100.00%> (ø)` | |
| [...s/google/marketing\_platform/operators/analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9vcGVyYXRvcnMvYW5hbHl0aWNzLnB5) | `100.00% <100.00%> (ø)` | |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: |
| [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `73.25% <0.00%> (-12.80%)` | :arrow_down: |
| [...oviders/amazon/aws/operators/sagemaker\_training.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9vcGVyYXRvcnMvc2FnZW1ha2VyX3RyYWluaW5nLnB5) | `97.29% <0.00%> (-2.71%)` | :arrow_down: |
| ... and [14 more](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=footer). Last update [137896f...b6997b3](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r389245961
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
+ def list_accounts(self, max_results: int, start_index: int) -> list:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ :param max_results: The maximum number of accounts to include in this response.
+ :type: max_result: int
+ :param start_index: An index of the first account to retrieve
+ Use this parameter as a pagination mechanism along with the max-results parameter.
+ :type: start_index: int
+ """
+
+ self.log.info("Retrieving accounts list")
+
+ # pylint: disable=no-member
+ accounts = (
+ self.get_conn()
+ .management()
+ .accounts()
+ .list()
+ .execute(num_retries=self.num_retries)
+ )
+ return self._list_emails_from_response(accounts)
+
+ @staticmethod
+ def _list_emails_from_response(accounts) -> list:
Review comment:
```suggestion
def _list_emails_from_response(accounts: Dict[str, Any]) -> List[?]:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7630:
[AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r391651107
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,80 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ def list_accounts(self) -> List[Dict[str, Any]]:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ More information about API for python client:
+ http://googleapis.github.io/google-api-python-client/docs/dyn/analytics_v3.management.accounts.html
+
+ """
+
+ self.log.info("Retrieving accounts list...")
+ result = [] # type: List[Dict]
+ conn = self.get_conn()
+ # pylint: disable=no-member
+ accounts = conn.management().accounts()
+ while True:
+ # start index has value 1
+ request = accounts.list(start_index=len(result) + 1)
+ response = request.execute(num_retries=self.num_retries)
+ result.extend(response.get('items', []))
+ # result is the number of fetched accounts from Analytics
+ # when all accounts will be add to the result
+ # the loop will be break
+ if response["totalResults"] == len(result):
Review comment:
```suggestion
if response["totalResults"] <= len(result):
```
Otherwise, we can fall into an infinite loop.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7630:
[AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#issuecomment-596135065
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=h1) Report
> Merging [#7630](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/137896f326cd29b59902a887e4c4e58f940ff62b?src=pr&el=desc) will **decrease** coverage by `27.15%`.
> The diff coverage is `54.16%`.
[](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7630 +/- ##
===========================================
- Coverage 87% 59.85% -27.16%
===========================================
Files 904 907 +3
Lines 43728 43776 +48
===========================================
- Hits 38046 26202 -11844
- Misses 5682 17574 +11892
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...rketing\_platform/example\_dags/example\_analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9leGFtcGxlX2RhZ3MvZXhhbXBsZV9hbmFseXRpY3MucHk=) | `100% <100%> (ø)` | |
| [...iders/google/marketing\_platform/hooks/analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9ob29rcy9hbmFseXRpY3MucHk=) | `26.92% <26.92%> (ø)` | |
| [...s/google/marketing\_platform/operators/analytics.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9vcGVyYXRvcnMvYW5hbHl0aWNzLnB5) | `80% <80%> (ø)` | |
| [airflow/providers/amazon/aws/hooks/kinesis.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9raW5lc2lzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/providers/apache/livy/sensors/livy.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2xpdnkvc2Vuc29ycy9saXZ5LnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [...providers/amazon/aws/sensors/sagemaker\_training.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9zZW5zb3JzL3NhZ2VtYWtlcl90cmFpbmluZy5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/providers/microsoft/azure/operators/adx.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbWljcm9zb2Z0L2F6dXJlL29wZXJhdG9ycy9hZHgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [...irflow/providers/amazon/aws/hooks/batch\_waiters.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9iYXRjaF93YWl0ZXJzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| ... and [306 more](https://codecov.io/gh/apache/airflow/pull/7630/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=footer). Last update [137896f...b6997b3](https://codecov.io/gh/apache/airflow/pull/7630?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r389245948
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
+ def list_accounts(self, max_results: int, start_index: int) -> list:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ :param max_results: The maximum number of accounts to include in this response.
+ :type: max_result: int
+ :param start_index: An index of the first account to retrieve
+ Use this parameter as a pagination mechanism along with the max-results parameter.
+ :type: start_index: int
+ """
+
+ self.log.info("Retrieving accounts list")
+
+ # pylint: disable=no-member
+ accounts = (
+ self.get_conn()
+ .management()
+ .accounts()
+ .list()
+ .execute(num_retries=self.num_retries)
+ )
+ return self._list_emails_from_response(accounts)
+
+ @staticmethod
+ def _list_emails_from_response(accounts) -> list:
+ """
+ :param accounts: GET request to Analytics service
+ :type accounts: list
+ """
+
+ result = [value for key, value in accounts.items() if key == "username"]
+ return result
Review comment:
```suggestion
return [value for key, value in accounts.items() if key == "username"]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r389305801
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
Review comment:
I agree, it is confusing. I will remove these args.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] michalslowikowski00 commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
michalslowikowski00 commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r392187313
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,80 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ def list_accounts(self) -> List[Dict[str, Any]]:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ More information about API for python client:
+ http://googleapis.github.io/google-api-python-client/docs/dyn/analytics_v3.management.accounts.html
+
+ """
+
+ self.log.info("Retrieving accounts list...")
+ result = [] # type: List[Dict]
+ conn = self.get_conn()
+ # pylint: disable=no-member
+ accounts = conn.management().accounts()
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request
#7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r389266999
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ # pylint: disable=unused-argument
Review comment:
I think we should remove those args if we are not using them. Otherwise, the function interface is misleading. We can add those args once we get info how to put them in API call :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7630:
[AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #7630: [AIRFLOW-6724] Add Google Analytics 360 Accounts Retrieve Operator
URL: https://github.com/apache/airflow/pull/7630#discussion_r391695761
##########
File path: airflow/providers/google/marketing_platform/hooks/analytics.py
##########
@@ -0,0 +1,80 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import Any, Dict, List
+
+from googleapiclient.discovery import Resource, build
+
+from airflow.providers.google.cloud.hooks.base import CloudBaseHook
+
+
+class GoogleAnalyticsHook(CloudBaseHook):
+ """
+ Hook for Google Analytics 360.
+ """
+
+ def __init__(
+ self,
+ api_version: str = "v3",
+ gcp_connection_id: str = "google cloud default",
+ *args,
+ **kwargs
+ ):
+ super().__init__(*args, **kwargs)
+ self.api_version = api_version
+ self.gcp_connection_is = gcp_connection_id
+ self._conn = None
+
+ def get_conn(self) -> Resource:
+ """
+ Retrieves connection to Google Analytics 360.
+ """
+ if not self._conn:
+ http_authorized = self._authorize()
+ self._conn = build(
+ "analytics",
+ self.api_version,
+ http=http_authorized,
+ cache_discovery=False,
+ )
+ return self._conn
+
+ def list_accounts(self) -> List[Dict[str, Any]]:
+ """
+ Lists accounts list from Google Analytics 360.
+
+ More information about API for python client:
+ http://googleapis.github.io/google-api-python-client/docs/dyn/analytics_v3.management.accounts.html
+
+ """
+
+ self.log.info("Retrieving accounts list...")
+ result = [] # type: List[Dict]
+ conn = self.get_conn()
+ # pylint: disable=no-member
+ accounts = conn.management().accounts()
+ while True:
+ # start index has value 1
+ request = accounts.list(start_index=len(result) + 1)
+ response = request.execute(num_retries=self.num_retries)
+ result.extend(response.get('items', []))
+ # result is the number of fetched accounts from Analytics
+ # when all accounts will be add to the result
+ # the loop will be break
+ if response["totalResults"] == len(result):
Review comment:
This may theoretically change because we don't use nextPageToken. It is better to avoid situations that require equality in the loop. This should not happen, but endless loops are a serious problem and it is better to avoid them also by avoiding certain situations.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services