You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/03/30 09:30:47 UTC
[GitHub] [airflow] randr97 opened a new pull request #8008: Custom Facebook
Ads Operator #7887
randr97 opened a new pull request #8008: Custom Facebook Ads Operator #7887
URL: https://github.com/apache/airflow/pull/8008
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r404602700
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,119 @@
+#
Review comment:
Will fix it according to the AIP. But what if tomorrow one would want integration with AWS or other cloud plf?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/cc9b1bc78df1e82da37b885d2a8aaf5779c6209d&el=desc) will **decrease** coverage by `0.33%`.
> The diff coverage is `82.14%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
==========================================
- Coverage 87.27% 86.94% -0.34%
==========================================
Files 937 940 +3
Lines 45549 45661 +112
==========================================
- Hits 39753 39700 -53
- Misses 5796 5961 +165
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `95.07% <ø> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `69.23% <69.23%> (ø)` | |
| [...providers/facebook/ads/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2V4YW1wbGVfZGFncy9leGFtcGxlX2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [airflow/providers/facebook/ads/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL29wZXJhdG9ycy9hZHMucHk=) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.36% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0.00%> (-45.66%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
| ... and [6 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [cc9b1bc...f22ea1c](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r404423523
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,108 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_ads_conn_id", "bucket_name", "object_name")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket_name: str,
+ object_name: str,
+ gcp_conn_id: str = "google_cloud_default",
+ facebook_ads_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ fields: Optional[List[str]] = None,
+ params: Optional[Dict[str, Any]] = None,
+ gzip: bool = False,
+ *args,
+ **kwargs,
+ ) -> None:
+ super().__init__(*args, **kwargs)
+ self.bucket_name = bucket_name
+ self.object_name = object_name
+ self.gcp_conn_id = gcp_conn_id
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.fields = fields
+ self.params = params # type:ignore
+ self.gzip = gzip
+
+ def execute(self, context: Dict):
+ service = FacebookAdsReportingHook(facebook_ads_conn_id=self.facebook_ads_conn_id,
+ api_version=self.api_version)
+ rows = service.bulk_facebook_report(params=self.params, fields=self.fields)
+
+ converted_rows = [dict(row) for row in rows]
Review comment:
Can you restore the ability to save as JSON? I think CSV and JSON are very useful and should be the option of choice.
Here is an example of the last operator I did:
https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/example_dags/example_presto_to_gcs.py#L135
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r404424465
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,119 @@
+#
Review comment:
The file name should be called facebook_ads_to_gcs and should be in the `airflow/providers/google/cloud/operators` package.
More information: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402736678
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,152 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from cached_property import cached_property
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookError, FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = "Job Completed"
+JOB_FAILED = "Job Failed"
+JOB_SKIPPED = "Job Skipped"
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ config = self.facebook_ads_config
+ try:
+ return FacebookAdsApi.init(app_id=config["app_id"],
+ app_secret=config["app_secret"],
+ access_token=config["access_token"],
+ account_id=config["account_id"],
+ api_version=self.api_version)
+ except AirflowException as error:
+ message = "Missing config data:\n {error}".format(error=error)
+ self._error_handler(message)
+ raise
+
+ @cached_property
+ def facebook_ads_config(self) -> None:
+ """
+ Gets Facebook ads connection from meta db and sets
+ facebook_ads_config attribute with returned config file
+ """
+ conn = self.get_connection(self.facebook_ads_conn_id)
+ if "facebook_ads_client" not in conn.extra_dejson:
+ raise AirflowException("facebook_ads_client not found in extra field")
+ return conn.extra_dejson.get("facebook_ads_client")
+
+ def bulk_facebook_report(
+ self,
+ params: Optional[Dict[str, Any]] = None,
+ fields: Optional[List[str]] = None,
+ ) -> List[AdsInsights]:
+ """
+ Pulls data from the Facebook Ads API
+
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+
+ :return: Facebook Ads API response, converted to Facebook Ads Row objects
+ :rtype: List[AdsInsights]
+ """
+ api = self._get_service()
+ params = params if params else self.default_params
+ ad_account = AdAccount(api.get_default_account_id(), api=api)
+ _async = ad_account.get_insights(params=params, fields=fields, is_async=True)
+ try:
+ while _async.api_get()["async_status"] != JOB_COMPLETED:
Review comment:
```
from facebookads.adobjects.adreportrun import AdReportRun
```
```suggestion
while _async.api_get()[AdReportRun.Field.async_status] != JOB_COMPLETED:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r404426263
##########
File path: docs/operators-and-hooks-ref.rst
##########
@@ -1014,6 +1014,12 @@ These integrations allow you to perform various operations within various servic
- :mod:`airflow.providers.discord.operators.discord_webhook`
-
+ * - `Facebook Ads <http://business.facebook.com>`__
+ -
+ - :mod:`airflow.providers.facebook.ads.hooks.ads`
+ - :mod:`airflow.providers.facebook.ads.operators.ads`
Review comment:
Can you add it in 2 sections (source and destination) for transfer operators?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6c273466d598d7bcfb7c21feafcccb07cc4230fb&el=desc) will **decrease** coverage by `27.96%`.
> The diff coverage is `57.25%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
===========================================
- Coverage 88.29% 60.33% -27.97%
===========================================
Files 935 938 +3
Lines 45170 45294 +124
===========================================
- Hits 39885 27328 -12557
- Misses 5285 17966 +12681
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `93.66% <ø> (-1.41%)` | :arrow_down: |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `37.09% <37.09%> (ø)` | |
| [airflow/providers/facebook/ads/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL29wZXJhdG9ycy9hZHMucHk=) | `62.16% <62.16%> (ø)` | |
| [...providers/facebook/ads/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2V4YW1wbGVfZGFncy9leGFtcGxlX2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.37% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/providers/exasol/operators/exasol.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZXhhc29sL29wZXJhdG9ycy9leGFzb2wucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/providers/amazon/aws/hooks/kinesis.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9ob29rcy9raW5lc2lzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/providers/apache/livy/sensors/livy.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2xpdnkvc2Vuc29ycy9saXZ5LnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/providers/amazon/aws/sensors/redshift.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9zZW5zb3JzL3JlZHNoaWZ0LnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/providers/mysql/operators/s3\_to\_mysql.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL3MzX3RvX215c3FsLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [328 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [6c27346...2344438](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401837518
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_ads_conn_id", "bucket", "obj")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket: str,
+ obj: str,
+ gcp_conn_id: str = "google_cloud_default",
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ fields: Optional[List[str]] = None,
+ params: Optional[Dict[str, Any]] = None,
+ gzip: bool = False,
+ *args,
+ **kwargs,
+ ) -> None:
+ super().__init__(*args, **kwargs)
+ self.bucket = bucket
+ self.obj = obj
+ self.gcp_conn_id = gcp_conn_id
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.fields = fields
+ self.params = params # type:ignore
+ self.gzip = gzip
+
+ def execute(self, context: Dict):
+ service = FacebookAdsReportingHook(facebook_ads_conn_id=self.facebook_ads_conn_id,
+ api_version=self.api_version)
+ rows = service.bulk_facebook_report(params=self.params, fields=self.fields)
+ try:
+ converted_rows = [dict(row) for row in rows]
+ except Exception as e:
+ self.log.error("An error occurred in converting the Facebook Ad Rows. \n Error %s", e)
+ raise
Review comment:
@turbaszek looks like I handling errors in the hook, thanks for pointing it out. I don't think the exception is required. Before I would iterate over the cursor in the operator, but now its just type casting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8008: Custom Facebook Ads
Operator
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-612833979
I think there was a change in MyPy in the meantime and you will have to wait until #8267 is merged (it fixes the problem) and rebase again I am afraid. Sorry for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401198133
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_ads_conn_id", "bucket", "obj")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket: str,
Review comment:
```suggestion
bucket_name: str,
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402748027
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,152 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from cached_property import cached_property
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookError, FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = "Job Completed"
+JOB_FAILED = "Job Failed"
+JOB_SKIPPED = "Job Skipped"
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.default_params = {
Review comment:
The exception is a better idea.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401824302
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ :return: list of Facebook Ads AdsInsights object(s)
+ :rtype: list[AdsInsights]
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.facebook_ads_config: Dict[str, Any] = {}
+ self.api_version = api_version
+ self.ad_account_id: str = ""
+ self.app_id: str = ""
+ self.app_secret: str = ""
+ self.access_token: str = ""
Review comment:
Actually i'm removing these class attrs. Can be done without them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/45c8983306ab1c54abdacd8f870e790fad25cb37&el=desc) will **decrease** coverage by `0.44%`.
> The diff coverage is `94.11%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
==========================================
- Coverage 88.45% 88.00% -0.45%
==========================================
Files 937 940 +3
Lines 45234 45353 +119
==========================================
- Hits 40011 39913 -98
- Misses 5223 5440 +217
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `95.07% <ø> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `87.93% <87.93%> (ø)` | |
| [...le/facebook\_ads\_to\_gcs/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3MvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYWRzLnB5) | `100.00% <100.00%> (ø)` | |
| [...viders/google/facebook\_ads\_to\_gcs/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3Mvb3BlcmF0b3JzL2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.37% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0.00%> (-45.66%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `56.00% <0.00%> (-44.00%)` | :arrow_down: |
| ... and [10 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [45c8983...ff2fcf6](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8008: Custom Facebook Ads
Operator
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-612984630
OK. You can rebase now - it should be fine :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401386387
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_ads_conn_id", "bucket", "obj")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket: str,
+ obj: str,
+ gcp_conn_id: str = "google_cloud_default",
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ fields: Optional[List[str]] = None,
+ params: Optional[Dict[str, Any]] = None,
+ gzip: bool = False,
+ *args,
+ **kwargs,
+ ) -> None:
+ super().__init__(*args, **kwargs)
+ self.bucket = bucket
+ self.obj = obj
+ self.gcp_conn_id = gcp_conn_id
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.fields = fields
+ self.params = params # type:ignore
+ self.gzip = gzip
+
+ def execute(self, context: Dict):
+ service = FacebookAdsReportingHook(facebook_ads_conn_id=self.facebook_ads_conn_id,
+ api_version=self.api_version)
+ rows = service.bulk_facebook_report(params=self.params, fields=self.fields)
+ try:
+ converted_rows = [dict(row) for row in rows]
+ except Exception as e:
+ self.log.error("An error occurred in converting the Facebook Ad Rows. \n Error %s", e)
+ raise
+
+ with tempfile.NamedTemporaryFile("w", suffix=".json") as json_file:
+ json.dump(converted_rows, json_file)
+ json_file.flush()
+ hook = GCSHook(gcp_conn_id=self.gcp_conn_id)
+ hook.upload(
+ bucket_name=self.bucket,
+ object_name=self.obj,
+ filename=json_file.name,
+ gzip=self.gzip,
+ )
+ self.log.info("%s uploaded to GCS", json_file.name)
Review comment:
Should we consider returning URI to the uploaded file for example `return f"gs://{self.bucket}/{self.obj}"`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on issue #8008: Custom Facebook Ads
Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-612806356
> Hey @randr97 -> I'd love to merge this one. Can you please rebase and regenerate the requirements?
Sure thing, will do
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8008: Custom Facebook Ads
Operator
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-613432339
Thanks @randr97 !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/510a377b50ae757ce3c5d5b455c4c77d7dc9068e&el=desc) will **decrease** coverage by `0.38%`.
> The diff coverage is `94.11%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
==========================================
- Coverage 88.30% 87.91% -0.39%
==========================================
Files 935 938 +3
Lines 45243 45362 +119
==========================================
- Hits 39951 39881 -70
- Misses 5292 5481 +189
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `95.07% <ø> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `87.93% <87.93%> (ø)` | |
| [...le/facebook\_ads\_to\_gcs/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3MvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYWRzLnB5) | `100.00% <100.00%> (ø)` | |
| [...viders/google/facebook\_ads\_to\_gcs/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3Mvb3BlcmF0b3JzL2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.37% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0.00%> (-45.66%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `56.00% <0.00%> (-44.00%)` | :arrow_down: |
| ... and [10 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [510a377...5234efa](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/510a377b50ae757ce3c5d5b455c4c77d7dc9068e&el=desc) will **decrease** coverage by `0.22%`.
> The diff coverage is `94.11%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
==========================================
- Coverage 88.30% 88.07% -0.23%
==========================================
Files 935 938 +3
Lines 45243 45362 +119
==========================================
+ Hits 39951 39952 +1
- Misses 5292 5410 +118
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `95.07% <ø> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `87.93% <87.93%> (ø)` | |
| [...le/facebook\_ads\_to\_gcs/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3MvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYWRzLnB5) | `100.00% <100.00%> (ø)` | |
| [...viders/google/facebook\_ads\_to\_gcs/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3Mvb3BlcmF0b3JzL2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.37% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: |
| ... and [6 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [510a377...5234efa](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/cc9b1bc78df1e82da37b885d2a8aaf5779c6209d&el=desc) will **decrease** coverage by `0.27%`.
> The diff coverage is `82.14%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
==========================================
- Coverage 87.27% 87.00% -0.28%
==========================================
Files 937 940 +3
Lines 45549 45661 +112
==========================================
- Hits 39753 39726 -27
- Misses 5796 5935 +139
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `95.07% <ø> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `69.23% <69.23%> (ø)` | |
| [...providers/facebook/ads/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2V4YW1wbGVfZGFncy9leGFtcGxlX2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [airflow/providers/facebook/ads/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL29wZXJhdG9ycy9hZHMucHk=) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.36% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: |
| ... and [3 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [cc9b1bc...f22ea1c](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401213486
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ :return: list of Facebook Ads AdsInsights object(s)
+ :rtype: list[AdsInsights]
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.facebook_ads_config: Dict[str, Any] = {}
+ self.api_version = api_version
+ self.ad_account_id: str = ""
+ self.app_id: str = ""
+ self.app_secret: str = ""
+ self.access_token: str = ""
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ self._get_config()
+ for key, val in self.facebook_ads_config.items():
Review comment:
Yep, will do so!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/510a377b50ae757ce3c5d5b455c4c77d7dc9068e&el=desc) will **decrease** coverage by `55.11%`.
> The diff coverage is `31.09%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
===========================================
- Coverage 88.30% 33.19% -55.12%
===========================================
Files 935 938 +3
Lines 45243 45362 +119
===========================================
- Hits 39951 15057 -24894
- Misses 5292 30305 +25013
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `53.52% <ø> (-41.55%)` | :arrow_down: |
| [...le/facebook\_ads\_to\_gcs/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3MvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYWRzLnB5) | `0.00% <0.00%> (ø)` | |
| [...viders/google/facebook\_ads\_to\_gcs/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3Mvb3BlcmF0b3JzL2Fkcy5weQ==) | `36.11% <36.11%> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `39.65% <39.65%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.37% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [815 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [510a377...5234efa](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8008: Custom Facebook Ads
Operator
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-612794535
Hey @randr97 -> I'd love to merge this one. Can you please rebase and regenerate the requirements?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402735132
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,152 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from cached_property import cached_property
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookError, FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = "Job Completed"
+JOB_FAILED = "Job Failed"
+JOB_SKIPPED = "Job Skipped"
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.default_params = {
Review comment:
Are these parameters described somewhere in the documentation? We should not make additional assumptions because users may not be aware of this. If I understand correctly, users will only get some data when they should get the error because they have not passed all the parameters.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402733145
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,152 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from cached_property import cached_property
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookError, FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = "Job Completed"
+JOB_FAILED = "Job Failed"
+JOB_SKIPPED = "Job Skipped"
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ config = self.facebook_ads_config
+ try:
+ return FacebookAdsApi.init(app_id=config["app_id"],
+ app_secret=config["app_secret"],
+ access_token=config["access_token"],
+ account_id=config["account_id"],
+ api_version=self.api_version)
+ except AirflowException as error:
+ message = "Missing config data:\n {error}".format(error=error)
+ self._error_handler(message)
+ raise
+
+ @cached_property
+ def facebook_ads_config(self) -> None:
+ """
+ Gets Facebook ads connection from meta db and sets
+ facebook_ads_config attribute with returned config file
+ """
+ conn = self.get_connection(self.facebook_ads_conn_id)
+ if "facebook_ads_client" not in conn.extra_dejson:
+ raise AirflowException("facebook_ads_client not found in extra field")
+ return conn.extra_dejson.get("facebook_ads_client")
+
+ def bulk_facebook_report(
+ self,
+ params: Optional[Dict[str, Any]] = None,
+ fields: Optional[List[str]] = None,
+ ) -> List[AdsInsights]:
+ """
+ Pulls data from the Facebook Ads API
+
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+
+ :return: Facebook Ads API response, converted to Facebook Ads Row objects
+ :rtype: List[AdsInsights]
+ """
+ api = self._get_service()
+ params = params if params else self.default_params
+ ad_account = AdAccount(api.get_default_account_id(), api=api)
+ _async = ad_account.get_insights(params=params, fields=fields, is_async=True)
+ try:
+ while _async.api_get()["async_status"] != JOB_COMPLETED:
Review comment:
Can you check the following 3 states here?
- Job Not Started,
- Job Started,
- Job Running
-
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401380068
##########
File path: airflow/providers/facebook/ads/example_dags/example_ads.py
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example Airflow DAG that shows how to use FacebookAdsReportToGcsOperator.
+"""
+import os
+
+from facebook_business.adobjects.adsinsights import AdsInsights
+
+from airflow import models
+from airflow.providers.facebook.ads.operators.ads import FacebookAdsReportToGcsOperator
+from airflow.utils.dates import days_ago
+
+# [START howto_GCS_BUCKET_env_variables]
+GCS_BUCKET = os.environ.get("FACEBOOK_ADS_BUCKET", "airflow_bucket_fb")
+GCS_OBJ_PATH = "Temp/this_is_my_report_json.json"
+GCS_CONN_ID = "google_cloud_default"
+# [END howto_GCS_BUCKET_env_variables]
+
+# [START howto_FB_ADS_env_variables]
+FACEBOOK_ADS_CONN_ID = os.environ.get("FACEBOOK_ADS_CONN_ID", "facebook_ads_default")
+FIELDS = [
+ AdsInsights.Field.campaign_name,
+ AdsInsights.Field.campaign_id,
+ AdsInsights.Field.ad_id,
+ AdsInsights.Field.clicks,
+ AdsInsights.Field.impressions,
+]
+PARAMS = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday'
+}
+# [END howto_FB_ADS_env_variables]
+
+default_args = {"start_date": days_ago(1)}
+
+with models.DAG(
+ "example_fb_operator",
+ default_args=default_args,
+ schedule_interval=None, # Override to match your needs
+) as dag:
+ # [START howto_FB_ADS_to_gcs_operator]
+ run_operator = FacebookAdsReportToGcsOperator(
+ task_id='run_fetch_data',
+ dag=dag,
Review comment:
It's not necessary, using `with` will set the DAG automatically
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r404641045
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,119 @@
+#
Review comment:
Hook should be in airflow/providers/facebook/ads/hooks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401197601
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ :return: list of Facebook Ads AdsInsights object(s)
+ :rtype: list[AdsInsights]
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.facebook_ads_config: Dict[str, Any] = {}
+ self.api_version = api_version
+ self.ad_account_id: str = ""
+ self.app_id: str = ""
+ self.app_secret: str = ""
+ self.access_token: str = ""
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ self._get_config()
+ for key, val in self.facebook_ads_config.items():
+ setattr(self, key, val)
+
+ return FacebookAdsApi.init(app_id=self.app_id,
+ app_secret=self.app_secret,
+ access_token=self.access_token,
+ api_version=self.api_version)
+
+ def _get_config(self) -> None:
+ """
+ Gets Facebook ads connection from meta db and sets
+ facebook_ads_config attribute with returned config file
+ """
+ conn = self.get_connection(self.facebook_ads_conn_id)
+ if "facebook_ads_client" not in conn.extra_dejson:
+ raise AirflowException("facebook_ads_client not found in extra field")
+
+ self.facebook_ads_config = conn.extra_dejson["facebook_ads_client"]
+
+ def bulk_facebook_report(
+ self,
+ params: Optional[Dict[str, Any]] = None,
+ fields: Optional[List[str]] = None,
+ ) -> List[AdsInsights]:
+ """
+ Pulls data from the Facebook Ads API
+
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+
+ :return: Facebook Ads API response, converted to Facebook Ads Row objects
+ :rtype: List[AdsInsights]
+ """
+ api = self._get_service()
+ params = params if params else self.default_params
+ ad_account = AdAccount(self.ad_account_id, api=api)
+ _async = ad_account.get_insights(params=params, fields=fields, is_async=True)
+ try:
+ while _async.api_get()["async_status"] != JOB_COMPLETED:
+ percent = _async.api_get()["async_percent_completion"]
+ self.log.info("%s %s completed", percent, "%")
+ time.sleep(10)
+ report_run_id = _async.api_get()["report_run_id"]
+ report_object = AdReportRun(report_run_id, api=api)
+ insights = report_object.get_insights()
+ return self._extract_rows(insights)
+ except FacebookRequestError as error:
+ self._error_handler(error)
+ raise
+
+ def _extract_rows(
+ self, iterators: Cursor
+ ) -> List[AdsInsights]:
+ """
+ Iterates over the cursor object and fetches the entire data from Facebook
+ :param iterators:
+ :return: List of all AdInsights objects
+ :rtype: List[AdsInsights]
+ """
+ try:
+ self.log.info("Extracting data from returned Facebook Ads Iterators")
+ return [iterator for iterator in iterators] # pylint: disable=unnecessary-comprehension
Review comment:
```suggestion
return list(iterators)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402731431
##########
File path: airflow/providers/facebook/ads/example_dags/example_ads.py
##########
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example Airflow DAG that shows how to use FacebookAdsReportToGcsOperator.
+"""
+import os
+
+from facebook_business.adobjects.adsinsights import AdsInsights
+
+from airflow import models
+from airflow.providers.facebook.ads.operators.ads import FacebookAdsReportToGcsOperator
+from airflow.utils.dates import days_ago
+
+# [START howto_GCS_BUCKET_env_variables]
+GCS_BUCKET = os.environ.get("FACEBOOK_ADS_BUCKET", "airflow_bucket_fb")
+GCS_OBJ_PATH = "Temp/this_is_my_report_json.json"
+GCS_CONN_ID = "google_cloud_default"
+# [END howto_GCS_BUCKET_env_variables]
+
+# [START howto_FB_ADS_env_variables]
+FACEBOOK_ADS_CONN_ID = os.environ.get("FACEBOOK_ADS_CONN_ID", "facebook_ads_default")
+FIELDS = [
+ AdsInsights.Field.campaign_name,
+ AdsInsights.Field.campaign_id,
+ AdsInsights.Field.ad_id,
+ AdsInsights.Field.clicks,
+ AdsInsights.Field.impressions,
+]
+PARAMS = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday'
+}
+# [END howto_FB_ADS_env_variables]
+
+default_args = {"start_date": days_ago(1)}
+
+with models.DAG(
+ "example_fb_operator",
+ default_args=default_args,
+ schedule_interval=None, # Override to match your needs
+) as dag:
+ # [START howto_FB_ADS_to_gcs_operator]
+ run_operator = FacebookAdsReportToGcsOperator(
+ task_id='run_fetch_data',
+ start_date=days_ago(2),
+ owner='airflow',
+ facebook_ads_conn_id=FACEBOOK_ADS_CONN_ID,
+ bucket_name=GCS_BUCKET,
+ params=PARAMS,
+ fields=FIELDS,
+ gcp_conn_id=GCS_CONN_ID,
+ object_name=GCS_OBJ_PATH,
+ )
Review comment:
Sounds good! Will have a look @mik-laj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/26b05bfe36d131ab19744514f9c9c69e9df126aa&el=desc) will **decrease** coverage by `0.67%`.
> The diff coverage is `85.45%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
==========================================
- Coverage 87.16% 86.49% -0.68%
==========================================
Files 932 935 +3
Lines 45331 45441 +110
==========================================
- Hits 39514 39302 -212
- Misses 5817 6139 +322
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `78.33% <78.33%> (ø)` | |
| [airflow/providers/facebook/ads/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL29wZXJhdG9ycy9hZHMucHk=) | `91.42% <91.42%> (ø)` | |
| [...providers/facebook/ads/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2V4YW1wbGVfZGFncy9leGFtcGxlX2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
| ... and [18 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [26b05bf...ed5efc0](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401380455
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ :return: list of Facebook Ads AdsInsights object(s)
+ :rtype: list[AdsInsights]
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.facebook_ads_config: Dict[str, Any] = {}
+ self.api_version = api_version
+ self.ad_account_id: str = ""
+ self.app_id: str = ""
+ self.app_secret: str = ""
+ self.access_token: str = ""
Review comment:
I would be in favor of using `: Optional[str] = None`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6c273466d598d7bcfb7c21feafcccb07cc4230fb&el=desc) will **decrease** coverage by `0.90%`.
> The diff coverage is `93.54%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
==========================================
- Coverage 88.29% 87.39% -0.91%
==========================================
Files 935 938 +3
Lines 45170 45294 +124
==========================================
- Hits 39885 39585 -300
- Misses 5285 5709 +424
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `95.07% <ø> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `87.09% <87.09%> (ø)` | |
| [...providers/facebook/ads/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2V4YW1wbGVfZGFncy9leGFtcGxlX2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [airflow/providers/facebook/ads/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL29wZXJhdG9ycy9hZHMucHk=) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.37% <100.00%> (+0.01%)` | :arrow_up: |
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0.00%> (-54.93%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
| ... and [23 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [6c27346...2344438](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r404639844
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,119 @@
+#
Review comment:
Cool, but the hook is consumed from google/FacebookAdsToGCS/hooks ??
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401201161
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ :return: list of Facebook Ads AdsInsights object(s)
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401381993
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ :return: list of Facebook Ads AdsInsights object(s)
+ :rtype: list[AdsInsights]
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.facebook_ads_config: Dict[str, Any] = {}
+ self.api_version = api_version
+ self.ad_account_id: str = ""
+ self.app_id: str = ""
+ self.app_secret: str = ""
+ self.access_token: str = ""
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ self._get_config()
+ for key, val in self.facebook_ads_config.items():
+ setattr(self, key, val)
+
+ return FacebookAdsApi.init(app_id=self.app_id,
+ app_secret=self.app_secret,
+ access_token=self.access_token,
+ api_version=self.api_version)
+
+ def _get_config(self) -> None:
+ """
+ Gets Facebook ads connection from meta db and sets
+ facebook_ads_config attribute with returned config file
+ """
+ conn = self.get_connection(self.facebook_ads_conn_id)
+ if "facebook_ads_client" not in conn.extra_dejson:
+ raise AirflowException("facebook_ads_client not found in extra field")
+
+ self.facebook_ads_config = conn.extra_dejson["facebook_ads_client"]
+
+ def bulk_facebook_report(
+ self,
+ params: Optional[Dict[str, Any]] = None,
+ fields: Optional[List[str]] = None,
+ ) -> List[AdsInsights]:
+ """
+ Pulls data from the Facebook Ads API
+
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+
+ :return: Facebook Ads API response, converted to Facebook Ads Row objects
+ :rtype: List[AdsInsights]
+ """
+ api = self._get_service()
+ params = params if params else self.default_params
+ ad_account = AdAccount(self.ad_account_id, api=api)
+ _async = ad_account.get_insights(params=params, fields=fields, is_async=True)
+ try:
+ while _async.api_get()["async_status"] != JOB_COMPLETED:
Review comment:
I'm do not know facebook API, is there possibility that the job will complete with other status than `JOB_COMPLETED`? For example with `JOB_FAILED` or something. I know others APIs have such behavior
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r407724327
##########
File path: airflow/utils/db.py
##########
@@ -240,6 +240,23 @@ def create_default_connections(session=None):
),
session
)
+ merge_conn(
+ Connection(
+ conn_id="facebook_default",
+ conn_type="facebook_social",
+ schema="""
+ {
+ "facebook_ads_client": {
+ "account_id": "act_123456789",
+ "app_id": "1234567890",
+ "app_secret": "1f45tghxxxx12345",
+ "access_token": "ABcdEfghiJKlmnoxxyz"
+ }
+ }
+ """,
Review comment:
Is there any better way to show the schema ?? Cus for GCP there is a tutorial on google's website. Integrating with airflow. Any thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401195142
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ :return: list of Facebook Ads AdsInsights object(s)
+ :rtype: list[AdsInsights]
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.facebook_ads_config: Dict[str, Any] = {}
+ self.api_version = api_version
+ self.ad_account_id: str = ""
+ self.app_id: str = ""
+ self.app_secret: str = ""
+ self.access_token: str = ""
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ self._get_config()
+ for key, val in self.facebook_ads_config.items():
Review comment:
It doesn't seem safe. Can you explicitly set attributes here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402153714
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_ads_conn_id", "bucket", "obj")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket: str,
+ obj: str,
+ gcp_conn_id: str = "google_cloud_default",
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ fields: Optional[List[str]] = None,
+ params: Optional[Dict[str, Any]] = None,
+ gzip: bool = False,
+ *args,
+ **kwargs,
+ ) -> None:
+ super().__init__(*args, **kwargs)
+ self.bucket = bucket
+ self.obj = obj
+ self.gcp_conn_id = gcp_conn_id
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.fields = fields
+ self.params = params # type:ignore
+ self.gzip = gzip
+
+ def execute(self, context: Dict):
+ service = FacebookAdsReportingHook(facebook_ads_conn_id=self.facebook_ads_conn_id,
+ api_version=self.api_version)
+ rows = service.bulk_facebook_report(params=self.params, fields=self.fields)
+ try:
+ converted_rows = [dict(row) for row in rows]
+ except Exception as e:
+ self.log.error("An error occurred in converting the Facebook Ad Rows. \n Error %s", e)
+ raise
+
+ with tempfile.NamedTemporaryFile("w", suffix=".json") as json_file:
+ json.dump(converted_rows, json_file)
+ json_file.flush()
+ hook = GCSHook(gcp_conn_id=self.gcp_conn_id)
+ hook.upload(
+ bucket_name=self.bucket,
+ object_name=self.obj,
+ filename=json_file.name,
+ gzip=self.gzip,
+ )
+ self.log.info("%s uploaded to GCS", json_file.name)
Review comment:
True, my idea is to add a line `return f"gs://{self.bucket}/{self.obj}"` so we can ccess this value using XCom. This will help in building DAGs :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401195872
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ :return: list of Facebook Ads AdsInsights object(s)
+ :rtype: list[AdsInsights]
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.facebook_ads_config: Dict[str, Any] = {}
+ self.api_version = api_version
+ self.ad_account_id: str = ""
+ self.app_id: str = ""
+ self.app_secret: str = ""
+ self.access_token: str = ""
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ self._get_config()
+ for key, val in self.facebook_ads_config.items():
+ setattr(self, key, val)
+
+ return FacebookAdsApi.init(app_id=self.app_id,
Review comment:
If these values are from the previous line, is it necessary to not add additional attributes to the class?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401196773
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ :return: list of Facebook Ads AdsInsights object(s)
+ :rtype: list[AdsInsights]
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.facebook_ads_config: Dict[str, Any] = {}
+ self.api_version = api_version
+ self.ad_account_id: str = ""
+ self.app_id: str = ""
+ self.app_secret: str = ""
+ self.access_token: str = ""
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ self._get_config()
+ for key, val in self.facebook_ads_config.items():
+ setattr(self, key, val)
+
+ return FacebookAdsApi.init(app_id=self.app_id,
+ app_secret=self.app_secret,
+ access_token=self.access_token,
+ api_version=self.api_version)
+
+ def _get_config(self) -> None:
+ """
+ Gets Facebook ads connection from meta db and sets
+ facebook_ads_config attribute with returned config file
+ """
+ conn = self.get_connection(self.facebook_ads_conn_id)
+ if "facebook_ads_client" not in conn.extra_dejson:
+ raise AirflowException("facebook_ads_client not found in extra field")
+
+ self.facebook_ads_config = conn.extra_dejson["facebook_ads_client"]
Review comment:
What do you think to use cached_property here? It will cause that the person who wants to use it will not have to know in which order the methods should be performed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r404420227
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,146 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from enum import Enum
+from typing import Any, Dict, List
+
+from cached_property import cached_property
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import FacebookAdsApi
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+
+class JobStatus(Enum):
+ """
+ Available options for facebook async task status
+ """
+ COMPLETED = 'Job Completed'
+ STARTED = 'Job Started'
+ RUNNING = 'Job Running'
+ FAILED = 'Job Failed'
+ SKIPPED = 'Job Skipped'
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ """
+
+ def __init__(
+ self,
+ facebook_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_conn_id = facebook_conn_id
+ self.api_version = api_version
+ self.client_required_fields = ["app_id",
+ "app_secret",
+ "access_token",
+ "account_id"]
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ config = self.facebook_ads_config
+ missings = []
+ for _each in self.client_required_fields:
Review comment:
This is very important, but can you inline this constant or make it a class attribute rather than an instance attribute? This is confusing,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401831272
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_ads_conn_id", "bucket", "obj")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket: str,
+ obj: str,
+ gcp_conn_id: str = "google_cloud_default",
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ fields: Optional[List[str]] = None,
+ params: Optional[Dict[str, Any]] = None,
+ gzip: bool = False,
+ *args,
+ **kwargs,
+ ) -> None:
+ super().__init__(*args, **kwargs)
+ self.bucket = bucket
+ self.obj = obj
+ self.gcp_conn_id = gcp_conn_id
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.fields = fields
+ self.params = params # type:ignore
+ self.gzip = gzip
+
+ def execute(self, context: Dict):
+ service = FacebookAdsReportingHook(facebook_ads_conn_id=self.facebook_ads_conn_id,
+ api_version=self.api_version)
+ rows = service.bulk_facebook_report(params=self.params, fields=self.fields)
+ try:
+ converted_rows = [dict(row) for row in rows]
+ except Exception as e:
+ self.log.error("An error occurred in converting the Facebook Ad Rows. \n Error %s", e)
+ raise
+
+ with tempfile.NamedTemporaryFile("w", suffix=".json") as json_file:
+ json.dump(converted_rows, json_file)
+ json_file.flush()
+ hook = GCSHook(gcp_conn_id=self.gcp_conn_id)
+ hook.upload(
+ bucket_name=self.bucket,
+ object_name=self.obj,
+ filename=json_file.name,
+ gzip=self.gzip,
+ )
+ self.log.info("%s uploaded to GCS", json_file.name)
Review comment:
@turbaszek Looks like the gsc hook accepts either filename->"Path to that file" or it excepts data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401837412
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_ads_conn_id", "bucket", "obj")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket: str,
+ obj: str,
+ gcp_conn_id: str = "google_cloud_default",
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ fields: Optional[List[str]] = None,
+ params: Optional[Dict[str, Any]] = None,
+ gzip: bool = False,
+ *args,
+ **kwargs,
+ ) -> None:
+ super().__init__(*args, **kwargs)
+ self.bucket = bucket
+ self.obj = obj
+ self.gcp_conn_id = gcp_conn_id
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.fields = fields
+ self.params = params # type:ignore
+ self.gzip = gzip
+
+ def execute(self, context: Dict):
+ service = FacebookAdsReportingHook(facebook_ads_conn_id=self.facebook_ads_conn_id,
+ api_version=self.api_version)
+ rows = service.bulk_facebook_report(params=self.params, fields=self.fields)
+ try:
+ converted_rows = [dict(row) for row in rows]
+ except Exception as e:
+ self.log.error("An error occurred in converting the Facebook Ad Rows. \n Error %s", e)
+ raise
Review comment:
@turbaszek looks like I handling errors in the hook, thanks for pointing it out. I don't think the exception is required. Before I would iterate over the cursor in the operator, but now its just type casting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402734563
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,152 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from cached_property import cached_property
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookError, FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = "Job Completed"
+JOB_FAILED = "Job Failed"
+JOB_SKIPPED = "Job Skipped"
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ config = self.facebook_ads_config
+ try:
+ return FacebookAdsApi.init(app_id=config["app_id"],
+ app_secret=config["app_secret"],
+ access_token=config["access_token"],
+ account_id=config["account_id"],
+ api_version=self.api_version)
+ except AirflowException as error:
+ message = "Missing config data:\n {error}".format(error=error)
+ self._error_handler(message)
+ raise
+
+ @cached_property
+ def facebook_ads_config(self) -> None:
+ """
+ Gets Facebook ads connection from meta db and sets
+ facebook_ads_config attribute with returned config file
+ """
+ conn = self.get_connection(self.facebook_ads_conn_id)
+ if "facebook_ads_client" not in conn.extra_dejson:
+ raise AirflowException("facebook_ads_client not found in extra field")
+ return conn.extra_dejson.get("facebook_ads_client")
+
+ def bulk_facebook_report(
+ self,
+ params: Optional[Dict[str, Any]] = None,
+ fields: Optional[List[str]] = None,
+ ) -> List[AdsInsights]:
+ """
+ Pulls data from the Facebook Ads API
+
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+
+ :return: Facebook Ads API response, converted to Facebook Ads Row objects
+ :rtype: List[AdsInsights]
+ """
+ api = self._get_service()
+ params = params if params else self.default_params
+ ad_account = AdAccount(api.get_default_account_id(), api=api)
+ _async = ad_account.get_insights(params=params, fields=fields, is_async=True)
+ try:
+ while _async.api_get()["async_status"] != JOB_COMPLETED:
+ percent = _async.api_get()["async_percent_completion"]
+ self.log.info("%s %s completed", percent, "%")
+ time.sleep(10)
+ if _async.api_get()["async_status"] in [JOB_SKIPPED, JOB_FAILED]:
+ self._error_handler("Task failed Retry...")
+ raise FacebookError
+ report_run_id = _async.api_get()["report_run_id"]
+ report_object = AdReportRun(report_run_id, api=api)
+ insights = report_object.get_insights()
+ return self._extract_rows(insights)
+ except FacebookRequestError as error:
+ self._error_handler(error)
Review comment:
This will cause the errors to be repeated in the log. The first time it will be displayed by you and the second time by Airflow, because it will be an unhandled exception. I am not convinced that we need special error handling here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402746919
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,152 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from cached_property import cached_property
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookError, FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = "Job Completed"
+JOB_FAILED = "Job Failed"
+JOB_SKIPPED = "Job Skipped"
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.default_params = {
Review comment:
Yes, what would you suggest? Not to have default params and throw error if user does not enter data. I can make it a required field.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/cc9b1bc78df1e82da37b885d2a8aaf5779c6209d&el=desc) will **decrease** coverage by `54.38%`.
> The diff coverage is `68.75%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
===========================================
- Coverage 87.27% 32.89% -54.39%
===========================================
Files 937 940 +3
Lines 45549 45661 +112
===========================================
- Hits 39753 15019 -24734
- Misses 5796 30642 +24846
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `53.52% <ø> (-41.55%)` | :arrow_down: |
| [...providers/facebook/ads/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2V4YW1wbGVfZGFncy9leGFtcGxlX2Fkcy5weQ==) | `0.00% <0.00%> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `33.84% <69.23%> (ø)` | |
| [airflow/providers/facebook/ads/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL29wZXJhdG9ycy9hZHMucHk=) | `41.93% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.36% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [511 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [cc9b1bc...f22ea1c](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r407666808
##########
File path: airflow/utils/db.py
##########
@@ -240,6 +240,23 @@ def create_default_connections(session=None):
),
session
)
+ merge_conn(
+ Connection(
+ conn_id="facebook_default",
+ conn_type="facebook_social",
+ schema="""
+ {
+ "facebook_ads_client": {
+ "account_id": "act_123456789",
+ "app_id": "1234567890",
+ "app_secret": "1f45tghxxxx12345",
+ "access_token": "ABcdEfghiJKlmnoxxyz"
+ }
+ }
+ """,
Review comment:
I am not sure if we should keep dummy values here like that. Can we do this likewise gcp? `schema="default"`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401385823
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_ads_conn_id", "bucket", "obj")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket: str,
+ obj: str,
+ gcp_conn_id: str = "google_cloud_default",
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ fields: Optional[List[str]] = None,
+ params: Optional[Dict[str, Any]] = None,
+ gzip: bool = False,
+ *args,
+ **kwargs,
+ ) -> None:
+ super().__init__(*args, **kwargs)
+ self.bucket = bucket
+ self.obj = obj
+ self.gcp_conn_id = gcp_conn_id
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.fields = fields
+ self.params = params # type:ignore
+ self.gzip = gzip
+
+ def execute(self, context: Dict):
+ service = FacebookAdsReportingHook(facebook_ads_conn_id=self.facebook_ads_conn_id,
+ api_version=self.api_version)
+ rows = service.bulk_facebook_report(params=self.params, fields=self.fields)
+ try:
+ converted_rows = [dict(row) for row in rows]
+ except Exception as e:
+ self.log.error("An error occurred in converting the Facebook Ad Rows. \n Error %s", e)
+ raise
Review comment:
Do we need this exception clause? @mik-laj will the error appear in logs anyway?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402736678
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,152 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from cached_property import cached_property
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookError, FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = "Job Completed"
+JOB_FAILED = "Job Failed"
+JOB_SKIPPED = "Job Skipped"
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ config = self.facebook_ads_config
+ try:
+ return FacebookAdsApi.init(app_id=config["app_id"],
+ app_secret=config["app_secret"],
+ access_token=config["access_token"],
+ account_id=config["account_id"],
+ api_version=self.api_version)
+ except AirflowException as error:
+ message = "Missing config data:\n {error}".format(error=error)
+ self._error_handler(message)
+ raise
+
+ @cached_property
+ def facebook_ads_config(self) -> None:
+ """
+ Gets Facebook ads connection from meta db and sets
+ facebook_ads_config attribute with returned config file
+ """
+ conn = self.get_connection(self.facebook_ads_conn_id)
+ if "facebook_ads_client" not in conn.extra_dejson:
+ raise AirflowException("facebook_ads_client not found in extra field")
+ return conn.extra_dejson.get("facebook_ads_client")
+
+ def bulk_facebook_report(
+ self,
+ params: Optional[Dict[str, Any]] = None,
+ fields: Optional[List[str]] = None,
+ ) -> List[AdsInsights]:
+ """
+ Pulls data from the Facebook Ads API
+
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+
+ :return: Facebook Ads API response, converted to Facebook Ads Row objects
+ :rtype: List[AdsInsights]
+ """
+ api = self._get_service()
+ params = params if params else self.default_params
+ ad_account = AdAccount(api.get_default_account_id(), api=api)
+ _async = ad_account.get_insights(params=params, fields=fields, is_async=True)
+ try:
+ while _async.api_get()["async_status"] != JOB_COMPLETED:
Review comment:
```
from facebookads.adobjects.adreportrun import AdReportRun
```
```suggestion
while _async.api_get()[AdReportRun.Field.async_status] != JOB_COMPLETED:
```
The library defined these constants, so it's worth using them. https://github.com/facebook/facebook-python-business-sdk/blob/master/facebook_business/adobjects/adreportrun.py#L45-L78
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402747195
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,108 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_ads_conn_id", "bucket_name", "object_name")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket_name: str,
+ object_name: str,
+ gcp_conn_id: str = "google_cloud_default",
+ facebook_ads_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ fields: Optional[List[str]] = None,
+ params: Optional[Dict[str, Any]] = None,
+ gzip: bool = False,
+ *args,
+ **kwargs,
+ ) -> None:
+ super().__init__(*args, **kwargs)
+ self.bucket_name = bucket_name
+ self.object_name = object_name
+ self.gcp_conn_id = gcp_conn_id
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.fields = fields
+ self.params = params # type:ignore
+ self.gzip = gzip
+
+ def execute(self, context: Dict):
+ service = FacebookAdsReportingHook(facebook_ads_conn_id=self.facebook_ads_conn_id,
+ api_version=self.api_version)
+ rows = service.bulk_facebook_report(params=self.params, fields=self.fields)
+
+ converted_rows = [dict(row) for row in rows]
Review comment:
yep cool, we can pass it as a arg! let the user decide.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r404420761
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,119 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import csv
+import tempfile
+from typing import Any, Dict, List
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param sleep_time: Time to sleep when async call is happening
+ :type sleep_time: int
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_conn_id", "bucket_name", "object_name")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket_name: str,
+ object_name: str,
+ fields: List[str],
+ params: Dict[str, Any],
+ gcp_conn_id: str = "google_cloud_default",
+ facebook_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ sleep_time: int = 5,
Review comment:
I don't think this is needed as part of the operator. The default value in hook is sufficient.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/cc9b1bc78df1e82da37b885d2a8aaf5779c6209d&el=desc) will **decrease** coverage by `54.41%`.
> The diff coverage is `32.14%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
===========================================
- Coverage 87.27% 32.86% -54.42%
===========================================
Files 937 940 +3
Lines 45549 45661 +112
===========================================
- Hits 39753 15006 -24747
- Misses 5796 30655 +24859
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `53.52% <ø> (-41.55%)` | :arrow_down: |
| [...providers/facebook/ads/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2V4YW1wbGVfZGFncy9leGFtcGxlX2Fkcy5weQ==) | `0.00% <0.00%> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `33.84% <33.84%> (ø)` | |
| [airflow/providers/facebook/ads/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL29wZXJhdG9ycy9hZHMucHk=) | `41.93% <41.93%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.36% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [813 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [cc9b1bc...f22ea1c](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/45c8983306ab1c54abdacd8f870e790fad25cb37&el=desc) will **decrease** coverage by `0.76%`.
> The diff coverage is `94.11%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
==========================================
- Coverage 88.45% 87.68% -0.77%
==========================================
Files 937 940 +3
Lines 45234 45353 +119
==========================================
- Hits 40011 39768 -243
- Misses 5223 5585 +362
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `95.07% <ø> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `87.93% <87.93%> (ø)` | |
| [...le/facebook\_ads\_to\_gcs/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3MvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYWRzLnB5) | `100.00% <100.00%> (ø)` | |
| [...viders/google/facebook\_ads\_to\_gcs/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3Mvb3BlcmF0b3JzL2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.37% <100.00%> (+0.01%)` | :arrow_up: |
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| ... and [18 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [45c8983...ff2fcf6](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402729596
##########
File path: CONTRIBUTING.rst
##########
@@ -316,11 +316,11 @@ This is the full list of those extras:
.. START EXTRAS HERE
all, all_dbs, async, atlas, aws, azure, cassandra, celery, cgroups, cloudant, dask, databricks,
-datadog, devel, devel_ci, devel_hadoop, doc, docker, druid, elasticsearch, exasol, gcp, gcp_api,
-github_enterprise, google_auth, grpc, hashicorp, hdfs, hive, jdbc, jira, kerberos, kubernetes, ldap,
-mongo, mssql, mysql, odbc, oracle, pagerduty, papermill, password, pinot, postgres, presto, qds,
-rabbitmq, redis, salesforce, samba, segment, sendgrid, sentry, singularity, slack, snowflake, ssh,
-statsd, tableau, vertica, virtualenv, webhdfs, winrm, yandexcloud
+datadog, devel, devel_ci, devel_hadoop, doc, docker, druid, elasticsearch, exasol, facebook, gcp,
+gcp_api, github_enterprise, google_auth, grpc, hashicorp, hdfs, hive, jdbc, jira, kerberos,
Review comment:
Can you also update instaltion.rst?
https://airflow.readthedocs.io/en/latest/installation.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/cc9b1bc78df1e82da37b885d2a8aaf5779c6209d&el=desc) will **decrease** coverage by `22.30%`.
> The diff coverage is `68.75%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
===========================================
- Coverage 87.27% 64.97% -22.31%
===========================================
Files 937 940 +3
Lines 45549 45661 +112
===========================================
- Hits 39753 29668 -10085
- Misses 5796 15993 +10197
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `75.35% <ø> (-19.72%)` | :arrow_down: |
| [...providers/facebook/ads/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2V4YW1wbGVfZGFncy9leGFtcGxlX2Fkcy5weQ==) | `0.00% <0.00%> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `69.23% <69.23%> (ø)` | |
| [airflow/providers/facebook/ads/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL29wZXJhdG9ycy9hZHMucHk=) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.36% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [508 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [cc9b1bc...f22ea1c](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401201092
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ :return: list of Facebook Ads AdsInsights object(s)
+ :rtype: list[AdsInsights]
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6c273466d598d7bcfb7c21feafcccb07cc4230fb&el=desc) will **decrease** coverage by `0.73%`.
> The diff coverage is `93.54%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
==========================================
- Coverage 88.29% 87.56% -0.74%
==========================================
Files 935 938 +3
Lines 45170 45294 +124
==========================================
- Hits 39885 39661 -224
- Misses 5285 5633 +348
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `95.07% <ø> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `87.09% <87.09%> (ø)` | |
| [...providers/facebook/ads/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2V4YW1wbGVfZGFncy9leGFtcGxlX2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [airflow/providers/facebook/ads/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL29wZXJhdG9ycy9hZHMucHk=) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.37% <100.00%> (+0.01%)` | :arrow_up: |
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| ... and [16 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [6c27346...2344438](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401198075
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_ads_conn_id", "bucket", "obj")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket: str,
+ obj: str,
Review comment:
```suggestion
object_naame: str,
```
obj is the word reserved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r404421633
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,119 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import csv
+import tempfile
+from typing import Any, Dict, List
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param sleep_time: Time to sleep when async call is happening
+ :type sleep_time: int
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_conn_id", "bucket_name", "object_name")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket_name: str,
+ object_name: str,
+ fields: List[str],
+ params: Dict[str, Any],
+ gcp_conn_id: str = "google_cloud_default",
+ facebook_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ sleep_time: int = 5,
+ gzip: bool = False,
Review comment:
Connection parameters, i.e. conn_id, api_version should be last. At least, that's what we're trying to do for GCP.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r404426868
##########
File path: airflow/providers/facebook/ads/example_dags/example_ads.py
##########
@@ -0,0 +1,110 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example Airflow DAG that shows how to use FacebookAdsReportToGcsOperator.
+"""
+from facebook_business.adobjects.adsinsights import AdsInsights
+
+from airflow import models
+from airflow.providers.facebook.ads.operators.ads import FacebookAdsReportToGcsOperator
+from airflow.providers.google.cloud.operators.bigquery import (
+ BigQueryCreateEmptyDatasetOperator, BigQueryCreateEmptyTableOperator, BigQueryExecuteQueryOperator,
+)
+from airflow.providers.google.cloud.operators.gcs_to_bigquery import GCSToBigQueryOperator
+from airflow.utils.dates import days_ago
+
+# [START howto_GCS_env_variables]
+GCP_PROJECT_ID = "free-tier-1997"
+GCS_BUCKET = "airflow_bucket_fb"
+GCS_OBJ_PATH = "Temp/this_is_my_report_csv.csv"
+GCS_CONN_ID = "google_cloud_default"
+DATASET_NAME = "airflow_test_dataset"
+TABLE_NAME = "airflow_test_datatable"
+# [END howto_GCS_env_variables]
+
+# [START howto_FB_ADS_env_variables]
+FACEBOOK_ADS_CONN_ID = "facebook_default"
+FIELDS = [
+ AdsInsights.Field.campaign_name,
+ AdsInsights.Field.campaign_id,
+ AdsInsights.Field.ad_id,
+ AdsInsights.Field.clicks,
+ AdsInsights.Field.impressions,
+]
+PARAMS = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday'
+}
+# [END howto_FB_ADS_env_variables]
+
+default_args = {"start_date": days_ago(1)}
+
+with models.DAG(
+ "example_fb_operator",
+ default_args=default_args,
+ schedule_interval=None, # Override to match your needs
+) as dag:
+
+ create_dataset = BigQueryCreateEmptyDatasetOperator(task_id="create-dataset",
+ dataset_id=DATASET_NAME)
+
+ create_table = BigQueryCreateEmptyTableOperator(
+ task_id="create_table",
+ dataset_id=DATASET_NAME,
+ table_id=TABLE_NAME,
+ schema_fields=[
+ {'name': 'campaign_name', 'type': 'STRING', 'mode': 'NULLABLE'},
+ {'name': 'campaign_id', 'type': 'STRING', 'mode': 'NULLABLE'},
+ {'name': 'ad_id', 'type': 'STRING', 'mode': 'NULLABLE'},
+ {'name': 'clicks', 'type': 'STRING', 'mode': 'NULLABLE'},
+ {'name': 'impressions', 'type': 'STRING', 'mode': 'NULLABLE'},
+ ],
+ )
+
+ # [START howto_FB_ADS_to_gcs_operator]
+ run_operator = FacebookAdsReportToGcsOperator(
+ task_id='run_fetch_data',
+ start_date=days_ago(2),
+ owner='airflow',
+ facebook_conn_id=FACEBOOK_ADS_CONN_ID,
+ bucket_name=GCS_BUCKET,
+ params=PARAMS,
+ fields=FIELDS,
+ gcp_conn_id=GCS_CONN_ID,
+ object_name=GCS_OBJ_PATH,
+ )
+ # [END howto_FB_ADS_to_gcs_operator]
+
+ # [START howto_operator_gcs_to_bq]
+ load_csv = GCSToBigQueryOperator(
+ task_id='gcs_to_bq_example',
+ bucket=GCS_BUCKET,
+ source_objects=[GCS_OBJ_PATH],
+ destination_project_dataset_table=f"{DATASET_NAME}.{TABLE_NAME}",
+ write_disposition='WRITE_TRUNCATE')
+ # [END howto_operator_gcs_to_bq]
+
+ # [START howto_operator_read_data_from_gcs]
+ read_data_from_gcs_many_chunks = BigQueryExecuteQueryOperator(
Review comment:
Thank you very much for the BigQuery example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402730359
##########
File path: airflow/providers/facebook/ads/example_dags/example_ads.py
##########
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example Airflow DAG that shows how to use FacebookAdsReportToGcsOperator.
+"""
+import os
+
+from facebook_business.adobjects.adsinsights import AdsInsights
+
+from airflow import models
+from airflow.providers.facebook.ads.operators.ads import FacebookAdsReportToGcsOperator
+from airflow.utils.dates import days_ago
+
+# [START howto_GCS_BUCKET_env_variables]
+GCS_BUCKET = os.environ.get("FACEBOOK_ADS_BUCKET", "airflow_bucket_fb")
+GCS_OBJ_PATH = "Temp/this_is_my_report_json.json"
+GCS_CONN_ID = "google_cloud_default"
+# [END howto_GCS_BUCKET_env_variables]
+
+# [START howto_FB_ADS_env_variables]
+FACEBOOK_ADS_CONN_ID = os.environ.get("FACEBOOK_ADS_CONN_ID", "facebook_ads_default")
+FIELDS = [
+ AdsInsights.Field.campaign_name,
+ AdsInsights.Field.campaign_id,
+ AdsInsights.Field.ad_id,
+ AdsInsights.Field.clicks,
+ AdsInsights.Field.impressions,
+]
+PARAMS = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday'
+}
+# [END howto_FB_ADS_env_variables]
+
+default_args = {"start_date": days_ago(1)}
+
+with models.DAG(
+ "example_fb_operator",
+ default_args=default_args,
+ schedule_interval=None, # Override to match your needs
+) as dag:
+ # [START howto_FB_ADS_to_gcs_operator]
+ run_operator = FacebookAdsReportToGcsOperator(
+ task_id='run_fetch_data',
+ start_date=days_ago(2),
+ owner='airflow',
+ facebook_ads_conn_id=FACEBOOK_ADS_CONN_ID,
+ bucket_name=GCS_BUCKET,
+ params=PARAMS,
+ fields=FIELDS,
+ gcp_conn_id=GCS_CONN_ID,
+ object_name=GCS_OBJ_PATH,
+ )
Review comment:
Wouldn't you like to extend this example to a more real case, i.e. add data copy to BigQuery? Many users will use it this way, so testing this case will be helpful.
Here is an example:
https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/example_dags/example_presto_to_gcs.py
It can also be an interesting task for you, because you will have to get to know BigQuery.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/cc9b1bc78df1e82da37b885d2a8aaf5779c6209d&el=desc) will **decrease** coverage by `22.30%`.
> The diff coverage is `68.75%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
===========================================
- Coverage 87.27% 64.97% -22.31%
===========================================
Files 937 940 +3
Lines 45549 45661 +112
===========================================
- Hits 39753 29668 -10085
- Misses 5796 15993 +10197
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `75.35% <ø> (-19.72%)` | :arrow_down: |
| [...providers/facebook/ads/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2V4YW1wbGVfZGFncy9leGFtcGxlX2Fkcy5weQ==) | `0.00% <0.00%> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `69.23% <69.23%> (ø)` | |
| [airflow/providers/facebook/ads/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL29wZXJhdG9ycy9hZHMucHk=) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.36% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [508 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [cc9b1bc...f22ea1c](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-613432272
Awesome work, congrats on your first merged pull request!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/510a377b50ae757ce3c5d5b455c4c77d7dc9068e&el=desc) will **decrease** coverage by `0.28%`.
> The diff coverage is `94.11%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
==========================================
- Coverage 88.30% 88.01% -0.29%
==========================================
Files 935 938 +3
Lines 45243 45362 +119
==========================================
- Hits 39951 39926 -25
- Misses 5292 5436 +144
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `95.07% <ø> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `87.93% <87.93%> (ø)` | |
| [...le/facebook\_ads\_to\_gcs/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3MvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYWRzLnB5) | `100.00% <100.00%> (ø)` | |
| [...viders/google/facebook\_ads\_to\_gcs/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3Mvb3BlcmF0b3JzL2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.37% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0.00%> (-45.66%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
| ... and [8 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [510a377...5234efa](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/510a377b50ae757ce3c5d5b455c4c77d7dc9068e&el=desc) will **decrease** coverage by `55.09%`.
> The diff coverage is `31.09%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
===========================================
- Coverage 88.30% 33.20% -55.10%
===========================================
Files 935 938 +3
Lines 45243 45362 +119
===========================================
- Hits 39951 15062 -24889
- Misses 5292 30300 +25008
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `53.52% <ø> (-41.55%)` | :arrow_down: |
| [...le/facebook\_ads\_to\_gcs/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3MvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYWRzLnB5) | `0.00% <0.00%> (ø)` | |
| [...viders/google/facebook\_ads\_to\_gcs/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3Mvb3BlcmF0b3JzL2Fkcy5weQ==) | `36.11% <36.11%> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `39.65% <39.65%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.37% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [815 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [510a377...5234efa](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401824549
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ :return: list of Facebook Ads AdsInsights object(s)
+ :rtype: list[AdsInsights]
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.facebook_ads_config: Dict[str, Any] = {}
+ self.api_version = api_version
+ self.ad_account_id: str = ""
+ self.app_id: str = ""
+ self.app_secret: str = ""
+ self.access_token: str = ""
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ self._get_config()
+ for key, val in self.facebook_ads_config.items():
+ setattr(self, key, val)
+
+ return FacebookAdsApi.init(app_id=self.app_id,
+ app_secret=self.app_secret,
+ access_token=self.access_token,
+ api_version=self.api_version)
+
+ def _get_config(self) -> None:
+ """
+ Gets Facebook ads connection from meta db and sets
+ facebook_ads_config attribute with returned config file
+ """
+ conn = self.get_connection(self.facebook_ads_conn_id)
+ if "facebook_ads_client" not in conn.extra_dejson:
+ raise AirflowException("facebook_ads_client not found in extra field")
+
+ self.facebook_ads_config = conn.extra_dejson["facebook_ads_client"]
+
+ def bulk_facebook_report(
+ self,
+ params: Optional[Dict[str, Any]] = None,
+ fields: Optional[List[str]] = None,
+ ) -> List[AdsInsights]:
+ """
+ Pulls data from the Facebook Ads API
+
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+
+ :return: Facebook Ads API response, converted to Facebook Ads Row objects
+ :rtype: List[AdsInsights]
+ """
+ api = self._get_service()
+ params = params if params else self.default_params
+ ad_account = AdAccount(self.ad_account_id, api=api)
+ _async = ad_account.get_insights(params=params, fields=fields, is_async=True)
+ try:
+ while _async.api_get()["async_status"] != JOB_COMPLETED:
Review comment:
Yep, have solved will, re commit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #8008: Custom Facebook Ads
Operator
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/26b05bfe36d131ab19744514f9c9c69e9df126aa&el=desc) will **decrease** coverage by `0.90%`.
> The diff coverage is `85.45%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
==========================================
- Coverage 87.16% 86.26% -0.91%
==========================================
Files 932 935 +3
Lines 45331 45441 +110
==========================================
- Hits 39514 39199 -315
- Misses 5817 6242 +425
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `78.33% <78.33%> (ø)` | |
| [airflow/providers/facebook/ads/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL29wZXJhdG9ycy9hZHMucHk=) | `91.42% <91.42%> (ø)` | |
| [...providers/facebook/ads/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2V4YW1wbGVfZGFncy9leGFtcGxlX2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `39.28% <0.00%> (-60.72%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0.00%> (-54.93%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| ... and [23 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [26b05bf...ed5efc0](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402738197
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,108 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_ads_conn_id", "bucket_name", "object_name")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket_name: str,
+ object_name: str,
+ gcp_conn_id: str = "google_cloud_default",
+ facebook_ads_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ fields: Optional[List[str]] = None,
+ params: Optional[Dict[str, Any]] = None,
+ gzip: bool = False,
+ *args,
+ **kwargs,
+ ) -> None:
+ super().__init__(*args, **kwargs)
+ self.bucket_name = bucket_name
+ self.object_name = object_name
+ self.gcp_conn_id = gcp_conn_id
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.fields = fields
+ self.params = params # type:ignore
+ self.gzip = gzip
+
+ def execute(self, context: Dict):
+ service = FacebookAdsReportingHook(facebook_ads_conn_id=self.facebook_ads_conn_id,
+ api_version=self.api_version)
+ rows = service.bulk_facebook_report(params=self.params, fields=self.fields)
+
+ converted_rows = [dict(row) for row in rows]
Review comment:
Do you think that it is worth adding the option to save in CSV format? This format is very often needed as an input format for various older tools.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk merged pull request #8008: Custom Facebook Ads
Operator
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r404424465
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,119 @@
+#
Review comment:
The file name should be called facebook_ads_to_gcs and should be in the `airflow/providers/google/cloud/operators` package. Transfer operators should be grouped by destination. Unless the source is a large cloud provider. Then it should be in cloud provider package. As long as the source and destination is a cloud provider. In this case, it's by destination.
More information: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401837518
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_ads_conn_id", "bucket", "obj")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket: str,
+ obj: str,
+ gcp_conn_id: str = "google_cloud_default",
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ fields: Optional[List[str]] = None,
+ params: Optional[Dict[str, Any]] = None,
+ gzip: bool = False,
+ *args,
+ **kwargs,
+ ) -> None:
+ super().__init__(*args, **kwargs)
+ self.bucket = bucket
+ self.obj = obj
+ self.gcp_conn_id = gcp_conn_id
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.fields = fields
+ self.params = params # type:ignore
+ self.gzip = gzip
+
+ def execute(self, context: Dict):
+ service = FacebookAdsReportingHook(facebook_ads_conn_id=self.facebook_ads_conn_id,
+ api_version=self.api_version)
+ rows = service.bulk_facebook_report(params=self.params, fields=self.fields)
+ try:
+ converted_rows = [dict(row) for row in rows]
+ except Exception as e:
+ self.log.error("An error occurred in converting the Facebook Ad Rows. \n Error %s", e)
+ raise
Review comment:
@turbaszek looks like I handling errors in the hook, thanks for pointing it out. I don't think the exception is required. Before I would iterate over the cursor in the operator, but now its just type casting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401215099
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = 'Job Completed'
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ :return: list of Facebook Ads AdsInsights object(s)
+ :rtype: list[AdsInsights]
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.facebook_ads_config: Dict[str, Any] = {}
+ self.api_version = api_version
+ self.ad_account_id: str = ""
+ self.app_id: str = ""
+ self.app_secret: str = ""
+ self.access_token: str = ""
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ self._get_config()
+ for key, val in self.facebook_ads_config.items():
+ setattr(self, key, val)
+
+ return FacebookAdsApi.init(app_id=self.app_id,
Review comment:
Ya this can be done!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/510a377b50ae757ce3c5d5b455c4c77d7dc9068e&el=desc) will **decrease** coverage by `55.07%`.
> The diff coverage is `31.09%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
===========================================
- Coverage 88.30% 33.23% -55.08%
===========================================
Files 935 938 +3
Lines 45243 45362 +119
===========================================
- Hits 39951 15075 -24876
- Misses 5292 30287 +24995
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `53.52% <ø> (-41.55%)` | :arrow_down: |
| [...le/facebook\_ads\_to\_gcs/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3MvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYWRzLnB5) | `0.00% <0.00%> (ø)` | |
| [...viders/google/facebook\_ads\_to\_gcs/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3Mvb3BlcmF0b3JzL2Fkcy5weQ==) | `36.11% <36.11%> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `39.65% <39.65%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.37% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [814 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [510a377...5234efa](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r407665649
##########
File path: airflow/providers/google/facebook_ads_to_gcs/example_dags/example_ads.py
##########
@@ -0,0 +1,110 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example Airflow DAG that shows how to use FacebookAdsReportToGcsOperator.
+"""
+from facebook_business.adobjects.adsinsights import AdsInsights
+
+from airflow import models
+from airflow.providers.google.cloud.operators.bigquery import (
+ BigQueryCreateEmptyDatasetOperator, BigQueryCreateEmptyTableOperator, BigQueryExecuteQueryOperator,
+)
+from airflow.providers.google.cloud.operators.gcs_to_bigquery import GCSToBigQueryOperator
+from airflow.providers.google.facebook_ads_to_gcs.operators.ads import FacebookAdsReportToGcsOperator
+from airflow.utils.dates import days_ago
+
+# [START howto_GCS_env_variables]
+GCP_PROJECT_ID = "free-tier-1997"
+GCS_BUCKET = "airflow_bucket_fb"
+GCS_OBJ_PATH = "Temp/this_is_my_report_csv.csv"
+GCS_CONN_ID = "google_cloud_default"
+DATASET_NAME = "airflow_test_dataset"
+TABLE_NAME = "airflow_test_datatable"
+# [END howto_GCS_env_variables]
+
+# [START howto_FB_ADS_env_variables]
+FACEBOOK_ADS_CONN_ID = "facebook_default"
Review comment:
Can you retrieve these values from environment variables? This will show that you should not store those values here.
```
TABLE_NAME = os.environ.get("FB_TABLE_NAME", "airflow_test_datatable")
```
Also I would remove the `FACEBOOK_ADS_CONN_ID` to present fallback to default connection
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r404629717
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,119 @@
+#
Review comment:
Then he will create a new operator, which will not be called FacebookAdsTo**GCS**.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r401386387
##########
File path: airflow/providers/facebook/ads/operators/ads.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ad Reporting to GCS operators.
+"""
+import json
+import tempfile
+from typing import Any, Dict, List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.facebook.ads.hooks.ads import FacebookAdsReportingHook
+from airflow.providers.google.cloud.hooks.gcs import GCSHook
+from airflow.utils.decorators import apply_defaults
+
+
+class FacebookAdsReportToGcsOperator(BaseOperator):
+ """
+ Fetches the results from the Facebook Ads API as desired in the params
+ Converts and saves the data as a temporary JSON file
+ Uploads the JSON to Google Cloud Storage
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ .. seealso::
+ For more information on the Facebook Ads Python SDK, take a look at the docs:
+ https://github.com/facebook/facebook-python-business-sdk
+
+ :param bucket: The GCS bucket to upload to
+ :type bucket: str
+ :param obj: GCS path to save the object. Must be the full file path (ex. `path/to/file.txt`)
+ :type obj: str
+ :param gcp_conn_id: Airflow Google Cloud Platform connection ID
+ :type gcp_conn_id: str
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type params: Dict[str, Any]
+ :param gzip: Option to compress local file or file data for upload
+ :type gzip: bool
+ """
+
+ template_fields = ("facebook_ads_conn_id", "bucket", "obj")
+
+ @apply_defaults
+ def __init__(
+ self,
+ bucket: str,
+ obj: str,
+ gcp_conn_id: str = "google_cloud_default",
+ facebook_ads_conn_id: str = "facebook_ads_default",
+ api_version: str = "v6.0",
+ fields: Optional[List[str]] = None,
+ params: Optional[Dict[str, Any]] = None,
+ gzip: bool = False,
+ *args,
+ **kwargs,
+ ) -> None:
+ super().__init__(*args, **kwargs)
+ self.bucket = bucket
+ self.obj = obj
+ self.gcp_conn_id = gcp_conn_id
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.fields = fields
+ self.params = params # type:ignore
+ self.gzip = gzip
+
+ def execute(self, context: Dict):
+ service = FacebookAdsReportingHook(facebook_ads_conn_id=self.facebook_ads_conn_id,
+ api_version=self.api_version)
+ rows = service.bulk_facebook_report(params=self.params, fields=self.fields)
+ try:
+ converted_rows = [dict(row) for row in rows]
+ except Exception as e:
+ self.log.error("An error occurred in converting the Facebook Ad Rows. \n Error %s", e)
+ raise
+
+ with tempfile.NamedTemporaryFile("w", suffix=".json") as json_file:
+ json.dump(converted_rows, json_file)
+ json_file.flush()
+ hook = GCSHook(gcp_conn_id=self.gcp_conn_id)
+ hook.upload(
+ bucket_name=self.bucket,
+ object_name=self.obj,
+ filename=json_file.name,
+ gzip=self.gzip,
+ )
+ self.log.info("%s uploaded to GCS", json_file.name)
Review comment:
Should we consider return URI to the uploaded file for example `return f"gs://{self.bucket}/{self.obj}"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/45c8983306ab1c54abdacd8f870e790fad25cb37&el=desc) will **decrease** coverage by `0.76%`.
> The diff coverage is `94.11%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
==========================================
- Coverage 88.45% 87.68% -0.77%
==========================================
Files 937 940 +3
Lines 45234 45353 +119
==========================================
- Hits 40011 39768 -243
- Misses 5223 5585 +362
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `95.07% <ø> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `87.93% <87.93%> (ø)` | |
| [...le/facebook\_ads\_to\_gcs/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3MvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYWRzLnB5) | `100.00% <100.00%> (ø)` | |
| [...viders/google/facebook\_ads\_to\_gcs/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3Mvb3BlcmF0b3JzL2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.37% <100.00%> (+0.01%)` | :arrow_up: |
| [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
| [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
| [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| ... and [18 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [45c8983...ff2fcf6](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/510a377b50ae757ce3c5d5b455c4c77d7dc9068e&el=desc) will **decrease** coverage by `22.68%`.
> The diff coverage is `73.94%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
===========================================
- Coverage 88.30% 65.61% -22.69%
===========================================
Files 935 938 +3
Lines 45243 45362 +119
===========================================
- Hits 39951 29764 -10187
- Misses 5292 15598 +10306
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `75.35% <ø> (-19.72%)` | :arrow_down: |
| [...le/facebook\_ads\_to\_gcs/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3MvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYWRzLnB5) | `0.00% <0.00%> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `87.93% <87.93%> (ø)` | |
| [...viders/google/facebook\_ads\_to\_gcs/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3Mvb3BlcmF0b3JzL2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.37% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [510 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [510a377...5234efa](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] randr97 commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
randr97 commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402746519
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,152 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from cached_property import cached_property
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookError, FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = "Job Completed"
+JOB_FAILED = "Job Failed"
+JOB_SKIPPED = "Job Skipped"
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ config = self.facebook_ads_config
+ try:
+ return FacebookAdsApi.init(app_id=config["app_id"],
+ app_secret=config["app_secret"],
+ access_token=config["access_token"],
+ account_id=config["account_id"],
+ api_version=self.api_version)
+ except AirflowException as error:
+ message = "Missing config data:\n {error}".format(error=error)
+ self._error_handler(message)
+ raise
+
+ @cached_property
+ def facebook_ads_config(self) -> None:
+ """
+ Gets Facebook ads connection from meta db and sets
+ facebook_ads_config attribute with returned config file
+ """
+ conn = self.get_connection(self.facebook_ads_conn_id)
+ if "facebook_ads_client" not in conn.extra_dejson:
+ raise AirflowException("facebook_ads_client not found in extra field")
+ return conn.extra_dejson.get("facebook_ads_client")
+
+ def bulk_facebook_report(
+ self,
+ params: Optional[Dict[str, Any]] = None,
+ fields: Optional[List[str]] = None,
+ ) -> List[AdsInsights]:
+ """
+ Pulls data from the Facebook Ads API
+
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+
+ :return: Facebook Ads API response, converted to Facebook Ads Row objects
+ :rtype: List[AdsInsights]
+ """
+ api = self._get_service()
+ params = params if params else self.default_params
+ ad_account = AdAccount(api.get_default_account_id(), api=api)
+ _async = ad_account.get_insights(params=params, fields=fields, is_async=True)
+ try:
+ while _async.api_get()["async_status"] != JOB_COMPLETED:
+ percent = _async.api_get()["async_percent_completion"]
+ self.log.info("%s %s completed", percent, "%")
+ time.sleep(10)
+ if _async.api_get()["async_status"] in [JOB_SKIPPED, JOB_FAILED]:
+ self._error_handler("Task failed Retry...")
+ raise FacebookError
+ report_run_id = _async.api_get()["report_run_id"]
+ report_object = AdReportRun(report_run_id, api=api)
+ insights = report_object.get_insights()
+ return self._extract_rows(insights)
+ except FacebookRequestError as error:
+ self._error_handler(error)
Review comment:
So the reason I put error handling was for formatted logs, but i think such changes done at the base level would make more sense and not in inherited hooks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/cc9b1bc78df1e82da37b885d2a8aaf5779c6209d&el=desc) will **decrease** coverage by `0.42%`.
> The diff coverage is `82.14%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
==========================================
- Coverage 87.27% 86.85% -0.43%
==========================================
Files 937 940 +3
Lines 45549 45661 +112
==========================================
- Hits 39753 39657 -96
- Misses 5796 6004 +208
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `95.07% <ø> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `69.23% <69.23%> (ø)` | |
| [...providers/facebook/ads/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2V4YW1wbGVfZGFncy9leGFtcGxlX2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [airflow/providers/facebook/ads/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL29wZXJhdG9ycy9hZHMucHk=) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.36% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `39.28% <0.00%> (-60.72%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0.00%> (-45.66%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| ... and [12 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [cc9b1bc...f22ea1c](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402733708
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,152 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from cached_property import cached_property
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookError, FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = "Job Completed"
+JOB_FAILED = "Job Failed"
+JOB_SKIPPED = "Job Skipped"
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ config = self.facebook_ads_config
+ try:
+ return FacebookAdsApi.init(app_id=config["app_id"],
+ app_secret=config["app_secret"],
+ access_token=config["access_token"],
+ account_id=config["account_id"],
+ api_version=self.api_version)
+ except AirflowException as error:
+ message = "Missing config data:\n {error}".format(error=error)
+ self._error_handler(message)
+ raise
+
+ @cached_property
+ def facebook_ads_config(self) -> None:
+ """
+ Gets Facebook ads connection from meta db and sets
+ facebook_ads_config attribute with returned config file
+ """
+ conn = self.get_connection(self.facebook_ads_conn_id)
+ if "facebook_ads_client" not in conn.extra_dejson:
+ raise AirflowException("facebook_ads_client not found in extra field")
+ return conn.extra_dejson.get("facebook_ads_client")
+
+ def bulk_facebook_report(
+ self,
+ params: Optional[Dict[str, Any]] = None,
+ fields: Optional[List[str]] = None,
+ ) -> List[AdsInsights]:
+ """
+ Pulls data from the Facebook Ads API
+
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+
+ :return: Facebook Ads API response, converted to Facebook Ads Row objects
+ :rtype: List[AdsInsights]
+ """
+ api = self._get_service()
+ params = params if params else self.default_params
+ ad_account = AdAccount(api.get_default_account_id(), api=api)
+ _async = ad_account.get_insights(params=params, fields=fields, is_async=True)
+ try:
+ while _async.api_get()["async_status"] != JOB_COMPLETED:
+ percent = _async.api_get()["async_percent_completion"]
+ self.log.info("%s %s completed", percent, "%")
+ time.sleep(10)
+ if _async.api_get()["async_status"] in [JOB_SKIPPED, JOB_FAILED]:
Review comment:
This condition should be after the loop if you enter my suggestion. You will also need to check the success of the surgery.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8008: Custom
Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#issuecomment-606854674
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=h1) Report
> Merging [#8008](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/510a377b50ae757ce3c5d5b455c4c77d7dc9068e&el=desc) will **decrease** coverage by `22.57%`.
> The diff coverage is `73.94%`.
[](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8008 +/- ##
===========================================
- Coverage 88.30% 65.72% -22.58%
===========================================
Files 935 938 +3
Lines 45243 45362 +119
===========================================
- Hits 39951 29814 -10137
- Misses 5292 15548 +10256
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `75.35% <ø> (-19.72%)` | :arrow_down: |
| [...le/facebook\_ads\_to\_gcs/example\_dags/example\_ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3MvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYWRzLnB5) | `0.00% <0.00%> (ø)` | |
| [airflow/providers/facebook/ads/hooks/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZmFjZWJvb2svYWRzL2hvb2tzL2Fkcy5weQ==) | `87.93% <87.93%> (ø)` | |
| [...viders/google/facebook\_ads\_to\_gcs/operators/ads.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2ZhY2Vib29rX2Fkc190b19nY3Mvb3BlcmF0b3JzL2Fkcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.37% <100.00%> (+0.01%)` | :arrow_up: |
| [airflow/hooks/S3\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9TM19ob29rLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/pig\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9waWdfaG9vay5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/http\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9odHRwX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [507 more](https://codecov.io/gh/apache/airflow/pull/8008/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=footer). Last update [510a377...5234efa](https://codecov.io/gh/apache/airflow/pull/8008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8008:
Custom Facebook Ads Operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8008: Custom Facebook Ads Operator
URL: https://github.com/apache/airflow/pull/8008#discussion_r402733145
##########
File path: airflow/providers/facebook/ads/hooks/ads.py
##########
@@ -0,0 +1,152 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+This module contains Facebook Ads Reporting hooks
+"""
+import time
+from typing import Any, Dict, List, Optional
+
+from cached_property import cached_property
+from facebook_business.adobjects.adaccount import AdAccount
+from facebook_business.adobjects.adreportrun import AdReportRun
+from facebook_business.adobjects.adsinsights import AdsInsights
+from facebook_business.api import Cursor, FacebookAdsApi
+from facebook_business.exceptions import FacebookError, FacebookRequestError
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+
+JOB_COMPLETED = "Job Completed"
+JOB_FAILED = "Job Failed"
+JOB_SKIPPED = "Job Skipped"
+
+
+class FacebookAdsReportingHook(BaseHook):
+ """
+ Hook for the Facebook Ads API
+
+ .. seealso::
+ For more information on the Facebook Ads API, take a look at the API docs:
+ https://developers.facebook.com/docs/marketing-apis/
+
+ :param facebook_ads_conn_id: Airflow Facebook Ads connection ID
+ :type facebook_ads_conn_id: str
+ :param api_version: The version of Facebook API. Default to v6.0
+ :type api_version: str
+
+ """
+
+ def __init__(
+ self,
+ facebook_ads_conn_id: str = "facebook_default",
+ api_version: str = "v6.0",
+ ) -> None:
+ super().__init__()
+ self.facebook_ads_conn_id = facebook_ads_conn_id
+ self.api_version = api_version
+ self.default_params = {
+ 'level': 'ad',
+ 'date_preset': 'yesterday',
+ }
+
+ def _get_service(self) -> FacebookAdsApi:
+ """ Returns Facebook Ads Client using a service account"""
+ config = self.facebook_ads_config
+ try:
+ return FacebookAdsApi.init(app_id=config["app_id"],
+ app_secret=config["app_secret"],
+ access_token=config["access_token"],
+ account_id=config["account_id"],
+ api_version=self.api_version)
+ except AirflowException as error:
+ message = "Missing config data:\n {error}".format(error=error)
+ self._error_handler(message)
+ raise
+
+ @cached_property
+ def facebook_ads_config(self) -> None:
+ """
+ Gets Facebook ads connection from meta db and sets
+ facebook_ads_config attribute with returned config file
+ """
+ conn = self.get_connection(self.facebook_ads_conn_id)
+ if "facebook_ads_client" not in conn.extra_dejson:
+ raise AirflowException("facebook_ads_client not found in extra field")
+ return conn.extra_dejson.get("facebook_ads_client")
+
+ def bulk_facebook_report(
+ self,
+ params: Optional[Dict[str, Any]] = None,
+ fields: Optional[List[str]] = None,
+ ) -> List[AdsInsights]:
+ """
+ Pulls data from the Facebook Ads API
+
+ :param fields: List of fields that is obtained from Facebook. Found in AdsInsights.Field class.
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+ :type fields: List[str]
+ :param params: Parameters that determine the query for Facebook
+ https://developers.facebook.com/docs/marketing-api/insights/parameters/v6.0
+
+ :return: Facebook Ads API response, converted to Facebook Ads Row objects
+ :rtype: List[AdsInsights]
+ """
+ api = self._get_service()
+ params = params if params else self.default_params
+ ad_account = AdAccount(api.get_default_account_id(), api=api)
+ _async = ad_account.get_insights(params=params, fields=fields, is_async=True)
+ try:
+ while _async.api_get()["async_status"] != JOB_COMPLETED:
Review comment:
Can you check the following 3 states here?
- Job Not Started,
- Job Started,
- Job Running
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services