You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Felix Uellendall (Jira)" <ji...@apache.org> on 2019/09/11 17:05:00 UTC
[jira] [Resolved] (AIRFLOW-4588) Add GoogleDiscoveryApiHook and
GoogleApiToS3Transfer
[ https://issues.apache.org/jira/browse/AIRFLOW-4588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Felix Uellendall resolved AIRFLOW-4588.
---------------------------------------
Resolution: Fixed
> Add GoogleDiscoveryApiHook and GoogleApiToS3Transfer
> ----------------------------------------------------
>
> Key: AIRFLOW-4588
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4588
> Project: Apache Airflow
> Issue Type: New Feature
> Components: contrib, gcp
> Affects Versions: 1.10.3
> Reporter: Felix Uellendall
> Assignee: Felix Uellendall
> Priority: Major
> Fix For: 1.10.6
>
>
> This PR adds a hook based on Google's Discovery API to communicate to any Google Services.
> And also a implementation of this hook in an operator that transfers data from Google to S3.
> - add documentation to integration.rst
> The hook provides:
> - a get_conn function to authenticate to the Google API via an airflow connection
> - a query function to dynamically query all data available for a specific endpoint and given parameters. (You are able to either retrieve one page of data or all data)
> The transfer operator provides:
> - basic transfer between google api and s3
> - passing an xcom variable to dynamically set the endpoint params for a request
> - exposing the response data to xcom, but raises exception when it exceeds MAX_XCOM_SIZE
--
This message was sent by Atlassian Jira
(v8.3.2#803003)