[jira] [Updated] (AIRFLOW-4588) Add GoogleDiscoveryApiHook and GoogleApiToS3Transfer

["Ash Berlin-Taylor (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-4588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ash Berlin-Taylor updated AIRFLOW-4588:
---------------------------------------
    Fix Version/s:     (was: 1.10.6)
                   2.0.0

> Add GoogleDiscoveryApiHook and GoogleApiToS3Transfer
> ----------------------------------------------------
>
>                 Key: AIRFLOW-4588
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4588
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: contrib, gcp
>    Affects Versions: 1.10.3
>            Reporter: Felix Uellendall
>            Assignee: Felix Uellendall
>            Priority: Major
>             Fix For: 2.0.0
>
>
> This PR adds a hook based on Google's Discovery API to communicate to any Google Services.
> And also a implementation of this hook in an operator that transfers data from Google to S3.
> - add documentation to integration.rst
> The hook provides:
> - a get_conn function to authenticate to the Google API via an airflow connection
> - a query function to dynamically query all data available for a specific endpoint and given parameters. (You are able to either retrieve one page of data or all data)
> The transfer operator provides:
> - basic transfer between google api and s3
> - passing an xcom variable to dynamically set the endpoint params for a request
> - exposing the response data to xcom, but raises exception when it exceeds MAX_XCOM_SIZE



--
This message was sent by Atlassian Jira
(v8.3.4#803005)