[jira] [Commented] (FLINK-22665) Add a Flink BigQuery Connector

["Matt Brown (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-22665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17517873#comment-17517873 ] 

Matt Brown commented on FLINK-22665:
------------------------------------

I wrote a very simple source using the BigQuery Storage API as a bit of a PoC. [https://github.com/mnbbrown/bigquery-connector] 

It needs a lot of testing with different data sources and formats (and some tests written).

Writing a sink shouldn't be too difficult either. 

> Add a Flink BigQuery Connector
> ------------------------------
>
>                 Key: FLINK-22665
>                 URL: https://issues.apache.org/jira/browse/FLINK-22665
>             Project: Flink
>          Issue Type: Improvement
>            Reporter: Jörn Kottmann
>            Priority: Minor
>
> BigQuery is a serverless big data warehouse which can easily manage the amounts of data typically processed by a Flink streaming job over a longer period of time. Some companies heavily use BigQuery to make their data available within their organization. Also writing data to BigQuery can help to debug and analyze the outputs from a Flink pipeline.
>  
> I would like to propose to add a BigQuery connector directly to Flink and ship it with the distribution as it is done for other stores. 
> The google cloud java client is licensed under AL 2.0 and can be a direct dependency to Apache Flink.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)