Pyflink -> Redshift/S3/Firehose

[Zain Haider Nemati <za...@retailo.co>]

Hi,
I am working on writing a flink processor which has to send transformed
data to redshift/S3.
I do not find any sort of documentation for pyflink in reference to how to
send data to firehose,s3 or redshift. Would appreciate some help here.

Re: Pyflink -> Redshift/S3/Firehose

[Dian Fu <di...@gmail.com>]

It uses connectors to send data to external storages. It should be noted
that it shares the connector implementations between Java API and Python
API and so if you could find a Java connector, usually it could be also be
used in PyFlink.

For firehose, it has provided a firehose sink connector in Flink 1.15
(still not released yet and should be released this week). You can refer to
Table API & SQL firehose connector [1] and DataStream firehose connector
[2] for more details. Regarding to how to use these connectors in PyFlink,
could refer to [3] [4] [5] for more details.

For s3, I guess you could use FileSystem connector. See [6][7] for more
details. Note that FileSystem connectors are introduced in earlier releases
and so you don't need to wait for Flink 1.15.

Regards,
Dian

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/table/firehose/
[2]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/firehose/
[3]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/python/table/intro_to_table_api/#create-using-ddl-statements
[4]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/python/datastream/intro_to_datastream_api/#create-using-datastream-connectors
[5]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/python/datastream/intro_to_datastream_api/#create-using-table--sql-connectors
[6]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/table/filesystem/
[7]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/filesystem/


On Mon, May 2, 2022 at 9:37 PM Zain Haider Nemati <za...@retailo.co>
wrote:

> Hi,
> I am working on writing a flink processor which has to send transformed
> data to redshift/S3.
> I do not find any sort of documentation for pyflink in reference to how to
> send data to firehose,s3 or redshift. Would appreciate some help here.
>