You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Brahmachaitanya Wajapey <br...@gmail.com> on 2020/06/06 18:03:13 UTC

Using KinesisIO to put records in Firehose

Hi,

Apache beam supports KinesisIO connector. Is this only for Kinesis streams or it can be used to put records in Kinesis firehose as well?

I have a requirement where i read events from kafka and I have to put the records in kinesis firehose.

So, can we use KinesisIO connector for this?

Thanks
Brahma

Re: Using KinesisIO to put records in Firehose

Posted by Jonothan Farr <jo...@gmail.com>.
I believe KinesisIO only works with Kinesis Data Streams.

> On Jun 8, 2020, at 9:42 AM, Brahmachaitanya Wajapey <br...@gmail.com> wrote:
> 
> Hi,
> 
> Apache beam supports KinesisIO connector. Is this only for Kinesis streams or it can be used to put records in Kinesis firehose as well?
> 
> I have a requirement where i read events from kafka and I have to put the records in kinesis firehose.
> 
> So, can we use KinesisIO connector for this?
> 
> Thanks
> Brahma

Re: Using KinesisIO to put records in Firehose

Posted by Alexey Romanenko <ar...@gmail.com>.
+ cross+posting to user@beam.apache.org <ma...@beam.apache.org>

Under the hood, KinesisIO uses IKinesisProducer to write to Kinesis stream. I’m not very familiar with Firehose API, so I’m not sure that KPL supports writes directly to Firehose. 

In the same time, Kinesis Stream can be used as a source for Kinesis Firehose. 

So, I guess the chain could be like this:
KinesisIO → Kinesis Stream → Kinesis Firehose

> On 6 Jun 2020, at 20:03, Brahmachaitanya Wajapey <br...@gmail.com> wrote:
> 
> Hi,
> 
> Apache beam supports KinesisIO connector. Is this only for Kinesis streams or it can be used to put records in Kinesis firehose as well?
> 
> I have a requirement where i read events from kafka and I have to put the records in kinesis firehose.
> 
> So, can we use KinesisIO connector for this?
> 
> Thanks
> Brahma