You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Jeyhun Karimov <je...@gmail.com> on 2016/08/04 16:11:20 UTC

Kafka Streams input

Hi community,

Can Kafka Streams use input sources other than Kafka? For example can it
read data from HDFS? You can assume that for each Streams instance
initiating, I will have separate  partition (in this case file path
containing data) and each instance will read from one file path.

Cheers
Jeyhun
-- 
-Cheers

Jeyhun

Re: Kafka Streams input

Posted by Jeyhun Karimov <je...@gmail.com>.

Thanks in advance Eno.

-Jeyhun


On Thu, Aug 4, 2016 at 6:25 PM Eno Thereska <en...@gmail.com> wrote:

> Hi Jeyhun,
>
> You can use Kafka Connect to bring data into Kafka from HDFS then use
> Kafka Streams as usual. I believe there is a desire to make the Connect +
> Streams more integrated so it doesn't feel like you're running two
> different things, and hopefully that integration happens soon.
>
> Thanks
> Eno
>
> > On 4 Aug 2016, at 17:11, Jeyhun Karimov <je...@gmail.com> wrote:
> >
> > Hi community,
> >
> > Can Kafka Streams use input sources other than Kafka? For example can it
> > read data from HDFS? You can assume that for each Streams instance
> > initiating, I will have separate  partition (in this case file path
> > containing data) and each instance will read from one file path.
> >
> > Cheers
> > Jeyhun
> > --
> > -Cheers
> >
> > Jeyhun
>
> --
-Cheers

Jeyhun

Re: Kafka Streams input

Posted by Eno Thereska <en...@gmail.com>.

Hi Jeyhun,

You can use Kafka Connect to bring data into Kafka from HDFS then use Kafka Streams as usual. I believe there is a desire to make the Connect + Streams more integrated so it doesn't feel like you're running two different things, and hopefully that integration happens soon. 

Thanks
Eno

> On 4 Aug 2016, at 17:11, Jeyhun Karimov <je...@gmail.com> wrote:
> 
> Hi community,
> 
> Can Kafka Streams use input sources other than Kafka? For example can it
> read data from HDFS? You can assume that for each Streams instance
> initiating, I will have separate  partition (in this case file path
> containing data) and each instance will read from one file path.
> 
> Cheers
> Jeyhun
> -- 
> -Cheers
> 
> Jeyhun