You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Thiago Souza <tc...@gmail.com> on 2014/10/07 16:35:54 UTC
Samza and Amazon Kinesis
Hi there,
I'm quite new to Samza and currently evaluating it for a new project.
Does it make any sense to implement an alternative to Kafka based on Amazon
Kinesis?
Thanks,
Thiago Souza
Re: Samza and Amazon Kinesis
Posted by Martin Kleppmann <ma...@kleppmann.com>.
Hi Thiago,
Have you had a chance to work on the Kinesis support for Samza? I've had a few people ask me about it, so perhaps there's an opportunity for folks to collaborate. There's definitely interest.
I've also got an engineering contact for Kinesis at Amazon who is interested in helping. I don't have time to work on this myself, but if you want to ping me off-list I can put you in touch. (Anyone else lurking on this list who's interested in Kinesis?)
Best,
Martin
On 7 Oct 2014, at 21:40, Thiago Souza <tc...@gmail.com> wrote:
> Hi Chris,
>
> Thanks for the clarification! I'll provide more information as soon as I
> get something implemented.
>
> Regards
>
> On Tue, Oct 7, 2014 at 5:31 PM, Chris Riccomini <
> criccomini@linkedin.com.invalid> wrote:
>
>> Hey Thiago,
>>
>> You can return null for these APIs. If you do so, it's up to the
>> underlying SystemConsumer implementation to determine what to do, since it
>> doesn't know where to start reading from. In Kafka's case, it just falls
>> back to the auto.offset.reset setting in Kafka's consumer config API,
>> which defaults to the most recent message. In Kinesis' case, you could
>> have the SystemConsumer start reading from anywhere that makes sense.
>>
>> Cheers,
>> Chris
>>
>> On 10/7/14 1:26 PM, "Thiago Souza" <tc...@gmail.com> wrote:
>>
>>> Hello all,
>>>
>>> I've been doing a little research on this.
>>>
>>> It seems that the first obstacle would be with
>>> "SystemAdmin.getOffsetsAfter" as it looks like that Amazon Kinesis does
>>> not
>>> provide this information out of the box.
>>>
>>> Regards
>>>
>>> On Tue, Oct 7, 2014 at 5:02 PM, Scott Clasen <sc...@heroku.com> wrote:
>>>
>>>> +1
>>>>
>>>> On Tue, Oct 7, 2014 at 12:51 PM, Martin Kleppmann <martin@kleppmann.com
>>>
>>>> wrote:
>>>>
>>>>> I've personally not used Kinesis before, but as far as I know it's
>>>> very
>>>>> similar to Kafka (it is also based on a total order of messages per
>>>>> partition), so Samza should be able to support it well.
>>>>>
>>>>> If you want to develop a SystemConsumer/SystemProducer for Kinesis, it
>>>>> would be an awesome contribution to Samza :)
>>>>>
>>>>> Thanks,
>>>>> Martin
>>>>>
>>>>> On 7 Oct 2014, at 15:35, Thiago Souza <tc...@gmail.com> wrote:
>>>>>
>>>>>> Hi there,
>>>>>>
>>>>>> I'm quite new to Samza and currently evaluating it for a new
>>>> project.
>>>>>>
>>>>>> Does it make any sense to implement an alternative to Kafka based on
>>>>> Amazon
>>>>>> Kinesis?
>>>>>>
>>>>>> Thanks,
>>>>>> Thiago Souza
>>>>>
>>>>>
>>>>
>>
>>
Re: Samza and Amazon Kinesis
Posted by Thiago Souza <tc...@gmail.com>.
Hi Chris,
Thanks for the clarification! I'll provide more information as soon as I
get something implemented.
Regards
On Tue, Oct 7, 2014 at 5:31 PM, Chris Riccomini <
criccomini@linkedin.com.invalid> wrote:
> Hey Thiago,
>
> You can return null for these APIs. If you do so, it's up to the
> underlying SystemConsumer implementation to determine what to do, since it
> doesn't know where to start reading from. In Kafka's case, it just falls
> back to the auto.offset.reset setting in Kafka's consumer config API,
> which defaults to the most recent message. In Kinesis' case, you could
> have the SystemConsumer start reading from anywhere that makes sense.
>
> Cheers,
> Chris
>
> On 10/7/14 1:26 PM, "Thiago Souza" <tc...@gmail.com> wrote:
>
> >Hello all,
> >
> >I've been doing a little research on this.
> >
> >It seems that the first obstacle would be with
> >"SystemAdmin.getOffsetsAfter" as it looks like that Amazon Kinesis does
> >not
> >provide this information out of the box.
> >
> >Regards
> >
> >On Tue, Oct 7, 2014 at 5:02 PM, Scott Clasen <sc...@heroku.com> wrote:
> >
> >> +1
> >>
> >> On Tue, Oct 7, 2014 at 12:51 PM, Martin Kleppmann <martin@kleppmann.com
> >
> >> wrote:
> >>
> >> > I've personally not used Kinesis before, but as far as I know it's
> >>very
> >> > similar to Kafka (it is also based on a total order of messages per
> >> > partition), so Samza should be able to support it well.
> >> >
> >> > If you want to develop a SystemConsumer/SystemProducer for Kinesis, it
> >> > would be an awesome contribution to Samza :)
> >> >
> >> > Thanks,
> >> > Martin
> >> >
> >> > On 7 Oct 2014, at 15:35, Thiago Souza <tc...@gmail.com> wrote:
> >> >
> >> > > Hi there,
> >> > >
> >> > > I'm quite new to Samza and currently evaluating it for a new
> >>project.
> >> > >
> >> > > Does it make any sense to implement an alternative to Kafka based on
> >> > Amazon
> >> > > Kinesis?
> >> > >
> >> > > Thanks,
> >> > > Thiago Souza
> >> >
> >> >
> >>
>
>
Re: Samza and Amazon Kinesis
Posted by Chris Riccomini <cr...@linkedin.com.INVALID>.
Hey Thiago,
You can return null for these APIs. If you do so, it's up to the
underlying SystemConsumer implementation to determine what to do, since it
doesn't know where to start reading from. In Kafka's case, it just falls
back to the auto.offset.reset setting in Kafka's consumer config API,
which defaults to the most recent message. In Kinesis' case, you could
have the SystemConsumer start reading from anywhere that makes sense.
Cheers,
Chris
On 10/7/14 1:26 PM, "Thiago Souza" <tc...@gmail.com> wrote:
>Hello all,
>
>I've been doing a little research on this.
>
>It seems that the first obstacle would be with
>"SystemAdmin.getOffsetsAfter" as it looks like that Amazon Kinesis does
>not
>provide this information out of the box.
>
>Regards
>
>On Tue, Oct 7, 2014 at 5:02 PM, Scott Clasen <sc...@heroku.com> wrote:
>
>> +1
>>
>> On Tue, Oct 7, 2014 at 12:51 PM, Martin Kleppmann <ma...@kleppmann.com>
>> wrote:
>>
>> > I've personally not used Kinesis before, but as far as I know it's
>>very
>> > similar to Kafka (it is also based on a total order of messages per
>> > partition), so Samza should be able to support it well.
>> >
>> > If you want to develop a SystemConsumer/SystemProducer for Kinesis, it
>> > would be an awesome contribution to Samza :)
>> >
>> > Thanks,
>> > Martin
>> >
>> > On 7 Oct 2014, at 15:35, Thiago Souza <tc...@gmail.com> wrote:
>> >
>> > > Hi there,
>> > >
>> > > I'm quite new to Samza and currently evaluating it for a new
>>project.
>> > >
>> > > Does it make any sense to implement an alternative to Kafka based on
>> > Amazon
>> > > Kinesis?
>> > >
>> > > Thanks,
>> > > Thiago Souza
>> >
>> >
>>
Re: Samza and Amazon Kinesis
Posted by Thiago Souza <tc...@gmail.com>.
Hello all,
I've been doing a little research on this.
It seems that the first obstacle would be with
"SystemAdmin.getOffsetsAfter" as it looks like that Amazon Kinesis does not
provide this information out of the box.
Regards
On Tue, Oct 7, 2014 at 5:02 PM, Scott Clasen <sc...@heroku.com> wrote:
> +1
>
> On Tue, Oct 7, 2014 at 12:51 PM, Martin Kleppmann <ma...@kleppmann.com>
> wrote:
>
> > I've personally not used Kinesis before, but as far as I know it's very
> > similar to Kafka (it is also based on a total order of messages per
> > partition), so Samza should be able to support it well.
> >
> > If you want to develop a SystemConsumer/SystemProducer for Kinesis, it
> > would be an awesome contribution to Samza :)
> >
> > Thanks,
> > Martin
> >
> > On 7 Oct 2014, at 15:35, Thiago Souza <tc...@gmail.com> wrote:
> >
> > > Hi there,
> > >
> > > I'm quite new to Samza and currently evaluating it for a new project.
> > >
> > > Does it make any sense to implement an alternative to Kafka based on
> > Amazon
> > > Kinesis?
> > >
> > > Thanks,
> > > Thiago Souza
> >
> >
>
Re: Samza and Amazon Kinesis
Posted by Scott Clasen <sc...@heroku.com>.
+1
On Tue, Oct 7, 2014 at 12:51 PM, Martin Kleppmann <ma...@kleppmann.com>
wrote:
> I've personally not used Kinesis before, but as far as I know it's very
> similar to Kafka (it is also based on a total order of messages per
> partition), so Samza should be able to support it well.
>
> If you want to develop a SystemConsumer/SystemProducer for Kinesis, it
> would be an awesome contribution to Samza :)
>
> Thanks,
> Martin
>
> On 7 Oct 2014, at 15:35, Thiago Souza <tc...@gmail.com> wrote:
>
> > Hi there,
> >
> > I'm quite new to Samza and currently evaluating it for a new project.
> >
> > Does it make any sense to implement an alternative to Kafka based on
> Amazon
> > Kinesis?
> >
> > Thanks,
> > Thiago Souza
>
>
Re: Samza and Amazon Kinesis
Posted by Martin Kleppmann <ma...@kleppmann.com>.
I've personally not used Kinesis before, but as far as I know it's very similar to Kafka (it is also based on a total order of messages per partition), so Samza should be able to support it well.
If you want to develop a SystemConsumer/SystemProducer for Kinesis, it would be an awesome contribution to Samza :)
Thanks,
Martin
On 7 Oct 2014, at 15:35, Thiago Souza <tc...@gmail.com> wrote:
> Hi there,
>
> I'm quite new to Samza and currently evaluating it for a new project.
>
> Does it make any sense to implement an alternative to Kafka based on Amazon
> Kinesis?
>
> Thanks,
> Thiago Souza