Right Tool

[Patrick Barker <pa...@gmail.com>]

Hey, I'm new to kafka and I'm trying to get a handle on how it all works. I
want to integrate polyglot persistence into my application. Kafka looks
like exactly what I want just on a smaller scale. I am currently only
dealing with about 2,000 users, which may grow,  but is kafka a good use
case here, or is there another technology thats better suited?

Thanks

Re: Right Tool

[Patrick Barker <pa...@gmail.com>]

Yeah, I would want to know they made it there. I like to use polyglot for
the availability of data, I build my recommendation engine in graph, my
bulk data is in mongo, and sql is kind of my default/ad hoc store. This is
working really well for me, but I want to ease up on the payload within my
app and provide a more streamlined synchronization.

On Fri, Sep 12, 2014 at 8:42 PM, Steve Morin <st...@stevemorin.com> wrote:

> You would need make sure they were all persisted down properly to each
> database?  Why are you persisting it to three different databases (sql,
> mongo, graph)?
> -Steve
>
> On Fri, Sep 12, 2014 at 7:35 PM, Patrick Barker <patrickbarkerco@gmail.com
> >
> wrote:
>
> > I'm just getting familiar with kafka, currently I just save everything to
> > all my db's in a single transaction, if any of them fail I roll them all
> > back. However, this is slowing my app down. So, as I understand it I
> could
> > write to kafka, close the transaction, and then it would keep on
> publishing
> > out to my databases. I'm not sure what format I would write it in yet, I
> > guess json
> >
> > On Fri, Sep 12, 2014 at 7:00 PM, Steve Morin <st...@gmail.com>
> > wrote:
> >
> > > What record format are you writing to Kafka with?
> > >
> > > > On Sep 12, 2014, at 17:45, Patrick Barker <patrickbarkerco@gmail.com
> >
> > > wrote:
> > > >
> > > > O, I'm not trying to use it for persistence, I'm wanting to sync 3
> > > > databases: sql, mongo, graph. I want to publish to kafka and then
> have
> > it
> > > > update the db's. I'm wanting to keep this as efficient as possible.
> > > >
> > > >> On Fri, Sep 12, 2014 at 6:39 PM, cacsar@gmail.com <cacsar@gmail.com
> >
> > > wrote:
> > > >>
> > > >> I would say that it depends upon what you mean by persistence. I
> don't
> > > >> believe Kafka is intended to be your permanent data store, but it
> > would
> > > >> work if you were basically write once with appropriate query
> patterns.
> > > It
> > > >> would be an odd way to describe it though.
> > > >>
> > > >> Christian
> > > >>
> > > >>> On Fri, Sep 12, 2014 at 4:05 PM, Stephen Boesch <javadba@gmail.com
> >
> > > wrote:
> > > >>>
> > > >>> Hi Patrick,   Kafka can be used at any scale including small ones
> > > >>> (initially anyways). The issues I ran into personally various
> issues
> > > with
> > > >>> ZooKeeper management and a bug in deleting topics (is that fixed
> > yet?)
> > > >> In
> > > >>> any case you might try out Kafka  - given its highly performant,
> > > >> scalable,
> > > >>> and flexible backbone.   After that you will have little worry
> about
> > > >> scale
> > > >>> - given Kafka's use within massive web scale deployments.
> > > >>>
> > > >>> 2014-09-12 15:18 GMT-07:00 Patrick Barker <
> patrickbarkerco@gmail.com
> > >:
> > > >>>
> > > >>>> Hey, I'm new to kafka and I'm trying to get a handle on how it all
> > > >>> works. I
> > > >>>> want to integrate polyglot persistence into my application. Kafka
> > > looks
> > > >>>> like exactly what I want just on a smaller scale. I am currently
> > only
> > > >>>> dealing with about 2,000 users, which may grow,  but is kafka a
> good
> > > >> use
> > > >>>> case here, or is there another technology thats better suited?
> > > >>>>
> > > >>>> Thanks
> > > >>
> > >
> >
>

Re: Right Tool

[Steve Morin <st...@stevemorin.com>]

You would need make sure they were all persisted down properly to each
database?  Why are you persisting it to three different databases (sql,
mongo, graph)?
-Steve

On Fri, Sep 12, 2014 at 7:35 PM, Patrick Barker <pa...@gmail.com>
wrote:

> I'm just getting familiar with kafka, currently I just save everything to
> all my db's in a single transaction, if any of them fail I roll them all
> back. However, this is slowing my app down. So, as I understand it I could
> write to kafka, close the transaction, and then it would keep on publishing
> out to my databases. I'm not sure what format I would write it in yet, I
> guess json
>
> On Fri, Sep 12, 2014 at 7:00 PM, Steve Morin <st...@gmail.com>
> wrote:
>
> > What record format are you writing to Kafka with?
> >
> > > On Sep 12, 2014, at 17:45, Patrick Barker <pa...@gmail.com>
> > wrote:
> > >
> > > O, I'm not trying to use it for persistence, I'm wanting to sync 3
> > > databases: sql, mongo, graph. I want to publish to kafka and then have
> it
> > > update the db's. I'm wanting to keep this as efficient as possible.
> > >
> > >> On Fri, Sep 12, 2014 at 6:39 PM, cacsar@gmail.com <ca...@gmail.com>
> > wrote:
> > >>
> > >> I would say that it depends upon what you mean by persistence. I don't
> > >> believe Kafka is intended to be your permanent data store, but it
> would
> > >> work if you were basically write once with appropriate query patterns.
> > It
> > >> would be an odd way to describe it though.
> > >>
> > >> Christian
> > >>
> > >>> On Fri, Sep 12, 2014 at 4:05 PM, Stephen Boesch <ja...@gmail.com>
> > wrote:
> > >>>
> > >>> Hi Patrick,   Kafka can be used at any scale including small ones
> > >>> (initially anyways). The issues I ran into personally various issues
> > with
> > >>> ZooKeeper management and a bug in deleting topics (is that fixed
> yet?)
> > >> In
> > >>> any case you might try out Kafka  - given its highly performant,
> > >> scalable,
> > >>> and flexible backbone.   After that you will have little worry about
> > >> scale
> > >>> - given Kafka's use within massive web scale deployments.
> > >>>
> > >>> 2014-09-12 15:18 GMT-07:00 Patrick Barker <patrickbarkerco@gmail.com
> >:
> > >>>
> > >>>> Hey, I'm new to kafka and I'm trying to get a handle on how it all
> > >>> works. I
> > >>>> want to integrate polyglot persistence into my application. Kafka
> > looks
> > >>>> like exactly what I want just on a smaller scale. I am currently
> only
> > >>>> dealing with about 2,000 users, which may grow,  but is kafka a good
> > >> use
> > >>>> case here, or is there another technology thats better suited?
> > >>>>
> > >>>> Thanks
> > >>
> >
>

Re: Right Tool

[Patrick Barker <pa...@gmail.com>]

I'm just getting familiar with kafka, currently I just save everything to
all my db's in a single transaction, if any of them fail I roll them all
back. However, this is slowing my app down. So, as I understand it I could
write to kafka, close the transaction, and then it would keep on publishing
out to my databases. I'm not sure what format I would write it in yet, I
guess json

On Fri, Sep 12, 2014 at 7:00 PM, Steve Morin <st...@gmail.com> wrote:

> What record format are you writing to Kafka with?
>
> > On Sep 12, 2014, at 17:45, Patrick Barker <pa...@gmail.com>
> wrote:
> >
> > O, I'm not trying to use it for persistence, I'm wanting to sync 3
> > databases: sql, mongo, graph. I want to publish to kafka and then have it
> > update the db's. I'm wanting to keep this as efficient as possible.
> >
> >> On Fri, Sep 12, 2014 at 6:39 PM, cacsar@gmail.com <ca...@gmail.com>
> wrote:
> >>
> >> I would say that it depends upon what you mean by persistence. I don't
> >> believe Kafka is intended to be your permanent data store, but it would
> >> work if you were basically write once with appropriate query patterns.
> It
> >> would be an odd way to describe it though.
> >>
> >> Christian
> >>
> >>> On Fri, Sep 12, 2014 at 4:05 PM, Stephen Boesch <ja...@gmail.com>
> wrote:
> >>>
> >>> Hi Patrick,   Kafka can be used at any scale including small ones
> >>> (initially anyways). The issues I ran into personally various issues
> with
> >>> ZooKeeper management and a bug in deleting topics (is that fixed yet?)
> >> In
> >>> any case you might try out Kafka  - given its highly performant,
> >> scalable,
> >>> and flexible backbone.   After that you will have little worry about
> >> scale
> >>> - given Kafka's use within massive web scale deployments.
> >>>
> >>> 2014-09-12 15:18 GMT-07:00 Patrick Barker <pa...@gmail.com>:
> >>>
> >>>> Hey, I'm new to kafka and I'm trying to get a handle on how it all
> >>> works. I
> >>>> want to integrate polyglot persistence into my application. Kafka
> looks
> >>>> like exactly what I want just on a smaller scale. I am currently only
> >>>> dealing with about 2,000 users, which may grow,  but is kafka a good
> >> use
> >>>> case here, or is there another technology thats better suited?
> >>>>
> >>>> Thanks
> >>
>

Re: Right Tool

[Steve Morin <st...@gmail.com>]

What record format are you writing to Kafka with?

> On Sep 12, 2014, at 17:45, Patrick Barker <pa...@gmail.com> wrote:
> 
> O, I'm not trying to use it for persistence, I'm wanting to sync 3
> databases: sql, mongo, graph. I want to publish to kafka and then have it
> update the db's. I'm wanting to keep this as efficient as possible.
> 
>> On Fri, Sep 12, 2014 at 6:39 PM, cacsar@gmail.com <ca...@gmail.com> wrote:
>> 
>> I would say that it depends upon what you mean by persistence. I don't
>> believe Kafka is intended to be your permanent data store, but it would
>> work if you were basically write once with appropriate query patterns. It
>> would be an odd way to describe it though.
>> 
>> Christian
>> 
>>> On Fri, Sep 12, 2014 at 4:05 PM, Stephen Boesch <ja...@gmail.com> wrote:
>>> 
>>> Hi Patrick,   Kafka can be used at any scale including small ones
>>> (initially anyways). The issues I ran into personally various issues with
>>> ZooKeeper management and a bug in deleting topics (is that fixed yet?)
>> In
>>> any case you might try out Kafka  - given its highly performant,
>> scalable,
>>> and flexible backbone.   After that you will have little worry about
>> scale
>>> - given Kafka's use within massive web scale deployments.
>>> 
>>> 2014-09-12 15:18 GMT-07:00 Patrick Barker <pa...@gmail.com>:
>>> 
>>>> Hey, I'm new to kafka and I'm trying to get a handle on how it all
>>> works. I
>>>> want to integrate polyglot persistence into my application. Kafka looks
>>>> like exactly what I want just on a smaller scale. I am currently only
>>>> dealing with about 2,000 users, which may grow,  but is kafka a good
>> use
>>>> case here, or is there another technology thats better suited?
>>>> 
>>>> Thanks
>> 

Re: Right Tool

["cacsar@gmail.com" <ca...@gmail.com>]

Right that makes much more sense. You will probably want to make sure that
your updates are idempotent (or you could just accept the risk), though in
the SQL case you could commit your offset to the DB as part of the same
transaction (requires more custom stuff).

Christian

On Fri, Sep 12, 2014 at 5:45 PM, Patrick Barker <pa...@gmail.com>
wrote:

> O, I'm not trying to use it for persistence, I'm wanting to sync 3
> databases: sql, mongo, graph. I want to publish to kafka and then have it
> update the db's. I'm wanting to keep this as efficient as possible.
>
> On Fri, Sep 12, 2014 at 6:39 PM, cacsar@gmail.com <ca...@gmail.com>
> wrote:
>
> > I would say that it depends upon what you mean by persistence. I don't
> > believe Kafka is intended to be your permanent data store, but it would
> > work if you were basically write once with appropriate query patterns. It
> > would be an odd way to describe it though.
> >
> > Christian
> >
> > On Fri, Sep 12, 2014 at 4:05 PM, Stephen Boesch <ja...@gmail.com>
> wrote:
> >
> > > Hi Patrick,   Kafka can be used at any scale including small ones
> > > (initially anyways). The issues I ran into personally various issues
> with
> > > ZooKeeper management and a bug in deleting topics (is that fixed yet?)
> > In
> > > any case you might try out Kafka  - given its highly performant,
> > scalable,
> > > and flexible backbone.   After that you will have little worry about
> > scale
> > > - given Kafka's use within massive web scale deployments.
> > >
> > > 2014-09-12 15:18 GMT-07:00 Patrick Barker <pa...@gmail.com>:
> > >
> > > > Hey, I'm new to kafka and I'm trying to get a handle on how it all
> > > works. I
> > > > want to integrate polyglot persistence into my application. Kafka
> looks
> > > > like exactly what I want just on a smaller scale. I am currently only
> > > > dealing with about 2,000 users, which may grow,  but is kafka a good
> > use
> > > > case here, or is there another technology thats better suited?
> > > >
> > > > Thanks
> > > >
> > >
> >
>

Re: Right Tool

[Patrick Barker <pa...@gmail.com>]

O, I'm not trying to use it for persistence, I'm wanting to sync 3
databases: sql, mongo, graph. I want to publish to kafka and then have it
update the db's. I'm wanting to keep this as efficient as possible.

On Fri, Sep 12, 2014 at 6:39 PM, cacsar@gmail.com <ca...@gmail.com> wrote:

> I would say that it depends upon what you mean by persistence. I don't
> believe Kafka is intended to be your permanent data store, but it would
> work if you were basically write once with appropriate query patterns. It
> would be an odd way to describe it though.
>
> Christian
>
> On Fri, Sep 12, 2014 at 4:05 PM, Stephen Boesch <ja...@gmail.com> wrote:
>
> > Hi Patrick,   Kafka can be used at any scale including small ones
> > (initially anyways). The issues I ran into personally various issues with
> > ZooKeeper management and a bug in deleting topics (is that fixed yet?)
> In
> > any case you might try out Kafka  - given its highly performant,
> scalable,
> > and flexible backbone.   After that you will have little worry about
> scale
> > - given Kafka's use within massive web scale deployments.
> >
> > 2014-09-12 15:18 GMT-07:00 Patrick Barker <pa...@gmail.com>:
> >
> > > Hey, I'm new to kafka and I'm trying to get a handle on how it all
> > works. I
> > > want to integrate polyglot persistence into my application. Kafka looks
> > > like exactly what I want just on a smaller scale. I am currently only
> > > dealing with about 2,000 users, which may grow,  but is kafka a good
> use
> > > case here, or is there another technology thats better suited?
> > >
> > > Thanks
> > >
> >
>

Re: Right Tool

["cacsar@gmail.com" <ca...@gmail.com>]

I would say that it depends upon what you mean by persistence. I don't
believe Kafka is intended to be your permanent data store, but it would
work if you were basically write once with appropriate query patterns. It
would be an odd way to describe it though.

Christian

On Fri, Sep 12, 2014 at 4:05 PM, Stephen Boesch <ja...@gmail.com> wrote:

> Hi Patrick,   Kafka can be used at any scale including small ones
> (initially anyways). The issues I ran into personally various issues with
> ZooKeeper management and a bug in deleting topics (is that fixed yet?)  In
> any case you might try out Kafka  - given its highly performant, scalable,
> and flexible backbone.   After that you will have little worry about scale
> - given Kafka's use within massive web scale deployments.
>
> 2014-09-12 15:18 GMT-07:00 Patrick Barker <pa...@gmail.com>:
>
> > Hey, I'm new to kafka and I'm trying to get a handle on how it all
> works. I
> > want to integrate polyglot persistence into my application. Kafka looks
> > like exactly what I want just on a smaller scale. I am currently only
> > dealing with about 2,000 users, which may grow,  but is kafka a good use
> > case here, or is there another technology thats better suited?
> >
> > Thanks
> >
>

Re: Right Tool

[Stephen Boesch <ja...@gmail.com>]

Hi Patrick,   Kafka can be used at any scale including small ones
(initially anyways). The issues I ran into personally various issues with
ZooKeeper management and a bug in deleting topics (is that fixed yet?)  In
any case you might try out Kafka  - given its highly performant, scalable,
and flexible backbone.   After that you will have little worry about scale
- given Kafka's use within massive web scale deployments.

2014-09-12 15:18 GMT-07:00 Patrick Barker <pa...@gmail.com>:

> Hey, I'm new to kafka and I'm trying to get a handle on how it all works. I
> want to integrate polyglot persistence into my application. Kafka looks
> like exactly what I want just on a smaller scale. I am currently only
> dealing with about 2,000 users, which may grow,  but is kafka a good use
> case here, or is there another technology thats better suited?
>
> Thanks
>