Kafka data can be stored on a Dbase instead of file system?

[siddharth ubale <si...@gmail.com>]

Hi Guys,

I wanted to know if we can store data sent to consumers via brokers on to
any database like how ActiveMQ persists data to kahaDB or any other Db like
PostGRE,SQL etc.

Also,when the broker is sending message to consumer and consumer fails, so
when the consumer is up again will the consumer pick data from where it
last left or will it start reading from beginning if i have set the
consumer  from beginning  while coding the consumer.

Thanks,
Siddharth Ubale

Re: Kafka data can be stored on a Dbase instead of file system?

["cacsar@gmail.com" <ca...@gmail.com>]

Based on the phrasing of your first question I might recommend taking
either a closer look at how Kafka works. Kafka stores data on its broker
servers in its own fashion as that is a key part of what makes it useful.
It is not written to use another database for message storage. Consumers of
the data in topics are free to make as many copies as they wish (and could
store it in a database), but such behavior is specific to the application
using it.

For the second question it will depend on what consumer library you use. As
you can see on
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchAPI
it is possible to have a consumer written to behave differently, and there
are building blocks provided (the simple consumer) which might assist in
that endeavour. For example if you were to have a consumer storing all
messages in a database instance for some reason you could write your
consumer to store its offset in that same database as part of the
transaction.

Christian

On Tue, Sep 9, 2014 at 10:40 PM, siddharth ubale <si...@gmail.com>
wrote:

> Hi,
>
> Thanks Sharninder!!
> Does everyone else share the same view as sharninder that it is not
> possible to store data in database and we can only store data in configured
> folder at consumer end  .please do let me know your views.
>
> Thanks,
> Siddharth
>
> On Tue, Sep 9, 2014 at 9:10 PM, Sharninder <sh...@gmail.com> wrote:
>
> > For your first question, I don't know of a way to do that.
> >
> > For the second question, consumers commit the offset after every/a few
> > reads. When the consumer restarts, it starts from the last committed
> > offset.
> >
> >
> > On Tue, Sep 9, 2014 at 7:23 PM, siddharth ubale <
> siddharth.ubale@gmail.com
> > >
> > wrote:
> >
> > > Hi Guys,
> > >
> > > I wanted to know if we can store data sent to consumers via brokers on
> to
> > > any database like how ActiveMQ persists data to kahaDB or any other Db
> > like
> > > PostGRE,SQL etc.
> > >
> > > Also,when the broker is sending message to consumer and consumer fails,
> > so
> > > when the consumer is up again will the consumer pick data from where it
> > > last left or will it start reading from beginning if i have set the
> > > consumer  from beginning  while coding the consumer.
> > >
> > > Thanks,
> > > Siddharth Ubale
> > >
> >
>

Re: Kafka data can be stored on a Dbase instead of file system?

[siddharth ubale <si...@gmail.com>]

Hi,

Thanks Sharninder!!
Does everyone else share the same view as sharninder that it is not
possible to store data in database and we can only store data in configured
folder at consumer end  .please do let me know your views.

Thanks,
Siddharth

On Tue, Sep 9, 2014 at 9:10 PM, Sharninder <sh...@gmail.com> wrote:

> For your first question, I don't know of a way to do that.
>
> For the second question, consumers commit the offset after every/a few
> reads. When the consumer restarts, it starts from the last committed
> offset.
>
>
> On Tue, Sep 9, 2014 at 7:23 PM, siddharth ubale <siddharth.ubale@gmail.com
> >
> wrote:
>
> > Hi Guys,
> >
> > I wanted to know if we can store data sent to consumers via brokers on to
> > any database like how ActiveMQ persists data to kahaDB or any other Db
> like
> > PostGRE,SQL etc.
> >
> > Also,when the broker is sending message to consumer and consumer fails,
> so
> > when the consumer is up again will the consumer pick data from where it
> > last left or will it start reading from beginning if i have set the
> > consumer  from beginning  while coding the consumer.
> >
> > Thanks,
> > Siddharth Ubale
> >
>

Re: Kafka data can be stored on a Dbase instead of file system?

[Sharninder <sh...@gmail.com>]

For your first question, I don't know of a way to do that.

For the second question, consumers commit the offset after every/a few
reads. When the consumer restarts, it starts from the last committed offset.


On Tue, Sep 9, 2014 at 7:23 PM, siddharth ubale <si...@gmail.com>
wrote:

> Hi Guys,
>
> I wanted to know if we can store data sent to consumers via brokers on to
> any database like how ActiveMQ persists data to kahaDB or any other Db like
> PostGRE,SQL etc.
>
> Also,when the broker is sending message to consumer and consumer fails, so
> when the consumer is up again will the consumer pick data from where it
> last left or will it start reading from beginning if i have set the
> consumer  from beginning  while coding the consumer.
>
> Thanks,
> Siddharth Ubale
>