It would be nice to be able to read Kafka topics in reverse.

[Jonathan Colt <jo...@jivesoftware.com>]

Hi,

Here is a use case that we would like to see kafka’s client support in the future. Currently reading a topic is FIFO. It would be awesome to read a topic in LIFO order. Put another way we would like to be able to read a topic in reverse.


Why? Basically we have per user streams which we have to keep up to date. When an event comes into the system it is determined to be cheap enough to fan out the user stream updates at write time or to expensive and thus deferred until read time. This is where kafka comes into play. We push all the events that were to expensive to fan out at write time onto a kafka topic. When a given user logs on we read through this topic and update the users stream. If it has been a while since the user has logged in it takes a while to catch up. If we could read a topic in reverse order we could catch the most recent events in the stream first. Which in our case is what the user cares about the most. Once the 20 most resent items have been added to a user's stream we can return the head of the stream and continue the catch up process in the background.


Thanks,

Jonathan

Re: It would be nice to be able to read Kafka topics in reverse.

[Otis Gospodnetic <ot...@gmail.com>]

Hi,

We have a similar use case.  In our case it's about performance metrics and
situations where we plan catch up.  Instead of processing data
chronologically, it would make sense to process newest data first, so
performance metrics graphs for "now" have data.

Here is a related thread: http://search-hadoop.com/m/4TaT4HaM6f1

Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/



On Wed, Feb 5, 2014 at 6:45 PM, Jay Kreps <ja...@gmail.com> wrote:

> Technically this is possible with the existing server and protocol and
> could be implemented using the "low level" network client. The high-level
> client doesn't really allow this. This would be a good thing to think about
> as we start on the redesign of that client. I don't think it has to be
> terribly convenient because it is not a very common use case, but just
> being able to read chunks and reverse them in memory would allow this, so I
> think the only thing needed is the ability to set your offset easily.
>
> -Jay
>
>
> On Wed, Feb 5, 2014 at 2:41 PM, Jonathan Colt <
> jonathan.colt@jivesoftware.com> wrote:
>
> > Hi,
> >
> > Here is a use case that we would like to see kafka's client support in
> the
> > future. Currently reading a topic is FIFO. It would be awesome to read a
> > topic in LIFO order. Put another way we would like to be able to read a
> > topic in reverse.
> >
> >
> > Why? Basically we have per user streams which we have to keep up to date.
> > When an event comes into the system it is determined to be cheap enough
> to
> > fan out the user stream updates at write time or to expensive and thus
> > deferred until read time. This is where kafka comes into play. We push
> all
> > the events that were to expensive to fan out at write time onto a kafka
> > topic. When a given user logs on we read through this topic and update
> the
> > users stream. If it has been a while since the user has logged in it
> takes
> > a while to catch up. If we could read a topic in reverse order we could
> > catch the most recent events in the stream first. Which in our case is
> what
> > the user cares about the most. Once the 20 most resent items have been
> > added to a user's stream we can return the head of the stream and
> continue
> > the catch up process in the background.
> >
> >
> > Thanks,
> >
> > Jonathan
> >
> >
>

Re: It would be nice to be able to read Kafka topics in reverse.

[Jay Kreps <ja...@gmail.com>]

Technically this is possible with the existing server and protocol and
could be implemented using the "low level" network client. The high-level
client doesn't really allow this. This would be a good thing to think about
as we start on the redesign of that client. I don't think it has to be
terribly convenient because it is not a very common use case, but just
being able to read chunks and reverse them in memory would allow this, so I
think the only thing needed is the ability to set your offset easily.

-Jay


On Wed, Feb 5, 2014 at 2:41 PM, Jonathan Colt <
jonathan.colt@jivesoftware.com> wrote:

> Hi,
>
> Here is a use case that we would like to see kafka's client support in the
> future. Currently reading a topic is FIFO. It would be awesome to read a
> topic in LIFO order. Put another way we would like to be able to read a
> topic in reverse.
>
>
> Why? Basically we have per user streams which we have to keep up to date.
> When an event comes into the system it is determined to be cheap enough to
> fan out the user stream updates at write time or to expensive and thus
> deferred until read time. This is where kafka comes into play. We push all
> the events that were to expensive to fan out at write time onto a kafka
> topic. When a given user logs on we read through this topic and update the
> users stream. If it has been a while since the user has logged in it takes
> a while to catch up. If we could read a topic in reverse order we could
> catch the most recent events in the stream first. Which in our case is what
> the user cares about the most. Once the 20 most resent items have been
> added to a user's stream we can return the head of the stream and continue
> the catch up process in the background.
>
>
> Thanks,
>
> Jonathan
>
>