You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Michael D. Spence" <ms...@gmail.com> on 2016/03/22 23:32:00 UTC
Using Kafka for persistence
We have to construct a messaging application that functions as a
switch between other applications in the enterprise. Since our switch
need only have a few days worth of messages, we are considering
using Kafka as the means for persistence of the messages themselves,
using something else such as Oracle or Elasticsearch to do inquiries
and analytics. A query would resolve to a Kafka message ID which we
would use to get the actual text if we needed it (many queries are for
aggregates rather than the messages themselves). We value durability
more than speed.
Has anyone ever done this?
Michael D. Spence
Mockingbird Data Systems, Inc
Re: Using Kafka for persistence
Posted by Rick Mangi <ri...@chartbeat.com>.
This sounds like a square peg in a round hole sort of solution. That said, you might want to look at the work being done with kafka-streams to expose a topic as a table.
> On Mar 30, 2016, at 3:23 PM, Michael D. Spence <ms...@gmail.com> wrote:
>
>
> Any advice on using Kafka to store the actual messages?
>
> On 3/22/2016 6:32 PM, Michael D. Spence wrote:
>> We have to construct a messaging application that functions as a
>> switch between other applications in the enterprise. Since our switch
>> need only have a few days worth of messages, we are considering
>> using Kafka as the means for persistence of the messages themselves,
>> using something else such as Oracle or Elasticsearch to do inquiries
>> and analytics. A query would resolve to a Kafka message ID which we
>> would use to get the actual text if we needed it (many queries are for
>> aggregates rather than the messages themselves). We value durability
>> more than speed.
>>
>> Has anyone ever done this?
>>
>> Michael D. Spence
>> Mockingbird Data Systems, Inc
>
Re: Using Kafka for persistence
Posted by "Michael D. Spence" <ms...@gmail.com>.
Any advice on using Kafka to store the actual messages?
On 3/22/2016 6:32 PM, Michael D. Spence wrote:
> We have to construct a messaging application that functions as a
> switch between other applications in the enterprise. Since our switch
> need only have a few days worth of messages, we are considering
> using Kafka as the means for persistence of the messages themselves,
> using something else such as Oracle or Elasticsearch to do inquiries
> and analytics. A query would resolve to a Kafka message ID which we
> would use to get the actual text if we needed it (many queries are for
> aggregates rather than the messages themselves). We value durability
> more than speed.
>
> Has anyone ever done this?
>
> Michael D. Spence
> Mockingbird Data Systems, Inc