You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Michael Noll <mi...@confluent.io> on 2016/08/02 13:54:04 UTC
Re: Reactive Kafka performance
David,
you wrote:
> Each task would effectively block on DB-io for every history-set
retrieval;
> obviously we would use a TTL cache (KTable could be useful here, but it
> wouldn’t be able to hold “all” of the history for every user)
Can you elaborate a bit why you think a KTable wouldn't be able to hold the
full history for every user? (Not implying I have a different opinion,
simply trying to understand.)
One advantage of using a KTable is data locality for any such lookups
during your stream-table join, i.e. when processing a new incoming record,
you wouldn't incur increased per-record processing latency because of
network round trips (while talking to a remote DB). I'd expect this would
significantly reduce the time needed to finish processing of one new
incoming message. There are further advantages for this setup, see [1].
So if a KTable-based approach would be suitable for your use case, I'd
consider giving that a try.
[1] http://www.confluent.io/blog/elastic-scaling-in-kafka-streams
On Sat, Jul 30, 2016 at 5:44 PM, David Garcia <da...@spiceworks.com> wrote:
> Hey Guozhang, our basic road block is asynchronous processing (this is
> actually related to my previous post about asynchronous processing). Here
> is the simplest use-case:
>
> The streaming job receives messages. Each message is a user-event and
> needs to randomly look up that user’s history (for 100’s-of-thousands of
> users, and each user can have 10’s-of-thousands of events in their
> history). Once the history is retrieved, processing can continue. We need
> processing to be as fast as possible and we need the ability to easily
> accommodate increases in incoming message traffic. Here are the two
> designs (with KStreams, and then with Reactive Kafka)
>
> KStreams Approach:
>
> KStreams depth-first approach requires finishing processing of one message
> before the next one becomes available. So, we would have to first estimate
> the average input message rate (and measure the performance of our app) and
> then partition our topic/s appropriately. Each task would effectively
> block on DB-io for every history-set retrieval; obviously we would use a
> TTL cache (KTable could be useful here, but it wouldn’t be able to hold
> “all” of the history for every user). If we need to “scale” our
> application, we would add more partitions and application processing
> instances. Please suggest any other design choice we could go with. I’m
> might be missing something.
>
> Reactive Kafka Approach:
>
> Reactive Kafka allows out-of-order processing. So, while we are fetching
> history for event-1, we can start processing event-2. In a nutshell
> Reactive-Kafka parallelism is not tightly-coupled to the number of
> partitions in the topic (obviously this doesn’t apply to the input…we can
> only receive events as fast as current partition configuration allows…but
> we don’t’ have to block on io before we receive the next message)
>
>
> I’m new to both technologies, so any and all suggestions are welcome.
>
> -David
>
> On 7/30/16, 9:24 AM, "Guozhang Wang" <wa...@gmail.com> wrote:
>
> Hello David,
>
> I'd love to learn details about the "flexibility" of Reactive Kafka
> compared with KStreams in order to see if KStreams can improve on that
> end.
> Would you like to elaborate a bit more on that end?
>
> Guozhang
>
>
> On Thu, Jul 28, 2016 at 12:16 PM, David Garcia <da...@spiceworks.com>
> wrote:
>
> > Our team is evaluating KStreams and Reactive Kafka (version
> 0.11-M4) on a
> > confluent 3.0 cluster. Our testing is very simple (pulling from one
> topic,
> > doing a simple transform) and then writing out to another topic.
> >
> > The performance for the two systems is night and day. Both
> applications
> > were running on a laptop and connecting to kafka over a wifi
> network. Here
> > are the numbers:
> >
> > KStreams: ~14K messages per second
> > Reactive Kafka: ~110 messages per second
> >
> > Both the input, and output topic had 54 partitions. I’m fairly
> certain
> > I’m not using Reactive kafka with good configuration. Here is some
> stubbed
> > out code:
> https://gist.github.com/anduill/2e17cd7a40d4a86fefe19870d1270f5b
> >
> > One note, I am using the confluent stack (hence the
> > CachedSchemaRegistryClient)
> >
> > I like the flexibility of Reactive Kafka, so we’d really like to use
> > it…but if performance is going to be like this, I can’t really
> justify it.
> > I’m a scala/akka/streaming-akka newbie, so I’m sure there are better
> ways
> > to use the API. Any help is appreciated.
> >
> > -David
> >
>
>
>
> --
> -- Guozhang
>
>
>