consistent consumption rate

[Jamie Park <ja...@gmail.com>]

Hi,

I am developing Kafka stream-application using low level processor API.

I am consuming records from multiple topics, applying some joining logics,
and then  sending them off another topics.

1) How can I make sure my application consume records at consistent rate
across different topics?  If I were to distribute data across multiple
topics, would that ensure that data will be consumed consistently?  For
example,  let's say topicA, topicB,topicC has ~1000 records in 5 minute
window.  Can I assume these records will be consumed in the stream app at
the same time?  - meaning all 3000 records will land in the streaming
application consistently.

2) Is there a way to consume some records faster than others in streaming
application?  Let's say topicA has 12X-15X times records being produced at
a rate topicC produces records.  Can I consume topicA somehow faster than I
do for topicC? I am trying to overcome this topic volume imbalance by
breaking down topicA into 12 topics ( so each sub-topicA has similar volume
with topicC).

3)  My join logic depends on the same event-time window of records from
multiple topics I consume from.  How can I make sure the same event-time
window gets processed for each topic?

4) Is there a way to "hold" or "pause" on consuming records in Processor
temporarily?  I thought that if somehow one Processor consuming certain
records takes too long, or "joinable" records arrive late,  I was hoping I
could delay another Processor that is "waiting" for it's joinable records.
 -- I am achieving this by hold the records in StateStore but this wait
seems arbitrary.  I am wondering if there is better solution than this.

5)  Let's say I have 40 partitions on 10 topics I consume from and I have
40 instances of stream-application consuming from these.  In this case how
does setting "num.stream.threads" work?   I see that is still uses one
thread for "fetching" which doesn't really help the cause on "fetching"
stuff but some of these threads are being used for "consuming" from
monitoring JMX.  Since each application handles 10 partitions ( one
partition for each of 10 topics), I was setting num.stream.threads=10 but I
don't have a very clear understanding of this value.  Would you recommend
increasing num of threads here?


I'd very much appreciate any help on any of these questions.

Thank you