priority of topic

[xiaoyu wang <xi...@gmail.com>]

Hi all,

Just want to bring up some discussion, does it make sense to have priority
defined over topics? In case of resource contention, topic with higher
priority get delivered first.


-Xiaoyu

Re: priority of topic

[Jun Rao <ju...@gmail.com>]

Xiaoyu,

Are you mostly concerned with throughput or latency?

Thanks,

Jun

On Mon, Jun 11, 2012 at 5:08 PM, xiaoyu wang <xi...@gmail.com> wrote:

> Hi all,
>
> Just want to bring up some discussion, does it make sense to have priority
> defined over topics? In case of resource contention, topic with higher
> priority get delivered first.
>
>
> -Xiaoyu
>

Re: priority of topic

[Jay Kreps <ja...@gmail.com>]

This does make sense, but I wonder if it would be very usable or worth the
effort? Maybe the solution is usually to just have good monitoring and have
sufficient capacity for your workload--that is rather than trying to handle
overload more gracefully, just avoid it by adding servers.

In any case, here would be one implementation approach:
Currently (as of 0.8) we keep a queue of requests waiting for processing by
one of N I/O threads (RequestChannel.scala). Currently this is just a
linear blocking queue. We could make it into a priority queue and assign a
priority to each request (either based on the topic or on a per-request
basis). Instead of processing requests on a FIFO basis we would always
process the highest priority request. The problem with this, of course, is
that it introduces all the scheduling problems common to OS resource
schedulers. For example if there is a continuous stream of high priority
requests it will starve out the lower priority request from EVER getting
processed, which may or may not be desired. This also complicates the
ordering guarantees. Since now two messages for the same topic/partition
would only be appended in-order if they are of exactly the same priority.

-Jay

On Mon, Jun 11, 2012 at 5:08 PM, xiaoyu wang <xi...@gmail.com> wrote:

> Hi all,
>
> Just want to bring up some discussion, does it make sense to have priority
> defined over topics? In case of resource contention, topic with higher
> priority get delivered first.
>
>
> -Xiaoyu
>