You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@kafka.apache.org by "Bala Prassanna I (JIRA)" <ji...@apache.org> on 2018/10/23 03:32:00 UTC

[jira] [Comment Edited] (KAFKA-6690) Priorities for Source Topics

    [ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659806#comment-16659806 ] 

Bala Prassanna I edited comment on KAFKA-6690 at 10/23/18 3:31 AM:
-------------------------------------------------------------------

[~nafshartous] Sorry about the delay in reply. We use Kafka to process the asynchronous events of our Document Management System such as preview generation, indexing for search etc. The traffic gets generated via Web and Desktop Sync application. In such cases, we had to prioritize the traffic from web and consume them first. But this might lead to the starvation of events from sync if the consumer speed is slow and the event rate is high from web. A solution to handle the starvation with a timeout after which the events are consumed normally for a specified period of time would be great and help us use our resources effectively.


was (Author: balaprassanna):
[~nafshartous]  Sorry about the delay in reply. We use Kafka to process the aynchronous events of our Document Management System such as preview generation, indexing for search etc. The traffic gets generated via Web and Desktop Sync application. In such cases, we had to prioritise the traffic from web and consume them first. Though this may lead to starvation of events from sync, an API as such will help us effectively manage the available resources and prioritise our traffic without allocating separate server to handle the sync load.

> Priorities for Source Topics
> ----------------------------
>
>                 Key: KAFKA-6690
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6690
>             Project: Kafka
>          Issue Type: New Feature
>          Components: consumer
>            Reporter: Bala Prassanna I
>            Assignee: Nick Afshartous
>            Priority: Major
>
> We often encounter use cases where we need to prioritise source topics. If a consumer is listening more than one topic, say, HighPriorityTopic and LowPriorityTopic, it should consume events from LowPriorityTopic only when all the events from HighPriorityTopic are consumed. This is needed in Kafka Streams processor topologies as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)