[jira] [Created] (KAFKA-8799) Support ability to pass global user data to consumers during Assignment

["Vinoth Chandar (JIRA)" <ji...@apache.org>]

Vinoth Chandar created KAFKA-8799:
-------------------------------------

             Summary: Support ability to pass global user data to consumers during Assignment
                 Key: KAFKA-8799
                 URL: https://issues.apache.org/jira/browse/KAFKA-8799
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Vinoth Chandar


This is a follow up from KAFKA-7149 

*Background :* 

Although we reduced the size of the AssignmentInfo object sent during each rebalance from leader to all followers in KAFKA-7149, we still repeat the same _partitionsByHost_ map for each host (all this when interactive queries are enabled) and thus still end up sending redundant bytes to the broker and also logging a large kafka message.

With 100s of streams instances, this overhead can grow into tens of megabytes easily.  

*Proposal :*

Extend the group assignment protocol to be able to support passing of an additional byte[], which can now contain the HostInfo -> partitions/partitionsByHost data just one time. 

{code}
final class GroupAssignment {
        private final Map<String, Assignment> assignments;

        // bytes sent to each consumer from leader
        private final byte[] globalUserData
        ...
    }
{code}
 
This can generally be handy to any other application like Streams, that does some stateful processing or lightweight cluster management 
 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)