[GitHub] metamodel issue #178: A new module for Apache Kafka "querying" using MetaMod...

[kaspersorensen <gi...@git.apache.org>]

Github user kaspersorensen commented on the issue:

    https://github.com/apache/metamodel/pull/178
  
    A bit of clarification to answer my own question/worry: It seems that the `Consumer.unsubscribe(..)` call is the best thing I can reasonably do. This seems to remove any Zookeeper entry for the consumer group. I think. At least on my own system where I'm testing, I can't see any trace of my consumers in Zookeeper. But the things that I don't think we can ever get rid of are the entries/records in the consumer_offsets topic that is internal to Kafka.
    
    I've been thinking that the best possibility would be to try and alleviate the problem by reusing existing Consumer group IDs. Implemented by using some pool of group IDs where leases for a group ID are granted when querying and returned when the DataSet is closed. But then again, that would make concurrent queries (or even worse - queries from multiple processes or nodes) very hard to manage because we would never know for sure that another MetaModel client wasn't _also_ firing a query with the same Consumer group ID. So my conclusion so far is that it isn't really improvable, except maybe by giving some hooks back to the user in case he really worries about storing too much consumer group information.


---