You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Murilo Tavares <mu...@gmail.com> on 2021/10/07 18:55:05 UTC
Tracking topic consumers
Hi. Looking for some insights here.
We use Kafka at a large scale, and have lots of microservices using Kafka
for all sorts of things.
Our biggest challenge nowadays is to track which topics are used and which
are not.
I have considered looking at consumer groups to identify which applications
consume from each topic, but then I realized we have lots of microservices
that do not use consumer groups (lots of goka views on top of Kafka topics).
Is there any architecture solution that can help track consumers of a Kafka
topic?
Thought that I could use ACL to restrict the creation of Kafka Topics and
use a gatekeeper service to centralize and log who creates the topics, but
still that does not block anyone from consuming from them.
Any thoughts on that?
Thanks
Murilo
Re: Tracking topic consumers
Posted by Murilo Tavares <mu...@gmail.com>.
Oh cool.
I had the wrong impression that ACL would not control read/writes, but just
topic operations like CREATE and DELETE.
Thanks for pointing that out.
On Thu, 7 Oct 2021 at 16:09, Boyang Chen <re...@gmail.com> wrote:
> Thanks, for standalone consumers I do not have any useful monitoring tool
> on top of my head, maybe this KIP looks promising to you, but it's still
> WIP
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability
>
> Also regarding ACL setup, I think that's also a good idea, and you should
> be able to restrict data reads
>
> https://docs.confluent.io/platform/current/kafka/authorization.html#operations
> or I'm missing some context here.
>
> On Thu, Oct 7, 2021 at 12:44 PM Murilo Tavares <mu...@gmail.com>
> wrote:
>
> > Hi Boyang
> > Thanks for your response.
> > Yes, I'm talking about that framework.
> > goka has a few different things. What they call `views` are similar to
> > Global KTables in KafkaStreams. It's a full snapshot of the Kafka topic,
> > held in memory. So every time you spin up your service, the goka view
> will
> > read the topic from beginning, thus no need to keep track of consumer
> > offset or consumer groups.
> > Thanks
> > Murilo
> >
> > On Thu, 7 Oct 2021 at 15:33, Boyang Chen <re...@gmail.com>
> > wrote:
> >
> > > Hey Murilo, could you explain what you mean by `goka views`? If you are
> > > talking about https://github.com/lovoo/goka, they should use consumer
> > > groups as well IIUC.
> > >
> > > Boyang
> > >
> > > On Thu, Oct 7, 2021 at 11:55 AM Murilo Tavares <mu...@gmail.com>
> > > wrote:
> > >
> > > > Hi. Looking for some insights here.
> > > > We use Kafka at a large scale, and have lots of microservices using
> > Kafka
> > > > for all sorts of things.
> > > > Our biggest challenge nowadays is to track which topics are used and
> > > which
> > > > are not.
> > > > I have considered looking at consumer groups to identify which
> > > applications
> > > > consume from each topic, but then I realized we have lots of
> > > microservices
> > > > that do not use consumer groups (lots of goka views on top of Kafka
> > > > topics).
> > > >
> > > > Is there any architecture solution that can help track consumers of a
> > > Kafka
> > > > topic?
> > > > Thought that I could use ACL to restrict the creation of Kafka Topics
> > and
> > > > use a gatekeeper service to centralize and log who creates the
> topics,
> > > but
> > > > still that does not block anyone from consuming from them.
> > > >
> > > > Any thoughts on that?
> > > > Thanks
> > > > Murilo
> > > >
> > >
> >
>
Re: Tracking topic consumers
Posted by Boyang Chen <re...@gmail.com>.
Thanks, for standalone consumers I do not have any useful monitoring tool
on top of my head, maybe this KIP looks promising to you, but it's still
WIP
https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability
Also regarding ACL setup, I think that's also a good idea, and you should
be able to restrict data reads
https://docs.confluent.io/platform/current/kafka/authorization.html#operations
or I'm missing some context here.
On Thu, Oct 7, 2021 at 12:44 PM Murilo Tavares <mu...@gmail.com> wrote:
> Hi Boyang
> Thanks for your response.
> Yes, I'm talking about that framework.
> goka has a few different things. What they call `views` are similar to
> Global KTables in KafkaStreams. It's a full snapshot of the Kafka topic,
> held in memory. So every time you spin up your service, the goka view will
> read the topic from beginning, thus no need to keep track of consumer
> offset or consumer groups.
> Thanks
> Murilo
>
> On Thu, 7 Oct 2021 at 15:33, Boyang Chen <re...@gmail.com>
> wrote:
>
> > Hey Murilo, could you explain what you mean by `goka views`? If you are
> > talking about https://github.com/lovoo/goka, they should use consumer
> > groups as well IIUC.
> >
> > Boyang
> >
> > On Thu, Oct 7, 2021 at 11:55 AM Murilo Tavares <mu...@gmail.com>
> > wrote:
> >
> > > Hi. Looking for some insights here.
> > > We use Kafka at a large scale, and have lots of microservices using
> Kafka
> > > for all sorts of things.
> > > Our biggest challenge nowadays is to track which topics are used and
> > which
> > > are not.
> > > I have considered looking at consumer groups to identify which
> > applications
> > > consume from each topic, but then I realized we have lots of
> > microservices
> > > that do not use consumer groups (lots of goka views on top of Kafka
> > > topics).
> > >
> > > Is there any architecture solution that can help track consumers of a
> > Kafka
> > > topic?
> > > Thought that I could use ACL to restrict the creation of Kafka Topics
> and
> > > use a gatekeeper service to centralize and log who creates the topics,
> > but
> > > still that does not block anyone from consuming from them.
> > >
> > > Any thoughts on that?
> > > Thanks
> > > Murilo
> > >
> >
>
Re: Tracking topic consumers
Posted by Murilo Tavares <mu...@gmail.com>.
Hi Boyang
Thanks for your response.
Yes, I'm talking about that framework.
goka has a few different things. What they call `views` are similar to
Global KTables in KafkaStreams. It's a full snapshot of the Kafka topic,
held in memory. So every time you spin up your service, the goka view will
read the topic from beginning, thus no need to keep track of consumer
offset or consumer groups.
Thanks
Murilo
On Thu, 7 Oct 2021 at 15:33, Boyang Chen <re...@gmail.com> wrote:
> Hey Murilo, could you explain what you mean by `goka views`? If you are
> talking about https://github.com/lovoo/goka, they should use consumer
> groups as well IIUC.
>
> Boyang
>
> On Thu, Oct 7, 2021 at 11:55 AM Murilo Tavares <mu...@gmail.com>
> wrote:
>
> > Hi. Looking for some insights here.
> > We use Kafka at a large scale, and have lots of microservices using Kafka
> > for all sorts of things.
> > Our biggest challenge nowadays is to track which topics are used and
> which
> > are not.
> > I have considered looking at consumer groups to identify which
> applications
> > consume from each topic, but then I realized we have lots of
> microservices
> > that do not use consumer groups (lots of goka views on top of Kafka
> > topics).
> >
> > Is there any architecture solution that can help track consumers of a
> Kafka
> > topic?
> > Thought that I could use ACL to restrict the creation of Kafka Topics and
> > use a gatekeeper service to centralize and log who creates the topics,
> but
> > still that does not block anyone from consuming from them.
> >
> > Any thoughts on that?
> > Thanks
> > Murilo
> >
>
Re: Tracking topic consumers
Posted by Boyang Chen <re...@gmail.com>.
Hey Murilo, could you explain what you mean by `goka views`? If you are
talking about https://github.com/lovoo/goka, they should use consumer
groups as well IIUC.
Boyang
On Thu, Oct 7, 2021 at 11:55 AM Murilo Tavares <mu...@gmail.com> wrote:
> Hi. Looking for some insights here.
> We use Kafka at a large scale, and have lots of microservices using Kafka
> for all sorts of things.
> Our biggest challenge nowadays is to track which topics are used and which
> are not.
> I have considered looking at consumer groups to identify which applications
> consume from each topic, but then I realized we have lots of microservices
> that do not use consumer groups (lots of goka views on top of Kafka
> topics).
>
> Is there any architecture solution that can help track consumers of a Kafka
> topic?
> Thought that I could use ACL to restrict the creation of Kafka Topics and
> use a gatekeeper service to centralize and log who creates the topics, but
> still that does not block anyone from consuming from them.
>
> Any thoughts on that?
> Thanks
> Murilo
>