Re: Kafka Advisory Topic

[Christopher Shannon <ch...@gmail.com>]

After discussing this a bit more and looking into what it would take to
implement this feature Knowles and I are going to hold off until we see
what the removal of Zookeeper will look like. We are still interested in
this feature but we figure that things will change significantly enough
that it's probably a good idea to wait to see how things change with Kafka
3.0.

On Tue, Jan 26, 2021 at 1:21 PM Knowles Atchison Jr <ka...@gmail.com>
wrote:

> Gwen,
>
> Yes, Chris and I work together; we can put together a KIP.
>
> If someone would please grant me wiki permissions, username is katchison.
>
> Thank you.
>
> On Mon, Jan 25, 2021 at 2:58 PM Gwen Shapira <gw...@confluent.io> wrote:
>
> > Agree that this sounds like a good idea.
> >
> > Would be good to have a more formal proposal (i.e. a KIP) with the
> details.
> > I can think of about 100 different questions (will there be "levels"
> > like in logs, what type of events are in or out of scope, rate
> > limiting, data formats, etc).
> > I am also curious on whether the notifications are intended for
> > humans, automated processes or even the Kafka client applications
> > themselves. I hope the proposal can include a few example scenarios to
> > help us reason about the experience.
> >
> > Knowlton, is this something you want to pick up?
> >
> > Gwen
> >
> > On Thu, Jan 21, 2021 at 6:05 AM Christopher Shannon
> > <ch...@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I am on the ActiveMQ PMC and I think this is a very good idea to have a
> > way
> > > to do advisories/notifications/events (whatever you want to call it).
> In
> > > ActiveMQ classic you have advisories and in Artemis you have
> > notifications.
> > > Having management messages that can be subscribed to in real time is
> > > actually a major feature that is missing from Kafka that many other
> > brokers
> > > have.
> > >
> > > The idea here would be to publish notifications of different
> configurable
> > > events when something important happens so a consumer can listen in on
> > > things it cares about and be able to do something instead of having to
> > poll
> > > the admin API. There are many events that happen in a broker that would
> > be
> > > useful to be notified about. Events such as new connections to the
> > cluster,
> > > new topics created or destroyed, consumer group creation, authorization
> > > errors, new leader election, etc. The list is pretty much endless.
> > >
> > > The metadata topic that will exist is probably not going to have all of
> > > this information so some other mechanism would be needed to handle
> > > publishing these messages to a specific management topic that would be
> > > useful for a consumer.
> > >
> > > Chris
> > >
> > >
> > > On Wed, Jan 20, 2021 at 4:12 PM Boyang Chen <
> reluctanthero104@gmail.com>
> > > wrote:
> > >
> > > > Hey Knowles,
> > > >
> > > > in Kafka people normally use admin clients to get those metadata. I'm
> > not
> > > > sure why you mentioned specifically that having a topic to manage
> these
> > > > information is useful, but a good news is that in KIP-500
> > > > <
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> > > > >
> > > > we
> > > > are trying to deprecate Zookeeper and migrate to a self-managed
> > metadata
> > > > topic quorum. At the time this feature is fully done, you should be
> > able to
> > > > use consumers to pull the metadata log.
> > > >
> > > > Best,
> > > > Boyang
> > > >
> > > > On Wed, Jan 20, 2021 at 11:22 AM Knowles Atchison Jr <
> > > > katchisonjr@gmail.com>
> > > > wrote:
> > > >
> > > > > Good afternoon all,
> > > > >
> > > > > In our Kafka clusters we have a need to know when certain
> activities
> > are
> > > > > performed, mainly topics being created, but brokers coming up/down
> is
> > > > also
> > > > > useful. This would be akin to what ActiveMQ does via advisory
> > messages (
> > > > > https://activemq.apache.org/advisory-message).
> > > > >
> > > > > Since there did not appear to be anything in the ecosystem
> > currently, I
> > > > > wrote a standalone Java program that watches the various ZooKeeper
> > > > > locations that the Kafka broker writes to and deltas can tell us
> > > > > topic/broker actions etc... and writes to a kafka topic for
> > downstream
> > > > > consumption.
> > > > >
> > > > > Ideally, we would rather have the broker handle this internally
> > rather
> > > > > than yet another service stood up in our systems. I began digging
> > through
> > > > > the broker source (my Scala is basically hello world level) and
> there
> > > > does
> > > > > not appear to be any mechanism in which this could be easily
> patched
> > into
> > > > > the broker.
> > > > >
> > > > > Specifically, a producer or consumer acting upon an nonexistent
> > topic or
> > > > a
> > > > > manual CreateTopic would trigger a Produce to this advisory topic
> > and the
> > > > > KafkaApis framework would handle it like any other request.
> However,
> > by
> > > > the
> > > > > time we are inside the getTopicMetadata call there doesn't seem to
> > be a
> > > > > clean way to fire off another message that would make its way
> through
> > > > > KafkaApis. Perhaps another XManager type object is required?
> > > > >
> > > > > Looking for alternative ideas or guidance (or I missed something in
> > the
> > > > > broker).
> > > > >
> > > > > Thank you.
> > > > >
> > > > > Knowles
> > > > >
> > > >
> >
> >
> >
> > --
> > Gwen Shapira
> > Engineering Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>