Broker Notification API for Kafka?

[Dominic Chambers <do...@gmail.com>]

I'd like to add a notification API to Kafka so that I can receive the
following notifications for each broker in the cluster:

  1. An event notification when a broker is promoted from follower to
leader, or when a follower catches up with the leader or falls behind the
leader.
  2. An event notification each time a new event is written to the leader
partition.
  3. An event notification each time an event is received by a partition
follower.

My motivation for having this API is two-fold:

  1. I want to be able to do 'stream processing' without having to run a
separate cluster for my app logic.
  2. I want to keep the processing load next to the data, avoiding all
unnecessary network communication.

This makes sense for me because I want to use Kafka as an event-sourcing
platform, rather than as a general streaming backbone between legacy
applications.

Could somebody with knowledge of the code base please provide me some
pointers as to how I should approach adding an API like this, or provide
further information if this is considered to be a bad idea.

Thanks in advance, Dominic Chambers.

Re: Broker Notification API for Kafka?

[Dominic Chambers <do...@gmail.com>]

Might this message be more appropriate for the developer mailing list? Is
it okay to double post in this case?

Thanks again, Dominic.
On Jun 12, 2016 00:59, "Dominic Chambers" <do...@gmail.com>
wrote:

I'd like to add a notification API to Kafka so that I can receive the
following notifications for each broker in the cluster:

  1. An event notification when a broker is promoted from follower to
leader, or when a follower catches up with the leader or falls behind the
leader.
  2. An event notification each time a new event is written to the leader
partition.
  3. An event notification each time an event is received by a partition
follower.

My motivation for having this API is two-fold:

  1. I want to be able to do 'stream processing' without having to run a
separate cluster for my app logic.
  2. I want to keep the processing load next to the data, avoiding all
unnecessary network communication.

This makes sense for me because I want to use Kafka as an event-sourcing
platform, rather than as a general streaming backbone between legacy
applications.

Could somebody with knowledge of the code base please provide me some
pointers as to how I should approach adding an API like this, or provide
further information if this is considered to be a bad idea.

Thanks in advance, Dominic Chambers.