A Kafka cluster health api idea, does it makes sense?

[Pere Urbón Bayes <pe...@gmail.com>]

Good night,

   After being for a long time working with Elasticsearch one gets used to
it’s great REST api where you can check all sort of things from the server.
As I usually get ask during my day life helping companies with their Kafka
I decided to build an small Kafka Health API POC.

This is a very small play app (scala) available from
https://github.com/purbon/kafka-health-api

Right now the available endpoints is limited, but the ones I find more
interesting are the root (/) and the /cluster/guarantees.

The root (/) resource include important kafka configuration as well as a
semaphore like status flag.

{

"hello": "Welcome to the Kafka Health API",

"versions": {

"brokersConfig": [{

"brokerId": "2",

"config": [....]

},

{

"brokerId": "0",

"config": [...]

}

]

},

"status": {

"semaphore": "Green",

"errors": []

},

"time": 1546370755336

}

The status flag is a very interesting and configurable feature as it lets
you define a quality measure for the overall cluster health. Basically if
there are some errors it will report them and flag the cluster health as
Orange, if the cluster is not operational it will flag the health as red.

Another interesting endpoint is the  /cluster/guarantees.

{

"producer": "true",

"broker": {

"allInSyncReplicas": true,

"replicaList": [{

"partitionName": "topicA-0",

"isrList": [ "2", "0" ],

"replicaList": [ "2", "0" ],

"inSync": true

},

{

"partitionName": "topicA-1",

"isrList": [ "0", "2" ],

"replicaList": [ "0", "2" ],

"inSync": true

},

{

"partitionName": "topicA-2",

"isrList": [ "2","0" ],

"replicaList": [ "2", "0" ],

"inSync": true

}

]

}

}

this resource will inform you about the different important metrics for the
topic guarantees such as in sync replicas, shown per partition and for the
overall cluster, and if the producer is using ack=all or a less secure
guarantee.

This project is in a very green status, but I would love to share to know
if you find it interesting?. Latest project status is available from
https://github.com/purbon/kafka-health-api, all sorts of collaborations and
feedback are very welcome.

Best wishes for 2019!

-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/

Re: A Kafka cluster health api idea, does it makes sense?

[André Missaglia <an...@arquivei.com.br>]

I think this project is indeed interesting and very promising. Is it
supposed to run inside the broker, or in a separate node, like a support
worker?

What I mostly miss in kafka is a handy way of checking if the broker is up
and working, or if it is still doing some replication. If this info were in
the API, it would really improve the health check in our kafka cluster.

Best regards,

Em ter, 1 de jan de 2019 às 17:43, Pere Urbón Bayes <pe...@gmail.com>
escreveu:

> Good night,
>
>    After being for a long time working with Elasticsearch one gets used to
> it’s great REST api where you can check all sort of things from the server.
> As I usually get ask during my day life helping companies with their Kafka
> I decided to build an small Kafka Health API POC.
>
> This is a very small play app (scala) available from
> https://github.com/purbon/kafka-health-api
>
> Right now the available endpoints is limited, but the ones I find more
> interesting are the root (/) and the /cluster/guarantees.
>
> The root (/) resource include important kafka configuration as well as a
> semaphore like status flag.
>
> {
>
> "hello": "Welcome to the Kafka Health API",
>
> "versions": {
>
> "brokersConfig": [{
>
> "brokerId": "2",
>
> "config": [....]
>
> },
>
> {
>
> "brokerId": "0",
>
> "config": [...]
>
> }
>
> ]
>
> },
>
> "status": {
>
> "semaphore": "Green",
>
> "errors": []
>
> },
>
> "time": 1546370755336
>
> }
>
> The status flag is a very interesting and configurable feature as it lets
> you define a quality measure for the overall cluster health. Basically if
> there are some errors it will report them and flag the cluster health as
> Orange, if the cluster is not operational it will flag the health as red.
>
> Another interesting endpoint is the  /cluster/guarantees.
>
> {
>
> "producer": "true",
>
> "broker": {
>
> "allInSyncReplicas": true,
>
> "replicaList": [{
>
> "partitionName": "topicA-0",
>
> "isrList": [ "2", "0" ],
>
> "replicaList": [ "2", "0" ],
>
> "inSync": true
>
> },
>
> {
>
> "partitionName": "topicA-1",
>
> "isrList": [ "0", "2" ],
>
> "replicaList": [ "0", "2" ],
>
> "inSync": true
>
> },
>
> {
>
> "partitionName": "topicA-2",
>
> "isrList": [ "2","0" ],
>
> "replicaList": [ "2", "0" ],
>
> "inSync": true
>
> }
>
> ]
>
> }
>
> }
>
> this resource will inform you about the different important metrics for the
> topic guarantees such as in sync replicas, shown per partition and for the
> overall cluster, and if the producer is using ack=all or a less secure
> guarantee.
>
> This project is in a very green status, but I would love to share to know
> if you find it interesting?. Latest project status is available from
> https://github.com/purbon/kafka-health-api, all sorts of collaborations
> and
> feedback are very welcome.
>
> Best wishes for 2019!
>
> --
> Pere Urbon-Bayes
> Software Architect
> http://www.purbon.com
> https://twitter.com/purbon
> https://www.linkedin.com/in/purbon/
>


-- 
*André Badawi Missaglia*
Data Engineer
(16) 3509-5515 *|* www.arquivei.com.br
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Arquivei.com.br – Inteligência em Notas Fiscais]
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]
<https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
<https://www.facebook.com/arquivei>
<https://www.linkedin.com/company/arquivei>
<https://www.youtube.com/watch?v=KJFrh8h4Zds&yt%3Acc=on>