You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Evan Chan (Created) (JIRA)" <ji...@apache.org> on 2012/02/07 23:51:03 UTC
[jira] [Created] (KAFKA-266) Kafka web console design
Kafka web console design
------------------------
Key: KAFKA-266
URL: https://issues.apache.org/jira/browse/KAFKA-266
Project: Kafka
Issue Type: New Feature
Components: contrib
Reporter: Evan Chan
This issue is created to track a community-contributed Kafka Web UI.
Here is an initial list of goals:
- Be able to easily see which brokers are up
- Be able to see lists of topics, connected producers, consumer groups, connected consumers
- Be able to see, for each consumer/partition, its offset, and more importantly, # of bytes unconsumed (== largest offset for partition - current offset)
- (Wish list) have a graphical view of the offsets
- (Wish list) be able to clean up consumer state, such as stale claimed partitions
List of challenges/questions:
- Which framework? Play! for Scala?
- Is all the data available from JMX and ZK? Hopefully, watching the files on the filesystem can be avoided....
- How to handle large numbers of topics, partitions, consumers, etc. efficiently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-266) Kafka web console design
Posted by "Jay Kreps (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226543#comment-13226543 ]
Jay Kreps commented on KAFKA-266:
---------------------------------
yeah i will post a patch.
> Kafka web console design
> ------------------------
>
> Key: KAFKA-266
> URL: https://issues.apache.org/jira/browse/KAFKA-266
> Project: Kafka
> Issue Type: New Feature
> Components: contrib
> Reporter: Evan Chan
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> This issue is created to track a community-contributed Kafka Web UI.
> Here is an initial list of goals:
> - Be able to easily see which brokers are up
> - Be able to see lists of topics, connected producers, consumer groups, connected consumers
> - Be able to see, for each consumer/partition, its offset, and more importantly, # of bytes unconsumed (== largest offset for partition - current offset)
> - (Wish list) have a graphical view of the offsets
> - (Wish list) be able to clean up consumer state, such as stale claimed partitions
> List of challenges/questions:
> - Which framework? Play! for Scala?
> - Is all the data available from JMX and ZK? Hopefully, watching the files on the filesystem can be avoided....
> - How to handle large numbers of topics, partitions, consumers, etc. efficiently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-266) Kafka web console design
Posted by "Joe Stein (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229285#comment-13229285 ]
Joe Stein commented on KAFKA-266:
---------------------------------
JMX, HTTP are both fine but also hooking into Graphite, Ganglia etc make for a more ubuitious friendly system (for the ops folk) I think KAFKA-203 really address this and a web console that is going to be for monitoring only I think should just pull the JSON from the http connector of the coda hale metrics implementation IMHO
> Kafka web console design
> ------------------------
>
> Key: KAFKA-266
> URL: https://issues.apache.org/jira/browse/KAFKA-266
> Project: Kafka
> Issue Type: New Feature
> Components: contrib
> Reporter: Evan Chan
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> This issue is created to track a community-contributed Kafka Web UI.
> Here is an initial list of goals:
> - Be able to easily see which brokers are up
> - Be able to see lists of topics, connected producers, consumer groups, connected consumers
> - Be able to see, for each consumer/partition, its offset, and more importantly, # of bytes unconsumed (== largest offset for partition - current offset)
> - (Wish list) have a graphical view of the offsets
> - (Wish list) be able to clean up consumer state, such as stale claimed partitions
> List of challenges/questions:
> - Which framework? Play! for Scala?
> - Is all the data available from JMX and ZK? Hopefully, watching the files on the filesystem can be avoided....
> - How to handle large numbers of topics, partitions, consumers, etc. efficiently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-266) Kafka web console design
Posted by "Evan Chan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396503#comment-13396503 ]
Evan Chan commented on KAFKA-266:
---------------------------------
BTW, just some thoughts. I'm thinking of not so much a web console for individual Kafka brokers, but a web console for an entire Kafka cluster, targetting mostly smaller clusters. This would grab information from ZK and from JMX (or JMX-over-HTTP) for individual brokers to put together reports like:
- Status of different Kafka brokers in the cluster, including overall read and write rates, etc.
- Information on consumers (at least the high level consumers registered in ZK), outstanding messages or how far behind they are, etc.
> Kafka web console design
> ------------------------
>
> Key: KAFKA-266
> URL: https://issues.apache.org/jira/browse/KAFKA-266
> Project: Kafka
> Issue Type: New Feature
> Components: contrib
> Reporter: Evan Chan
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> This issue is created to track a community-contributed Kafka Web UI.
> Here is an initial list of goals:
> - Be able to easily see which brokers are up
> - Be able to see lists of topics, connected producers, consumer groups, connected consumers
> - Be able to see, for each consumer/partition, its offset, and more importantly, # of bytes unconsumed (== largest offset for partition - current offset)
> - (Wish list) have a graphical view of the offsets
> - (Wish list) be able to clean up consumer state, such as stale claimed partitions
> List of challenges/questions:
> - Which framework? Play! for Scala?
> - Is all the data available from JMX and ZK? Hopefully, watching the files on the filesystem can be avoided....
> - How to handle large numbers of topics, partitions, consumers, etc. efficiently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-266) Kafka web console design
Posted by "Eric Hauser (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229233#comment-13229233 ]
Eric Hauser commented on KAFKA-266:
-----------------------------------
I would like to see a simple monitoring console as something that is embedded within the brokers themselves (ActiveMQ does something similar). It should have minimal dependencies (jetty, servlets, and Javascript sound appropriate). The simple HTTP interface should allow for quick checking to see if:
1) Brokers are up
2) Topic are being written to
3) Topics are being consumed
While most of this data is available via JMX already, HTTP is a preferable alternative. An external application that provides comprehensive management, audit solution seems to make sense as well.
> Kafka web console design
> ------------------------
>
> Key: KAFKA-266
> URL: https://issues.apache.org/jira/browse/KAFKA-266
> Project: Kafka
> Issue Type: New Feature
> Components: contrib
> Reporter: Evan Chan
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> This issue is created to track a community-contributed Kafka Web UI.
> Here is an initial list of goals:
> - Be able to easily see which brokers are up
> - Be able to see lists of topics, connected producers, consumer groups, connected consumers
> - Be able to see, for each consumer/partition, its offset, and more importantly, # of bytes unconsumed (== largest offset for partition - current offset)
> - (Wish list) have a graphical view of the offsets
> - (Wish list) be able to clean up consumer state, such as stale claimed partitions
> List of challenges/questions:
> - Which framework? Play! for Scala?
> - Is all the data available from JMX and ZK? Hopefully, watching the files on the filesystem can be avoided....
> - How to handle large numbers of topics, partitions, consumers, etc. efficiently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-266) Kafka web console design
Posted by "Jay Kreps (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229629#comment-13229629 ]
Jay Kreps commented on KAFKA-266:
---------------------------------
My take is that, as Joe says, most serious folks already have some system they use for monitoring across their stack (ganglia or whatever). To really work operationally you can't try to replace this or add a new system. For this reason, I kind of prefer the web app to be an optional stand-alone thing since it may not be of use to everyone, though i think that complicates its design (potentially). I think the advantage of the web app is custom display of very kafka-specific things (the audit, cluster status, etc).
> Kafka web console design
> ------------------------
>
> Key: KAFKA-266
> URL: https://issues.apache.org/jira/browse/KAFKA-266
> Project: Kafka
> Issue Type: New Feature
> Components: contrib
> Reporter: Evan Chan
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> This issue is created to track a community-contributed Kafka Web UI.
> Here is an initial list of goals:
> - Be able to easily see which brokers are up
> - Be able to see lists of topics, connected producers, consumer groups, connected consumers
> - Be able to see, for each consumer/partition, its offset, and more importantly, # of bytes unconsumed (== largest offset for partition - current offset)
> - (Wish list) have a graphical view of the offsets
> - (Wish list) be able to clean up consumer state, such as stale claimed partitions
> List of challenges/questions:
> - Which framework? Play! for Scala?
> - Is all the data available from JMX and ZK? Hopefully, watching the files on the filesystem can be avoided....
> - How to handle large numbers of topics, partitions, consumers, etc. efficiently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-266) Kafka web console design
Posted by "Jay Kreps (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202937#comment-13202937 ]
Jay Kreps commented on KAFKA-266:
---------------------------------
I think this would be fantastic to have. We don't have such a thing at LinkedIn, we just have hooks into our monitoring system, but I think this would be better.
We do have an application that does a data audit. This basically proves that all data that is sent is received by all interested consumers. See KAFKA-260 for a detailed description and screenshot. We had planned to fold this functionality into the open source project. Since both of these are kind of monitoring/health functionality are pretty closely related, maybe it would make sense to combine them.
The existing app is a java app that implements runs a few jetty servlets to implement a JSON api. The UI just runs off this api and is implemented in hmtl/css/js. The charts are dygraph and it uses jquery for javascript helpers. I am not married to this setup (the original code was written by someone else and just refactored by me), so I would be okay moving it into another mvc framework if there is something much better out there as long as it isn't too complex.
> Kafka web console design
> ------------------------
>
> Key: KAFKA-266
> URL: https://issues.apache.org/jira/browse/KAFKA-266
> Project: Kafka
> Issue Type: New Feature
> Components: contrib
> Reporter: Evan Chan
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> This issue is created to track a community-contributed Kafka Web UI.
> Here is an initial list of goals:
> - Be able to easily see which brokers are up
> - Be able to see lists of topics, connected producers, consumer groups, connected consumers
> - Be able to see, for each consumer/partition, its offset, and more importantly, # of bytes unconsumed (== largest offset for partition - current offset)
> - (Wish list) have a graphical view of the offsets
> - (Wish list) be able to clean up consumer state, such as stale claimed partitions
> List of challenges/questions:
> - Which framework? Play! for Scala?
> - Is all the data available from JMX and ZK? Hopefully, watching the files on the filesystem can be avoided....
> - How to handle large numbers of topics, partitions, consumers, etc. efficiently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-266) Kafka web console design
Posted by "Evan Chan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229953#comment-13229953 ]
Evan Chan commented on KAFKA-266:
---------------------------------
The web console would not primarily be for monitoring -- for us anyways -- and I agree it should not replace Ganglia etc. (Although at some shops - most notably Google - having web routes for every single service and app is a policy). I think the web console would be invaluable for debugging and status though. For looking at a snapshot of the system easily.
Also, standard JMX just doesnt work in EC2, you don't know what ports need to be opened up.
I think the metric Im most interested in -- number of messages or MB outstanding -- is not available directly from Kafka server itself anyways.
What is the timeframe for KAFKA-203?
> Kafka web console design
> ------------------------
>
> Key: KAFKA-266
> URL: https://issues.apache.org/jira/browse/KAFKA-266
> Project: Kafka
> Issue Type: New Feature
> Components: contrib
> Reporter: Evan Chan
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> This issue is created to track a community-contributed Kafka Web UI.
> Here is an initial list of goals:
> - Be able to easily see which brokers are up
> - Be able to see lists of topics, connected producers, consumer groups, connected consumers
> - Be able to see, for each consumer/partition, its offset, and more importantly, # of bytes unconsumed (== largest offset for partition - current offset)
> - (Wish list) have a graphical view of the offsets
> - (Wish list) be able to clean up consumer state, such as stale claimed partitions
> List of challenges/questions:
> - Which framework? Play! for Scala?
> - Is all the data available from JMX and ZK? Hopefully, watching the files on the filesystem can be avoided....
> - How to handle large numbers of topics, partitions, consumers, etc. efficiently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-266) Kafka web console design
Posted by "Evan Chan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229952#comment-13229952 ]
Evan Chan commented on KAFKA-266:
---------------------------------
The web console would not primarily be for monitoring -- for us anyways -- and I agree it should not replace Ganglia etc. (Although at some shops - most notably Google - having web routes for every single service and app is a policy). I think the web console would be invaluable for debugging and status though. For looking at a snapshot of the system easily.
Also, standard JMX just doesnt work in EC2, you don't know what ports need to be opened up.
I think the metric Im most interested in -- number of messages or MB outstanding -- is not available directly from Kafka server itself anyways.
What is the timeframe for KAFKA-203?
> Kafka web console design
> ------------------------
>
> Key: KAFKA-266
> URL: https://issues.apache.org/jira/browse/KAFKA-266
> Project: Kafka
> Issue Type: New Feature
> Components: contrib
> Reporter: Evan Chan
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> This issue is created to track a community-contributed Kafka Web UI.
> Here is an initial list of goals:
> - Be able to easily see which brokers are up
> - Be able to see lists of topics, connected producers, consumer groups, connected consumers
> - Be able to see, for each consumer/partition, its offset, and more importantly, # of bytes unconsumed (== largest offset for partition - current offset)
> - (Wish list) have a graphical view of the offsets
> - (Wish list) be able to clean up consumer state, such as stale claimed partitions
> List of challenges/questions:
> - Which framework? Play! for Scala?
> - Is all the data available from JMX and ZK? Hopefully, watching the files on the filesystem can be avoided....
> - How to handle large numbers of topics, partitions, consumers, etc. efficiently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-266) Kafka web console design
Posted by "Evan Chan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219381#comment-13219381 ]
Evan Chan commented on KAFKA-266:
---------------------------------
Jay,
Would it be possible to look at the source of the java app you guys have, or open source it somehow? It would be great to see how you guys are pulling out the stats, and might save lots of work in doing the web console.
thanks.
> Kafka web console design
> ------------------------
>
> Key: KAFKA-266
> URL: https://issues.apache.org/jira/browse/KAFKA-266
> Project: Kafka
> Issue Type: New Feature
> Components: contrib
> Reporter: Evan Chan
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> This issue is created to track a community-contributed Kafka Web UI.
> Here is an initial list of goals:
> - Be able to easily see which brokers are up
> - Be able to see lists of topics, connected producers, consumer groups, connected consumers
> - Be able to see, for each consumer/partition, its offset, and more importantly, # of bytes unconsumed (== largest offset for partition - current offset)
> - (Wish list) have a graphical view of the offsets
> - (Wish list) be able to clean up consumer state, such as stale claimed partitions
> List of challenges/questions:
> - Which framework? Play! for Scala?
> - Is all the data available from JMX and ZK? Hopefully, watching the files on the filesystem can be avoided....
> - How to handle large numbers of topics, partitions, consumers, etc. efficiently
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira