Pet Project - Web based consumer for Kafka

[Stephen P <st...@gmail.com>]

Hello all!

I've been working on a pet project over the last couple of months building
out a web based interface for consuming from Kafka called Kafka Web View
<https://github.com/SourceLabOrg/kafka-webview>.

I often found myself going to kafka-console-consumer to manually inspect or
find data in Kafka.  Between wanting to filter for specific records, using
custom deserializers, and seeking to specific offsets or timestamps, the
command line tool starts to become a bit unwieldy to use.  On top of that
in production environments access to consoles or the clusters are typically
limited.

This project allows you to define "Views" over Kafka topics which include
configuring key and value deserializers as well as server side filtering
logic.  Using a view you can then "page" through records starting from the
head or tail of the topic, seek to specific offsets or timestamps, and
enable customizable filters.  The tool also allows you to stream a
view/records to your browser using web sockets.

Hoping someone else out there will find it useful as well, and definitely
open to suggestions or contributions.

Stephen

Re: Pet Project - Web based consumer for Kafka

[Stephen P <st...@gmail.com>]

Similar tool, but implemented very differently I believe.  Full disclosure
I'm not very familiar w/ Landoop's tool, but I'll try to explain the main
differences from reading through that project.

   - Landoop's tool has a hard dependency on Confluent's Rest Proxy
   service, where as Kafka Web View has no such requirement.
   - Indirectly, Landoop's tool seems to only support a limited set of data
   formats as Rest Proxy only appears to support:  "JSON, raw bytes encoded
   with base64 or using JSON-encoded Avro. With Avro, schemas are registered
   and validated against the Schema Registry."


   - Kafka Web View allows you to use any of Kafka's officially supported
   Deserializers along with uploading your own Deserializer implementations.
   - Landoop's tool only appears to support client side
   filtering/searching, where as Kafka Web View supports server side
   filtering.  This means if you're searching a large topic for a small set of
   data, you will not need to transfer the entire data set over the wire to
   your browser just to filter or "throw away" the records that do not match.
   The assumption here is that your Kafka Web View installation is co-located
   near your Kafka clusters, where as your web browser is a much longer and
   slower hop away.
   - Kafka Web View supports web socket streaming of topics in addition to
   being able to page through the results.

On the flip side I believe Landoop's tool aims to solve many problems that
Kafka Web View doesn't try to touch, such as administration tasks.

Hope this helps!

Stephen


On Mon, Jan 8, 2018 at 11:08 PM, Jacob Sheck <sh...@gmail.com> wrote:

> Is this tool similar to this: https://github.com/Landoop/kafka-topics-ui ?
>
> On Mon, Jan 8, 2018 at 7:09 AM Stephen P <st...@gmail.com> wrote:
>
> > Hello all!
> >
> > I've been working on a pet project over the last couple of months
> building
> > out a web based interface for consuming from Kafka called Kafka Web View
> > <https://github.com/SourceLabOrg/kafka-webview>.
> >
> > I often found myself going to kafka-console-consumer to manually inspect
> or
> > find data in Kafka.  Between wanting to filter for specific records,
> using
> > custom deserializers, and seeking to specific offsets or timestamps, the
> > command line tool starts to become a bit unwieldy to use.  On top of that
> > in production environments access to consoles or the clusters are
> typically
> > limited.
> >
> > This project allows you to define "Views" over Kafka topics which include
> > configuring key and value deserializers as well as server side filtering
> > logic.  Using a view you can then "page" through records starting from
> the
> > head or tail of the topic, seek to specific offsets or timestamps, and
> > enable customizable filters.  The tool also allows you to stream a
> > view/records to your browser using web sockets.
> >
> > Hoping someone else out there will find it useful as well, and definitely
> > open to suggestions or contributions.
> >
> > Stephen
> >
>

Re: Pet Project - Web based consumer for Kafka

[Jacob Sheck <sh...@gmail.com>]

Is this tool similar to this: https://github.com/Landoop/kafka-topics-ui ?

On Mon, Jan 8, 2018 at 7:09 AM Stephen P <st...@gmail.com> wrote:

> Hello all!
>
> I've been working on a pet project over the last couple of months building
> out a web based interface for consuming from Kafka called Kafka Web View
> <https://github.com/SourceLabOrg/kafka-webview>.
>
> I often found myself going to kafka-console-consumer to manually inspect or
> find data in Kafka.  Between wanting to filter for specific records, using
> custom deserializers, and seeking to specific offsets or timestamps, the
> command line tool starts to become a bit unwieldy to use.  On top of that
> in production environments access to consoles or the clusters are typically
> limited.
>
> This project allows you to define "Views" over Kafka topics which include
> configuring key and value deserializers as well as server side filtering
> logic.  Using a view you can then "page" through records starting from the
> head or tail of the topic, seek to specific offsets or timestamps, and
> enable customizable filters.  The tool also allows you to stream a
> view/records to your browser using web sockets.
>
> Hoping someone else out there will find it useful as well, and definitely
> open to suggestions or contributions.
>
> Stephen
>