Re: PIP-40: Contribute Pulsar Manager

[Sijie Guo <gu...@gmail.com>]

If there is no objection about this PIP, I'd like to start a vote thread
for accepting this project as part of Pulsar.
So we can continue the development of this project in the Apache way under
the governance of Pulsar PMC.

Let me know if you do see problems.

Thanks,
Sijie

On Thu, Aug 29, 2019 at 6:39 PM Sijie Guo <gu...@gmail.com> wrote:

> > I see there are dependencies on websockets, I would a prefer a simple
> polling model of the http, also it's enable by default in pulsar.
>
> I don't think we depend don pulsar websocket. All are http restful
> requests.
>
> > For the ui the default persistence should be sqlite. Potentially packaged
> by default.
>
> Initially the default is sqlite. But sqlite doesn't work if there are a
> lot of topic metrics.
> We switched to MySQL as default for supporting production traffic. We can
> switch default to PostgresSQL (as pulsar dashboard).
>
> - Sijie
>
> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <al...@gmail.com> wrote:
>
>> I see there are dependencies on websockets, I would a prefer a simple
>> polling model of the http, also it's enable by default in pulsar.
>>
>> For the ui the default persistence should be sqlite. Potentially packaged
>> by default.
>>
>> -Ali
>>
>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <eo...@gmail.com>
>> wrote:
>>
>> > Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha scritto:
>> >
>> > > That sounds an interesting idea!
>> >
>> >
>> > Awesome
>> >
>> > Does HerdDB support JDBC? If so, it should
>> > > be pretty straightforward to enable HerdDB.
>> > >
>> >
>> > Sure, as far as I know, the JDBC client is the only client really used
>> in
>> > production.
>> >
>> > In a replicated environment it uses Zookeeper for metadata and service
>> > discovery and Bookkeeper as WAL and if you have a pulsar cluster you
>> > already have both of them (zk cluster an bookies)
>> >
>> > I will take a look to how Pulsar Manager   uses JDBC, maybe it will be
>> very
>> > easy.
>> >
>> > I will be back with news
>> >
>> >
>> > Enrico
>> >
>> >
>> >
>> >
>> >
>> > > Thanks,
>> > > Sijie
>> > >
>> > > On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <eolivelli@gmail.com
>> >
>> > > wrote:
>> > >
>> > > > Maybe you can try to use HerdDB (1), it is a replicated SQL database
>> > that
>> > > > can be run embedded in the JVM.
>> > > > It is an SQL database that we developed in my company, in order to
>> > > > explicitly replace MySQL.
>> > > > It uses Bookkeeper to store the WAL and implement replication, but
>> it
>> > can
>> > > > run in standalone mode or in memory (for tests/dev).
>> > > >
>> > > > I can help in setting up a demo.
>> > > >
>> > > > Enrico
>> > > >
>> > > > (1) https://github.com/diennea/herddb
>> > > >
>> > > > Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha
>> scritto:
>> > > >
>> > > > > Hi Dave,
>> > > > >
>> > > > > It doesn't require MySQL. You can use any JDBC supported
>> databases.
>> > > > > Initially we would use SQLite for development and change to MySQL
>> for
>> > > > more
>> > > > > production-ready workloads.
>> > > > >
>> > > > > Hope this clarify your question.
>> > > > >
>> > > > > Thanks,
>> > > > > Sijie
>> > > > >
>> > > > > On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org>
>> wrote:
>> > > > >
>> > > > > > Hi -
>> > > > > >
>> > > > > > I see that MySQL is used. This would be a GPL dependency. Does
>> the
>> > > > Pulsar
>> > > > > > Manager require MySQL, or can other Databases that are compliant
>> > with
>> > > > > > Apache Release Policy be used instead?
>> > > > > >
>> > > > > > Regards,
>> > > > > > Dave
>> > > > > >
>> > > > > > > On Aug 29, 2019, at 7:49 AM, Guangning E <
>> eguangning@gmail.com>
>> > > > wrote:
>> > > > > > >
>> > > > > > > Hi all,
>> > > > > > >
>> > > > > > > We have developed a new Pulsar web UI - pulsar-manager,
>> aiming at
>> > > > > > > supporting managing Pulsar clusters running in different
>> > > environments
>> > > > > > > (on-premise data centers, cloud, and so on). We’d like to
>> > > contribute
>> > > > > the
>> > > > > > > project back to the Pulsar community.
>> > > > > > >
>> > > > > > > ---
>> > > > > > >
>> > > > > > > ## Motivation
>> > > > > > >
>> > > > > > > Currently, Pulsar has a monitoring tool - [Pulsar Dashboard](
>> [
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
>> > > > > > > ).
>> > > > > > > However, it only focuses on the simple monitoring of Pulsar -
>> > > > > collecting
>> > > > > > > and displaying information such as statistics of tenants,
>> > > namespaces,
>> > > > > > > topics, subscriptions, and so on. It doesn’t provide any
>> > management
>> > > > > > > operations such as add, delete and update tenants, namespaces,
>> > > > topics,
>> > > > > > and
>> > > > > > > so on. When the scale of Pulsar cluster increases or the
>> number
>> > of
>> > > > > > clusters
>> > > > > > > grows, using `pulsar-admin` to manage Pulsar can not satisfy
>> > > demands.
>> > > > > > > Therefore, Pulsar needs a simple and easy-to-use management
>> > console
>> > > > for
>> > > > > > > administrators.
>> > > > > > >
>> > > > > > > Pulsar Manager is a web-based GUI management and monitoring
>> tool
>> > > that
>> > > > > > helps
>> > > > > > > administrators and users manage and monitor tenants,
>> namespaces,
>> > > > > topics,
>> > > > > > > subscriptions, brokers, clusters, and so on, and supports
>> dynamic
>> > > > > > > configuration of multiple environments.
>> > > > > > >
>> > > > > > > ## Features
>> > > > > > >
>> > > > > > > Pulsar manager provides two main features: management and
>> > > monitoring.
>> > > > > > >
>> > > > > > > ### Management
>> > > > > > >
>> > > > > > > 1. Environment : (operations on the environment)
>> > > > > > > Create / Delete / Update / List / Get
>> > > > > > > 2. Cluster : (operations on clusters)
>> > > > > > > Create / Delete / Update / List / Get
>> > > > > > > 3. Brokers : (operations on brokers)
>> > > > > > > Heartbeat / Unload
>> > > > > > > 4. Tenants:
>> > > > > > > Create / Delete / Update / List
>> > > > > > > 5. Namespaces:
>> > > > > > > Create / Delete / List
>> > > > > > > Manage Namespace Policy
>> > > > > > > Unload
>> > > > > > > Operations on Namespace Bundles
>> > > > > > > 6. Topics
>> > > > > > > Create / Delete / List
>> > > > > > > Unload / Terminate / Offload / Compact
>> > > > > > > Topic Details
>> > > > > > > 7. Subscriptions
>> > > > > > > Create / Delete / List
>> > > > > > > Reset Cursor / Skip / Clear backlog / Unsubscribe
>> > > > > > > 8. Namespace Isolation Policies
>> > > > > > > Create / Delete / Update / Get / List
>> > > > > > > 9. Failure Domains
>> > > > > > > Create / Delete / Update / Get / List
>> > > > > > >
>> > > > > > > ### Monitoring: (display *aggregated* stats at different
>> levels)
>> > > > > > >
>> > > > > > > 1. Tenants
>> > > > > > > List the total number of namespaces per tenant (both
>> aggregated
>> > and
>> > > > > > > per-cluster basis)
>> > > > > > >
>> > > > > > > 2. Namespaces
>> > > > > > > List the total number of topics per namespace
>> > > > > > > List the aggregated stats per namespace (such as rate-in,
>> > rate-out,
>> > > > > > > throughput-in, and throughput-out)
>> > > > > > > List the distributions of namespace bundles
>> > > > > > >
>> > > > > > > 3. Topics
>> > > > > > > List the total number of partitions per topic
>> > > > > > > List the aggregated stats per topic (such as rate-in,
>> rate-out,
>> > > > > > > throughput-in, and throughput-out)
>> > > > > > > Detailed stats per topic partition
>> > > > > > > Detailed stats of storage per topic partition
>> > > > > > > Detailed stats of subscription per topic and per partition
>> > > > > > > Detailed stats of producers
>> > > > > > > Detailed stats of consumers
>> > > > > > >
>> > > > > > > The detailed design proposal is in
>> > > > > > > [
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
>> > > > > > >
>> > > > > > > Looking forward to any feedback.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Guangning
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>