PIP-40: Contribute Pulsar Manager

[Guangning E <eg...@gmail.com>]

Hi all,

We have developed a new Pulsar web UI - pulsar-manager, aiming at
supporting managing Pulsar clusters running in different environments
(on-premise data centers, cloud, and so on). We’d like to contribute the
project back to the Pulsar community.

---

## Motivation

Currently, Pulsar has a monitoring tool - [Pulsar Dashboard]( [
http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
).
However, it only focuses on the simple monitoring of Pulsar - collecting
and displaying information such as statistics of tenants, namespaces,
topics, subscriptions, and so on. It doesn’t provide any management
operations such as add, delete and update tenants, namespaces, topics, and
so on. When the scale of Pulsar cluster increases or the number of clusters
grows, using `pulsar-admin` to manage Pulsar can not satisfy demands.
Therefore, Pulsar needs a simple and easy-to-use management console for
administrators.

Pulsar Manager is a web-based GUI management and monitoring tool that helps
administrators and users manage and monitor tenants, namespaces, topics,
subscriptions, brokers, clusters, and so on, and supports dynamic
configuration of multiple environments.

## Features

Pulsar manager provides two main features: management and monitoring.

### Management

1. Environment : (operations on the environment)
Create / Delete / Update / List / Get
2. Cluster : (operations on clusters)
Create / Delete / Update / List / Get
3. Brokers : (operations on brokers)
Heartbeat / Unload
4. Tenants:
Create / Delete / Update / List
5. Namespaces:
Create / Delete / List
Manage Namespace Policy
Unload
Operations on Namespace Bundles
6. Topics
Create / Delete / List
Unload / Terminate / Offload / Compact
Topic Details
7. Subscriptions
Create / Delete / List
Reset Cursor / Skip / Clear backlog / Unsubscribe
8. Namespace Isolation Policies
Create / Delete / Update / Get / List
9. Failure Domains
Create / Delete / Update / Get / List

### Monitoring: (display *aggregated* stats at different levels)

1. Tenants
List the total number of namespaces per tenant (both aggregated and
per-cluster basis)

2. Namespaces
List the total number of topics per namespace
List the aggregated stats per namespace (such as rate-in, rate-out,
throughput-in, and throughput-out)
List the distributions of namespace bundles

3. Topics
List the total number of partitions per topic
List the aggregated stats per topic (such as rate-in, rate-out,
throughput-in, and throughput-out)
Detailed stats per topic partition
Detailed stats of storage per topic partition
Detailed stats of subscription per topic and per partition
Detailed stats of producers
Detailed stats of consumers

The detailed design proposal is in
 [
https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)

Looking forward to any feedback.

Thanks,
Guangning

Re: PIP-40: Contribute Pulsar Manager

[Sijie Guo <gu...@gmail.com>]

If there is no objection about this PIP, I'd like to start a vote thread
for accepting this project as part of Pulsar.
So we can continue the development of this project in the Apache way under
the governance of Pulsar PMC.

Let me know if you do see problems.

Thanks,
Sijie

On Thu, Aug 29, 2019 at 6:39 PM Sijie Guo <gu...@gmail.com> wrote:

> > I see there are dependencies on websockets, I would a prefer a simple
> polling model of the http, also it's enable by default in pulsar.
>
> I don't think we depend don pulsar websocket. All are http restful
> requests.
>
> > For the ui the default persistence should be sqlite. Potentially packaged
> by default.
>
> Initially the default is sqlite. But sqlite doesn't work if there are a
> lot of topic metrics.
> We switched to MySQL as default for supporting production traffic. We can
> switch default to PostgresSQL (as pulsar dashboard).
>
> - Sijie
>
> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <al...@gmail.com> wrote:
>
>> I see there are dependencies on websockets, I would a prefer a simple
>> polling model of the http, also it's enable by default in pulsar.
>>
>> For the ui the default persistence should be sqlite. Potentially packaged
>> by default.
>>
>> -Ali
>>
>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <eo...@gmail.com>
>> wrote:
>>
>> > Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha scritto:
>> >
>> > > That sounds an interesting idea!
>> >
>> >
>> > Awesome
>> >
>> > Does HerdDB support JDBC? If so, it should
>> > > be pretty straightforward to enable HerdDB.
>> > >
>> >
>> > Sure, as far as I know, the JDBC client is the only client really used
>> in
>> > production.
>> >
>> > In a replicated environment it uses Zookeeper for metadata and service
>> > discovery and Bookkeeper as WAL and if you have a pulsar cluster you
>> > already have both of them (zk cluster an bookies)
>> >
>> > I will take a look to how Pulsar Manager   uses JDBC, maybe it will be
>> very
>> > easy.
>> >
>> > I will be back with news
>> >
>> >
>> > Enrico
>> >
>> >
>> >
>> >
>> >
>> > > Thanks,
>> > > Sijie
>> > >
>> > > On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <eolivelli@gmail.com
>> >
>> > > wrote:
>> > >
>> > > > Maybe you can try to use HerdDB (1), it is a replicated SQL database
>> > that
>> > > > can be run embedded in the JVM.
>> > > > It is an SQL database that we developed in my company, in order to
>> > > > explicitly replace MySQL.
>> > > > It uses Bookkeeper to store the WAL and implement replication, but
>> it
>> > can
>> > > > run in standalone mode or in memory (for tests/dev).
>> > > >
>> > > > I can help in setting up a demo.
>> > > >
>> > > > Enrico
>> > > >
>> > > > (1) https://github.com/diennea/herddb
>> > > >
>> > > > Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha
>> scritto:
>> > > >
>> > > > > Hi Dave,
>> > > > >
>> > > > > It doesn't require MySQL. You can use any JDBC supported
>> databases.
>> > > > > Initially we would use SQLite for development and change to MySQL
>> for
>> > > > more
>> > > > > production-ready workloads.
>> > > > >
>> > > > > Hope this clarify your question.
>> > > > >
>> > > > > Thanks,
>> > > > > Sijie
>> > > > >
>> > > > > On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org>
>> wrote:
>> > > > >
>> > > > > > Hi -
>> > > > > >
>> > > > > > I see that MySQL is used. This would be a GPL dependency. Does
>> the
>> > > > Pulsar
>> > > > > > Manager require MySQL, or can other Databases that are compliant
>> > with
>> > > > > > Apache Release Policy be used instead?
>> > > > > >
>> > > > > > Regards,
>> > > > > > Dave
>> > > > > >
>> > > > > > > On Aug 29, 2019, at 7:49 AM, Guangning E <
>> eguangning@gmail.com>
>> > > > wrote:
>> > > > > > >
>> > > > > > > Hi all,
>> > > > > > >
>> > > > > > > We have developed a new Pulsar web UI - pulsar-manager,
>> aiming at
>> > > > > > > supporting managing Pulsar clusters running in different
>> > > environments
>> > > > > > > (on-premise data centers, cloud, and so on). We’d like to
>> > > contribute
>> > > > > the
>> > > > > > > project back to the Pulsar community.
>> > > > > > >
>> > > > > > > ---
>> > > > > > >
>> > > > > > > ## Motivation
>> > > > > > >
>> > > > > > > Currently, Pulsar has a monitoring tool - [Pulsar Dashboard](
>> [
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
>> > > > > > > ).
>> > > > > > > However, it only focuses on the simple monitoring of Pulsar -
>> > > > > collecting
>> > > > > > > and displaying information such as statistics of tenants,
>> > > namespaces,
>> > > > > > > topics, subscriptions, and so on. It doesn’t provide any
>> > management
>> > > > > > > operations such as add, delete and update tenants, namespaces,
>> > > > topics,
>> > > > > > and
>> > > > > > > so on. When the scale of Pulsar cluster increases or the
>> number
>> > of
>> > > > > > clusters
>> > > > > > > grows, using `pulsar-admin` to manage Pulsar can not satisfy
>> > > demands.
>> > > > > > > Therefore, Pulsar needs a simple and easy-to-use management
>> > console
>> > > > for
>> > > > > > > administrators.
>> > > > > > >
>> > > > > > > Pulsar Manager is a web-based GUI management and monitoring
>> tool
>> > > that
>> > > > > > helps
>> > > > > > > administrators and users manage and monitor tenants,
>> namespaces,
>> > > > > topics,
>> > > > > > > subscriptions, brokers, clusters, and so on, and supports
>> dynamic
>> > > > > > > configuration of multiple environments.
>> > > > > > >
>> > > > > > > ## Features
>> > > > > > >
>> > > > > > > Pulsar manager provides two main features: management and
>> > > monitoring.
>> > > > > > >
>> > > > > > > ### Management
>> > > > > > >
>> > > > > > > 1. Environment : (operations on the environment)
>> > > > > > > Create / Delete / Update / List / Get
>> > > > > > > 2. Cluster : (operations on clusters)
>> > > > > > > Create / Delete / Update / List / Get
>> > > > > > > 3. Brokers : (operations on brokers)
>> > > > > > > Heartbeat / Unload
>> > > > > > > 4. Tenants:
>> > > > > > > Create / Delete / Update / List
>> > > > > > > 5. Namespaces:
>> > > > > > > Create / Delete / List
>> > > > > > > Manage Namespace Policy
>> > > > > > > Unload
>> > > > > > > Operations on Namespace Bundles
>> > > > > > > 6. Topics
>> > > > > > > Create / Delete / List
>> > > > > > > Unload / Terminate / Offload / Compact
>> > > > > > > Topic Details
>> > > > > > > 7. Subscriptions
>> > > > > > > Create / Delete / List
>> > > > > > > Reset Cursor / Skip / Clear backlog / Unsubscribe
>> > > > > > > 8. Namespace Isolation Policies
>> > > > > > > Create / Delete / Update / Get / List
>> > > > > > > 9. Failure Domains
>> > > > > > > Create / Delete / Update / Get / List
>> > > > > > >
>> > > > > > > ### Monitoring: (display *aggregated* stats at different
>> levels)
>> > > > > > >
>> > > > > > > 1. Tenants
>> > > > > > > List the total number of namespaces per tenant (both
>> aggregated
>> > and
>> > > > > > > per-cluster basis)
>> > > > > > >
>> > > > > > > 2. Namespaces
>> > > > > > > List the total number of topics per namespace
>> > > > > > > List the aggregated stats per namespace (such as rate-in,
>> > rate-out,
>> > > > > > > throughput-in, and throughput-out)
>> > > > > > > List the distributions of namespace bundles
>> > > > > > >
>> > > > > > > 3. Topics
>> > > > > > > List the total number of partitions per topic
>> > > > > > > List the aggregated stats per topic (such as rate-in,
>> rate-out,
>> > > > > > > throughput-in, and throughput-out)
>> > > > > > > Detailed stats per topic partition
>> > > > > > > Detailed stats of storage per topic partition
>> > > > > > > Detailed stats of subscription per topic and per partition
>> > > > > > > Detailed stats of producers
>> > > > > > > Detailed stats of consumers
>> > > > > > >
>> > > > > > > The detailed design proposal is in
>> > > > > > > [
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
>> > > > > > >
>> > > > > > > Looking forward to any feedback.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Guangning
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: PIP-40: Contribute Pulsar Manager

[Sijie Guo <gu...@gmail.com>]

FYI.

We have completed transferring pulsar-manager repo to the ASF. It is at :
https://github.com/apache/pulsar-manager

Thanks,
Sijie

On Mon, Sep 16, 2019 at 4:35 PM Sijie Guo <gu...@gmail.com> wrote:

> FYI.  IP clearance is done in general@incubator.apache.org. I am moving
> forward with transferring the pulsar-manager repo.
>
> INFRA Jira is created: https://issues.apache.org/jira/browse/INFRA-19030
>
> - Sijie
>
> On Fri, Sep 6, 2019 at 5:29 AM Jia Zhai <zh...@gmail.com> wrote:
>
>> +1
>>
>>
>> Best Regards.
>>
>>
>> Jia Zhai
>>
>> Beijing, China
>>
>> Mobile: +86 15810491983
>>
>>
>>
>>
>> On Thu, Sep 5, 2019 at 3:34 AM Sijie Guo <gu...@gmail.com> wrote:
>>
>> > Dave, thank you! That’s super helpful!
>> >
>> > Thanks,
>> > Sijie
>> >
>> > On Wed, Sep 4, 2019 at 10:29 AM Dave Fisher <wa...@apache.org> wrote:
>> >
>> > >
>> > >
>> > > > On Sep 4, 2019, at 10:10 AM, Sijie Guo <gu...@gmail.com> wrote:
>> > > >
>> > > > Hi Dave,
>> > > >
>> > > > Thank you for the reminder. I was about to ping you yesterday about
>> > this.
>> > > > The IP clearance is the last piece that I wasn’t sure about. Just to
>> > make
>> > > > sure I understand correctly, so the Pulsar PMC has to fill out the
>> IP
>> > > > clearance template, check in the file to the SVN repo and start a
>> > thread
>> > > in
>> > > > general@incubator to get the approval from IPMC. Is that correct?
>> > >
>> > > Yes. The approval is via LAZY CONSENSUS of 3 days.
>> > >
>> > > StreamNative ought to provide an SGA to secretary@ unless all of the
>> IP
>> > > is already covered by ICLAs.
>> > >
>> > > Infra can then help transfer the repository.
>> > >
>> > > Regards,
>> > > Dave
>> > >
>> > > >
>> > > > Thanks,
>> > > > Sijie
>> > > >
>> > > > On Wed, Sep 4, 2019 at 9:41 AM Dave Fisher <wa...@apache.org> wrote:
>> > > >
>> > > >> BTW - Once the VOTE is complete to accept the codebase there is
>> some
>> > > >> paperwork.
>> > > >>
>> > > >> http://incubator.apache.org/ip-clearance/
>> > > >>
>> > > >> While this is the Incubator the Board has requested that the IPMC
>> > track
>> > > IP
>> > > >> Clearance. Everything is in SVN and the pages are updated every
>> > evening
>> > > >> PDT. If help is needed then let me know.
>> > > >>
>> > > >> Regards,
>> > > >> Dave
>> > > >>
>> > > >>> On Sep 4, 2019, at 9:01 AM, Enrico Olivelli <eo...@gmail.com>
>> > > wrote:
>> > > >>>
>> > > >>> For whom who is interested I am working on adding support for
>> HerdDB
>> > in
>> > > >>> Pulsar Manager.
>> > > >>>
>> > > >>> The pull request is still very raw, but Pulsar Manager seems to
>> work
>> > > >>> https://github.com/streamnative/pulsar-manager/pull/183
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <
>> uvaraj6@gmail.com>
>> > > ha
>> > > >>> scritto:
>> > > >>>
>> > > >>>> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <gu...@gmail.com>
>> wrote:
>> > > >>>>
>> > > >>>>>> I see there are dependencies on websockets, I would a prefer a
>> > > simple
>> > > >>>>> polling model of the http, also it's enable by default in
>> pulsar.
>> > > >>>>>
>> > > >>>>> I don't think we depend don pulsar websocket. All are http
>> restful
>> > > >>>>> requests.
>> > > >>>>>
>> > > >>>>>> For the ui the default persistence should be sqlite.
>> Potentially
>> > > >>>> packaged
>> > > >>>>> by default.
>> > > >>>>>
>> > > >>>>> Initially the default is sqlite. But sqlite doesn't work if
>> there
>> > > are a
>> > > >>>> lot
>> > > >>>>> of topic metrics.
>> > > >>>>> We switched to MySQL as default for supporting production
>> traffic.
>> > We
>> > > >> can
>> > > >>>>> switch default to PostgresSQL (as pulsar dashboard).
>> > > >>>>>
>> > > >>>> +1
>> > > >>>> Yes, Switching to Postgres Would be great. Because Postgres
>> license
>> > is
>> > > >> more
>> > > >>>> liberal and works well with Apache license ecosystem.
>> > > >>>>
>> > > >>>>>
>> > > >>>>> - Sijie
>> > > >>>>>
>> > > >>>>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <alahmed.se@gmail.com
>> >
>> > > >> wrote:
>> > > >>>>>
>> > > >>>>>> I see there are dependencies on websockets, I would a prefer a
>> > > simple
>> > > >>>>>> polling model of the http, also it's enable by default in
>> pulsar.
>> > > >>>>>>
>> > > >>>>>> For the ui the default persistence should be sqlite.
>> Potentially
>> > > >>>> packaged
>> > > >>>>>> by default.
>> > > >>>>>>
>> > > >>>>>> -Ali
>> > > >>>>>>
>> > > >>>>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <
>> > > eolivelli@gmail.com>
>> > > >>>>>> wrote:
>> > > >>>>>>
>> > > >>>>>>> Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha
>> > > scritto:
>> > > >>>>>>>
>> > > >>>>>>>> That sounds an interesting idea!
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> Awesome
>> > > >>>>>>>
>> > > >>>>>>> Does HerdDB support JDBC? If so, it should
>> > > >>>>>>>> be pretty straightforward to enable HerdDB.
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> Sure, as far as I know, the JDBC client is the only client
>> really
>> > > >>>> used
>> > > >>>>> in
>> > > >>>>>>> production.
>> > > >>>>>>>
>> > > >>>>>>> In a replicated environment it uses Zookeeper for metadata and
>> > > >>>> service
>> > > >>>>>>> discovery and Bookkeeper as WAL and if you have a pulsar
>> cluster
>> > > you
>> > > >>>>>>> already have both of them (zk cluster an bookies)
>> > > >>>>>>>
>> > > >>>>>>> I will take a look to how Pulsar Manager   uses JDBC, maybe it
>> > will
>> > > >>>> be
>> > > >>>>>> very
>> > > >>>>>>> easy.
>> > > >>>>>>>
>> > > >>>>>>> I will be back with news
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> Enrico
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>>> Thanks,
>> > > >>>>>>>> Sijie
>> > > >>>>>>>>
>> > > >>>>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <
>> > > >>>>> eolivelli@gmail.com>
>> > > >>>>>>>> wrote:
>> > > >>>>>>>>
>> > > >>>>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL
>> > > >>>>> database
>> > > >>>>>>> that
>> > > >>>>>>>>> can be run embedded in the JVM.
>> > > >>>>>>>>> It is an SQL database that we developed in my company, in
>> order
>> > > >>>> to
>> > > >>>>>>>>> explicitly replace MySQL.
>> > > >>>>>>>>> It uses Bookkeeper to store the WAL and implement
>> replication,
>> > > >>>> but
>> > > >>>>> it
>> > > >>>>>>> can
>> > > >>>>>>>>> run in standalone mode or in memory (for tests/dev).
>> > > >>>>>>>>>
>> > > >>>>>>>>> I can help in setting up a demo.
>> > > >>>>>>>>>
>> > > >>>>>>>>> Enrico
>> > > >>>>>>>>>
>> > > >>>>>>>>> (1) https://github.com/diennea/herddb
>> > > >>>>>>>>>
>> > > >>>>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha
>> > > >>>>> scritto:
>> > > >>>>>>>>>
>> > > >>>>>>>>>> Hi Dave,
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> It doesn't require MySQL. You can use any JDBC supported
>> > > >>>>> databases.
>> > > >>>>>>>>>> Initially we would use SQLite for development and change to
>> > > >>>> MySQL
>> > > >>>>>> for
>> > > >>>>>>>>> more
>> > > >>>>>>>>>> production-ready workloads.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Hope this clarify your question.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Thanks,
>> > > >>>>>>>>>> Sijie
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <
>> wave@apache.org>
>> > > >>>>>> wrote:
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> Hi -
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> I see that MySQL is used. This would be a GPL dependency.
>> > > >>>> Does
>> > > >>>>>> the
>> > > >>>>>>>>> Pulsar
>> > > >>>>>>>>>>> Manager require MySQL, or can other Databases that are
>> > > >>>>> compliant
>> > > >>>>>>> with
>> > > >>>>>>>>>>> Apache Release Policy be used instead?
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Regards,
>> > > >>>>>>>>>>> Dave
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> On Aug 29, 2019, at 7:49 AM, Guangning E <
>> > > >>>>> eguangning@gmail.com
>> > > >>>>>>>
>> > > >>>>>>>>> wrote:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Hi all,
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> We have developed a new Pulsar web UI - pulsar-manager,
>> > > >>>>> aiming
>> > > >>>>>> at
>> > > >>>>>>>>>>>> supporting managing Pulsar clusters running in different
>> > > >>>>>>>> environments
>> > > >>>>>>>>>>>> (on-premise data centers, cloud, and so on). We’d like to
>> > > >>>>>>>> contribute
>> > > >>>>>>>>>> the
>> > > >>>>>>>>>>>> project back to the Pulsar community.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> ---
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> ## Motivation
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Currently, Pulsar has a monitoring tool - [Pulsar
>> > > >>>>> Dashboard]( [
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>
>> > >
>> >
>> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
>> > > >>>>>>>>>>>> ).
>> > > >>>>>>>>>>>> However, it only focuses on the simple monitoring of
>> > > >>>> Pulsar -
>> > > >>>>>>>>>> collecting
>> > > >>>>>>>>>>>> and displaying information such as statistics of tenants,
>> > > >>>>>>>> namespaces,
>> > > >>>>>>>>>>>> topics, subscriptions, and so on. It doesn’t provide any
>> > > >>>>>>> management
>> > > >>>>>>>>>>>> operations such as add, delete and update tenants,
>> > > >>>>> namespaces,
>> > > >>>>>>>>> topics,
>> > > >>>>>>>>>>> and
>> > > >>>>>>>>>>>> so on. When the scale of Pulsar cluster increases or the
>> > > >>>>> number
>> > > >>>>>>> of
>> > > >>>>>>>>>>> clusters
>> > > >>>>>>>>>>>> grows, using `pulsar-admin` to manage Pulsar can not
>> > > >>>> satisfy
>> > > >>>>>>>> demands.
>> > > >>>>>>>>>>>> Therefore, Pulsar needs a simple and easy-to-use
>> management
>> > > >>>>>>> console
>> > > >>>>>>>>> for
>> > > >>>>>>>>>>>> administrators.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Pulsar Manager is a web-based GUI management and
>> monitoring
>> > > >>>>>> tool
>> > > >>>>>>>> that
>> > > >>>>>>>>>>> helps
>> > > >>>>>>>>>>>> administrators and users manage and monitor tenants,
>> > > >>>>>> namespaces,
>> > > >>>>>>>>>> topics,
>> > > >>>>>>>>>>>> subscriptions, brokers, clusters, and so on, and supports
>> > > >>>>>> dynamic
>> > > >>>>>>>>>>>> configuration of multiple environments.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> ## Features
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Pulsar manager provides two main features: management and
>> > > >>>>>>>> monitoring.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> ### Management
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> 1. Environment : (operations on the environment)
>> > > >>>>>>>>>>>> Create / Delete / Update / List / Get
>> > > >>>>>>>>>>>> 2. Cluster : (operations on clusters)
>> > > >>>>>>>>>>>> Create / Delete / Update / List / Get
>> > > >>>>>>>>>>>> 3. Brokers : (operations on brokers)
>> > > >>>>>>>>>>>> Heartbeat / Unload
>> > > >>>>>>>>>>>> 4. Tenants:
>> > > >>>>>>>>>>>> Create / Delete / Update / List
>> > > >>>>>>>>>>>> 5. Namespaces:
>> > > >>>>>>>>>>>> Create / Delete / List
>> > > >>>>>>>>>>>> Manage Namespace Policy
>> > > >>>>>>>>>>>> Unload
>> > > >>>>>>>>>>>> Operations on Namespace Bundles
>> > > >>>>>>>>>>>> 6. Topics
>> > > >>>>>>>>>>>> Create / Delete / List
>> > > >>>>>>>>>>>> Unload / Terminate / Offload / Compact
>> > > >>>>>>>>>>>> Topic Details
>> > > >>>>>>>>>>>> 7. Subscriptions
>> > > >>>>>>>>>>>> Create / Delete / List
>> > > >>>>>>>>>>>> Reset Cursor / Skip / Clear backlog / Unsubscribe
>> > > >>>>>>>>>>>> 8. Namespace Isolation Policies
>> > > >>>>>>>>>>>> Create / Delete / Update / Get / List
>> > > >>>>>>>>>>>> 9. Failure Domains
>> > > >>>>>>>>>>>> Create / Delete / Update / Get / List
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> ### Monitoring: (display *aggregated* stats at different
>> > > >>>>>> levels)
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> 1. Tenants
>> > > >>>>>>>>>>>> List the total number of namespaces per tenant (both
>> > > >>>>> aggregated
>> > > >>>>>>> and
>> > > >>>>>>>>>>>> per-cluster basis)
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> 2. Namespaces
>> > > >>>>>>>>>>>> List the total number of topics per namespace
>> > > >>>>>>>>>>>> List the aggregated stats per namespace (such as rate-in,
>> > > >>>>>>> rate-out,
>> > > >>>>>>>>>>>> throughput-in, and throughput-out)
>> > > >>>>>>>>>>>> List the distributions of namespace bundles
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> 3. Topics
>> > > >>>>>>>>>>>> List the total number of partitions per topic
>> > > >>>>>>>>>>>> List the aggregated stats per topic (such as rate-in,
>> > > >>>>> rate-out,
>> > > >>>>>>>>>>>> throughput-in, and throughput-out)
>> > > >>>>>>>>>>>> Detailed stats per topic partition
>> > > >>>>>>>>>>>> Detailed stats of storage per topic partition
>> > > >>>>>>>>>>>> Detailed stats of subscription per topic and per
>> partition
>> > > >>>>>>>>>>>> Detailed stats of producers
>> > > >>>>>>>>>>>> Detailed stats of consumers
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> The detailed design proposal is in
>> > > >>>>>>>>>>>> [
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>
>> > >
>> >
>> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Looking forward to any feedback.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Thanks,
>> > > >>>>>>>>>>>> Guangning
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>>
>> > > >>>> --
>> > > >>>> *Thanks*
>> > > >>>>
>> > > >>>> *Yuvaraj L*
>> > > >>>>
>> > > >>
>> > > >>
>> > >
>> > >
>> >
>>
>

Re: PIP-40: Contribute Pulsar Manager

[Sijie Guo <gu...@gmail.com>]

FYI.  IP clearance is done in general@incubator.apache.org. I am moving
forward with transferring the pulsar-manager repo.

INFRA Jira is created: https://issues.apache.org/jira/browse/INFRA-19030

- Sijie

On Fri, Sep 6, 2019 at 5:29 AM Jia Zhai <zh...@gmail.com> wrote:

> +1
>
>
> Best Regards.
>
>
> Jia Zhai
>
> Beijing, China
>
> Mobile: +86 15810491983
>
>
>
>
> On Thu, Sep 5, 2019 at 3:34 AM Sijie Guo <gu...@gmail.com> wrote:
>
> > Dave, thank you! That’s super helpful!
> >
> > Thanks,
> > Sijie
> >
> > On Wed, Sep 4, 2019 at 10:29 AM Dave Fisher <wa...@apache.org> wrote:
> >
> > >
> > >
> > > > On Sep 4, 2019, at 10:10 AM, Sijie Guo <gu...@gmail.com> wrote:
> > > >
> > > > Hi Dave,
> > > >
> > > > Thank you for the reminder. I was about to ping you yesterday about
> > this.
> > > > The IP clearance is the last piece that I wasn’t sure about. Just to
> > make
> > > > sure I understand correctly, so the Pulsar PMC has to fill out the IP
> > > > clearance template, check in the file to the SVN repo and start a
> > thread
> > > in
> > > > general@incubator to get the approval from IPMC. Is that correct?
> > >
> > > Yes. The approval is via LAZY CONSENSUS of 3 days.
> > >
> > > StreamNative ought to provide an SGA to secretary@ unless all of the
> IP
> > > is already covered by ICLAs.
> > >
> > > Infra can then help transfer the repository.
> > >
> > > Regards,
> > > Dave
> > >
> > > >
> > > > Thanks,
> > > > Sijie
> > > >
> > > > On Wed, Sep 4, 2019 at 9:41 AM Dave Fisher <wa...@apache.org> wrote:
> > > >
> > > >> BTW - Once the VOTE is complete to accept the codebase there is some
> > > >> paperwork.
> > > >>
> > > >> http://incubator.apache.org/ip-clearance/
> > > >>
> > > >> While this is the Incubator the Board has requested that the IPMC
> > track
> > > IP
> > > >> Clearance. Everything is in SVN and the pages are updated every
> > evening
> > > >> PDT. If help is needed then let me know.
> > > >>
> > > >> Regards,
> > > >> Dave
> > > >>
> > > >>> On Sep 4, 2019, at 9:01 AM, Enrico Olivelli <eo...@gmail.com>
> > > wrote:
> > > >>>
> > > >>> For whom who is interested I am working on adding support for
> HerdDB
> > in
> > > >>> Pulsar Manager.
> > > >>>
> > > >>> The pull request is still very raw, but Pulsar Manager seems to
> work
> > > >>> https://github.com/streamnative/pulsar-manager/pull/183
> > > >>>
> > > >>>
> > > >>>
> > > >>> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <
> uvaraj6@gmail.com>
> > > ha
> > > >>> scritto:
> > > >>>
> > > >>>> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <gu...@gmail.com>
> wrote:
> > > >>>>
> > > >>>>>> I see there are dependencies on websockets, I would a prefer a
> > > simple
> > > >>>>> polling model of the http, also it's enable by default in pulsar.
> > > >>>>>
> > > >>>>> I don't think we depend don pulsar websocket. All are http
> restful
> > > >>>>> requests.
> > > >>>>>
> > > >>>>>> For the ui the default persistence should be sqlite. Potentially
> > > >>>> packaged
> > > >>>>> by default.
> > > >>>>>
> > > >>>>> Initially the default is sqlite. But sqlite doesn't work if there
> > > are a
> > > >>>> lot
> > > >>>>> of topic metrics.
> > > >>>>> We switched to MySQL as default for supporting production
> traffic.
> > We
> > > >> can
> > > >>>>> switch default to PostgresSQL (as pulsar dashboard).
> > > >>>>>
> > > >>>> +1
> > > >>>> Yes, Switching to Postgres Would be great. Because Postgres
> license
> > is
> > > >> more
> > > >>>> liberal and works well with Apache license ecosystem.
> > > >>>>
> > > >>>>>
> > > >>>>> - Sijie
> > > >>>>>
> > > >>>>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <al...@gmail.com>
> > > >> wrote:
> > > >>>>>
> > > >>>>>> I see there are dependencies on websockets, I would a prefer a
> > > simple
> > > >>>>>> polling model of the http, also it's enable by default in
> pulsar.
> > > >>>>>>
> > > >>>>>> For the ui the default persistence should be sqlite. Potentially
> > > >>>> packaged
> > > >>>>>> by default.
> > > >>>>>>
> > > >>>>>> -Ali
> > > >>>>>>
> > > >>>>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <
> > > eolivelli@gmail.com>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha
> > > scritto:
> > > >>>>>>>
> > > >>>>>>>> That sounds an interesting idea!
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Awesome
> > > >>>>>>>
> > > >>>>>>> Does HerdDB support JDBC? If so, it should
> > > >>>>>>>> be pretty straightforward to enable HerdDB.
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>> Sure, as far as I know, the JDBC client is the only client
> really
> > > >>>> used
> > > >>>>> in
> > > >>>>>>> production.
> > > >>>>>>>
> > > >>>>>>> In a replicated environment it uses Zookeeper for metadata and
> > > >>>> service
> > > >>>>>>> discovery and Bookkeeper as WAL and if you have a pulsar
> cluster
> > > you
> > > >>>>>>> already have both of them (zk cluster an bookies)
> > > >>>>>>>
> > > >>>>>>> I will take a look to how Pulsar Manager   uses JDBC, maybe it
> > will
> > > >>>> be
> > > >>>>>> very
> > > >>>>>>> easy.
> > > >>>>>>>
> > > >>>>>>> I will be back with news
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Enrico
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>> Thanks,
> > > >>>>>>>> Sijie
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <
> > > >>>>> eolivelli@gmail.com>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL
> > > >>>>> database
> > > >>>>>>> that
> > > >>>>>>>>> can be run embedded in the JVM.
> > > >>>>>>>>> It is an SQL database that we developed in my company, in
> order
> > > >>>> to
> > > >>>>>>>>> explicitly replace MySQL.
> > > >>>>>>>>> It uses Bookkeeper to store the WAL and implement
> replication,
> > > >>>> but
> > > >>>>> it
> > > >>>>>>> can
> > > >>>>>>>>> run in standalone mode or in memory (for tests/dev).
> > > >>>>>>>>>
> > > >>>>>>>>> I can help in setting up a demo.
> > > >>>>>>>>>
> > > >>>>>>>>> Enrico
> > > >>>>>>>>>
> > > >>>>>>>>> (1) https://github.com/diennea/herddb
> > > >>>>>>>>>
> > > >>>>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha
> > > >>>>> scritto:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi Dave,
> > > >>>>>>>>>>
> > > >>>>>>>>>> It doesn't require MySQL. You can use any JDBC supported
> > > >>>>> databases.
> > > >>>>>>>>>> Initially we would use SQLite for development and change to
> > > >>>> MySQL
> > > >>>>>> for
> > > >>>>>>>>> more
> > > >>>>>>>>>> production-ready workloads.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Hope this clarify your question.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks,
> > > >>>>>>>>>> Sijie
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <
> wave@apache.org>
> > > >>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hi -
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I see that MySQL is used. This would be a GPL dependency.
> > > >>>> Does
> > > >>>>>> the
> > > >>>>>>>>> Pulsar
> > > >>>>>>>>>>> Manager require MySQL, or can other Databases that are
> > > >>>>> compliant
> > > >>>>>>> with
> > > >>>>>>>>>>> Apache Release Policy be used instead?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Regards,
> > > >>>>>>>>>>> Dave
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> On Aug 29, 2019, at 7:49 AM, Guangning E <
> > > >>>>> eguangning@gmail.com
> > > >>>>>>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> We have developed a new Pulsar web UI - pulsar-manager,
> > > >>>>> aiming
> > > >>>>>> at
> > > >>>>>>>>>>>> supporting managing Pulsar clusters running in different
> > > >>>>>>>> environments
> > > >>>>>>>>>>>> (on-premise data centers, cloud, and so on). We’d like to
> > > >>>>>>>> contribute
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> project back to the Pulsar community.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> ---
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> ## Motivation
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Currently, Pulsar has a monitoring tool - [Pulsar
> > > >>>>> Dashboard]( [
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > >
> >
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> > > >>>>>>>>>>>> ).
> > > >>>>>>>>>>>> However, it only focuses on the simple monitoring of
> > > >>>> Pulsar -
> > > >>>>>>>>>> collecting
> > > >>>>>>>>>>>> and displaying information such as statistics of tenants,
> > > >>>>>>>> namespaces,
> > > >>>>>>>>>>>> topics, subscriptions, and so on. It doesn’t provide any
> > > >>>>>>> management
> > > >>>>>>>>>>>> operations such as add, delete and update tenants,
> > > >>>>> namespaces,
> > > >>>>>>>>> topics,
> > > >>>>>>>>>>> and
> > > >>>>>>>>>>>> so on. When the scale of Pulsar cluster increases or the
> > > >>>>> number
> > > >>>>>>> of
> > > >>>>>>>>>>> clusters
> > > >>>>>>>>>>>> grows, using `pulsar-admin` to manage Pulsar can not
> > > >>>> satisfy
> > > >>>>>>>> demands.
> > > >>>>>>>>>>>> Therefore, Pulsar needs a simple and easy-to-use
> management
> > > >>>>>>> console
> > > >>>>>>>>> for
> > > >>>>>>>>>>>> administrators.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Pulsar Manager is a web-based GUI management and
> monitoring
> > > >>>>>> tool
> > > >>>>>>>> that
> > > >>>>>>>>>>> helps
> > > >>>>>>>>>>>> administrators and users manage and monitor tenants,
> > > >>>>>> namespaces,
> > > >>>>>>>>>> topics,
> > > >>>>>>>>>>>> subscriptions, brokers, clusters, and so on, and supports
> > > >>>>>> dynamic
> > > >>>>>>>>>>>> configuration of multiple environments.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> ## Features
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Pulsar manager provides two main features: management and
> > > >>>>>>>> monitoring.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> ### Management
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1. Environment : (operations on the environment)
> > > >>>>>>>>>>>> Create / Delete / Update / List / Get
> > > >>>>>>>>>>>> 2. Cluster : (operations on clusters)
> > > >>>>>>>>>>>> Create / Delete / Update / List / Get
> > > >>>>>>>>>>>> 3. Brokers : (operations on brokers)
> > > >>>>>>>>>>>> Heartbeat / Unload
> > > >>>>>>>>>>>> 4. Tenants:
> > > >>>>>>>>>>>> Create / Delete / Update / List
> > > >>>>>>>>>>>> 5. Namespaces:
> > > >>>>>>>>>>>> Create / Delete / List
> > > >>>>>>>>>>>> Manage Namespace Policy
> > > >>>>>>>>>>>> Unload
> > > >>>>>>>>>>>> Operations on Namespace Bundles
> > > >>>>>>>>>>>> 6. Topics
> > > >>>>>>>>>>>> Create / Delete / List
> > > >>>>>>>>>>>> Unload / Terminate / Offload / Compact
> > > >>>>>>>>>>>> Topic Details
> > > >>>>>>>>>>>> 7. Subscriptions
> > > >>>>>>>>>>>> Create / Delete / List
> > > >>>>>>>>>>>> Reset Cursor / Skip / Clear backlog / Unsubscribe
> > > >>>>>>>>>>>> 8. Namespace Isolation Policies
> > > >>>>>>>>>>>> Create / Delete / Update / Get / List
> > > >>>>>>>>>>>> 9. Failure Domains
> > > >>>>>>>>>>>> Create / Delete / Update / Get / List
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> ### Monitoring: (display *aggregated* stats at different
> > > >>>>>> levels)
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 1. Tenants
> > > >>>>>>>>>>>> List the total number of namespaces per tenant (both
> > > >>>>> aggregated
> > > >>>>>>> and
> > > >>>>>>>>>>>> per-cluster basis)
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 2. Namespaces
> > > >>>>>>>>>>>> List the total number of topics per namespace
> > > >>>>>>>>>>>> List the aggregated stats per namespace (such as rate-in,
> > > >>>>>>> rate-out,
> > > >>>>>>>>>>>> throughput-in, and throughput-out)
> > > >>>>>>>>>>>> List the distributions of namespace bundles
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 3. Topics
> > > >>>>>>>>>>>> List the total number of partitions per topic
> > > >>>>>>>>>>>> List the aggregated stats per topic (such as rate-in,
> > > >>>>> rate-out,
> > > >>>>>>>>>>>> throughput-in, and throughput-out)
> > > >>>>>>>>>>>> Detailed stats per topic partition
> > > >>>>>>>>>>>> Detailed stats of storage per topic partition
> > > >>>>>>>>>>>> Detailed stats of subscription per topic and per partition
> > > >>>>>>>>>>>> Detailed stats of producers
> > > >>>>>>>>>>>> Detailed stats of consumers
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> The detailed design proposal is in
> > > >>>>>>>>>>>> [
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > >
> >
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Looking forward to any feedback.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>> Guangning
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> *Thanks*
> > > >>>>
> > > >>>> *Yuvaraj L*
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
>

Re: PIP-40: Contribute Pulsar Manager

[Jia Zhai <zh...@gmail.com>]

+1


Best Regards.


Jia Zhai

Beijing, China

Mobile: +86 15810491983




On Thu, Sep 5, 2019 at 3:34 AM Sijie Guo <gu...@gmail.com> wrote:

> Dave, thank you! That’s super helpful!
>
> Thanks,
> Sijie
>
> On Wed, Sep 4, 2019 at 10:29 AM Dave Fisher <wa...@apache.org> wrote:
>
> >
> >
> > > On Sep 4, 2019, at 10:10 AM, Sijie Guo <gu...@gmail.com> wrote:
> > >
> > > Hi Dave,
> > >
> > > Thank you for the reminder. I was about to ping you yesterday about
> this.
> > > The IP clearance is the last piece that I wasn’t sure about. Just to
> make
> > > sure I understand correctly, so the Pulsar PMC has to fill out the IP
> > > clearance template, check in the file to the SVN repo and start a
> thread
> > in
> > > general@incubator to get the approval from IPMC. Is that correct?
> >
> > Yes. The approval is via LAZY CONSENSUS of 3 days.
> >
> > StreamNative ought to provide an SGA to secretary@ unless all of the IP
> > is already covered by ICLAs.
> >
> > Infra can then help transfer the repository.
> >
> > Regards,
> > Dave
> >
> > >
> > > Thanks,
> > > Sijie
> > >
> > > On Wed, Sep 4, 2019 at 9:41 AM Dave Fisher <wa...@apache.org> wrote:
> > >
> > >> BTW - Once the VOTE is complete to accept the codebase there is some
> > >> paperwork.
> > >>
> > >> http://incubator.apache.org/ip-clearance/
> > >>
> > >> While this is the Incubator the Board has requested that the IPMC
> track
> > IP
> > >> Clearance. Everything is in SVN and the pages are updated every
> evening
> > >> PDT. If help is needed then let me know.
> > >>
> > >> Regards,
> > >> Dave
> > >>
> > >>> On Sep 4, 2019, at 9:01 AM, Enrico Olivelli <eo...@gmail.com>
> > wrote:
> > >>>
> > >>> For whom who is interested I am working on adding support for HerdDB
> in
> > >>> Pulsar Manager.
> > >>>
> > >>> The pull request is still very raw, but Pulsar Manager seems to work
> > >>> https://github.com/streamnative/pulsar-manager/pull/183
> > >>>
> > >>>
> > >>>
> > >>> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <uv...@gmail.com>
> > ha
> > >>> scritto:
> > >>>
> > >>>> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <gu...@gmail.com> wrote:
> > >>>>
> > >>>>>> I see there are dependencies on websockets, I would a prefer a
> > simple
> > >>>>> polling model of the http, also it's enable by default in pulsar.
> > >>>>>
> > >>>>> I don't think we depend don pulsar websocket. All are http restful
> > >>>>> requests.
> > >>>>>
> > >>>>>> For the ui the default persistence should be sqlite. Potentially
> > >>>> packaged
> > >>>>> by default.
> > >>>>>
> > >>>>> Initially the default is sqlite. But sqlite doesn't work if there
> > are a
> > >>>> lot
> > >>>>> of topic metrics.
> > >>>>> We switched to MySQL as default for supporting production traffic.
> We
> > >> can
> > >>>>> switch default to PostgresSQL (as pulsar dashboard).
> > >>>>>
> > >>>> +1
> > >>>> Yes, Switching to Postgres Would be great. Because Postgres license
> is
> > >> more
> > >>>> liberal and works well with Apache license ecosystem.
> > >>>>
> > >>>>>
> > >>>>> - Sijie
> > >>>>>
> > >>>>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <al...@gmail.com>
> > >> wrote:
> > >>>>>
> > >>>>>> I see there are dependencies on websockets, I would a prefer a
> > simple
> > >>>>>> polling model of the http, also it's enable by default in pulsar.
> > >>>>>>
> > >>>>>> For the ui the default persistence should be sqlite. Potentially
> > >>>> packaged
> > >>>>>> by default.
> > >>>>>>
> > >>>>>> -Ali
> > >>>>>>
> > >>>>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <
> > eolivelli@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha
> > scritto:
> > >>>>>>>
> > >>>>>>>> That sounds an interesting idea!
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Awesome
> > >>>>>>>
> > >>>>>>> Does HerdDB support JDBC? If so, it should
> > >>>>>>>> be pretty straightforward to enable HerdDB.
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>> Sure, as far as I know, the JDBC client is the only client really
> > >>>> used
> > >>>>> in
> > >>>>>>> production.
> > >>>>>>>
> > >>>>>>> In a replicated environment it uses Zookeeper for metadata and
> > >>>> service
> > >>>>>>> discovery and Bookkeeper as WAL and if you have a pulsar cluster
> > you
> > >>>>>>> already have both of them (zk cluster an bookies)
> > >>>>>>>
> > >>>>>>> I will take a look to how Pulsar Manager   uses JDBC, maybe it
> will
> > >>>> be
> > >>>>>> very
> > >>>>>>> easy.
> > >>>>>>>
> > >>>>>>> I will be back with news
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Enrico
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> Sijie
> > >>>>>>>>
> > >>>>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <
> > >>>>> eolivelli@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL
> > >>>>> database
> > >>>>>>> that
> > >>>>>>>>> can be run embedded in the JVM.
> > >>>>>>>>> It is an SQL database that we developed in my company, in order
> > >>>> to
> > >>>>>>>>> explicitly replace MySQL.
> > >>>>>>>>> It uses Bookkeeper to store the WAL and implement replication,
> > >>>> but
> > >>>>> it
> > >>>>>>> can
> > >>>>>>>>> run in standalone mode or in memory (for tests/dev).
> > >>>>>>>>>
> > >>>>>>>>> I can help in setting up a demo.
> > >>>>>>>>>
> > >>>>>>>>> Enrico
> > >>>>>>>>>
> > >>>>>>>>> (1) https://github.com/diennea/herddb
> > >>>>>>>>>
> > >>>>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha
> > >>>>> scritto:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Dave,
> > >>>>>>>>>>
> > >>>>>>>>>> It doesn't require MySQL. You can use any JDBC supported
> > >>>>> databases.
> > >>>>>>>>>> Initially we would use SQLite for development and change to
> > >>>> MySQL
> > >>>>>> for
> > >>>>>>>>> more
> > >>>>>>>>>> production-ready workloads.
> > >>>>>>>>>>
> > >>>>>>>>>> Hope this clarify your question.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>> Sijie
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org>
> > >>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi -
> > >>>>>>>>>>>
> > >>>>>>>>>>> I see that MySQL is used. This would be a GPL dependency.
> > >>>> Does
> > >>>>>> the
> > >>>>>>>>> Pulsar
> > >>>>>>>>>>> Manager require MySQL, or can other Databases that are
> > >>>>> compliant
> > >>>>>>> with
> > >>>>>>>>>>> Apache Release Policy be used instead?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Regards,
> > >>>>>>>>>>> Dave
> > >>>>>>>>>>>
> > >>>>>>>>>>>> On Aug 29, 2019, at 7:49 AM, Guangning E <
> > >>>>> eguangning@gmail.com
> > >>>>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> We have developed a new Pulsar web UI - pulsar-manager,
> > >>>>> aiming
> > >>>>>> at
> > >>>>>>>>>>>> supporting managing Pulsar clusters running in different
> > >>>>>>>> environments
> > >>>>>>>>>>>> (on-premise data centers, cloud, and so on). We’d like to
> > >>>>>>>> contribute
> > >>>>>>>>>> the
> > >>>>>>>>>>>> project back to the Pulsar community.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ---
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ## Motivation
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Currently, Pulsar has a monitoring tool - [Pulsar
> > >>>>> Dashboard]( [
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> > >>>>>>>>>>>> ).
> > >>>>>>>>>>>> However, it only focuses on the simple monitoring of
> > >>>> Pulsar -
> > >>>>>>>>>> collecting
> > >>>>>>>>>>>> and displaying information such as statistics of tenants,
> > >>>>>>>> namespaces,
> > >>>>>>>>>>>> topics, subscriptions, and so on. It doesn’t provide any
> > >>>>>>> management
> > >>>>>>>>>>>> operations such as add, delete and update tenants,
> > >>>>> namespaces,
> > >>>>>>>>> topics,
> > >>>>>>>>>>> and
> > >>>>>>>>>>>> so on. When the scale of Pulsar cluster increases or the
> > >>>>> number
> > >>>>>>> of
> > >>>>>>>>>>> clusters
> > >>>>>>>>>>>> grows, using `pulsar-admin` to manage Pulsar can not
> > >>>> satisfy
> > >>>>>>>> demands.
> > >>>>>>>>>>>> Therefore, Pulsar needs a simple and easy-to-use management
> > >>>>>>> console
> > >>>>>>>>> for
> > >>>>>>>>>>>> administrators.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Pulsar Manager is a web-based GUI management and monitoring
> > >>>>>> tool
> > >>>>>>>> that
> > >>>>>>>>>>> helps
> > >>>>>>>>>>>> administrators and users manage and monitor tenants,
> > >>>>>> namespaces,
> > >>>>>>>>>> topics,
> > >>>>>>>>>>>> subscriptions, brokers, clusters, and so on, and supports
> > >>>>>> dynamic
> > >>>>>>>>>>>> configuration of multiple environments.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ## Features
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Pulsar manager provides two main features: management and
> > >>>>>>>> monitoring.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ### Management
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1. Environment : (operations on the environment)
> > >>>>>>>>>>>> Create / Delete / Update / List / Get
> > >>>>>>>>>>>> 2. Cluster : (operations on clusters)
> > >>>>>>>>>>>> Create / Delete / Update / List / Get
> > >>>>>>>>>>>> 3. Brokers : (operations on brokers)
> > >>>>>>>>>>>> Heartbeat / Unload
> > >>>>>>>>>>>> 4. Tenants:
> > >>>>>>>>>>>> Create / Delete / Update / List
> > >>>>>>>>>>>> 5. Namespaces:
> > >>>>>>>>>>>> Create / Delete / List
> > >>>>>>>>>>>> Manage Namespace Policy
> > >>>>>>>>>>>> Unload
> > >>>>>>>>>>>> Operations on Namespace Bundles
> > >>>>>>>>>>>> 6. Topics
> > >>>>>>>>>>>> Create / Delete / List
> > >>>>>>>>>>>> Unload / Terminate / Offload / Compact
> > >>>>>>>>>>>> Topic Details
> > >>>>>>>>>>>> 7. Subscriptions
> > >>>>>>>>>>>> Create / Delete / List
> > >>>>>>>>>>>> Reset Cursor / Skip / Clear backlog / Unsubscribe
> > >>>>>>>>>>>> 8. Namespace Isolation Policies
> > >>>>>>>>>>>> Create / Delete / Update / Get / List
> > >>>>>>>>>>>> 9. Failure Domains
> > >>>>>>>>>>>> Create / Delete / Update / Get / List
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ### Monitoring: (display *aggregated* stats at different
> > >>>>>> levels)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1. Tenants
> > >>>>>>>>>>>> List the total number of namespaces per tenant (both
> > >>>>> aggregated
> > >>>>>>> and
> > >>>>>>>>>>>> per-cluster basis)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 2. Namespaces
> > >>>>>>>>>>>> List the total number of topics per namespace
> > >>>>>>>>>>>> List the aggregated stats per namespace (such as rate-in,
> > >>>>>>> rate-out,
> > >>>>>>>>>>>> throughput-in, and throughput-out)
> > >>>>>>>>>>>> List the distributions of namespace bundles
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 3. Topics
> > >>>>>>>>>>>> List the total number of partitions per topic
> > >>>>>>>>>>>> List the aggregated stats per topic (such as rate-in,
> > >>>>> rate-out,
> > >>>>>>>>>>>> throughput-in, and throughput-out)
> > >>>>>>>>>>>> Detailed stats per topic partition
> > >>>>>>>>>>>> Detailed stats of storage per topic partition
> > >>>>>>>>>>>> Detailed stats of subscription per topic and per partition
> > >>>>>>>>>>>> Detailed stats of producers
> > >>>>>>>>>>>> Detailed stats of consumers
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The detailed design proposal is in
> > >>>>>>>>>>>> [
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Looking forward to any feedback.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>> Guangning
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> *Thanks*
> > >>>>
> > >>>> *Yuvaraj L*
> > >>>>
> > >>
> > >>
> >
> >
>

Re: PIP-40: Contribute Pulsar Manager

[Sijie Guo <gu...@gmail.com>]

Dave, thank you! That’s super helpful!

Thanks,
Sijie

On Wed, Sep 4, 2019 at 10:29 AM Dave Fisher <wa...@apache.org> wrote:

>
>
> > On Sep 4, 2019, at 10:10 AM, Sijie Guo <gu...@gmail.com> wrote:
> >
> > Hi Dave,
> >
> > Thank you for the reminder. I was about to ping you yesterday about this.
> > The IP clearance is the last piece that I wasn’t sure about. Just to make
> > sure I understand correctly, so the Pulsar PMC has to fill out the IP
> > clearance template, check in the file to the SVN repo and start a thread
> in
> > general@incubator to get the approval from IPMC. Is that correct?
>
> Yes. The approval is via LAZY CONSENSUS of 3 days.
>
> StreamNative ought to provide an SGA to secretary@ unless all of the IP
> is already covered by ICLAs.
>
> Infra can then help transfer the repository.
>
> Regards,
> Dave
>
> >
> > Thanks,
> > Sijie
> >
> > On Wed, Sep 4, 2019 at 9:41 AM Dave Fisher <wa...@apache.org> wrote:
> >
> >> BTW - Once the VOTE is complete to accept the codebase there is some
> >> paperwork.
> >>
> >> http://incubator.apache.org/ip-clearance/
> >>
> >> While this is the Incubator the Board has requested that the IPMC track
> IP
> >> Clearance. Everything is in SVN and the pages are updated every evening
> >> PDT. If help is needed then let me know.
> >>
> >> Regards,
> >> Dave
> >>
> >>> On Sep 4, 2019, at 9:01 AM, Enrico Olivelli <eo...@gmail.com>
> wrote:
> >>>
> >>> For whom who is interested I am working on adding support for HerdDB in
> >>> Pulsar Manager.
> >>>
> >>> The pull request is still very raw, but Pulsar Manager seems to work
> >>> https://github.com/streamnative/pulsar-manager/pull/183
> >>>
> >>>
> >>>
> >>> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <uv...@gmail.com>
> ha
> >>> scritto:
> >>>
> >>>> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <gu...@gmail.com> wrote:
> >>>>
> >>>>>> I see there are dependencies on websockets, I would a prefer a
> simple
> >>>>> polling model of the http, also it's enable by default in pulsar.
> >>>>>
> >>>>> I don't think we depend don pulsar websocket. All are http restful
> >>>>> requests.
> >>>>>
> >>>>>> For the ui the default persistence should be sqlite. Potentially
> >>>> packaged
> >>>>> by default.
> >>>>>
> >>>>> Initially the default is sqlite. But sqlite doesn't work if there
> are a
> >>>> lot
> >>>>> of topic metrics.
> >>>>> We switched to MySQL as default for supporting production traffic. We
> >> can
> >>>>> switch default to PostgresSQL (as pulsar dashboard).
> >>>>>
> >>>> +1
> >>>> Yes, Switching to Postgres Would be great. Because Postgres license is
> >> more
> >>>> liberal and works well with Apache license ecosystem.
> >>>>
> >>>>>
> >>>>> - Sijie
> >>>>>
> >>>>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <al...@gmail.com>
> >> wrote:
> >>>>>
> >>>>>> I see there are dependencies on websockets, I would a prefer a
> simple
> >>>>>> polling model of the http, also it's enable by default in pulsar.
> >>>>>>
> >>>>>> For the ui the default persistence should be sqlite. Potentially
> >>>> packaged
> >>>>>> by default.
> >>>>>>
> >>>>>> -Ali
> >>>>>>
> >>>>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <
> eolivelli@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha
> scritto:
> >>>>>>>
> >>>>>>>> That sounds an interesting idea!
> >>>>>>>
> >>>>>>>
> >>>>>>> Awesome
> >>>>>>>
> >>>>>>> Does HerdDB support JDBC? If so, it should
> >>>>>>>> be pretty straightforward to enable HerdDB.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Sure, as far as I know, the JDBC client is the only client really
> >>>> used
> >>>>> in
> >>>>>>> production.
> >>>>>>>
> >>>>>>> In a replicated environment it uses Zookeeper for metadata and
> >>>> service
> >>>>>>> discovery and Bookkeeper as WAL and if you have a pulsar cluster
> you
> >>>>>>> already have both of them (zk cluster an bookies)
> >>>>>>>
> >>>>>>> I will take a look to how Pulsar Manager   uses JDBC, maybe it will
> >>>> be
> >>>>>> very
> >>>>>>> easy.
> >>>>>>>
> >>>>>>> I will be back with news
> >>>>>>>
> >>>>>>>
> >>>>>>> Enrico
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Sijie
> >>>>>>>>
> >>>>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <
> >>>>> eolivelli@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL
> >>>>> database
> >>>>>>> that
> >>>>>>>>> can be run embedded in the JVM.
> >>>>>>>>> It is an SQL database that we developed in my company, in order
> >>>> to
> >>>>>>>>> explicitly replace MySQL.
> >>>>>>>>> It uses Bookkeeper to store the WAL and implement replication,
> >>>> but
> >>>>> it
> >>>>>>> can
> >>>>>>>>> run in standalone mode or in memory (for tests/dev).
> >>>>>>>>>
> >>>>>>>>> I can help in setting up a demo.
> >>>>>>>>>
> >>>>>>>>> Enrico
> >>>>>>>>>
> >>>>>>>>> (1) https://github.com/diennea/herddb
> >>>>>>>>>
> >>>>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha
> >>>>> scritto:
> >>>>>>>>>
> >>>>>>>>>> Hi Dave,
> >>>>>>>>>>
> >>>>>>>>>> It doesn't require MySQL. You can use any JDBC supported
> >>>>> databases.
> >>>>>>>>>> Initially we would use SQLite for development and change to
> >>>> MySQL
> >>>>>> for
> >>>>>>>>> more
> >>>>>>>>>> production-ready workloads.
> >>>>>>>>>>
> >>>>>>>>>> Hope this clarify your question.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Sijie
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org>
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi -
> >>>>>>>>>>>
> >>>>>>>>>>> I see that MySQL is used. This would be a GPL dependency.
> >>>> Does
> >>>>>> the
> >>>>>>>>> Pulsar
> >>>>>>>>>>> Manager require MySQL, or can other Databases that are
> >>>>> compliant
> >>>>>>> with
> >>>>>>>>>>> Apache Release Policy be used instead?
> >>>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>> Dave
> >>>>>>>>>>>
> >>>>>>>>>>>> On Aug 29, 2019, at 7:49 AM, Guangning E <
> >>>>> eguangning@gmail.com
> >>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> We have developed a new Pulsar web UI - pulsar-manager,
> >>>>> aiming
> >>>>>> at
> >>>>>>>>>>>> supporting managing Pulsar clusters running in different
> >>>>>>>> environments
> >>>>>>>>>>>> (on-premise data centers, cloud, and so on). We’d like to
> >>>>>>>> contribute
> >>>>>>>>>> the
> >>>>>>>>>>>> project back to the Pulsar community.
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>>
> >>>>>>>>>>>> ## Motivation
> >>>>>>>>>>>>
> >>>>>>>>>>>> Currently, Pulsar has a monitoring tool - [Pulsar
> >>>>> Dashboard]( [
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> >>>>>>>>>>>> ).
> >>>>>>>>>>>> However, it only focuses on the simple monitoring of
> >>>> Pulsar -
> >>>>>>>>>> collecting
> >>>>>>>>>>>> and displaying information such as statistics of tenants,
> >>>>>>>> namespaces,
> >>>>>>>>>>>> topics, subscriptions, and so on. It doesn’t provide any
> >>>>>>> management
> >>>>>>>>>>>> operations such as add, delete and update tenants,
> >>>>> namespaces,
> >>>>>>>>> topics,
> >>>>>>>>>>> and
> >>>>>>>>>>>> so on. When the scale of Pulsar cluster increases or the
> >>>>> number
> >>>>>>> of
> >>>>>>>>>>> clusters
> >>>>>>>>>>>> grows, using `pulsar-admin` to manage Pulsar can not
> >>>> satisfy
> >>>>>>>> demands.
> >>>>>>>>>>>> Therefore, Pulsar needs a simple and easy-to-use management
> >>>>>>> console
> >>>>>>>>> for
> >>>>>>>>>>>> administrators.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Pulsar Manager is a web-based GUI management and monitoring
> >>>>>> tool
> >>>>>>>> that
> >>>>>>>>>>> helps
> >>>>>>>>>>>> administrators and users manage and monitor tenants,
> >>>>>> namespaces,
> >>>>>>>>>> topics,
> >>>>>>>>>>>> subscriptions, brokers, clusters, and so on, and supports
> >>>>>> dynamic
> >>>>>>>>>>>> configuration of multiple environments.
> >>>>>>>>>>>>
> >>>>>>>>>>>> ## Features
> >>>>>>>>>>>>
> >>>>>>>>>>>> Pulsar manager provides two main features: management and
> >>>>>>>> monitoring.
> >>>>>>>>>>>>
> >>>>>>>>>>>> ### Management
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. Environment : (operations on the environment)
> >>>>>>>>>>>> Create / Delete / Update / List / Get
> >>>>>>>>>>>> 2. Cluster : (operations on clusters)
> >>>>>>>>>>>> Create / Delete / Update / List / Get
> >>>>>>>>>>>> 3. Brokers : (operations on brokers)
> >>>>>>>>>>>> Heartbeat / Unload
> >>>>>>>>>>>> 4. Tenants:
> >>>>>>>>>>>> Create / Delete / Update / List
> >>>>>>>>>>>> 5. Namespaces:
> >>>>>>>>>>>> Create / Delete / List
> >>>>>>>>>>>> Manage Namespace Policy
> >>>>>>>>>>>> Unload
> >>>>>>>>>>>> Operations on Namespace Bundles
> >>>>>>>>>>>> 6. Topics
> >>>>>>>>>>>> Create / Delete / List
> >>>>>>>>>>>> Unload / Terminate / Offload / Compact
> >>>>>>>>>>>> Topic Details
> >>>>>>>>>>>> 7. Subscriptions
> >>>>>>>>>>>> Create / Delete / List
> >>>>>>>>>>>> Reset Cursor / Skip / Clear backlog / Unsubscribe
> >>>>>>>>>>>> 8. Namespace Isolation Policies
> >>>>>>>>>>>> Create / Delete / Update / Get / List
> >>>>>>>>>>>> 9. Failure Domains
> >>>>>>>>>>>> Create / Delete / Update / Get / List
> >>>>>>>>>>>>
> >>>>>>>>>>>> ### Monitoring: (display *aggregated* stats at different
> >>>>>> levels)
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. Tenants
> >>>>>>>>>>>> List the total number of namespaces per tenant (both
> >>>>> aggregated
> >>>>>>> and
> >>>>>>>>>>>> per-cluster basis)
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2. Namespaces
> >>>>>>>>>>>> List the total number of topics per namespace
> >>>>>>>>>>>> List the aggregated stats per namespace (such as rate-in,
> >>>>>>> rate-out,
> >>>>>>>>>>>> throughput-in, and throughput-out)
> >>>>>>>>>>>> List the distributions of namespace bundles
> >>>>>>>>>>>>
> >>>>>>>>>>>> 3. Topics
> >>>>>>>>>>>> List the total number of partitions per topic
> >>>>>>>>>>>> List the aggregated stats per topic (such as rate-in,
> >>>>> rate-out,
> >>>>>>>>>>>> throughput-in, and throughput-out)
> >>>>>>>>>>>> Detailed stats per topic partition
> >>>>>>>>>>>> Detailed stats of storage per topic partition
> >>>>>>>>>>>> Detailed stats of subscription per topic and per partition
> >>>>>>>>>>>> Detailed stats of producers
> >>>>>>>>>>>> Detailed stats of consumers
> >>>>>>>>>>>>
> >>>>>>>>>>>> The detailed design proposal is in
> >>>>>>>>>>>> [
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Looking forward to any feedback.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Guangning
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> *Thanks*
> >>>>
> >>>> *Yuvaraj L*
> >>>>
> >>
> >>
>
>

Re: PIP-40: Contribute Pulsar Manager

[Sijie Guo <gu...@gmail.com>]

Hi Dave,

I have committed an ip-clearance form to the SVN repo. but it doesn't seem
to show up in the website (after one day):
https://incubator.apache.org/ip-clearance/

I am wondering how can I trigger the build of incubator website. so that I
can proceed sending out the ip-clearance email.

Thanks,
Sijie

On Wed, Sep 4, 2019 at 10:29 AM Dave Fisher <wa...@apache.org> wrote:

>
>
> > On Sep 4, 2019, at 10:10 AM, Sijie Guo <gu...@gmail.com> wrote:
> >
> > Hi Dave,
> >
> > Thank you for the reminder. I was about to ping you yesterday about this.
> > The IP clearance is the last piece that I wasn’t sure about. Just to make
> > sure I understand correctly, so the Pulsar PMC has to fill out the IP
> > clearance template, check in the file to the SVN repo and start a thread
> in
> > general@incubator to get the approval from IPMC. Is that correct?
>
> Yes. The approval is via LAZY CONSENSUS of 3 days.
>
> StreamNative ought to provide an SGA to secretary@ unless all of the IP
> is already covered by ICLAs.
>
> Infra can then help transfer the repository.
>
> Regards,
> Dave
>
> >
> > Thanks,
> > Sijie
> >
> > On Wed, Sep 4, 2019 at 9:41 AM Dave Fisher <wa...@apache.org> wrote:
> >
> >> BTW - Once the VOTE is complete to accept the codebase there is some
> >> paperwork.
> >>
> >> http://incubator.apache.org/ip-clearance/
> >>
> >> While this is the Incubator the Board has requested that the IPMC track
> IP
> >> Clearance. Everything is in SVN and the pages are updated every evening
> >> PDT. If help is needed then let me know.
> >>
> >> Regards,
> >> Dave
> >>
> >>> On Sep 4, 2019, at 9:01 AM, Enrico Olivelli <eo...@gmail.com>
> wrote:
> >>>
> >>> For whom who is interested I am working on adding support for HerdDB in
> >>> Pulsar Manager.
> >>>
> >>> The pull request is still very raw, but Pulsar Manager seems to work
> >>> https://github.com/streamnative/pulsar-manager/pull/183
> >>>
> >>>
> >>>
> >>> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <uv...@gmail.com>
> ha
> >>> scritto:
> >>>
> >>>> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <gu...@gmail.com> wrote:
> >>>>
> >>>>>> I see there are dependencies on websockets, I would a prefer a
> simple
> >>>>> polling model of the http, also it's enable by default in pulsar.
> >>>>>
> >>>>> I don't think we depend don pulsar websocket. All are http restful
> >>>>> requests.
> >>>>>
> >>>>>> For the ui the default persistence should be sqlite. Potentially
> >>>> packaged
> >>>>> by default.
> >>>>>
> >>>>> Initially the default is sqlite. But sqlite doesn't work if there
> are a
> >>>> lot
> >>>>> of topic metrics.
> >>>>> We switched to MySQL as default for supporting production traffic. We
> >> can
> >>>>> switch default to PostgresSQL (as pulsar dashboard).
> >>>>>
> >>>> +1
> >>>> Yes, Switching to Postgres Would be great. Because Postgres license is
> >> more
> >>>> liberal and works well with Apache license ecosystem.
> >>>>
> >>>>>
> >>>>> - Sijie
> >>>>>
> >>>>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <al...@gmail.com>
> >> wrote:
> >>>>>
> >>>>>> I see there are dependencies on websockets, I would a prefer a
> simple
> >>>>>> polling model of the http, also it's enable by default in pulsar.
> >>>>>>
> >>>>>> For the ui the default persistence should be sqlite. Potentially
> >>>> packaged
> >>>>>> by default.
> >>>>>>
> >>>>>> -Ali
> >>>>>>
> >>>>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <
> eolivelli@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha
> scritto:
> >>>>>>>
> >>>>>>>> That sounds an interesting idea!
> >>>>>>>
> >>>>>>>
> >>>>>>> Awesome
> >>>>>>>
> >>>>>>> Does HerdDB support JDBC? If so, it should
> >>>>>>>> be pretty straightforward to enable HerdDB.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Sure, as far as I know, the JDBC client is the only client really
> >>>> used
> >>>>> in
> >>>>>>> production.
> >>>>>>>
> >>>>>>> In a replicated environment it uses Zookeeper for metadata and
> >>>> service
> >>>>>>> discovery and Bookkeeper as WAL and if you have a pulsar cluster
> you
> >>>>>>> already have both of them (zk cluster an bookies)
> >>>>>>>
> >>>>>>> I will take a look to how Pulsar Manager   uses JDBC, maybe it will
> >>>> be
> >>>>>> very
> >>>>>>> easy.
> >>>>>>>
> >>>>>>> I will be back with news
> >>>>>>>
> >>>>>>>
> >>>>>>> Enrico
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Sijie
> >>>>>>>>
> >>>>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <
> >>>>> eolivelli@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL
> >>>>> database
> >>>>>>> that
> >>>>>>>>> can be run embedded in the JVM.
> >>>>>>>>> It is an SQL database that we developed in my company, in order
> >>>> to
> >>>>>>>>> explicitly replace MySQL.
> >>>>>>>>> It uses Bookkeeper to store the WAL and implement replication,
> >>>> but
> >>>>> it
> >>>>>>> can
> >>>>>>>>> run in standalone mode or in memory (for tests/dev).
> >>>>>>>>>
> >>>>>>>>> I can help in setting up a demo.
> >>>>>>>>>
> >>>>>>>>> Enrico
> >>>>>>>>>
> >>>>>>>>> (1) https://github.com/diennea/herddb
> >>>>>>>>>
> >>>>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha
> >>>>> scritto:
> >>>>>>>>>
> >>>>>>>>>> Hi Dave,
> >>>>>>>>>>
> >>>>>>>>>> It doesn't require MySQL. You can use any JDBC supported
> >>>>> databases.
> >>>>>>>>>> Initially we would use SQLite for development and change to
> >>>> MySQL
> >>>>>> for
> >>>>>>>>> more
> >>>>>>>>>> production-ready workloads.
> >>>>>>>>>>
> >>>>>>>>>> Hope this clarify your question.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Sijie
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org>
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi -
> >>>>>>>>>>>
> >>>>>>>>>>> I see that MySQL is used. This would be a GPL dependency.
> >>>> Does
> >>>>>> the
> >>>>>>>>> Pulsar
> >>>>>>>>>>> Manager require MySQL, or can other Databases that are
> >>>>> compliant
> >>>>>>> with
> >>>>>>>>>>> Apache Release Policy be used instead?
> >>>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>> Dave
> >>>>>>>>>>>
> >>>>>>>>>>>> On Aug 29, 2019, at 7:49 AM, Guangning E <
> >>>>> eguangning@gmail.com
> >>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> We have developed a new Pulsar web UI - pulsar-manager,
> >>>>> aiming
> >>>>>> at
> >>>>>>>>>>>> supporting managing Pulsar clusters running in different
> >>>>>>>> environments
> >>>>>>>>>>>> (on-premise data centers, cloud, and so on). We’d like to
> >>>>>>>> contribute
> >>>>>>>>>> the
> >>>>>>>>>>>> project back to the Pulsar community.
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>>
> >>>>>>>>>>>> ## Motivation
> >>>>>>>>>>>>
> >>>>>>>>>>>> Currently, Pulsar has a monitoring tool - [Pulsar
> >>>>> Dashboard]( [
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> >>>>>>>>>>>> ).
> >>>>>>>>>>>> However, it only focuses on the simple monitoring of
> >>>> Pulsar -
> >>>>>>>>>> collecting
> >>>>>>>>>>>> and displaying information such as statistics of tenants,
> >>>>>>>> namespaces,
> >>>>>>>>>>>> topics, subscriptions, and so on. It doesn’t provide any
> >>>>>>> management
> >>>>>>>>>>>> operations such as add, delete and update tenants,
> >>>>> namespaces,
> >>>>>>>>> topics,
> >>>>>>>>>>> and
> >>>>>>>>>>>> so on. When the scale of Pulsar cluster increases or the
> >>>>> number
> >>>>>>> of
> >>>>>>>>>>> clusters
> >>>>>>>>>>>> grows, using `pulsar-admin` to manage Pulsar can not
> >>>> satisfy
> >>>>>>>> demands.
> >>>>>>>>>>>> Therefore, Pulsar needs a simple and easy-to-use management
> >>>>>>> console
> >>>>>>>>> for
> >>>>>>>>>>>> administrators.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Pulsar Manager is a web-based GUI management and monitoring
> >>>>>> tool
> >>>>>>>> that
> >>>>>>>>>>> helps
> >>>>>>>>>>>> administrators and users manage and monitor tenants,
> >>>>>> namespaces,
> >>>>>>>>>> topics,
> >>>>>>>>>>>> subscriptions, brokers, clusters, and so on, and supports
> >>>>>> dynamic
> >>>>>>>>>>>> configuration of multiple environments.
> >>>>>>>>>>>>
> >>>>>>>>>>>> ## Features
> >>>>>>>>>>>>
> >>>>>>>>>>>> Pulsar manager provides two main features: management and
> >>>>>>>> monitoring.
> >>>>>>>>>>>>
> >>>>>>>>>>>> ### Management
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. Environment : (operations on the environment)
> >>>>>>>>>>>> Create / Delete / Update / List / Get
> >>>>>>>>>>>> 2. Cluster : (operations on clusters)
> >>>>>>>>>>>> Create / Delete / Update / List / Get
> >>>>>>>>>>>> 3. Brokers : (operations on brokers)
> >>>>>>>>>>>> Heartbeat / Unload
> >>>>>>>>>>>> 4. Tenants:
> >>>>>>>>>>>> Create / Delete / Update / List
> >>>>>>>>>>>> 5. Namespaces:
> >>>>>>>>>>>> Create / Delete / List
> >>>>>>>>>>>> Manage Namespace Policy
> >>>>>>>>>>>> Unload
> >>>>>>>>>>>> Operations on Namespace Bundles
> >>>>>>>>>>>> 6. Topics
> >>>>>>>>>>>> Create / Delete / List
> >>>>>>>>>>>> Unload / Terminate / Offload / Compact
> >>>>>>>>>>>> Topic Details
> >>>>>>>>>>>> 7. Subscriptions
> >>>>>>>>>>>> Create / Delete / List
> >>>>>>>>>>>> Reset Cursor / Skip / Clear backlog / Unsubscribe
> >>>>>>>>>>>> 8. Namespace Isolation Policies
> >>>>>>>>>>>> Create / Delete / Update / Get / List
> >>>>>>>>>>>> 9. Failure Domains
> >>>>>>>>>>>> Create / Delete / Update / Get / List
> >>>>>>>>>>>>
> >>>>>>>>>>>> ### Monitoring: (display *aggregated* stats at different
> >>>>>> levels)
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. Tenants
> >>>>>>>>>>>> List the total number of namespaces per tenant (both
> >>>>> aggregated
> >>>>>>> and
> >>>>>>>>>>>> per-cluster basis)
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2. Namespaces
> >>>>>>>>>>>> List the total number of topics per namespace
> >>>>>>>>>>>> List the aggregated stats per namespace (such as rate-in,
> >>>>>>> rate-out,
> >>>>>>>>>>>> throughput-in, and throughput-out)
> >>>>>>>>>>>> List the distributions of namespace bundles
> >>>>>>>>>>>>
> >>>>>>>>>>>> 3. Topics
> >>>>>>>>>>>> List the total number of partitions per topic
> >>>>>>>>>>>> List the aggregated stats per topic (such as rate-in,
> >>>>> rate-out,
> >>>>>>>>>>>> throughput-in, and throughput-out)
> >>>>>>>>>>>> Detailed stats per topic partition
> >>>>>>>>>>>> Detailed stats of storage per topic partition
> >>>>>>>>>>>> Detailed stats of subscription per topic and per partition
> >>>>>>>>>>>> Detailed stats of producers
> >>>>>>>>>>>> Detailed stats of consumers
> >>>>>>>>>>>>
> >>>>>>>>>>>> The detailed design proposal is in
> >>>>>>>>>>>> [
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Looking forward to any feedback.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Guangning
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> *Thanks*
> >>>>
> >>>> *Yuvaraj L*
> >>>>
> >>
> >>
>
>

Re: PIP-40: Contribute Pulsar Manager

[Dave Fisher <wa...@apache.org>]

> On Sep 4, 2019, at 10:10 AM, Sijie Guo <gu...@gmail.com> wrote:
> 
> Hi Dave,
> 
> Thank you for the reminder. I was about to ping you yesterday about this.
> The IP clearance is the last piece that I wasn’t sure about. Just to make
> sure I understand correctly, so the Pulsar PMC has to fill out the IP
> clearance template, check in the file to the SVN repo and start a thread in
> general@incubator to get the approval from IPMC. Is that correct?

Yes. The approval is via LAZY CONSENSUS of 3 days.

StreamNative ought to provide an SGA to secretary@ unless all of the IP is already covered by ICLAs.

Infra can then help transfer the repository. 

Regards,
Dave

> 
> Thanks,
> Sijie
> 
> On Wed, Sep 4, 2019 at 9:41 AM Dave Fisher <wa...@apache.org> wrote:
> 
>> BTW - Once the VOTE is complete to accept the codebase there is some
>> paperwork.
>> 
>> http://incubator.apache.org/ip-clearance/
>> 
>> While this is the Incubator the Board has requested that the IPMC track IP
>> Clearance. Everything is in SVN and the pages are updated every evening
>> PDT. If help is needed then let me know.
>> 
>> Regards,
>> Dave
>> 
>>> On Sep 4, 2019, at 9:01 AM, Enrico Olivelli <eo...@gmail.com> wrote:
>>> 
>>> For whom who is interested I am working on adding support for HerdDB in
>>> Pulsar Manager.
>>> 
>>> The pull request is still very raw, but Pulsar Manager seems to work
>>> https://github.com/streamnative/pulsar-manager/pull/183
>>> 
>>> 
>>> 
>>> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <uv...@gmail.com> ha
>>> scritto:
>>> 
>>>> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <gu...@gmail.com> wrote:
>>>> 
>>>>>> I see there are dependencies on websockets, I would a prefer a simple
>>>>> polling model of the http, also it's enable by default in pulsar.
>>>>> 
>>>>> I don't think we depend don pulsar websocket. All are http restful
>>>>> requests.
>>>>> 
>>>>>> For the ui the default persistence should be sqlite. Potentially
>>>> packaged
>>>>> by default.
>>>>> 
>>>>> Initially the default is sqlite. But sqlite doesn't work if there are a
>>>> lot
>>>>> of topic metrics.
>>>>> We switched to MySQL as default for supporting production traffic. We
>> can
>>>>> switch default to PostgresSQL (as pulsar dashboard).
>>>>> 
>>>> +1
>>>> Yes, Switching to Postgres Would be great. Because Postgres license is
>> more
>>>> liberal and works well with Apache license ecosystem.
>>>> 
>>>>> 
>>>>> - Sijie
>>>>> 
>>>>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <al...@gmail.com>
>> wrote:
>>>>> 
>>>>>> I see there are dependencies on websockets, I would a prefer a simple
>>>>>> polling model of the http, also it's enable by default in pulsar.
>>>>>> 
>>>>>> For the ui the default persistence should be sqlite. Potentially
>>>> packaged
>>>>>> by default.
>>>>>> 
>>>>>> -Ali
>>>>>> 
>>>>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <eo...@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha scritto:
>>>>>>> 
>>>>>>>> That sounds an interesting idea!
>>>>>>> 
>>>>>>> 
>>>>>>> Awesome
>>>>>>> 
>>>>>>> Does HerdDB support JDBC? If so, it should
>>>>>>>> be pretty straightforward to enable HerdDB.
>>>>>>>> 
>>>>>>> 
>>>>>>> Sure, as far as I know, the JDBC client is the only client really
>>>> used
>>>>> in
>>>>>>> production.
>>>>>>> 
>>>>>>> In a replicated environment it uses Zookeeper for metadata and
>>>> service
>>>>>>> discovery and Bookkeeper as WAL and if you have a pulsar cluster you
>>>>>>> already have both of them (zk cluster an bookies)
>>>>>>> 
>>>>>>> I will take a look to how Pulsar Manager   uses JDBC, maybe it will
>>>> be
>>>>>> very
>>>>>>> easy.
>>>>>>> 
>>>>>>> I will be back with news
>>>>>>> 
>>>>>>> 
>>>>>>> Enrico
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Sijie
>>>>>>>> 
>>>>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <
>>>>> eolivelli@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL
>>>>> database
>>>>>>> that
>>>>>>>>> can be run embedded in the JVM.
>>>>>>>>> It is an SQL database that we developed in my company, in order
>>>> to
>>>>>>>>> explicitly replace MySQL.
>>>>>>>>> It uses Bookkeeper to store the WAL and implement replication,
>>>> but
>>>>> it
>>>>>>> can
>>>>>>>>> run in standalone mode or in memory (for tests/dev).
>>>>>>>>> 
>>>>>>>>> I can help in setting up a demo.
>>>>>>>>> 
>>>>>>>>> Enrico
>>>>>>>>> 
>>>>>>>>> (1) https://github.com/diennea/herddb
>>>>>>>>> 
>>>>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha
>>>>> scritto:
>>>>>>>>> 
>>>>>>>>>> Hi Dave,
>>>>>>>>>> 
>>>>>>>>>> It doesn't require MySQL. You can use any JDBC supported
>>>>> databases.
>>>>>>>>>> Initially we would use SQLite for development and change to
>>>> MySQL
>>>>>> for
>>>>>>>>> more
>>>>>>>>>> production-ready workloads.
>>>>>>>>>> 
>>>>>>>>>> Hope this clarify your question.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> Sijie
>>>>>>>>>> 
>>>>>>>>>> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org>
>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi -
>>>>>>>>>>> 
>>>>>>>>>>> I see that MySQL is used. This would be a GPL dependency.
>>>> Does
>>>>>> the
>>>>>>>>> Pulsar
>>>>>>>>>>> Manager require MySQL, or can other Databases that are
>>>>> compliant
>>>>>>> with
>>>>>>>>>>> Apache Release Policy be used instead?
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> Dave
>>>>>>>>>>> 
>>>>>>>>>>>> On Aug 29, 2019, at 7:49 AM, Guangning E <
>>>>> eguangning@gmail.com
>>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>> 
>>>>>>>>>>>> We have developed a new Pulsar web UI - pulsar-manager,
>>>>> aiming
>>>>>> at
>>>>>>>>>>>> supporting managing Pulsar clusters running in different
>>>>>>>> environments
>>>>>>>>>>>> (on-premise data centers, cloud, and so on). We’d like to
>>>>>>>> contribute
>>>>>>>>>> the
>>>>>>>>>>>> project back to the Pulsar community.
>>>>>>>>>>>> 
>>>>>>>>>>>> ---
>>>>>>>>>>>> 
>>>>>>>>>>>> ## Motivation
>>>>>>>>>>>> 
>>>>>>>>>>>> Currently, Pulsar has a monitoring tool - [Pulsar
>>>>> Dashboard]( [
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
>>>>>>>>>>>> ).
>>>>>>>>>>>> However, it only focuses on the simple monitoring of
>>>> Pulsar -
>>>>>>>>>> collecting
>>>>>>>>>>>> and displaying information such as statistics of tenants,
>>>>>>>> namespaces,
>>>>>>>>>>>> topics, subscriptions, and so on. It doesn’t provide any
>>>>>>> management
>>>>>>>>>>>> operations such as add, delete and update tenants,
>>>>> namespaces,
>>>>>>>>> topics,
>>>>>>>>>>> and
>>>>>>>>>>>> so on. When the scale of Pulsar cluster increases or the
>>>>> number
>>>>>>> of
>>>>>>>>>>> clusters
>>>>>>>>>>>> grows, using `pulsar-admin` to manage Pulsar can not
>>>> satisfy
>>>>>>>> demands.
>>>>>>>>>>>> Therefore, Pulsar needs a simple and easy-to-use management
>>>>>>> console
>>>>>>>>> for
>>>>>>>>>>>> administrators.
>>>>>>>>>>>> 
>>>>>>>>>>>> Pulsar Manager is a web-based GUI management and monitoring
>>>>>> tool
>>>>>>>> that
>>>>>>>>>>> helps
>>>>>>>>>>>> administrators and users manage and monitor tenants,
>>>>>> namespaces,
>>>>>>>>>> topics,
>>>>>>>>>>>> subscriptions, brokers, clusters, and so on, and supports
>>>>>> dynamic
>>>>>>>>>>>> configuration of multiple environments.
>>>>>>>>>>>> 
>>>>>>>>>>>> ## Features
>>>>>>>>>>>> 
>>>>>>>>>>>> Pulsar manager provides two main features: management and
>>>>>>>> monitoring.
>>>>>>>>>>>> 
>>>>>>>>>>>> ### Management
>>>>>>>>>>>> 
>>>>>>>>>>>> 1. Environment : (operations on the environment)
>>>>>>>>>>>> Create / Delete / Update / List / Get
>>>>>>>>>>>> 2. Cluster : (operations on clusters)
>>>>>>>>>>>> Create / Delete / Update / List / Get
>>>>>>>>>>>> 3. Brokers : (operations on brokers)
>>>>>>>>>>>> Heartbeat / Unload
>>>>>>>>>>>> 4. Tenants:
>>>>>>>>>>>> Create / Delete / Update / List
>>>>>>>>>>>> 5. Namespaces:
>>>>>>>>>>>> Create / Delete / List
>>>>>>>>>>>> Manage Namespace Policy
>>>>>>>>>>>> Unload
>>>>>>>>>>>> Operations on Namespace Bundles
>>>>>>>>>>>> 6. Topics
>>>>>>>>>>>> Create / Delete / List
>>>>>>>>>>>> Unload / Terminate / Offload / Compact
>>>>>>>>>>>> Topic Details
>>>>>>>>>>>> 7. Subscriptions
>>>>>>>>>>>> Create / Delete / List
>>>>>>>>>>>> Reset Cursor / Skip / Clear backlog / Unsubscribe
>>>>>>>>>>>> 8. Namespace Isolation Policies
>>>>>>>>>>>> Create / Delete / Update / Get / List
>>>>>>>>>>>> 9. Failure Domains
>>>>>>>>>>>> Create / Delete / Update / Get / List
>>>>>>>>>>>> 
>>>>>>>>>>>> ### Monitoring: (display *aggregated* stats at different
>>>>>> levels)
>>>>>>>>>>>> 
>>>>>>>>>>>> 1. Tenants
>>>>>>>>>>>> List the total number of namespaces per tenant (both
>>>>> aggregated
>>>>>>> and
>>>>>>>>>>>> per-cluster basis)
>>>>>>>>>>>> 
>>>>>>>>>>>> 2. Namespaces
>>>>>>>>>>>> List the total number of topics per namespace
>>>>>>>>>>>> List the aggregated stats per namespace (such as rate-in,
>>>>>>> rate-out,
>>>>>>>>>>>> throughput-in, and throughput-out)
>>>>>>>>>>>> List the distributions of namespace bundles
>>>>>>>>>>>> 
>>>>>>>>>>>> 3. Topics
>>>>>>>>>>>> List the total number of partitions per topic
>>>>>>>>>>>> List the aggregated stats per topic (such as rate-in,
>>>>> rate-out,
>>>>>>>>>>>> throughput-in, and throughput-out)
>>>>>>>>>>>> Detailed stats per topic partition
>>>>>>>>>>>> Detailed stats of storage per topic partition
>>>>>>>>>>>> Detailed stats of subscription per topic and per partition
>>>>>>>>>>>> Detailed stats of producers
>>>>>>>>>>>> Detailed stats of consumers
>>>>>>>>>>>> 
>>>>>>>>>>>> The detailed design proposal is in
>>>>>>>>>>>> [
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
>>>>>>>>>>>> 
>>>>>>>>>>>> Looking forward to any feedback.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Guangning
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> *Thanks*
>>>> 
>>>> *Yuvaraj L*
>>>> 
>> 
>> 

Re: PIP-40: Contribute Pulsar Manager

[Sijie Guo <gu...@gmail.com>]

Hi Dave,

Thank you for the reminder. I was about to ping you yesterday about this.
The IP clearance is the last piece that I wasn’t sure about. Just to make
sure I understand correctly, so the Pulsar PMC has to fill out the IP
clearance template, check in the file to the SVN repo and start a thread in
general@incubator to get the approval from IPMC. Is that correct?

Thanks,
Sijie

On Wed, Sep 4, 2019 at 9:41 AM Dave Fisher <wa...@apache.org> wrote:

> BTW - Once the VOTE is complete to accept the codebase there is some
> paperwork.
>
> http://incubator.apache.org/ip-clearance/
>
> While this is the Incubator the Board has requested that the IPMC track IP
> Clearance. Everything is in SVN and the pages are updated every evening
> PDT. If help is needed then let me know.
>
> Regards,
> Dave
>
> > On Sep 4, 2019, at 9:01 AM, Enrico Olivelli <eo...@gmail.com> wrote:
> >
> > For whom who is interested I am working on adding support for HerdDB in
> > Pulsar Manager.
> >
> > The pull request is still very raw, but Pulsar Manager seems to work
> > https://github.com/streamnative/pulsar-manager/pull/183
> >
> >
> >
> > Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <uv...@gmail.com> ha
> > scritto:
> >
> >> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <gu...@gmail.com> wrote:
> >>
> >>>> I see there are dependencies on websockets, I would a prefer a simple
> >>> polling model of the http, also it's enable by default in pulsar.
> >>>
> >>> I don't think we depend don pulsar websocket. All are http restful
> >>> requests.
> >>>
> >>>> For the ui the default persistence should be sqlite. Potentially
> >> packaged
> >>> by default.
> >>>
> >>> Initially the default is sqlite. But sqlite doesn't work if there are a
> >> lot
> >>> of topic metrics.
> >>> We switched to MySQL as default for supporting production traffic. We
> can
> >>> switch default to PostgresSQL (as pulsar dashboard).
> >>>
> >> +1
> >> Yes, Switching to Postgres Would be great. Because Postgres license is
> more
> >> liberal and works well with Apache license ecosystem.
> >>
> >>>
> >>> - Sijie
> >>>
> >>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <al...@gmail.com>
> wrote:
> >>>
> >>>> I see there are dependencies on websockets, I would a prefer a simple
> >>>> polling model of the http, also it's enable by default in pulsar.
> >>>>
> >>>> For the ui the default persistence should be sqlite. Potentially
> >> packaged
> >>>> by default.
> >>>>
> >>>> -Ali
> >>>>
> >>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <eo...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha scritto:
> >>>>>
> >>>>>> That sounds an interesting idea!
> >>>>>
> >>>>>
> >>>>> Awesome
> >>>>>
> >>>>> Does HerdDB support JDBC? If so, it should
> >>>>>> be pretty straightforward to enable HerdDB.
> >>>>>>
> >>>>>
> >>>>> Sure, as far as I know, the JDBC client is the only client really
> >> used
> >>> in
> >>>>> production.
> >>>>>
> >>>>> In a replicated environment it uses Zookeeper for metadata and
> >> service
> >>>>> discovery and Bookkeeper as WAL and if you have a pulsar cluster you
> >>>>> already have both of them (zk cluster an bookies)
> >>>>>
> >>>>> I will take a look to how Pulsar Manager   uses JDBC, maybe it will
> >> be
> >>>> very
> >>>>> easy.
> >>>>>
> >>>>> I will be back with news
> >>>>>
> >>>>>
> >>>>> Enrico
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Thanks,
> >>>>>> Sijie
> >>>>>>
> >>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <
> >>> eolivelli@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL
> >>> database
> >>>>> that
> >>>>>>> can be run embedded in the JVM.
> >>>>>>> It is an SQL database that we developed in my company, in order
> >> to
> >>>>>>> explicitly replace MySQL.
> >>>>>>> It uses Bookkeeper to store the WAL and implement replication,
> >> but
> >>> it
> >>>>> can
> >>>>>>> run in standalone mode or in memory (for tests/dev).
> >>>>>>>
> >>>>>>> I can help in setting up a demo.
> >>>>>>>
> >>>>>>> Enrico
> >>>>>>>
> >>>>>>> (1) https://github.com/diennea/herddb
> >>>>>>>
> >>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha
> >>> scritto:
> >>>>>>>
> >>>>>>>> Hi Dave,
> >>>>>>>>
> >>>>>>>> It doesn't require MySQL. You can use any JDBC supported
> >>> databases.
> >>>>>>>> Initially we would use SQLite for development and change to
> >> MySQL
> >>>> for
> >>>>>>> more
> >>>>>>>> production-ready workloads.
> >>>>>>>>
> >>>>>>>> Hope this clarify your question.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Sijie
> >>>>>>>>
> >>>>>>>> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi -
> >>>>>>>>>
> >>>>>>>>> I see that MySQL is used. This would be a GPL dependency.
> >> Does
> >>>> the
> >>>>>>> Pulsar
> >>>>>>>>> Manager require MySQL, or can other Databases that are
> >>> compliant
> >>>>> with
> >>>>>>>>> Apache Release Policy be used instead?
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Dave
> >>>>>>>>>
> >>>>>>>>>> On Aug 29, 2019, at 7:49 AM, Guangning E <
> >>> eguangning@gmail.com
> >>>>>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi all,
> >>>>>>>>>>
> >>>>>>>>>> We have developed a new Pulsar web UI - pulsar-manager,
> >>> aiming
> >>>> at
> >>>>>>>>>> supporting managing Pulsar clusters running in different
> >>>>>> environments
> >>>>>>>>>> (on-premise data centers, cloud, and so on). We’d like to
> >>>>>> contribute
> >>>>>>>> the
> >>>>>>>>>> project back to the Pulsar community.
> >>>>>>>>>>
> >>>>>>>>>> ---
> >>>>>>>>>>
> >>>>>>>>>> ## Motivation
> >>>>>>>>>>
> >>>>>>>>>> Currently, Pulsar has a monitoring tool - [Pulsar
> >>> Dashboard]( [
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> >>>>>>>>>> ).
> >>>>>>>>>> However, it only focuses on the simple monitoring of
> >> Pulsar -
> >>>>>>>> collecting
> >>>>>>>>>> and displaying information such as statistics of tenants,
> >>>>>> namespaces,
> >>>>>>>>>> topics, subscriptions, and so on. It doesn’t provide any
> >>>>> management
> >>>>>>>>>> operations such as add, delete and update tenants,
> >>> namespaces,
> >>>>>>> topics,
> >>>>>>>>> and
> >>>>>>>>>> so on. When the scale of Pulsar cluster increases or the
> >>> number
> >>>>> of
> >>>>>>>>> clusters
> >>>>>>>>>> grows, using `pulsar-admin` to manage Pulsar can not
> >> satisfy
> >>>>>> demands.
> >>>>>>>>>> Therefore, Pulsar needs a simple and easy-to-use management
> >>>>> console
> >>>>>>> for
> >>>>>>>>>> administrators.
> >>>>>>>>>>
> >>>>>>>>>> Pulsar Manager is a web-based GUI management and monitoring
> >>>> tool
> >>>>>> that
> >>>>>>>>> helps
> >>>>>>>>>> administrators and users manage and monitor tenants,
> >>>> namespaces,
> >>>>>>>> topics,
> >>>>>>>>>> subscriptions, brokers, clusters, and so on, and supports
> >>>> dynamic
> >>>>>>>>>> configuration of multiple environments.
> >>>>>>>>>>
> >>>>>>>>>> ## Features
> >>>>>>>>>>
> >>>>>>>>>> Pulsar manager provides two main features: management and
> >>>>>> monitoring.
> >>>>>>>>>>
> >>>>>>>>>> ### Management
> >>>>>>>>>>
> >>>>>>>>>> 1. Environment : (operations on the environment)
> >>>>>>>>>> Create / Delete / Update / List / Get
> >>>>>>>>>> 2. Cluster : (operations on clusters)
> >>>>>>>>>> Create / Delete / Update / List / Get
> >>>>>>>>>> 3. Brokers : (operations on brokers)
> >>>>>>>>>> Heartbeat / Unload
> >>>>>>>>>> 4. Tenants:
> >>>>>>>>>> Create / Delete / Update / List
> >>>>>>>>>> 5. Namespaces:
> >>>>>>>>>> Create / Delete / List
> >>>>>>>>>> Manage Namespace Policy
> >>>>>>>>>> Unload
> >>>>>>>>>> Operations on Namespace Bundles
> >>>>>>>>>> 6. Topics
> >>>>>>>>>> Create / Delete / List
> >>>>>>>>>> Unload / Terminate / Offload / Compact
> >>>>>>>>>> Topic Details
> >>>>>>>>>> 7. Subscriptions
> >>>>>>>>>> Create / Delete / List
> >>>>>>>>>> Reset Cursor / Skip / Clear backlog / Unsubscribe
> >>>>>>>>>> 8. Namespace Isolation Policies
> >>>>>>>>>> Create / Delete / Update / Get / List
> >>>>>>>>>> 9. Failure Domains
> >>>>>>>>>> Create / Delete / Update / Get / List
> >>>>>>>>>>
> >>>>>>>>>> ### Monitoring: (display *aggregated* stats at different
> >>>> levels)
> >>>>>>>>>>
> >>>>>>>>>> 1. Tenants
> >>>>>>>>>> List the total number of namespaces per tenant (both
> >>> aggregated
> >>>>> and
> >>>>>>>>>> per-cluster basis)
> >>>>>>>>>>
> >>>>>>>>>> 2. Namespaces
> >>>>>>>>>> List the total number of topics per namespace
> >>>>>>>>>> List the aggregated stats per namespace (such as rate-in,
> >>>>> rate-out,
> >>>>>>>>>> throughput-in, and throughput-out)
> >>>>>>>>>> List the distributions of namespace bundles
> >>>>>>>>>>
> >>>>>>>>>> 3. Topics
> >>>>>>>>>> List the total number of partitions per topic
> >>>>>>>>>> List the aggregated stats per topic (such as rate-in,
> >>> rate-out,
> >>>>>>>>>> throughput-in, and throughput-out)
> >>>>>>>>>> Detailed stats per topic partition
> >>>>>>>>>> Detailed stats of storage per topic partition
> >>>>>>>>>> Detailed stats of subscription per topic and per partition
> >>>>>>>>>> Detailed stats of producers
> >>>>>>>>>> Detailed stats of consumers
> >>>>>>>>>>
> >>>>>>>>>> The detailed design proposal is in
> >>>>>>>>>> [
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> >>>>>>>>>>
> >>>>>>>>>> Looking forward to any feedback.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Guangning
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> *Thanks*
> >>
> >> *Yuvaraj L*
> >>
>
>

Re: PIP-40: Contribute Pulsar Manager

[Dave Fisher <wa...@apache.org>]

BTW - Once the VOTE is complete to accept the codebase there is some paperwork.

http://incubator.apache.org/ip-clearance/

While this is the Incubator the Board has requested that the IPMC track IP Clearance. Everything is in SVN and the pages are updated every evening PDT. If help is needed then let me know.

Regards,
Dave

> On Sep 4, 2019, at 9:01 AM, Enrico Olivelli <eo...@gmail.com> wrote:
> 
> For whom who is interested I am working on adding support for HerdDB in
> Pulsar Manager.
> 
> The pull request is still very raw, but Pulsar Manager seems to work
> https://github.com/streamnative/pulsar-manager/pull/183
> 
> 
> 
> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <uv...@gmail.com> ha
> scritto:
> 
>> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <gu...@gmail.com> wrote:
>> 
>>>> I see there are dependencies on websockets, I would a prefer a simple
>>> polling model of the http, also it's enable by default in pulsar.
>>> 
>>> I don't think we depend don pulsar websocket. All are http restful
>>> requests.
>>> 
>>>> For the ui the default persistence should be sqlite. Potentially
>> packaged
>>> by default.
>>> 
>>> Initially the default is sqlite. But sqlite doesn't work if there are a
>> lot
>>> of topic metrics.
>>> We switched to MySQL as default for supporting production traffic. We can
>>> switch default to PostgresSQL (as pulsar dashboard).
>>> 
>> +1
>> Yes, Switching to Postgres Would be great. Because Postgres license is more
>> liberal and works well with Apache license ecosystem.
>> 
>>> 
>>> - Sijie
>>> 
>>> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <al...@gmail.com> wrote:
>>> 
>>>> I see there are dependencies on websockets, I would a prefer a simple
>>>> polling model of the http, also it's enable by default in pulsar.
>>>> 
>>>> For the ui the default persistence should be sqlite. Potentially
>> packaged
>>>> by default.
>>>> 
>>>> -Ali
>>>> 
>>>> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <eo...@gmail.com>
>>>> wrote:
>>>> 
>>>>> Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha scritto:
>>>>> 
>>>>>> That sounds an interesting idea!
>>>>> 
>>>>> 
>>>>> Awesome
>>>>> 
>>>>> Does HerdDB support JDBC? If so, it should
>>>>>> be pretty straightforward to enable HerdDB.
>>>>>> 
>>>>> 
>>>>> Sure, as far as I know, the JDBC client is the only client really
>> used
>>> in
>>>>> production.
>>>>> 
>>>>> In a replicated environment it uses Zookeeper for metadata and
>> service
>>>>> discovery and Bookkeeper as WAL and if you have a pulsar cluster you
>>>>> already have both of them (zk cluster an bookies)
>>>>> 
>>>>> I will take a look to how Pulsar Manager   uses JDBC, maybe it will
>> be
>>>> very
>>>>> easy.
>>>>> 
>>>>> I will be back with news
>>>>> 
>>>>> 
>>>>> Enrico
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> Thanks,
>>>>>> Sijie
>>>>>> 
>>>>>> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <
>>> eolivelli@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Maybe you can try to use HerdDB (1), it is a replicated SQL
>>> database
>>>>> that
>>>>>>> can be run embedded in the JVM.
>>>>>>> It is an SQL database that we developed in my company, in order
>> to
>>>>>>> explicitly replace MySQL.
>>>>>>> It uses Bookkeeper to store the WAL and implement replication,
>> but
>>> it
>>>>> can
>>>>>>> run in standalone mode or in memory (for tests/dev).
>>>>>>> 
>>>>>>> I can help in setting up a demo.
>>>>>>> 
>>>>>>> Enrico
>>>>>>> 
>>>>>>> (1) https://github.com/diennea/herddb
>>>>>>> 
>>>>>>> Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha
>>> scritto:
>>>>>>> 
>>>>>>>> Hi Dave,
>>>>>>>> 
>>>>>>>> It doesn't require MySQL. You can use any JDBC supported
>>> databases.
>>>>>>>> Initially we would use SQLite for development and change to
>> MySQL
>>>> for
>>>>>>> more
>>>>>>>> production-ready workloads.
>>>>>>>> 
>>>>>>>> Hope this clarify your question.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Sijie
>>>>>>>> 
>>>>>>>> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org>
>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi -
>>>>>>>>> 
>>>>>>>>> I see that MySQL is used. This would be a GPL dependency.
>> Does
>>>> the
>>>>>>> Pulsar
>>>>>>>>> Manager require MySQL, or can other Databases that are
>>> compliant
>>>>> with
>>>>>>>>> Apache Release Policy be used instead?
>>>>>>>>> 
>>>>>>>>> Regards,
>>>>>>>>> Dave
>>>>>>>>> 
>>>>>>>>>> On Aug 29, 2019, at 7:49 AM, Guangning E <
>>> eguangning@gmail.com
>>>>> 
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi all,
>>>>>>>>>> 
>>>>>>>>>> We have developed a new Pulsar web UI - pulsar-manager,
>>> aiming
>>>> at
>>>>>>>>>> supporting managing Pulsar clusters running in different
>>>>>> environments
>>>>>>>>>> (on-premise data centers, cloud, and so on). We’d like to
>>>>>> contribute
>>>>>>>> the
>>>>>>>>>> project back to the Pulsar community.
>>>>>>>>>> 
>>>>>>>>>> ---
>>>>>>>>>> 
>>>>>>>>>> ## Motivation
>>>>>>>>>> 
>>>>>>>>>> Currently, Pulsar has a monitoring tool - [Pulsar
>>> Dashboard]( [
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
>>>>>>>>>> ).
>>>>>>>>>> However, it only focuses on the simple monitoring of
>> Pulsar -
>>>>>>>> collecting
>>>>>>>>>> and displaying information such as statistics of tenants,
>>>>>> namespaces,
>>>>>>>>>> topics, subscriptions, and so on. It doesn’t provide any
>>>>> management
>>>>>>>>>> operations such as add, delete and update tenants,
>>> namespaces,
>>>>>>> topics,
>>>>>>>>> and
>>>>>>>>>> so on. When the scale of Pulsar cluster increases or the
>>> number
>>>>> of
>>>>>>>>> clusters
>>>>>>>>>> grows, using `pulsar-admin` to manage Pulsar can not
>> satisfy
>>>>>> demands.
>>>>>>>>>> Therefore, Pulsar needs a simple and easy-to-use management
>>>>> console
>>>>>>> for
>>>>>>>>>> administrators.
>>>>>>>>>> 
>>>>>>>>>> Pulsar Manager is a web-based GUI management and monitoring
>>>> tool
>>>>>> that
>>>>>>>>> helps
>>>>>>>>>> administrators and users manage and monitor tenants,
>>>> namespaces,
>>>>>>>> topics,
>>>>>>>>>> subscriptions, brokers, clusters, and so on, and supports
>>>> dynamic
>>>>>>>>>> configuration of multiple environments.
>>>>>>>>>> 
>>>>>>>>>> ## Features
>>>>>>>>>> 
>>>>>>>>>> Pulsar manager provides two main features: management and
>>>>>> monitoring.
>>>>>>>>>> 
>>>>>>>>>> ### Management
>>>>>>>>>> 
>>>>>>>>>> 1. Environment : (operations on the environment)
>>>>>>>>>> Create / Delete / Update / List / Get
>>>>>>>>>> 2. Cluster : (operations on clusters)
>>>>>>>>>> Create / Delete / Update / List / Get
>>>>>>>>>> 3. Brokers : (operations on brokers)
>>>>>>>>>> Heartbeat / Unload
>>>>>>>>>> 4. Tenants:
>>>>>>>>>> Create / Delete / Update / List
>>>>>>>>>> 5. Namespaces:
>>>>>>>>>> Create / Delete / List
>>>>>>>>>> Manage Namespace Policy
>>>>>>>>>> Unload
>>>>>>>>>> Operations on Namespace Bundles
>>>>>>>>>> 6. Topics
>>>>>>>>>> Create / Delete / List
>>>>>>>>>> Unload / Terminate / Offload / Compact
>>>>>>>>>> Topic Details
>>>>>>>>>> 7. Subscriptions
>>>>>>>>>> Create / Delete / List
>>>>>>>>>> Reset Cursor / Skip / Clear backlog / Unsubscribe
>>>>>>>>>> 8. Namespace Isolation Policies
>>>>>>>>>> Create / Delete / Update / Get / List
>>>>>>>>>> 9. Failure Domains
>>>>>>>>>> Create / Delete / Update / Get / List
>>>>>>>>>> 
>>>>>>>>>> ### Monitoring: (display *aggregated* stats at different
>>>> levels)
>>>>>>>>>> 
>>>>>>>>>> 1. Tenants
>>>>>>>>>> List the total number of namespaces per tenant (both
>>> aggregated
>>>>> and
>>>>>>>>>> per-cluster basis)
>>>>>>>>>> 
>>>>>>>>>> 2. Namespaces
>>>>>>>>>> List the total number of topics per namespace
>>>>>>>>>> List the aggregated stats per namespace (such as rate-in,
>>>>> rate-out,
>>>>>>>>>> throughput-in, and throughput-out)
>>>>>>>>>> List the distributions of namespace bundles
>>>>>>>>>> 
>>>>>>>>>> 3. Topics
>>>>>>>>>> List the total number of partitions per topic
>>>>>>>>>> List the aggregated stats per topic (such as rate-in,
>>> rate-out,
>>>>>>>>>> throughput-in, and throughput-out)
>>>>>>>>>> Detailed stats per topic partition
>>>>>>>>>> Detailed stats of storage per topic partition
>>>>>>>>>> Detailed stats of subscription per topic and per partition
>>>>>>>>>> Detailed stats of producers
>>>>>>>>>> Detailed stats of consumers
>>>>>>>>>> 
>>>>>>>>>> The detailed design proposal is in
>>>>>>>>>> [
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
>>>>>>>>>> 
>>>>>>>>>> Looking forward to any feedback.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> Guangning
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
>> --
>> *Thanks*
>> 
>> *Yuvaraj L*
>> 

Re: PIP-40: Contribute Pulsar Manager

[Sijie Guo <gu...@gmail.com>]

That looks good, Enrico. Thank you for driving the work.

Sijie

On Wed, Sep 4, 2019 at 9:02 AM Enrico Olivelli <eo...@gmail.com> wrote:

> For whom who is interested I am working on adding support for HerdDB in
> Pulsar Manager.
>
> The pull request is still very raw, but Pulsar Manager seems to work
> https://github.com/streamnative/pulsar-manager/pull/183
>
>
>
> Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <uv...@gmail.com> ha
> scritto:
>
> > On Fri, 30 Aug 2019 at 07:09, Sijie Guo <gu...@gmail.com> wrote:
> >
> > > > I see there are dependencies on websockets, I would a prefer a simple
> > > polling model of the http, also it's enable by default in pulsar.
> > >
> > > I don't think we depend don pulsar websocket. All are http restful
> > > requests.
> > >
> > > > For the ui the default persistence should be sqlite. Potentially
> > packaged
> > > by default.
> > >
> > > Initially the default is sqlite. But sqlite doesn't work if there are a
> > lot
> > > of topic metrics.
> > > We switched to MySQL as default for supporting production traffic. We
> can
> > > switch default to PostgresSQL (as pulsar dashboard).
> > >
> > +1
> > Yes, Switching to Postgres Would be great. Because Postgres license is
> more
> > liberal and works well with Apache license ecosystem.
> >
> > >
> > > - Sijie
> > >
> > > On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <al...@gmail.com>
> wrote:
> > >
> > > > I see there are dependencies on websockets, I would a prefer a simple
> > > > polling model of the http, also it's enable by default in pulsar.
> > > >
> > > > For the ui the default persistence should be sqlite. Potentially
> > packaged
> > > > by default.
> > > >
> > > > -Ali
> > > >
> > > > On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <eolivelli@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha
> scritto:
> > > > >
> > > > > > That sounds an interesting idea!
> > > > >
> > > > >
> > > > > Awesome
> > > > >
> > > > > Does HerdDB support JDBC? If so, it should
> > > > > > be pretty straightforward to enable HerdDB.
> > > > > >
> > > > >
> > > > > Sure, as far as I know, the JDBC client is the only client really
> > used
> > > in
> > > > > production.
> > > > >
> > > > > In a replicated environment it uses Zookeeper for metadata and
> > service
> > > > > discovery and Bookkeeper as WAL and if you have a pulsar cluster
> you
> > > > > already have both of them (zk cluster an bookies)
> > > > >
> > > > > I will take a look to how Pulsar Manager   uses JDBC, maybe it will
> > be
> > > > very
> > > > > easy.
> > > > >
> > > > > I will be back with news
> > > > >
> > > > >
> > > > > Enrico
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > Thanks,
> > > > > > Sijie
> > > > > >
> > > > > > On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <
> > > eolivelli@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Maybe you can try to use HerdDB (1), it is a replicated SQL
> > > database
> > > > > that
> > > > > > > can be run embedded in the JVM.
> > > > > > > It is an SQL database that we developed in my company, in order
> > to
> > > > > > > explicitly replace MySQL.
> > > > > > > It uses Bookkeeper to store the WAL and implement replication,
> > but
> > > it
> > > > > can
> > > > > > > run in standalone mode or in memory (for tests/dev).
> > > > > > >
> > > > > > > I can help in setting up a demo.
> > > > > > >
> > > > > > > Enrico
> > > > > > >
> > > > > > > (1) https://github.com/diennea/herddb
> > > > > > >
> > > > > > > Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha
> > > scritto:
> > > > > > >
> > > > > > > > Hi Dave,
> > > > > > > >
> > > > > > > > It doesn't require MySQL. You can use any JDBC supported
> > > databases.
> > > > > > > > Initially we would use SQLite for development and change to
> > MySQL
> > > > for
> > > > > > > more
> > > > > > > > production-ready workloads.
> > > > > > > >
> > > > > > > > Hope this clarify your question.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Sijie
> > > > > > > >
> > > > > > > > On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wave@apache.org
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi -
> > > > > > > > >
> > > > > > > > > I see that MySQL is used. This would be a GPL dependency.
> > Does
> > > > the
> > > > > > > Pulsar
> > > > > > > > > Manager require MySQL, or can other Databases that are
> > > compliant
> > > > > with
> > > > > > > > > Apache Release Policy be used instead?
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Dave
> > > > > > > > >
> > > > > > > > > > On Aug 29, 2019, at 7:49 AM, Guangning E <
> > > eguangning@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > We have developed a new Pulsar web UI - pulsar-manager,
> > > aiming
> > > > at
> > > > > > > > > > supporting managing Pulsar clusters running in different
> > > > > > environments
> > > > > > > > > > (on-premise data centers, cloud, and so on). We’d like to
> > > > > > contribute
> > > > > > > > the
> > > > > > > > > > project back to the Pulsar community.
> > > > > > > > > >
> > > > > > > > > > ---
> > > > > > > > > >
> > > > > > > > > > ## Motivation
> > > > > > > > > >
> > > > > > > > > > Currently, Pulsar has a monitoring tool - [Pulsar
> > > Dashboard]( [
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> > > > > > > > > > ).
> > > > > > > > > > However, it only focuses on the simple monitoring of
> > Pulsar -
> > > > > > > > collecting
> > > > > > > > > > and displaying information such as statistics of tenants,
> > > > > > namespaces,
> > > > > > > > > > topics, subscriptions, and so on. It doesn’t provide any
> > > > > management
> > > > > > > > > > operations such as add, delete and update tenants,
> > > namespaces,
> > > > > > > topics,
> > > > > > > > > and
> > > > > > > > > > so on. When the scale of Pulsar cluster increases or the
> > > number
> > > > > of
> > > > > > > > > clusters
> > > > > > > > > > grows, using `pulsar-admin` to manage Pulsar can not
> > satisfy
> > > > > > demands.
> > > > > > > > > > Therefore, Pulsar needs a simple and easy-to-use
> management
> > > > > console
> > > > > > > for
> > > > > > > > > > administrators.
> > > > > > > > > >
> > > > > > > > > > Pulsar Manager is a web-based GUI management and
> monitoring
> > > > tool
> > > > > > that
> > > > > > > > > helps
> > > > > > > > > > administrators and users manage and monitor tenants,
> > > > namespaces,
> > > > > > > > topics,
> > > > > > > > > > subscriptions, brokers, clusters, and so on, and supports
> > > > dynamic
> > > > > > > > > > configuration of multiple environments.
> > > > > > > > > >
> > > > > > > > > > ## Features
> > > > > > > > > >
> > > > > > > > > > Pulsar manager provides two main features: management and
> > > > > > monitoring.
> > > > > > > > > >
> > > > > > > > > > ### Management
> > > > > > > > > >
> > > > > > > > > > 1. Environment : (operations on the environment)
> > > > > > > > > > Create / Delete / Update / List / Get
> > > > > > > > > > 2. Cluster : (operations on clusters)
> > > > > > > > > > Create / Delete / Update / List / Get
> > > > > > > > > > 3. Brokers : (operations on brokers)
> > > > > > > > > > Heartbeat / Unload
> > > > > > > > > > 4. Tenants:
> > > > > > > > > > Create / Delete / Update / List
> > > > > > > > > > 5. Namespaces:
> > > > > > > > > > Create / Delete / List
> > > > > > > > > > Manage Namespace Policy
> > > > > > > > > > Unload
> > > > > > > > > > Operations on Namespace Bundles
> > > > > > > > > > 6. Topics
> > > > > > > > > > Create / Delete / List
> > > > > > > > > > Unload / Terminate / Offload / Compact
> > > > > > > > > > Topic Details
> > > > > > > > > > 7. Subscriptions
> > > > > > > > > > Create / Delete / List
> > > > > > > > > > Reset Cursor / Skip / Clear backlog / Unsubscribe
> > > > > > > > > > 8. Namespace Isolation Policies
> > > > > > > > > > Create / Delete / Update / Get / List
> > > > > > > > > > 9. Failure Domains
> > > > > > > > > > Create / Delete / Update / Get / List
> > > > > > > > > >
> > > > > > > > > > ### Monitoring: (display *aggregated* stats at different
> > > > levels)
> > > > > > > > > >
> > > > > > > > > > 1. Tenants
> > > > > > > > > > List the total number of namespaces per tenant (both
> > > aggregated
> > > > > and
> > > > > > > > > > per-cluster basis)
> > > > > > > > > >
> > > > > > > > > > 2. Namespaces
> > > > > > > > > > List the total number of topics per namespace
> > > > > > > > > > List the aggregated stats per namespace (such as rate-in,
> > > > > rate-out,
> > > > > > > > > > throughput-in, and throughput-out)
> > > > > > > > > > List the distributions of namespace bundles
> > > > > > > > > >
> > > > > > > > > > 3. Topics
> > > > > > > > > > List the total number of partitions per topic
> > > > > > > > > > List the aggregated stats per topic (such as rate-in,
> > > rate-out,
> > > > > > > > > > throughput-in, and throughput-out)
> > > > > > > > > > Detailed stats per topic partition
> > > > > > > > > > Detailed stats of storage per topic partition
> > > > > > > > > > Detailed stats of subscription per topic and per
> partition
> > > > > > > > > > Detailed stats of producers
> > > > > > > > > > Detailed stats of consumers
> > > > > > > > > >
> > > > > > > > > > The detailed design proposal is in
> > > > > > > > > > [
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> > > > > > > > > >
> > > > > > > > > > Looking forward to any feedback.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Guangning
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > *Thanks*
> >
> > *Yuvaraj L*
> >
>

Re: PIP-40: Contribute Pulsar Manager

[Enrico Olivelli <eo...@gmail.com>]

For whom who is interested I am working on adding support for HerdDB in
Pulsar Manager.

The pull request is still very raw, but Pulsar Manager seems to work
https://github.com/streamnative/pulsar-manager/pull/183



Il giorno mar 3 set 2019 alle ore 08:37 Yuva raj <uv...@gmail.com> ha
scritto:

> On Fri, 30 Aug 2019 at 07:09, Sijie Guo <gu...@gmail.com> wrote:
>
> > > I see there are dependencies on websockets, I would a prefer a simple
> > polling model of the http, also it's enable by default in pulsar.
> >
> > I don't think we depend don pulsar websocket. All are http restful
> > requests.
> >
> > > For the ui the default persistence should be sqlite. Potentially
> packaged
> > by default.
> >
> > Initially the default is sqlite. But sqlite doesn't work if there are a
> lot
> > of topic metrics.
> > We switched to MySQL as default for supporting production traffic. We can
> > switch default to PostgresSQL (as pulsar dashboard).
> >
> +1
> Yes, Switching to Postgres Would be great. Because Postgres license is more
> liberal and works well with Apache license ecosystem.
>
> >
> > - Sijie
> >
> > On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <al...@gmail.com> wrote:
> >
> > > I see there are dependencies on websockets, I would a prefer a simple
> > > polling model of the http, also it's enable by default in pulsar.
> > >
> > > For the ui the default persistence should be sqlite. Potentially
> packaged
> > > by default.
> > >
> > > -Ali
> > >
> > > On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <eo...@gmail.com>
> > > wrote:
> > >
> > > > Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha scritto:
> > > >
> > > > > That sounds an interesting idea!
> > > >
> > > >
> > > > Awesome
> > > >
> > > > Does HerdDB support JDBC? If so, it should
> > > > > be pretty straightforward to enable HerdDB.
> > > > >
> > > >
> > > > Sure, as far as I know, the JDBC client is the only client really
> used
> > in
> > > > production.
> > > >
> > > > In a replicated environment it uses Zookeeper for metadata and
> service
> > > > discovery and Bookkeeper as WAL and if you have a pulsar cluster you
> > > > already have both of them (zk cluster an bookies)
> > > >
> > > > I will take a look to how Pulsar Manager   uses JDBC, maybe it will
> be
> > > very
> > > > easy.
> > > >
> > > > I will be back with news
> > > >
> > > >
> > > > Enrico
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > > Thanks,
> > > > > Sijie
> > > > >
> > > > > On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <
> > eolivelli@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Maybe you can try to use HerdDB (1), it is a replicated SQL
> > database
> > > > that
> > > > > > can be run embedded in the JVM.
> > > > > > It is an SQL database that we developed in my company, in order
> to
> > > > > > explicitly replace MySQL.
> > > > > > It uses Bookkeeper to store the WAL and implement replication,
> but
> > it
> > > > can
> > > > > > run in standalone mode or in memory (for tests/dev).
> > > > > >
> > > > > > I can help in setting up a demo.
> > > > > >
> > > > > > Enrico
> > > > > >
> > > > > > (1) https://github.com/diennea/herddb
> > > > > >
> > > > > > Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha
> > scritto:
> > > > > >
> > > > > > > Hi Dave,
> > > > > > >
> > > > > > > It doesn't require MySQL. You can use any JDBC supported
> > databases.
> > > > > > > Initially we would use SQLite for development and change to
> MySQL
> > > for
> > > > > > more
> > > > > > > production-ready workloads.
> > > > > > >
> > > > > > > Hope this clarify your question.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Sijie
> > > > > > >
> > > > > > > On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org>
> > > wrote:
> > > > > > >
> > > > > > > > Hi -
> > > > > > > >
> > > > > > > > I see that MySQL is used. This would be a GPL dependency.
> Does
> > > the
> > > > > > Pulsar
> > > > > > > > Manager require MySQL, or can other Databases that are
> > compliant
> > > > with
> > > > > > > > Apache Release Policy be used instead?
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Dave
> > > > > > > >
> > > > > > > > > On Aug 29, 2019, at 7:49 AM, Guangning E <
> > eguangning@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > We have developed a new Pulsar web UI - pulsar-manager,
> > aiming
> > > at
> > > > > > > > > supporting managing Pulsar clusters running in different
> > > > > environments
> > > > > > > > > (on-premise data centers, cloud, and so on). We’d like to
> > > > > contribute
> > > > > > > the
> > > > > > > > > project back to the Pulsar community.
> > > > > > > > >
> > > > > > > > > ---
> > > > > > > > >
> > > > > > > > > ## Motivation
> > > > > > > > >
> > > > > > > > > Currently, Pulsar has a monitoring tool - [Pulsar
> > Dashboard]( [
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> > > > > > > > > ).
> > > > > > > > > However, it only focuses on the simple monitoring of
> Pulsar -
> > > > > > > collecting
> > > > > > > > > and displaying information such as statistics of tenants,
> > > > > namespaces,
> > > > > > > > > topics, subscriptions, and so on. It doesn’t provide any
> > > > management
> > > > > > > > > operations such as add, delete and update tenants,
> > namespaces,
> > > > > > topics,
> > > > > > > > and
> > > > > > > > > so on. When the scale of Pulsar cluster increases or the
> > number
> > > > of
> > > > > > > > clusters
> > > > > > > > > grows, using `pulsar-admin` to manage Pulsar can not
> satisfy
> > > > > demands.
> > > > > > > > > Therefore, Pulsar needs a simple and easy-to-use management
> > > > console
> > > > > > for
> > > > > > > > > administrators.
> > > > > > > > >
> > > > > > > > > Pulsar Manager is a web-based GUI management and monitoring
> > > tool
> > > > > that
> > > > > > > > helps
> > > > > > > > > administrators and users manage and monitor tenants,
> > > namespaces,
> > > > > > > topics,
> > > > > > > > > subscriptions, brokers, clusters, and so on, and supports
> > > dynamic
> > > > > > > > > configuration of multiple environments.
> > > > > > > > >
> > > > > > > > > ## Features
> > > > > > > > >
> > > > > > > > > Pulsar manager provides two main features: management and
> > > > > monitoring.
> > > > > > > > >
> > > > > > > > > ### Management
> > > > > > > > >
> > > > > > > > > 1. Environment : (operations on the environment)
> > > > > > > > > Create / Delete / Update / List / Get
> > > > > > > > > 2. Cluster : (operations on clusters)
> > > > > > > > > Create / Delete / Update / List / Get
> > > > > > > > > 3. Brokers : (operations on brokers)
> > > > > > > > > Heartbeat / Unload
> > > > > > > > > 4. Tenants:
> > > > > > > > > Create / Delete / Update / List
> > > > > > > > > 5. Namespaces:
> > > > > > > > > Create / Delete / List
> > > > > > > > > Manage Namespace Policy
> > > > > > > > > Unload
> > > > > > > > > Operations on Namespace Bundles
> > > > > > > > > 6. Topics
> > > > > > > > > Create / Delete / List
> > > > > > > > > Unload / Terminate / Offload / Compact
> > > > > > > > > Topic Details
> > > > > > > > > 7. Subscriptions
> > > > > > > > > Create / Delete / List
> > > > > > > > > Reset Cursor / Skip / Clear backlog / Unsubscribe
> > > > > > > > > 8. Namespace Isolation Policies
> > > > > > > > > Create / Delete / Update / Get / List
> > > > > > > > > 9. Failure Domains
> > > > > > > > > Create / Delete / Update / Get / List
> > > > > > > > >
> > > > > > > > > ### Monitoring: (display *aggregated* stats at different
> > > levels)
> > > > > > > > >
> > > > > > > > > 1. Tenants
> > > > > > > > > List the total number of namespaces per tenant (both
> > aggregated
> > > > and
> > > > > > > > > per-cluster basis)
> > > > > > > > >
> > > > > > > > > 2. Namespaces
> > > > > > > > > List the total number of topics per namespace
> > > > > > > > > List the aggregated stats per namespace (such as rate-in,
> > > > rate-out,
> > > > > > > > > throughput-in, and throughput-out)
> > > > > > > > > List the distributions of namespace bundles
> > > > > > > > >
> > > > > > > > > 3. Topics
> > > > > > > > > List the total number of partitions per topic
> > > > > > > > > List the aggregated stats per topic (such as rate-in,
> > rate-out,
> > > > > > > > > throughput-in, and throughput-out)
> > > > > > > > > Detailed stats per topic partition
> > > > > > > > > Detailed stats of storage per topic partition
> > > > > > > > > Detailed stats of subscription per topic and per partition
> > > > > > > > > Detailed stats of producers
> > > > > > > > > Detailed stats of consumers
> > > > > > > > >
> > > > > > > > > The detailed design proposal is in
> > > > > > > > > [
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> > > > > > > > >
> > > > > > > > > Looking forward to any feedback.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Guangning
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> *Thanks*
>
> *Yuvaraj L*
>

Re: PIP-40: Contribute Pulsar Manager

[Yuva raj <uv...@gmail.com>]

On Fri, 30 Aug 2019 at 07:09, Sijie Guo <gu...@gmail.com> wrote:

> > I see there are dependencies on websockets, I would a prefer a simple
> polling model of the http, also it's enable by default in pulsar.
>
> I don't think we depend don pulsar websocket. All are http restful
> requests.
>
> > For the ui the default persistence should be sqlite. Potentially packaged
> by default.
>
> Initially the default is sqlite. But sqlite doesn't work if there are a lot
> of topic metrics.
> We switched to MySQL as default for supporting production traffic. We can
> switch default to PostgresSQL (as pulsar dashboard).
>
+1
Yes, Switching to Postgres Would be great. Because Postgres license is more
liberal and works well with Apache license ecosystem.

>
> - Sijie
>
> On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <al...@gmail.com> wrote:
>
> > I see there are dependencies on websockets, I would a prefer a simple
> > polling model of the http, also it's enable by default in pulsar.
> >
> > For the ui the default persistence should be sqlite. Potentially packaged
> > by default.
> >
> > -Ali
> >
> > On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha scritto:
> > >
> > > > That sounds an interesting idea!
> > >
> > >
> > > Awesome
> > >
> > > Does HerdDB support JDBC? If so, it should
> > > > be pretty straightforward to enable HerdDB.
> > > >
> > >
> > > Sure, as far as I know, the JDBC client is the only client really used
> in
> > > production.
> > >
> > > In a replicated environment it uses Zookeeper for metadata and service
> > > discovery and Bookkeeper as WAL and if you have a pulsar cluster you
> > > already have both of them (zk cluster an bookies)
> > >
> > > I will take a look to how Pulsar Manager   uses JDBC, maybe it will be
> > very
> > > easy.
> > >
> > > I will be back with news
> > >
> > >
> > > Enrico
> > >
> > >
> > >
> > >
> > >
> > > > Thanks,
> > > > Sijie
> > > >
> > > > On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <
> eolivelli@gmail.com>
> > > > wrote:
> > > >
> > > > > Maybe you can try to use HerdDB (1), it is a replicated SQL
> database
> > > that
> > > > > can be run embedded in the JVM.
> > > > > It is an SQL database that we developed in my company, in order to
> > > > > explicitly replace MySQL.
> > > > > It uses Bookkeeper to store the WAL and implement replication, but
> it
> > > can
> > > > > run in standalone mode or in memory (for tests/dev).
> > > > >
> > > > > I can help in setting up a demo.
> > > > >
> > > > > Enrico
> > > > >
> > > > > (1) https://github.com/diennea/herddb
> > > > >
> > > > > Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha
> scritto:
> > > > >
> > > > > > Hi Dave,
> > > > > >
> > > > > > It doesn't require MySQL. You can use any JDBC supported
> databases.
> > > > > > Initially we would use SQLite for development and change to MySQL
> > for
> > > > > more
> > > > > > production-ready workloads.
> > > > > >
> > > > > > Hope this clarify your question.
> > > > > >
> > > > > > Thanks,
> > > > > > Sijie
> > > > > >
> > > > > > On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org>
> > wrote:
> > > > > >
> > > > > > > Hi -
> > > > > > >
> > > > > > > I see that MySQL is used. This would be a GPL dependency. Does
> > the
> > > > > Pulsar
> > > > > > > Manager require MySQL, or can other Databases that are
> compliant
> > > with
> > > > > > > Apache Release Policy be used instead?
> > > > > > >
> > > > > > > Regards,
> > > > > > > Dave
> > > > > > >
> > > > > > > > On Aug 29, 2019, at 7:49 AM, Guangning E <
> eguangning@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > We have developed a new Pulsar web UI - pulsar-manager,
> aiming
> > at
> > > > > > > > supporting managing Pulsar clusters running in different
> > > > environments
> > > > > > > > (on-premise data centers, cloud, and so on). We’d like to
> > > > contribute
> > > > > > the
> > > > > > > > project back to the Pulsar community.
> > > > > > > >
> > > > > > > > ---
> > > > > > > >
> > > > > > > > ## Motivation
> > > > > > > >
> > > > > > > > Currently, Pulsar has a monitoring tool - [Pulsar
> Dashboard]( [
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> > > > > > > > ).
> > > > > > > > However, it only focuses on the simple monitoring of Pulsar -
> > > > > > collecting
> > > > > > > > and displaying information such as statistics of tenants,
> > > > namespaces,
> > > > > > > > topics, subscriptions, and so on. It doesn’t provide any
> > > management
> > > > > > > > operations such as add, delete and update tenants,
> namespaces,
> > > > > topics,
> > > > > > > and
> > > > > > > > so on. When the scale of Pulsar cluster increases or the
> number
> > > of
> > > > > > > clusters
> > > > > > > > grows, using `pulsar-admin` to manage Pulsar can not satisfy
> > > > demands.
> > > > > > > > Therefore, Pulsar needs a simple and easy-to-use management
> > > console
> > > > > for
> > > > > > > > administrators.
> > > > > > > >
> > > > > > > > Pulsar Manager is a web-based GUI management and monitoring
> > tool
> > > > that
> > > > > > > helps
> > > > > > > > administrators and users manage and monitor tenants,
> > namespaces,
> > > > > > topics,
> > > > > > > > subscriptions, brokers, clusters, and so on, and supports
> > dynamic
> > > > > > > > configuration of multiple environments.
> > > > > > > >
> > > > > > > > ## Features
> > > > > > > >
> > > > > > > > Pulsar manager provides two main features: management and
> > > > monitoring.
> > > > > > > >
> > > > > > > > ### Management
> > > > > > > >
> > > > > > > > 1. Environment : (operations on the environment)
> > > > > > > > Create / Delete / Update / List / Get
> > > > > > > > 2. Cluster : (operations on clusters)
> > > > > > > > Create / Delete / Update / List / Get
> > > > > > > > 3. Brokers : (operations on brokers)
> > > > > > > > Heartbeat / Unload
> > > > > > > > 4. Tenants:
> > > > > > > > Create / Delete / Update / List
> > > > > > > > 5. Namespaces:
> > > > > > > > Create / Delete / List
> > > > > > > > Manage Namespace Policy
> > > > > > > > Unload
> > > > > > > > Operations on Namespace Bundles
> > > > > > > > 6. Topics
> > > > > > > > Create / Delete / List
> > > > > > > > Unload / Terminate / Offload / Compact
> > > > > > > > Topic Details
> > > > > > > > 7. Subscriptions
> > > > > > > > Create / Delete / List
> > > > > > > > Reset Cursor / Skip / Clear backlog / Unsubscribe
> > > > > > > > 8. Namespace Isolation Policies
> > > > > > > > Create / Delete / Update / Get / List
> > > > > > > > 9. Failure Domains
> > > > > > > > Create / Delete / Update / Get / List
> > > > > > > >
> > > > > > > > ### Monitoring: (display *aggregated* stats at different
> > levels)
> > > > > > > >
> > > > > > > > 1. Tenants
> > > > > > > > List the total number of namespaces per tenant (both
> aggregated
> > > and
> > > > > > > > per-cluster basis)
> > > > > > > >
> > > > > > > > 2. Namespaces
> > > > > > > > List the total number of topics per namespace
> > > > > > > > List the aggregated stats per namespace (such as rate-in,
> > > rate-out,
> > > > > > > > throughput-in, and throughput-out)
> > > > > > > > List the distributions of namespace bundles
> > > > > > > >
> > > > > > > > 3. Topics
> > > > > > > > List the total number of partitions per topic
> > > > > > > > List the aggregated stats per topic (such as rate-in,
> rate-out,
> > > > > > > > throughput-in, and throughput-out)
> > > > > > > > Detailed stats per topic partition
> > > > > > > > Detailed stats of storage per topic partition
> > > > > > > > Detailed stats of subscription per topic and per partition
> > > > > > > > Detailed stats of producers
> > > > > > > > Detailed stats of consumers
> > > > > > > >
> > > > > > > > The detailed design proposal is in
> > > > > > > > [
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> > > > > > > >
> > > > > > > > Looking forward to any feedback.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Guangning
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>


-- 
*Thanks*

*Yuvaraj L*

Re: PIP-40: Contribute Pulsar Manager

[Sijie Guo <gu...@gmail.com>]

> I see there are dependencies on websockets, I would a prefer a simple
polling model of the http, also it's enable by default in pulsar.

I don't think we depend don pulsar websocket. All are http restful requests.

> For the ui the default persistence should be sqlite. Potentially packaged
by default.

Initially the default is sqlite. But sqlite doesn't work if there are a lot
of topic metrics.
We switched to MySQL as default for supporting production traffic. We can
switch default to PostgresSQL (as pulsar dashboard).

- Sijie

On Thu, Aug 29, 2019 at 6:23 PM Ali Ahmed <al...@gmail.com> wrote:

> I see there are dependencies on websockets, I would a prefer a simple
> polling model of the http, also it's enable by default in pulsar.
>
> For the ui the default persistence should be sqlite. Potentially packaged
> by default.
>
> -Ali
>
> On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha scritto:
> >
> > > That sounds an interesting idea!
> >
> >
> > Awesome
> >
> > Does HerdDB support JDBC? If so, it should
> > > be pretty straightforward to enable HerdDB.
> > >
> >
> > Sure, as far as I know, the JDBC client is the only client really used in
> > production.
> >
> > In a replicated environment it uses Zookeeper for metadata and service
> > discovery and Bookkeeper as WAL and if you have a pulsar cluster you
> > already have both of them (zk cluster an bookies)
> >
> > I will take a look to how Pulsar Manager   uses JDBC, maybe it will be
> very
> > easy.
> >
> > I will be back with news
> >
> >
> > Enrico
> >
> >
> >
> >
> >
> > > Thanks,
> > > Sijie
> > >
> > > On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <eo...@gmail.com>
> > > wrote:
> > >
> > > > Maybe you can try to use HerdDB (1), it is a replicated SQL database
> > that
> > > > can be run embedded in the JVM.
> > > > It is an SQL database that we developed in my company, in order to
> > > > explicitly replace MySQL.
> > > > It uses Bookkeeper to store the WAL and implement replication, but it
> > can
> > > > run in standalone mode or in memory (for tests/dev).
> > > >
> > > > I can help in setting up a demo.
> > > >
> > > > Enrico
> > > >
> > > > (1) https://github.com/diennea/herddb
> > > >
> > > > Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha scritto:
> > > >
> > > > > Hi Dave,
> > > > >
> > > > > It doesn't require MySQL. You can use any JDBC supported databases.
> > > > > Initially we would use SQLite for development and change to MySQL
> for
> > > > more
> > > > > production-ready workloads.
> > > > >
> > > > > Hope this clarify your question.
> > > > >
> > > > > Thanks,
> > > > > Sijie
> > > > >
> > > > > On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org>
> wrote:
> > > > >
> > > > > > Hi -
> > > > > >
> > > > > > I see that MySQL is used. This would be a GPL dependency. Does
> the
> > > > Pulsar
> > > > > > Manager require MySQL, or can other Databases that are compliant
> > with
> > > > > > Apache Release Policy be used instead?
> > > > > >
> > > > > > Regards,
> > > > > > Dave
> > > > > >
> > > > > > > On Aug 29, 2019, at 7:49 AM, Guangning E <eguangning@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We have developed a new Pulsar web UI - pulsar-manager, aiming
> at
> > > > > > > supporting managing Pulsar clusters running in different
> > > environments
> > > > > > > (on-premise data centers, cloud, and so on). We’d like to
> > > contribute
> > > > > the
> > > > > > > project back to the Pulsar community.
> > > > > > >
> > > > > > > ---
> > > > > > >
> > > > > > > ## Motivation
> > > > > > >
> > > > > > > Currently, Pulsar has a monitoring tool - [Pulsar Dashboard]( [
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> > > > > > > ).
> > > > > > > However, it only focuses on the simple monitoring of Pulsar -
> > > > > collecting
> > > > > > > and displaying information such as statistics of tenants,
> > > namespaces,
> > > > > > > topics, subscriptions, and so on. It doesn’t provide any
> > management
> > > > > > > operations such as add, delete and update tenants, namespaces,
> > > > topics,
> > > > > > and
> > > > > > > so on. When the scale of Pulsar cluster increases or the number
> > of
> > > > > > clusters
> > > > > > > grows, using `pulsar-admin` to manage Pulsar can not satisfy
> > > demands.
> > > > > > > Therefore, Pulsar needs a simple and easy-to-use management
> > console
> > > > for
> > > > > > > administrators.
> > > > > > >
> > > > > > > Pulsar Manager is a web-based GUI management and monitoring
> tool
> > > that
> > > > > > helps
> > > > > > > administrators and users manage and monitor tenants,
> namespaces,
> > > > > topics,
> > > > > > > subscriptions, brokers, clusters, and so on, and supports
> dynamic
> > > > > > > configuration of multiple environments.
> > > > > > >
> > > > > > > ## Features
> > > > > > >
> > > > > > > Pulsar manager provides two main features: management and
> > > monitoring.
> > > > > > >
> > > > > > > ### Management
> > > > > > >
> > > > > > > 1. Environment : (operations on the environment)
> > > > > > > Create / Delete / Update / List / Get
> > > > > > > 2. Cluster : (operations on clusters)
> > > > > > > Create / Delete / Update / List / Get
> > > > > > > 3. Brokers : (operations on brokers)
> > > > > > > Heartbeat / Unload
> > > > > > > 4. Tenants:
> > > > > > > Create / Delete / Update / List
> > > > > > > 5. Namespaces:
> > > > > > > Create / Delete / List
> > > > > > > Manage Namespace Policy
> > > > > > > Unload
> > > > > > > Operations on Namespace Bundles
> > > > > > > 6. Topics
> > > > > > > Create / Delete / List
> > > > > > > Unload / Terminate / Offload / Compact
> > > > > > > Topic Details
> > > > > > > 7. Subscriptions
> > > > > > > Create / Delete / List
> > > > > > > Reset Cursor / Skip / Clear backlog / Unsubscribe
> > > > > > > 8. Namespace Isolation Policies
> > > > > > > Create / Delete / Update / Get / List
> > > > > > > 9. Failure Domains
> > > > > > > Create / Delete / Update / Get / List
> > > > > > >
> > > > > > > ### Monitoring: (display *aggregated* stats at different
> levels)
> > > > > > >
> > > > > > > 1. Tenants
> > > > > > > List the total number of namespaces per tenant (both aggregated
> > and
> > > > > > > per-cluster basis)
> > > > > > >
> > > > > > > 2. Namespaces
> > > > > > > List the total number of topics per namespace
> > > > > > > List the aggregated stats per namespace (such as rate-in,
> > rate-out,
> > > > > > > throughput-in, and throughput-out)
> > > > > > > List the distributions of namespace bundles
> > > > > > >
> > > > > > > 3. Topics
> > > > > > > List the total number of partitions per topic
> > > > > > > List the aggregated stats per topic (such as rate-in, rate-out,
> > > > > > > throughput-in, and throughput-out)
> > > > > > > Detailed stats per topic partition
> > > > > > > Detailed stats of storage per topic partition
> > > > > > > Detailed stats of subscription per topic and per partition
> > > > > > > Detailed stats of producers
> > > > > > > Detailed stats of consumers
> > > > > > >
> > > > > > > The detailed design proposal is in
> > > > > > > [
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> > > > > > >
> > > > > > > Looking forward to any feedback.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Guangning
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: PIP-40: Contribute Pulsar Manager

[Ali Ahmed <al...@gmail.com>]

I see there are dependencies on websockets, I would a prefer a simple
polling model of the http, also it's enable by default in pulsar.

For the ui the default persistence should be sqlite. Potentially packaged
by default.

-Ali

On Thu, Aug 29, 2019 at 2:47 PM Enrico Olivelli <eo...@gmail.com> wrote:

> Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha scritto:
>
> > That sounds an interesting idea!
>
>
> Awesome
>
> Does HerdDB support JDBC? If so, it should
> > be pretty straightforward to enable HerdDB.
> >
>
> Sure, as far as I know, the JDBC client is the only client really used in
> production.
>
> In a replicated environment it uses Zookeeper for metadata and service
> discovery and Bookkeeper as WAL and if you have a pulsar cluster you
> already have both of them (zk cluster an bookies)
>
> I will take a look to how Pulsar Manager   uses JDBC, maybe it will be very
> easy.
>
> I will be back with news
>
>
> Enrico
>
>
>
>
>
> > Thanks,
> > Sijie
> >
> > On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > Maybe you can try to use HerdDB (1), it is a replicated SQL database
> that
> > > can be run embedded in the JVM.
> > > It is an SQL database that we developed in my company, in order to
> > > explicitly replace MySQL.
> > > It uses Bookkeeper to store the WAL and implement replication, but it
> can
> > > run in standalone mode or in memory (for tests/dev).
> > >
> > > I can help in setting up a demo.
> > >
> > > Enrico
> > >
> > > (1) https://github.com/diennea/herddb
> > >
> > > Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha scritto:
> > >
> > > > Hi Dave,
> > > >
> > > > It doesn't require MySQL. You can use any JDBC supported databases.
> > > > Initially we would use SQLite for development and change to MySQL for
> > > more
> > > > production-ready workloads.
> > > >
> > > > Hope this clarify your question.
> > > >
> > > > Thanks,
> > > > Sijie
> > > >
> > > > On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org> wrote:
> > > >
> > > > > Hi -
> > > > >
> > > > > I see that MySQL is used. This would be a GPL dependency. Does the
> > > Pulsar
> > > > > Manager require MySQL, or can other Databases that are compliant
> with
> > > > > Apache Release Policy be used instead?
> > > > >
> > > > > Regards,
> > > > > Dave
> > > > >
> > > > > > On Aug 29, 2019, at 7:49 AM, Guangning E <eg...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We have developed a new Pulsar web UI - pulsar-manager, aiming at
> > > > > > supporting managing Pulsar clusters running in different
> > environments
> > > > > > (on-premise data centers, cloud, and so on). We’d like to
> > contribute
> > > > the
> > > > > > project back to the Pulsar community.
> > > > > >
> > > > > > ---
> > > > > >
> > > > > > ## Motivation
> > > > > >
> > > > > > Currently, Pulsar has a monitoring tool - [Pulsar Dashboard]( [
> > > > > >
> > > > >
> > > >
> > >
> >
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> > > > > > ).
> > > > > > However, it only focuses on the simple monitoring of Pulsar -
> > > > collecting
> > > > > > and displaying information such as statistics of tenants,
> > namespaces,
> > > > > > topics, subscriptions, and so on. It doesn’t provide any
> management
> > > > > > operations such as add, delete and update tenants, namespaces,
> > > topics,
> > > > > and
> > > > > > so on. When the scale of Pulsar cluster increases or the number
> of
> > > > > clusters
> > > > > > grows, using `pulsar-admin` to manage Pulsar can not satisfy
> > demands.
> > > > > > Therefore, Pulsar needs a simple and easy-to-use management
> console
> > > for
> > > > > > administrators.
> > > > > >
> > > > > > Pulsar Manager is a web-based GUI management and monitoring tool
> > that
> > > > > helps
> > > > > > administrators and users manage and monitor tenants, namespaces,
> > > > topics,
> > > > > > subscriptions, brokers, clusters, and so on, and supports dynamic
> > > > > > configuration of multiple environments.
> > > > > >
> > > > > > ## Features
> > > > > >
> > > > > > Pulsar manager provides two main features: management and
> > monitoring.
> > > > > >
> > > > > > ### Management
> > > > > >
> > > > > > 1. Environment : (operations on the environment)
> > > > > > Create / Delete / Update / List / Get
> > > > > > 2. Cluster : (operations on clusters)
> > > > > > Create / Delete / Update / List / Get
> > > > > > 3. Brokers : (operations on brokers)
> > > > > > Heartbeat / Unload
> > > > > > 4. Tenants:
> > > > > > Create / Delete / Update / List
> > > > > > 5. Namespaces:
> > > > > > Create / Delete / List
> > > > > > Manage Namespace Policy
> > > > > > Unload
> > > > > > Operations on Namespace Bundles
> > > > > > 6. Topics
> > > > > > Create / Delete / List
> > > > > > Unload / Terminate / Offload / Compact
> > > > > > Topic Details
> > > > > > 7. Subscriptions
> > > > > > Create / Delete / List
> > > > > > Reset Cursor / Skip / Clear backlog / Unsubscribe
> > > > > > 8. Namespace Isolation Policies
> > > > > > Create / Delete / Update / Get / List
> > > > > > 9. Failure Domains
> > > > > > Create / Delete / Update / Get / List
> > > > > >
> > > > > > ### Monitoring: (display *aggregated* stats at different levels)
> > > > > >
> > > > > > 1. Tenants
> > > > > > List the total number of namespaces per tenant (both aggregated
> and
> > > > > > per-cluster basis)
> > > > > >
> > > > > > 2. Namespaces
> > > > > > List the total number of topics per namespace
> > > > > > List the aggregated stats per namespace (such as rate-in,
> rate-out,
> > > > > > throughput-in, and throughput-out)
> > > > > > List the distributions of namespace bundles
> > > > > >
> > > > > > 3. Topics
> > > > > > List the total number of partitions per topic
> > > > > > List the aggregated stats per topic (such as rate-in, rate-out,
> > > > > > throughput-in, and throughput-out)
> > > > > > Detailed stats per topic partition
> > > > > > Detailed stats of storage per topic partition
> > > > > > Detailed stats of subscription per topic and per partition
> > > > > > Detailed stats of producers
> > > > > > Detailed stats of consumers
> > > > > >
> > > > > > The detailed design proposal is in
> > > > > > [
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> > > > > >
> > > > > > Looking forward to any feedback.
> > > > > >
> > > > > > Thanks,
> > > > > > Guangning
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: PIP-40: Contribute Pulsar Manager

[Enrico Olivelli <eo...@gmail.com>]

Il gio 29 ago 2019, 23:28 Sijie Guo <gu...@gmail.com> ha scritto:

> That sounds an interesting idea!


Awesome

Does HerdDB support JDBC? If so, it should
> be pretty straightforward to enable HerdDB.
>

Sure, as far as I know, the JDBC client is the only client really used in
production.

In a replicated environment it uses Zookeeper for metadata and service
discovery and Bookkeeper as WAL and if you have a pulsar cluster you
already have both of them (zk cluster an bookies)

I will take a look to how Pulsar Manager   uses JDBC, maybe it will be very
easy.

I will be back with news


Enrico





> Thanks,
> Sijie
>
> On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Maybe you can try to use HerdDB (1), it is a replicated SQL database that
> > can be run embedded in the JVM.
> > It is an SQL database that we developed in my company, in order to
> > explicitly replace MySQL.
> > It uses Bookkeeper to store the WAL and implement replication, but it can
> > run in standalone mode or in memory (for tests/dev).
> >
> > I can help in setting up a demo.
> >
> > Enrico
> >
> > (1) https://github.com/diennea/herddb
> >
> > Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha scritto:
> >
> > > Hi Dave,
> > >
> > > It doesn't require MySQL. You can use any JDBC supported databases.
> > > Initially we would use SQLite for development and change to MySQL for
> > more
> > > production-ready workloads.
> > >
> > > Hope this clarify your question.
> > >
> > > Thanks,
> > > Sijie
> > >
> > > On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org> wrote:
> > >
> > > > Hi -
> > > >
> > > > I see that MySQL is used. This would be a GPL dependency. Does the
> > Pulsar
> > > > Manager require MySQL, or can other Databases that are compliant with
> > > > Apache Release Policy be used instead?
> > > >
> > > > Regards,
> > > > Dave
> > > >
> > > > > On Aug 29, 2019, at 7:49 AM, Guangning E <eg...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > We have developed a new Pulsar web UI - pulsar-manager, aiming at
> > > > > supporting managing Pulsar clusters running in different
> environments
> > > > > (on-premise data centers, cloud, and so on). We’d like to
> contribute
> > > the
> > > > > project back to the Pulsar community.
> > > > >
> > > > > ---
> > > > >
> > > > > ## Motivation
> > > > >
> > > > > Currently, Pulsar has a monitoring tool - [Pulsar Dashboard]( [
> > > > >
> > > >
> > >
> >
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> > > > > ).
> > > > > However, it only focuses on the simple monitoring of Pulsar -
> > > collecting
> > > > > and displaying information such as statistics of tenants,
> namespaces,
> > > > > topics, subscriptions, and so on. It doesn’t provide any management
> > > > > operations such as add, delete and update tenants, namespaces,
> > topics,
> > > > and
> > > > > so on. When the scale of Pulsar cluster increases or the number of
> > > > clusters
> > > > > grows, using `pulsar-admin` to manage Pulsar can not satisfy
> demands.
> > > > > Therefore, Pulsar needs a simple and easy-to-use management console
> > for
> > > > > administrators.
> > > > >
> > > > > Pulsar Manager is a web-based GUI management and monitoring tool
> that
> > > > helps
> > > > > administrators and users manage and monitor tenants, namespaces,
> > > topics,
> > > > > subscriptions, brokers, clusters, and so on, and supports dynamic
> > > > > configuration of multiple environments.
> > > > >
> > > > > ## Features
> > > > >
> > > > > Pulsar manager provides two main features: management and
> monitoring.
> > > > >
> > > > > ### Management
> > > > >
> > > > > 1. Environment : (operations on the environment)
> > > > > Create / Delete / Update / List / Get
> > > > > 2. Cluster : (operations on clusters)
> > > > > Create / Delete / Update / List / Get
> > > > > 3. Brokers : (operations on brokers)
> > > > > Heartbeat / Unload
> > > > > 4. Tenants:
> > > > > Create / Delete / Update / List
> > > > > 5. Namespaces:
> > > > > Create / Delete / List
> > > > > Manage Namespace Policy
> > > > > Unload
> > > > > Operations on Namespace Bundles
> > > > > 6. Topics
> > > > > Create / Delete / List
> > > > > Unload / Terminate / Offload / Compact
> > > > > Topic Details
> > > > > 7. Subscriptions
> > > > > Create / Delete / List
> > > > > Reset Cursor / Skip / Clear backlog / Unsubscribe
> > > > > 8. Namespace Isolation Policies
> > > > > Create / Delete / Update / Get / List
> > > > > 9. Failure Domains
> > > > > Create / Delete / Update / Get / List
> > > > >
> > > > > ### Monitoring: (display *aggregated* stats at different levels)
> > > > >
> > > > > 1. Tenants
> > > > > List the total number of namespaces per tenant (both aggregated and
> > > > > per-cluster basis)
> > > > >
> > > > > 2. Namespaces
> > > > > List the total number of topics per namespace
> > > > > List the aggregated stats per namespace (such as rate-in, rate-out,
> > > > > throughput-in, and throughput-out)
> > > > > List the distributions of namespace bundles
> > > > >
> > > > > 3. Topics
> > > > > List the total number of partitions per topic
> > > > > List the aggregated stats per topic (such as rate-in, rate-out,
> > > > > throughput-in, and throughput-out)
> > > > > Detailed stats per topic partition
> > > > > Detailed stats of storage per topic partition
> > > > > Detailed stats of subscription per topic and per partition
> > > > > Detailed stats of producers
> > > > > Detailed stats of consumers
> > > > >
> > > > > The detailed design proposal is in
> > > > > [
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> > > > >
> > > > > Looking forward to any feedback.
> > > > >
> > > > > Thanks,
> > > > > Guangning
> > > >
> > > >
> > >
> >
>

Re: PIP-40: Contribute Pulsar Manager

[Sijie Guo <gu...@gmail.com>]

That sounds an interesting idea! Does HerdDB support JDBC? If so, it should
be pretty straightforward to enable HerdDB.

Thanks,
Sijie

On Thu, Aug 29, 2019 at 12:08 PM Enrico Olivelli <eo...@gmail.com>
wrote:

> Maybe you can try to use HerdDB (1), it is a replicated SQL database that
> can be run embedded in the JVM.
> It is an SQL database that we developed in my company, in order to
> explicitly replace MySQL.
> It uses Bookkeeper to store the WAL and implement replication, but it can
> run in standalone mode or in memory (for tests/dev).
>
> I can help in setting up a demo.
>
> Enrico
>
> (1) https://github.com/diennea/herddb
>
> Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha scritto:
>
> > Hi Dave,
> >
> > It doesn't require MySQL. You can use any JDBC supported databases.
> > Initially we would use SQLite for development and change to MySQL for
> more
> > production-ready workloads.
> >
> > Hope this clarify your question.
> >
> > Thanks,
> > Sijie
> >
> > On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org> wrote:
> >
> > > Hi -
> > >
> > > I see that MySQL is used. This would be a GPL dependency. Does the
> Pulsar
> > > Manager require MySQL, or can other Databases that are compliant with
> > > Apache Release Policy be used instead?
> > >
> > > Regards,
> > > Dave
> > >
> > > > On Aug 29, 2019, at 7:49 AM, Guangning E <eg...@gmail.com>
> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > We have developed a new Pulsar web UI - pulsar-manager, aiming at
> > > > supporting managing Pulsar clusters running in different environments
> > > > (on-premise data centers, cloud, and so on). We’d like to contribute
> > the
> > > > project back to the Pulsar community.
> > > >
> > > > ---
> > > >
> > > > ## Motivation
> > > >
> > > > Currently, Pulsar has a monitoring tool - [Pulsar Dashboard]( [
> > > >
> > >
> >
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> > > > ).
> > > > However, it only focuses on the simple monitoring of Pulsar -
> > collecting
> > > > and displaying information such as statistics of tenants, namespaces,
> > > > topics, subscriptions, and so on. It doesn’t provide any management
> > > > operations such as add, delete and update tenants, namespaces,
> topics,
> > > and
> > > > so on. When the scale of Pulsar cluster increases or the number of
> > > clusters
> > > > grows, using `pulsar-admin` to manage Pulsar can not satisfy demands.
> > > > Therefore, Pulsar needs a simple and easy-to-use management console
> for
> > > > administrators.
> > > >
> > > > Pulsar Manager is a web-based GUI management and monitoring tool that
> > > helps
> > > > administrators and users manage and monitor tenants, namespaces,
> > topics,
> > > > subscriptions, brokers, clusters, and so on, and supports dynamic
> > > > configuration of multiple environments.
> > > >
> > > > ## Features
> > > >
> > > > Pulsar manager provides two main features: management and monitoring.
> > > >
> > > > ### Management
> > > >
> > > > 1. Environment : (operations on the environment)
> > > > Create / Delete / Update / List / Get
> > > > 2. Cluster : (operations on clusters)
> > > > Create / Delete / Update / List / Get
> > > > 3. Brokers : (operations on brokers)
> > > > Heartbeat / Unload
> > > > 4. Tenants:
> > > > Create / Delete / Update / List
> > > > 5. Namespaces:
> > > > Create / Delete / List
> > > > Manage Namespace Policy
> > > > Unload
> > > > Operations on Namespace Bundles
> > > > 6. Topics
> > > > Create / Delete / List
> > > > Unload / Terminate / Offload / Compact
> > > > Topic Details
> > > > 7. Subscriptions
> > > > Create / Delete / List
> > > > Reset Cursor / Skip / Clear backlog / Unsubscribe
> > > > 8. Namespace Isolation Policies
> > > > Create / Delete / Update / Get / List
> > > > 9. Failure Domains
> > > > Create / Delete / Update / Get / List
> > > >
> > > > ### Monitoring: (display *aggregated* stats at different levels)
> > > >
> > > > 1. Tenants
> > > > List the total number of namespaces per tenant (both aggregated and
> > > > per-cluster basis)
> > > >
> > > > 2. Namespaces
> > > > List the total number of topics per namespace
> > > > List the aggregated stats per namespace (such as rate-in, rate-out,
> > > > throughput-in, and throughput-out)
> > > > List the distributions of namespace bundles
> > > >
> > > > 3. Topics
> > > > List the total number of partitions per topic
> > > > List the aggregated stats per topic (such as rate-in, rate-out,
> > > > throughput-in, and throughput-out)
> > > > Detailed stats per topic partition
> > > > Detailed stats of storage per topic partition
> > > > Detailed stats of subscription per topic and per partition
> > > > Detailed stats of producers
> > > > Detailed stats of consumers
> > > >
> > > > The detailed design proposal is in
> > > > [
> > > >
> > >
> >
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> > > >
> > > > Looking forward to any feedback.
> > > >
> > > > Thanks,
> > > > Guangning
> > >
> > >
> >
>

Re: PIP-40: Contribute Pulsar Manager

[Enrico Olivelli <eo...@gmail.com>]

Maybe you can try to use HerdDB (1), it is a replicated SQL database that
can be run embedded in the JVM.
It is an SQL database that we developed in my company, in order to
explicitly replace MySQL.
It uses Bookkeeper to store the WAL and implement replication, but it can
run in standalone mode or in memory (for tests/dev).

I can help in setting up a demo.

Enrico

(1) https://github.com/diennea/herddb

Il gio 29 ago 2019, 16:59 Sijie Guo <gu...@gmail.com> ha scritto:

> Hi Dave,
>
> It doesn't require MySQL. You can use any JDBC supported databases.
> Initially we would use SQLite for development and change to MySQL for more
> production-ready workloads.
>
> Hope this clarify your question.
>
> Thanks,
> Sijie
>
> On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org> wrote:
>
> > Hi -
> >
> > I see that MySQL is used. This would be a GPL dependency. Does the Pulsar
> > Manager require MySQL, or can other Databases that are compliant with
> > Apache Release Policy be used instead?
> >
> > Regards,
> > Dave
> >
> > > On Aug 29, 2019, at 7:49 AM, Guangning E <eg...@gmail.com> wrote:
> > >
> > > Hi all,
> > >
> > > We have developed a new Pulsar web UI - pulsar-manager, aiming at
> > > supporting managing Pulsar clusters running in different environments
> > > (on-premise data centers, cloud, and so on). We’d like to contribute
> the
> > > project back to the Pulsar community.
> > >
> > > ---
> > >
> > > ## Motivation
> > >
> > > Currently, Pulsar has a monitoring tool - [Pulsar Dashboard]( [
> > >
> >
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> > > ).
> > > However, it only focuses on the simple monitoring of Pulsar -
> collecting
> > > and displaying information such as statistics of tenants, namespaces,
> > > topics, subscriptions, and so on. It doesn’t provide any management
> > > operations such as add, delete and update tenants, namespaces, topics,
> > and
> > > so on. When the scale of Pulsar cluster increases or the number of
> > clusters
> > > grows, using `pulsar-admin` to manage Pulsar can not satisfy demands.
> > > Therefore, Pulsar needs a simple and easy-to-use management console for
> > > administrators.
> > >
> > > Pulsar Manager is a web-based GUI management and monitoring tool that
> > helps
> > > administrators and users manage and monitor tenants, namespaces,
> topics,
> > > subscriptions, brokers, clusters, and so on, and supports dynamic
> > > configuration of multiple environments.
> > >
> > > ## Features
> > >
> > > Pulsar manager provides two main features: management and monitoring.
> > >
> > > ### Management
> > >
> > > 1. Environment : (operations on the environment)
> > > Create / Delete / Update / List / Get
> > > 2. Cluster : (operations on clusters)
> > > Create / Delete / Update / List / Get
> > > 3. Brokers : (operations on brokers)
> > > Heartbeat / Unload
> > > 4. Tenants:
> > > Create / Delete / Update / List
> > > 5. Namespaces:
> > > Create / Delete / List
> > > Manage Namespace Policy
> > > Unload
> > > Operations on Namespace Bundles
> > > 6. Topics
> > > Create / Delete / List
> > > Unload / Terminate / Offload / Compact
> > > Topic Details
> > > 7. Subscriptions
> > > Create / Delete / List
> > > Reset Cursor / Skip / Clear backlog / Unsubscribe
> > > 8. Namespace Isolation Policies
> > > Create / Delete / Update / Get / List
> > > 9. Failure Domains
> > > Create / Delete / Update / Get / List
> > >
> > > ### Monitoring: (display *aggregated* stats at different levels)
> > >
> > > 1. Tenants
> > > List the total number of namespaces per tenant (both aggregated and
> > > per-cluster basis)
> > >
> > > 2. Namespaces
> > > List the total number of topics per namespace
> > > List the aggregated stats per namespace (such as rate-in, rate-out,
> > > throughput-in, and throughput-out)
> > > List the distributions of namespace bundles
> > >
> > > 3. Topics
> > > List the total number of partitions per topic
> > > List the aggregated stats per topic (such as rate-in, rate-out,
> > > throughput-in, and throughput-out)
> > > Detailed stats per topic partition
> > > Detailed stats of storage per topic partition
> > > Detailed stats of subscription per topic and per partition
> > > Detailed stats of producers
> > > Detailed stats of consumers
> > >
> > > The detailed design proposal is in
> > > [
> > >
> >
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> > >
> > > Looking forward to any feedback.
> > >
> > > Thanks,
> > > Guangning
> >
> >
>

Re: PIP-40: Contribute Pulsar Manager

[Sijie Guo <gu...@gmail.com>]

Hi Dave,

It doesn't require MySQL. You can use any JDBC supported databases.
Initially we would use SQLite for development and change to MySQL for more
production-ready workloads.

Hope this clarify your question.

Thanks,
Sijie

On Thu, Aug 29, 2019 at 5:56 AM Dave Fisher <wa...@apache.org> wrote:

> Hi -
>
> I see that MySQL is used. This would be a GPL dependency. Does the Pulsar
> Manager require MySQL, or can other Databases that are compliant with
> Apache Release Policy be used instead?
>
> Regards,
> Dave
>
> > On Aug 29, 2019, at 7:49 AM, Guangning E <eg...@gmail.com> wrote:
> >
> > Hi all,
> >
> > We have developed a new Pulsar web UI - pulsar-manager, aiming at
> > supporting managing Pulsar clusters running in different environments
> > (on-premise data centers, cloud, and so on). We’d like to contribute the
> > project back to the Pulsar community.
> >
> > ---
> >
> > ## Motivation
> >
> > Currently, Pulsar has a monitoring tool - [Pulsar Dashboard]( [
> >
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> > ).
> > However, it only focuses on the simple monitoring of Pulsar - collecting
> > and displaying information such as statistics of tenants, namespaces,
> > topics, subscriptions, and so on. It doesn’t provide any management
> > operations such as add, delete and update tenants, namespaces, topics,
> and
> > so on. When the scale of Pulsar cluster increases or the number of
> clusters
> > grows, using `pulsar-admin` to manage Pulsar can not satisfy demands.
> > Therefore, Pulsar needs a simple and easy-to-use management console for
> > administrators.
> >
> > Pulsar Manager is a web-based GUI management and monitoring tool that
> helps
> > administrators and users manage and monitor tenants, namespaces, topics,
> > subscriptions, brokers, clusters, and so on, and supports dynamic
> > configuration of multiple environments.
> >
> > ## Features
> >
> > Pulsar manager provides two main features: management and monitoring.
> >
> > ### Management
> >
> > 1. Environment : (operations on the environment)
> > Create / Delete / Update / List / Get
> > 2. Cluster : (operations on clusters)
> > Create / Delete / Update / List / Get
> > 3. Brokers : (operations on brokers)
> > Heartbeat / Unload
> > 4. Tenants:
> > Create / Delete / Update / List
> > 5. Namespaces:
> > Create / Delete / List
> > Manage Namespace Policy
> > Unload
> > Operations on Namespace Bundles
> > 6. Topics
> > Create / Delete / List
> > Unload / Terminate / Offload / Compact
> > Topic Details
> > 7. Subscriptions
> > Create / Delete / List
> > Reset Cursor / Skip / Clear backlog / Unsubscribe
> > 8. Namespace Isolation Policies
> > Create / Delete / Update / Get / List
> > 9. Failure Domains
> > Create / Delete / Update / Get / List
> >
> > ### Monitoring: (display *aggregated* stats at different levels)
> >
> > 1. Tenants
> > List the total number of namespaces per tenant (both aggregated and
> > per-cluster basis)
> >
> > 2. Namespaces
> > List the total number of topics per namespace
> > List the aggregated stats per namespace (such as rate-in, rate-out,
> > throughput-in, and throughput-out)
> > List the distributions of namespace bundles
> >
> > 3. Topics
> > List the total number of partitions per topic
> > List the aggregated stats per topic (such as rate-in, rate-out,
> > throughput-in, and throughput-out)
> > Detailed stats per topic partition
> > Detailed stats of storage per topic partition
> > Detailed stats of subscription per topic and per partition
> > Detailed stats of producers
> > Detailed stats of consumers
> >
> > The detailed design proposal is in
> > [
> >
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> >
> > Looking forward to any feedback.
> >
> > Thanks,
> > Guangning
>
>

Re: PIP-40: Contribute Pulsar Manager

[Dave Fisher <wa...@apache.org>]

Hi -

I see that MySQL is used. This would be a GPL dependency. Does the Pulsar Manager require MySQL, or can other Databases that are compliant with Apache Release Policy be used instead?

Regards,
Dave

> On Aug 29, 2019, at 7:49 AM, Guangning E <eg...@gmail.com> wrote:
> 
> Hi all,
> 
> We have developed a new Pulsar web UI - pulsar-manager, aiming at
> supporting managing Pulsar clusters running in different environments
> (on-premise data centers, cloud, and so on). We’d like to contribute the
> project back to the Pulsar community.
> 
> ---
> 
> ## Motivation
> 
> Currently, Pulsar has a monitoring tool - [Pulsar Dashboard]( [
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> ).
> However, it only focuses on the simple monitoring of Pulsar - collecting
> and displaying information such as statistics of tenants, namespaces,
> topics, subscriptions, and so on. It doesn’t provide any management
> operations such as add, delete and update tenants, namespaces, topics, and
> so on. When the scale of Pulsar cluster increases or the number of clusters
> grows, using `pulsar-admin` to manage Pulsar can not satisfy demands.
> Therefore, Pulsar needs a simple and easy-to-use management console for
> administrators.
> 
> Pulsar Manager is a web-based GUI management and monitoring tool that helps
> administrators and users manage and monitor tenants, namespaces, topics,
> subscriptions, brokers, clusters, and so on, and supports dynamic
> configuration of multiple environments.
> 
> ## Features
> 
> Pulsar manager provides two main features: management and monitoring.
> 
> ### Management
> 
> 1. Environment : (operations on the environment)
> Create / Delete / Update / List / Get
> 2. Cluster : (operations on clusters)
> Create / Delete / Update / List / Get
> 3. Brokers : (operations on brokers)
> Heartbeat / Unload
> 4. Tenants:
> Create / Delete / Update / List
> 5. Namespaces:
> Create / Delete / List
> Manage Namespace Policy
> Unload
> Operations on Namespace Bundles
> 6. Topics
> Create / Delete / List
> Unload / Terminate / Offload / Compact
> Topic Details
> 7. Subscriptions
> Create / Delete / List
> Reset Cursor / Skip / Clear backlog / Unsubscribe
> 8. Namespace Isolation Policies
> Create / Delete / Update / Get / List
> 9. Failure Domains
> Create / Delete / Update / Get / List
> 
> ### Monitoring: (display *aggregated* stats at different levels)
> 
> 1. Tenants
> List the total number of namespaces per tenant (both aggregated and
> per-cluster basis)
> 
> 2. Namespaces
> List the total number of topics per namespace
> List the aggregated stats per namespace (such as rate-in, rate-out,
> throughput-in, and throughput-out)
> List the distributions of namespace bundles
> 
> 3. Topics
> List the total number of partitions per topic
> List the aggregated stats per topic (such as rate-in, rate-out,
> throughput-in, and throughput-out)
> Detailed stats per topic partition
> Detailed stats of storage per topic partition
> Detailed stats of subscription per topic and per partition
> Detailed stats of producers
> Detailed stats of consumers
> 
> The detailed design proposal is in
> [
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
> 
> Looking forward to any feedback.
> 
> Thanks,
> Guangning

Re: PIP-40: Contribute Pulsar Manager

[Yuva raj <uv...@gmail.com>]

This is excellent.

On Thu, 29 Aug 2019 at 17:19, Guangning E <eg...@gmail.com> wrote:

> Hi all,
>
> We have developed a new Pulsar web UI - pulsar-manager, aiming at
> supporting managing Pulsar clusters running in different environments
> (on-premise data centers, cloud, and so on). We’d like to contribute the
> project back to the Pulsar community.
>
> ---
>
> ## Motivation
>
> Currently, Pulsar has a monitoring tool - [Pulsar Dashboard]( [
>
> http://pulsar.apache.org/docs/en/administration-dashboard/](http://pulsar.apache.org/docs/en/administration-dashboard/)
> ).
> However, it only focuses on the simple monitoring of Pulsar - collecting
> and displaying information such as statistics of tenants, namespaces,
> topics, subscriptions, and so on. It doesn’t provide any management
> operations such as add, delete and update tenants, namespaces, topics, and
> so on. When the scale of Pulsar cluster increases or the number of clusters
> grows, using `pulsar-admin` to manage Pulsar can not satisfy demands.
> Therefore, Pulsar needs a simple and easy-to-use management console for
> administrators.
>
> Pulsar Manager is a web-based GUI management and monitoring tool that helps
> administrators and users manage and monitor tenants, namespaces, topics,
> subscriptions, brokers, clusters, and so on, and supports dynamic
> configuration of multiple environments.
>
> ## Features
>
> Pulsar manager provides two main features: management and monitoring.
>
> ### Management
>
> 1. Environment : (operations on the environment)
> Create / Delete / Update / List / Get
> 2. Cluster : (operations on clusters)
> Create / Delete / Update / List / Get
> 3. Brokers : (operations on brokers)
> Heartbeat / Unload
> 4. Tenants:
> Create / Delete / Update / List
> 5. Namespaces:
> Create / Delete / List
> Manage Namespace Policy
> Unload
> Operations on Namespace Bundles
> 6. Topics
> Create / Delete / List
> Unload / Terminate / Offload / Compact
> Topic Details
> 7. Subscriptions
> Create / Delete / List
> Reset Cursor / Skip / Clear backlog / Unsubscribe
> 8. Namespace Isolation Policies
> Create / Delete / Update / Get / List
> 9. Failure Domains
> Create / Delete / Update / Get / List
>
> ### Monitoring: (display *aggregated* stats at different levels)
>
> 1. Tenants
> List the total number of namespaces per tenant (both aggregated and
> per-cluster basis)
>
> 2. Namespaces
> List the total number of topics per namespace
> List the aggregated stats per namespace (such as rate-in, rate-out,
> throughput-in, and throughput-out)
> List the distributions of namespace bundles
>
> 3. Topics
> List the total number of partitions per topic
> List the aggregated stats per topic (such as rate-in, rate-out,
> throughput-in, and throughput-out)
> Detailed stats per topic partition
> Detailed stats of storage per topic partition
> Detailed stats of subscription per topic and per partition
> Detailed stats of producers
> Detailed stats of consumers
>
> The detailed design proposal is in
>  [
>
> https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#](https://docs.google.com/document/d/1C3meaHJzxX9wGDWQx-dC1b20yGeKsQtbwNiYuxZgOCE/edit#)
>
> Looking forward to any feedback.
>
> Thanks,
> Guangning
>


-- 
*Thanks*

*Yuvaraj L*