You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Matteo Merli <mm...@apache.org> on 2021/12/21 23:58:57 UTC
[VOTE] PIP-123: Introduce Pulsar metadata CLI tool
This is the voting thread for PIP-123. It will stay open for at least 48h.
https://github.com/apache/pulsar/issues/13346
---------
## Motivation
For a very long time, we have included a CLI command to start the ZooKeeper
shell utility: `pulsar zookeeper-shell`, which is essentially a repackaging
of the ZooKeeper tool `zkCli.sh`.
This is useful in some cases to either verify the content of metadata or
to perform cleanup and modification tasks for which there is not an available
option through the Pulsar REST APIs.
While it's very useful, there are some drawbacks with the `zookeeper-shell`
as it is today:
1. This is only a ZooKeeper tool (obviously). Since we are adding more
metadata backends, we should have a tool that works across all the
implementations and presents a single consistent interface.
2. ZooKeeper shell is designed to be an interactive shell and it's not very
good when trying to do non-interactive scriptable operations.
3. ZooKeeper is a bit clunky when using it and it requires the user to retype
paths many times. The commands are not very intuitive or documented.
It's not possible to update z-node with multi-lines content.
4. We cannot easily add functionality or improvements into ZooKeeper shell,
since it belongs to a different project and the tool has been stagnating
for many years.
5. In cases where the z-nodes content is binary (Protobuf) or compressed, there
is no easy way to inspect the content from the ZooKeeper shell.
Additionally, we can format and colorize JSON content to make it easier to
read.
6. The paths used for metadata resources are also often using encodings that
make it more difficult to construct on the shell tool.
Part of what is described here is in the `pulsar-managed-ledger-admin` CLI tool,
though that is a Python script that requires additional dependencies that are
not typically installed, it only works with ZooKeeper, and it only targets
accessing the managed ledger metadata.
## Goal
Introduce a new Java CLI tool to access, inspect and modify metadata that solves
all the issues described above.
We would leave the `zookeeper-shell` command for now. In the future, once the
new tool is proven, we can consider removing the `zookeeper-shell` command.
## Proposed changes
Add a new command:
```bash
bin/pulsar metadata
```
with several subcommands:
#### Get
Examples:
```bash
# General path get
$ pulsar metadata get /my-path
# Topic metadata
$ pulsar metadata get topic my-tenant/my-namespace/my-topic
{
# Managed ledger metadata
}
# Namespace get
$ pulsar metadata get namespace my-tenant/my-namespace
{
# Namespace metadata
}
$ pulsar metadata get ledger 12345
{
# BK ledger metadata
}
```
#### Delete
Examples:
```bash
# General path delete
$ pulsar metadata delete /my-path
# Topic metadata
$ pulsar metadata delete topic my-tenant/my-namespace/my-topic
```
#### Scan
Examples:
```bash
$ pulsar metadata scan /my-path
/my-path
/my-path/1
/my-path/2
/my-path/3
/my-path/3/1
$ pulsar metadata scan --values /my-path
/my-path
{value}
/my-path/1
{value}
/my-path/2
{value}
```
#### Shell
```bash
$ pulsar metadata shell
> get topic my-tenant/my-namespace/my-topic
{
# Managed ledger metadata
}
> delete topic my-tenant/my-namespace/my-topic
> cd /my-path
> ls
1
2
3
> delete 1 # Delete keys with relative paths
```
--
Matteo Merli
<mm...@apache.org>
Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool
Posted by Hang Chen <ch...@apache.org>.
+1
Thanks,
Hang
PengHui Li <pe...@apache.org> 于2021年12月22日周三 19:33写道：
>
> +1
>
> Penghui
>
> On Wed, Dec 22, 2021 at 3:31 PM Enrico Olivelli <eo...@gmail.com> wrote:
>
> > very good
> >
> > Enrico
> >
> > Il giorno mer 22 dic 2021 alle ore 03:37 mattison chao <
> > mattisonchao@gmail.com> ha scritto:
> >
> > > +1
> > >
> > > On Wed, 22 Dec 2021 at 07:59, Matteo Merli <mm...@apache.org> wrote:
> > >
> > > > This is the voting thread for PIP-123. It will stay open for at least
> > > 48h.
> > > >
> > > > https://github.com/apache/pulsar/issues/13346
> > > >
> > > >
> > > > ---------
> > > > ## Motivation
> > > >
> > > > For a very long time, we have included a CLI command to start the
> > > ZooKeeper
> > > > shell utility: `pulsar zookeeper-shell`, which is essentially a
> > > repackaging
> > > > of the ZooKeeper tool `zkCli.sh`.
> > > >
> > > > This is useful in some cases to either verify the content of metadata
> > or
> > > > to perform cleanup and modification tasks for which there is not an
> > > > available
> > > > option through the Pulsar REST APIs.
> > > >
> > > > While it's very useful, there are some drawbacks with the
> > > `zookeeper-shell`
> > > > as it is today:
> > > >
> > > > 1. This is only a ZooKeeper tool (obviously). Since we are adding more
> > > > metadata backends, we should have a tool that works across all the
> > > > implementations and presents a single consistent interface.
> > > >
> > > > 2. ZooKeeper shell is designed to be an interactive shell and it's not
> > > > very
> > > > good when trying to do non-interactive scriptable operations.
> > > >
> > > > 3. ZooKeeper is a bit clunky when using it and it requires the user to
> > > > retype
> > > > paths many times. The commands are not very intuitive or
> > documented.
> > > > It's not possible to update z-node with multi-lines content.
> > > >
> > > > 4. We cannot easily add functionality or improvements into ZooKeeper
> > > > shell,
> > > > since it belongs to a different project and the tool has been
> > > > stagnating
> > > > for many years.
> > > >
> > > > 5. In cases where the z-nodes content is binary (Protobuf) or
> > > compressed,
> > > > there
> > > > is no easy way to inspect the content from the ZooKeeper shell.
> > > > Additionally, we can format and colorize JSON content to make it
> > > > easier to
> > > > read.
> > > >
> > > > 6. The paths used for metadata resources are also often using
> > encodings
> > > > that
> > > > make it more difficult to construct on the shell tool.
> > > >
> > > > Part of what is described here is in the `pulsar-managed-ledger-admin`
> > > CLI
> > > > tool,
> > > > though that is a Python script that requires additional dependencies
> > that
> > > > are
> > > > not typically installed, it only works with ZooKeeper, and it only
> > > targets
> > > > accessing the managed ledger metadata.
> > > >
> > > > ## Goal
> > > >
> > > > Introduce a new Java CLI tool to access, inspect and modify metadata
> > that
> > > > solves
> > > > all the issues described above.
> > > >
> > > > We would leave the `zookeeper-shell` command for now. In the future,
> > once
> > > > the
> > > > new tool is proven, we can consider removing the `zookeeper-shell`
> > > command.
> > > >
> > > >
> > > > ## Proposed changes
> > > >
> > > > Add a new command:
> > > > ```bash
> > > > bin/pulsar metadata
> > > > ```
> > > >
> > > > with several subcommands:
> > > >
> > > >
> > > > #### Get
> > > >
> > > > Examples:
> > > > ```bash
> > > > # General path get
> > > > $ pulsar metadata get /my-path
> > > >
> > > >
> > > > # Topic metadata
> > > > $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> > > > {
> > > > # Managed ledger metadata
> > > > }
> > > >
> > > > # Namespace get
> > > > $ pulsar metadata get namespace my-tenant/my-namespace
> > > > {
> > > > # Namespace metadata
> > > > }
> > > >
> > > > $ pulsar metadata get ledger 12345
> > > > {
> > > > # BK ledger metadata
> > > > }
> > > > ```
> > > >
> > > > #### Delete
> > > >
> > > > Examples:
> > > > ```bash
> > > > # General path delete
> > > > $ pulsar metadata delete /my-path
> > > >
> > > > # Topic metadata
> > > > $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> > > > ```
> > > >
> > > > #### Scan
> > > >
> > > > Examples:
> > > > ```bash
> > > > $ pulsar metadata scan /my-path
> > > > /my-path
> > > > /my-path/1
> > > > /my-path/2
> > > > /my-path/3
> > > > /my-path/3/1
> > > >
> > > > $ pulsar metadata scan --values /my-path
> > > > /my-path
> > > > {value}
> > > >
> > > > /my-path/1
> > > > {value}
> > > >
> > > > /my-path/2
> > > > {value}
> > > > ```
> > > >
> > > > #### Shell
> > > >
> > > > ```bash
> > > > $ pulsar metadata shell
> > > > > get topic my-tenant/my-namespace/my-topic
> > > > {
> > > > # Managed ledger metadata
> > > > }
> > > >
> > > > > delete topic my-tenant/my-namespace/my-topic
> > > >
> > > > > cd /my-path
> > > > > ls
> > > > 1
> > > > 2
> > > > 3
> > > > > delete 1 # Delete keys with relative paths
> > > >
> > > > ```
> > > >
> > > >
> > > > --
> > > > Matteo Merli
> > > > <mm...@apache.org>
> > > >
> > >
> >
Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool
Posted by PengHui Li <pe...@apache.org>.
+1
Penghui
On Wed, Dec 22, 2021 at 3:31 PM Enrico Olivelli <eo...@gmail.com> wrote:
> very good
>
> Enrico
>
> Il giorno mer 22 dic 2021 alle ore 03:37 mattison chao <
> mattisonchao@gmail.com> ha scritto:
>
> > +1
> >
> > On Wed, 22 Dec 2021 at 07:59, Matteo Merli <mm...@apache.org> wrote:
> >
> > > This is the voting thread for PIP-123. It will stay open for at least
> > 48h.
> > >
> > > https://github.com/apache/pulsar/issues/13346
> > >
> > >
> > > ---------
> > > ## Motivation
> > >
> > > For a very long time, we have included a CLI command to start the
> > ZooKeeper
> > > shell utility: `pulsar zookeeper-shell`, which is essentially a
> > repackaging
> > > of the ZooKeeper tool `zkCli.sh`.
> > >
> > > This is useful in some cases to either verify the content of metadata
> or
> > > to perform cleanup and modification tasks for which there is not an
> > > available
> > > option through the Pulsar REST APIs.
> > >
> > > While it's very useful, there are some drawbacks with the
> > `zookeeper-shell`
> > > as it is today:
> > >
> > > 1. This is only a ZooKeeper tool (obviously). Since we are adding more
> > > metadata backends, we should have a tool that works across all the
> > > implementations and presents a single consistent interface.
> > >
> > > 2. ZooKeeper shell is designed to be an interactive shell and it's not
> > > very
> > > good when trying to do non-interactive scriptable operations.
> > >
> > > 3. ZooKeeper is a bit clunky when using it and it requires the user to
> > > retype
> > > paths many times. The commands are not very intuitive or
> documented.
> > > It's not possible to update z-node with multi-lines content.
> > >
> > > 4. We cannot easily add functionality or improvements into ZooKeeper
> > > shell,
> > > since it belongs to a different project and the tool has been
> > > stagnating
> > > for many years.
> > >
> > > 5. In cases where the z-nodes content is binary (Protobuf) or
> > compressed,
> > > there
> > > is no easy way to inspect the content from the ZooKeeper shell.
> > > Additionally, we can format and colorize JSON content to make it
> > > easier to
> > > read.
> > >
> > > 6. The paths used for metadata resources are also often using
> encodings
> > > that
> > > make it more difficult to construct on the shell tool.
> > >
> > > Part of what is described here is in the `pulsar-managed-ledger-admin`
> > CLI
> > > tool,
> > > though that is a Python script that requires additional dependencies
> that
> > > are
> > > not typically installed, it only works with ZooKeeper, and it only
> > targets
> > > accessing the managed ledger metadata.
> > >
> > > ## Goal
> > >
> > > Introduce a new Java CLI tool to access, inspect and modify metadata
> that
> > > solves
> > > all the issues described above.
> > >
> > > We would leave the `zookeeper-shell` command for now. In the future,
> once
> > > the
> > > new tool is proven, we can consider removing the `zookeeper-shell`
> > command.
> > >
> > >
> > > ## Proposed changes
> > >
> > > Add a new command:
> > > ```bash
> > > bin/pulsar metadata
> > > ```
> > >
> > > with several subcommands:
> > >
> > >
> > > #### Get
> > >
> > > Examples:
> > > ```bash
> > > # General path get
> > > $ pulsar metadata get /my-path
> > >
> > >
> > > # Topic metadata
> > > $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> > > {
> > > # Managed ledger metadata
> > > }
> > >
> > > # Namespace get
> > > $ pulsar metadata get namespace my-tenant/my-namespace
> > > {
> > > # Namespace metadata
> > > }
> > >
> > > $ pulsar metadata get ledger 12345
> > > {
> > > # BK ledger metadata
> > > }
> > > ```
> > >
> > > #### Delete
> > >
> > > Examples:
> > > ```bash
> > > # General path delete
> > > $ pulsar metadata delete /my-path
> > >
> > > # Topic metadata
> > > $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> > > ```
> > >
> > > #### Scan
> > >
> > > Examples:
> > > ```bash
> > > $ pulsar metadata scan /my-path
> > > /my-path
> > > /my-path/1
> > > /my-path/2
> > > /my-path/3
> > > /my-path/3/1
> > >
> > > $ pulsar metadata scan --values /my-path
> > > /my-path
> > > {value}
> > >
> > > /my-path/1
> > > {value}
> > >
> > > /my-path/2
> > > {value}
> > > ```
> > >
> > > #### Shell
> > >
> > > ```bash
> > > $ pulsar metadata shell
> > > > get topic my-tenant/my-namespace/my-topic
> > > {
> > > # Managed ledger metadata
> > > }
> > >
> > > > delete topic my-tenant/my-namespace/my-topic
> > >
> > > > cd /my-path
> > > > ls
> > > 1
> > > 2
> > > 3
> > > > delete 1 # Delete keys with relative paths
> > >
> > > ```
> > >
> > >
> > > --
> > > Matteo Merli
> > > <mm...@apache.org>
> > >
> >
>
Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool
Posted by Enrico Olivelli <eo...@gmail.com>.
very good
Enrico
Il giorno mer 22 dic 2021 alle ore 03:37 mattison chao <
mattisonchao@gmail.com> ha scritto:
> +1
>
> On Wed, 22 Dec 2021 at 07:59, Matteo Merli <mm...@apache.org> wrote:
>
> > This is the voting thread for PIP-123. It will stay open for at least
> 48h.
> >
> > https://github.com/apache/pulsar/issues/13346
> >
> >
> > ---------
> > ## Motivation
> >
> > For a very long time, we have included a CLI command to start the
> ZooKeeper
> > shell utility: `pulsar zookeeper-shell`, which is essentially a
> repackaging
> > of the ZooKeeper tool `zkCli.sh`.
> >
> > This is useful in some cases to either verify the content of metadata or
> > to perform cleanup and modification tasks for which there is not an
> > available
> > option through the Pulsar REST APIs.
> >
> > While it's very useful, there are some drawbacks with the
> `zookeeper-shell`
> > as it is today:
> >
> > 1. This is only a ZooKeeper tool (obviously). Since we are adding more
> > metadata backends, we should have a tool that works across all the
> > implementations and presents a single consistent interface.
> >
> > 2. ZooKeeper shell is designed to be an interactive shell and it's not
> > very
> > good when trying to do non-interactive scriptable operations.
> >
> > 3. ZooKeeper is a bit clunky when using it and it requires the user to
> > retype
> > paths many times. The commands are not very intuitive or documented.
> > It's not possible to update z-node with multi-lines content.
> >
> > 4. We cannot easily add functionality or improvements into ZooKeeper
> > shell,
> > since it belongs to a different project and the tool has been
> > stagnating
> > for many years.
> >
> > 5. In cases where the z-nodes content is binary (Protobuf) or
> compressed,
> > there
> > is no easy way to inspect the content from the ZooKeeper shell.
> > Additionally, we can format and colorize JSON content to make it
> > easier to
> > read.
> >
> > 6. The paths used for metadata resources are also often using encodings
> > that
> > make it more difficult to construct on the shell tool.
> >
> > Part of what is described here is in the `pulsar-managed-ledger-admin`
> CLI
> > tool,
> > though that is a Python script that requires additional dependencies that
> > are
> > not typically installed, it only works with ZooKeeper, and it only
> targets
> > accessing the managed ledger metadata.
> >
> > ## Goal
> >
> > Introduce a new Java CLI tool to access, inspect and modify metadata that
> > solves
> > all the issues described above.
> >
> > We would leave the `zookeeper-shell` command for now. In the future, once
> > the
> > new tool is proven, we can consider removing the `zookeeper-shell`
> command.
> >
> >
> > ## Proposed changes
> >
> > Add a new command:
> > ```bash
> > bin/pulsar metadata
> > ```
> >
> > with several subcommands:
> >
> >
> > #### Get
> >
> > Examples:
> > ```bash
> > # General path get
> > $ pulsar metadata get /my-path
> >
> >
> > # Topic metadata
> > $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> > {
> > # Managed ledger metadata
> > }
> >
> > # Namespace get
> > $ pulsar metadata get namespace my-tenant/my-namespace
> > {
> > # Namespace metadata
> > }
> >
> > $ pulsar metadata get ledger 12345
> > {
> > # BK ledger metadata
> > }
> > ```
> >
> > #### Delete
> >
> > Examples:
> > ```bash
> > # General path delete
> > $ pulsar metadata delete /my-path
> >
> > # Topic metadata
> > $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> > ```
> >
> > #### Scan
> >
> > Examples:
> > ```bash
> > $ pulsar metadata scan /my-path
> > /my-path
> > /my-path/1
> > /my-path/2
> > /my-path/3
> > /my-path/3/1
> >
> > $ pulsar metadata scan --values /my-path
> > /my-path
> > {value}
> >
> > /my-path/1
> > {value}
> >
> > /my-path/2
> > {value}
> > ```
> >
> > #### Shell
> >
> > ```bash
> > $ pulsar metadata shell
> > > get topic my-tenant/my-namespace/my-topic
> > {
> > # Managed ledger metadata
> > }
> >
> > > delete topic my-tenant/my-namespace/my-topic
> >
> > > cd /my-path
> > > ls
> > 1
> > 2
> > 3
> > > delete 1 # Delete keys with relative paths
> >
> > ```
> >
> >
> > --
> > Matteo Merli
> > <mm...@apache.org>
> >
>
Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool
Posted by mattison chao <ma...@gmail.com>.
+1
On Wed, 22 Dec 2021 at 07:59, Matteo Merli <mm...@apache.org> wrote:
> This is the voting thread for PIP-123. It will stay open for at least 48h.
>
> https://github.com/apache/pulsar/issues/13346
>
>
> ---------
> ## Motivation
>
> For a very long time, we have included a CLI command to start the ZooKeeper
> shell utility: `pulsar zookeeper-shell`, which is essentially a repackaging
> of the ZooKeeper tool `zkCli.sh`.
>
> This is useful in some cases to either verify the content of metadata or
> to perform cleanup and modification tasks for which there is not an
> available
> option through the Pulsar REST APIs.
>
> While it's very useful, there are some drawbacks with the `zookeeper-shell`
> as it is today:
>
> 1. This is only a ZooKeeper tool (obviously). Since we are adding more
> metadata backends, we should have a tool that works across all the
> implementations and presents a single consistent interface.
>
> 2. ZooKeeper shell is designed to be an interactive shell and it's not
> very
> good when trying to do non-interactive scriptable operations.
>
> 3. ZooKeeper is a bit clunky when using it and it requires the user to
> retype
> paths many times. The commands are not very intuitive or documented.
> It's not possible to update z-node with multi-lines content.
>
> 4. We cannot easily add functionality or improvements into ZooKeeper
> shell,
> since it belongs to a different project and the tool has been
> stagnating
> for many years.
>
> 5. In cases where the z-nodes content is binary (Protobuf) or compressed,
> there
> is no easy way to inspect the content from the ZooKeeper shell.
> Additionally, we can format and colorize JSON content to make it
> easier to
> read.
>
> 6. The paths used for metadata resources are also often using encodings
> that
> make it more difficult to construct on the shell tool.
>
> Part of what is described here is in the `pulsar-managed-ledger-admin` CLI
> tool,
> though that is a Python script that requires additional dependencies that
> are
> not typically installed, it only works with ZooKeeper, and it only targets
> accessing the managed ledger metadata.
>
> ## Goal
>
> Introduce a new Java CLI tool to access, inspect and modify metadata that
> solves
> all the issues described above.
>
> We would leave the `zookeeper-shell` command for now. In the future, once
> the
> new tool is proven, we can consider removing the `zookeeper-shell` command.
>
>
> ## Proposed changes
>
> Add a new command:
> ```bash
> bin/pulsar metadata
> ```
>
> with several subcommands:
>
>
> #### Get
>
> Examples:
> ```bash
> # General path get
> $ pulsar metadata get /my-path
>
>
> # Topic metadata
> $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> {
> # Managed ledger metadata
> }
>
> # Namespace get
> $ pulsar metadata get namespace my-tenant/my-namespace
> {
> # Namespace metadata
> }
>
> $ pulsar metadata get ledger 12345
> {
> # BK ledger metadata
> }
> ```
>
> #### Delete
>
> Examples:
> ```bash
> # General path delete
> $ pulsar metadata delete /my-path
>
> # Topic metadata
> $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> ```
>
> #### Scan
>
> Examples:
> ```bash
> $ pulsar metadata scan /my-path
> /my-path
> /my-path/1
> /my-path/2
> /my-path/3
> /my-path/3/1
>
> $ pulsar metadata scan --values /my-path
> /my-path
> {value}
>
> /my-path/1
> {value}
>
> /my-path/2
> {value}
> ```
>
> #### Shell
>
> ```bash
> $ pulsar metadata shell
> > get topic my-tenant/my-namespace/my-topic
> {
> # Managed ledger metadata
> }
>
> > delete topic my-tenant/my-namespace/my-topic
>
> > cd /my-path
> > ls
> 1
> 2
> 3
> > delete 1 # Delete keys with relative paths
>
> ```
>
>
> --
> Matteo Merli
> <mm...@apache.org>
>
Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool
Posted by Christophe Bornet <bo...@gmail.com>.
+1
Le mer. 22 déc. 2021 à 00:59, Matteo Merli <mm...@apache.org> a écrit :
> This is the voting thread for PIP-123. It will stay open for at least 48h.
>
> https://github.com/apache/pulsar/issues/13346
>
>
> ---------
> ## Motivation
>
> For a very long time, we have included a CLI command to start the ZooKeeper
> shell utility: `pulsar zookeeper-shell`, which is essentially a repackaging
> of the ZooKeeper tool `zkCli.sh`.
>
> This is useful in some cases to either verify the content of metadata or
> to perform cleanup and modification tasks for which there is not an
> available
> option through the Pulsar REST APIs.
>
> While it's very useful, there are some drawbacks with the `zookeeper-shell`
> as it is today:
>
> 1. This is only a ZooKeeper tool (obviously). Since we are adding more
> metadata backends, we should have a tool that works across all the
> implementations and presents a single consistent interface.
>
> 2. ZooKeeper shell is designed to be an interactive shell and it's not
> very
> good when trying to do non-interactive scriptable operations.
>
> 3. ZooKeeper is a bit clunky when using it and it requires the user to
> retype
> paths many times. The commands are not very intuitive or documented.
> It's not possible to update z-node with multi-lines content.
>
> 4. We cannot easily add functionality or improvements into ZooKeeper
> shell,
> since it belongs to a different project and the tool has been
> stagnating
> for many years.
>
> 5. In cases where the z-nodes content is binary (Protobuf) or compressed,
> there
> is no easy way to inspect the content from the ZooKeeper shell.
> Additionally, we can format and colorize JSON content to make it
> easier to
> read.
>
> 6. The paths used for metadata resources are also often using encodings
> that
> make it more difficult to construct on the shell tool.
>
> Part of what is described here is in the `pulsar-managed-ledger-admin` CLI
> tool,
> though that is a Python script that requires additional dependencies that
> are
> not typically installed, it only works with ZooKeeper, and it only targets
> accessing the managed ledger metadata.
>
> ## Goal
>
> Introduce a new Java CLI tool to access, inspect and modify metadata that
> solves
> all the issues described above.
>
> We would leave the `zookeeper-shell` command for now. In the future, once
> the
> new tool is proven, we can consider removing the `zookeeper-shell` command.
>
>
> ## Proposed changes
>
> Add a new command:
> ```bash
> bin/pulsar metadata
> ```
>
> with several subcommands:
>
>
> #### Get
>
> Examples:
> ```bash
> # General path get
> $ pulsar metadata get /my-path
>
>
> # Topic metadata
> $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> {
> # Managed ledger metadata
> }
>
> # Namespace get
> $ pulsar metadata get namespace my-tenant/my-namespace
> {
> # Namespace metadata
> }
>
> $ pulsar metadata get ledger 12345
> {
> # BK ledger metadata
> }
> ```
>
> #### Delete
>
> Examples:
> ```bash
> # General path delete
> $ pulsar metadata delete /my-path
>
> # Topic metadata
> $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> ```
>
> #### Scan
>
> Examples:
> ```bash
> $ pulsar metadata scan /my-path
> /my-path
> /my-path/1
> /my-path/2
> /my-path/3
> /my-path/3/1
>
> $ pulsar metadata scan --values /my-path
> /my-path
> {value}
>
> /my-path/1
> {value}
>
> /my-path/2
> {value}
> ```
>
> #### Shell
>
> ```bash
> $ pulsar metadata shell
> > get topic my-tenant/my-namespace/my-topic
> {
> # Managed ledger metadata
> }
>
> > delete topic my-tenant/my-namespace/my-topic
>
> > cd /my-path
> > ls
> 1
> 2
> 3
> > delete 1 # Delete keys with relative paths
>
> ```
>
>
> --
> Matteo Merli
> <mm...@apache.org>
>
Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool
Posted by Haiting Jiang <ji...@apache.org>.
+1
Thanks,
Haiting
On 2021/12/23 05:38:14 Michael Marshall wrote:
> +1
>
> - Michael
>
> On Wed, Dec 22, 2021 at 6:18 PM Sijie Guo <gu...@gmail.com> wrote:
> >
> > +1
> >
> > On Tue, Dec 21, 2021 at 3:59 PM Matteo Merli <mm...@apache.org> wrote:
> >
> > > This is the voting thread for PIP-123. It will stay open for at least 48h.
> > >
> > > https://github.com/apache/pulsar/issues/13346
> > >
> > >
> > > ---------
> > > ## Motivation
> > >
> > > For a very long time, we have included a CLI command to start the ZooKeeper
> > > shell utility: `pulsar zookeeper-shell`, which is essentially a repackaging
> > > of the ZooKeeper tool `zkCli.sh`.
> > >
> > > This is useful in some cases to either verify the content of metadata or
> > > to perform cleanup and modification tasks for which there is not an
> > > available
> > > option through the Pulsar REST APIs.
> > >
> > > While it's very useful, there are some drawbacks with the `zookeeper-shell`
> > > as it is today:
> > >
> > > 1. This is only a ZooKeeper tool (obviously). Since we are adding more
> > > metadata backends, we should have a tool that works across all the
> > > implementations and presents a single consistent interface.
> > >
> > > 2. ZooKeeper shell is designed to be an interactive shell and it's not
> > > very
> > > good when trying to do non-interactive scriptable operations.
> > >
> > > 3. ZooKeeper is a bit clunky when using it and it requires the user to
> > > retype
> > > paths many times. The commands are not very intuitive or documented.
> > > It's not possible to update z-node with multi-lines content.
> > >
> > > 4. We cannot easily add functionality or improvements into ZooKeeper
> > > shell,
> > > since it belongs to a different project and the tool has been
> > > stagnating
> > > for many years.
> > >
> > > 5. In cases where the z-nodes content is binary (Protobuf) or compressed,
> > > there
> > > is no easy way to inspect the content from the ZooKeeper shell.
> > > Additionally, we can format and colorize JSON content to make it
> > > easier to
> > > read.
> > >
> > > 6. The paths used for metadata resources are also often using encodings
> > > that
> > > make it more difficult to construct on the shell tool.
> > >
> > > Part of what is described here is in the `pulsar-managed-ledger-admin` CLI
> > > tool,
> > > though that is a Python script that requires additional dependencies that
> > > are
> > > not typically installed, it only works with ZooKeeper, and it only targets
> > > accessing the managed ledger metadata.
> > >
> > > ## Goal
> > >
> > > Introduce a new Java CLI tool to access, inspect and modify metadata that
> > > solves
> > > all the issues described above.
> > >
> > > We would leave the `zookeeper-shell` command for now. In the future, once
> > > the
> > > new tool is proven, we can consider removing the `zookeeper-shell` command.
> > >
> > >
> > > ## Proposed changes
> > >
> > > Add a new command:
> > > ```bash
> > > bin/pulsar metadata
> > > ```
> > >
> > > with several subcommands:
> > >
> > >
> > > #### Get
> > >
> > > Examples:
> > > ```bash
> > > # General path get
> > > $ pulsar metadata get /my-path
> > >
> > >
> > > # Topic metadata
> > > $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> > > {
> > > # Managed ledger metadata
> > > }
> > >
> > > # Namespace get
> > > $ pulsar metadata get namespace my-tenant/my-namespace
> > > {
> > > # Namespace metadata
> > > }
> > >
> > > $ pulsar metadata get ledger 12345
> > > {
> > > # BK ledger metadata
> > > }
> > > ```
> > >
> > > #### Delete
> > >
> > > Examples:
> > > ```bash
> > > # General path delete
> > > $ pulsar metadata delete /my-path
> > >
> > > # Topic metadata
> > > $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> > > ```
> > >
> > > #### Scan
> > >
> > > Examples:
> > > ```bash
> > > $ pulsar metadata scan /my-path
> > > /my-path
> > > /my-path/1
> > > /my-path/2
> > > /my-path/3
> > > /my-path/3/1
> > >
> > > $ pulsar metadata scan --values /my-path
> > > /my-path
> > > {value}
> > >
> > > /my-path/1
> > > {value}
> > >
> > > /my-path/2
> > > {value}
> > > ```
> > >
> > > #### Shell
> > >
> > > ```bash
> > > $ pulsar metadata shell
> > > > get topic my-tenant/my-namespace/my-topic
> > > {
> > > # Managed ledger metadata
> > > }
> > >
> > > > delete topic my-tenant/my-namespace/my-topic
> > >
> > > > cd /my-path
> > > > ls
> > > 1
> > > 2
> > > 3
> > > > delete 1 # Delete keys with relative paths
> > >
> > > ```
> > >
> > >
> > > --
> > > Matteo Merli
> > > <mm...@apache.org>
> > >
>
Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool
Posted by Michael Marshall <mm...@apache.org>.
+1
- Michael
On Wed, Dec 22, 2021 at 6:18 PM Sijie Guo <gu...@gmail.com> wrote:
>
> +1
>
> On Tue, Dec 21, 2021 at 3:59 PM Matteo Merli <mm...@apache.org> wrote:
>
> > This is the voting thread for PIP-123. It will stay open for at least 48h.
> >
> > https://github.com/apache/pulsar/issues/13346
> >
> >
> > ---------
> > ## Motivation
> >
> > For a very long time, we have included a CLI command to start the ZooKeeper
> > shell utility: `pulsar zookeeper-shell`, which is essentially a repackaging
> > of the ZooKeeper tool `zkCli.sh`.
> >
> > This is useful in some cases to either verify the content of metadata or
> > to perform cleanup and modification tasks for which there is not an
> > available
> > option through the Pulsar REST APIs.
> >
> > While it's very useful, there are some drawbacks with the `zookeeper-shell`
> > as it is today:
> >
> > 1. This is only a ZooKeeper tool (obviously). Since we are adding more
> > metadata backends, we should have a tool that works across all the
> > implementations and presents a single consistent interface.
> >
> > 2. ZooKeeper shell is designed to be an interactive shell and it's not
> > very
> > good when trying to do non-interactive scriptable operations.
> >
> > 3. ZooKeeper is a bit clunky when using it and it requires the user to
> > retype
> > paths many times. The commands are not very intuitive or documented.
> > It's not possible to update z-node with multi-lines content.
> >
> > 4. We cannot easily add functionality or improvements into ZooKeeper
> > shell,
> > since it belongs to a different project and the tool has been
> > stagnating
> > for many years.
> >
> > 5. In cases where the z-nodes content is binary (Protobuf) or compressed,
> > there
> > is no easy way to inspect the content from the ZooKeeper shell.
> > Additionally, we can format and colorize JSON content to make it
> > easier to
> > read.
> >
> > 6. The paths used for metadata resources are also often using encodings
> > that
> > make it more difficult to construct on the shell tool.
> >
> > Part of what is described here is in the `pulsar-managed-ledger-admin` CLI
> > tool,
> > though that is a Python script that requires additional dependencies that
> > are
> > not typically installed, it only works with ZooKeeper, and it only targets
> > accessing the managed ledger metadata.
> >
> > ## Goal
> >
> > Introduce a new Java CLI tool to access, inspect and modify metadata that
> > solves
> > all the issues described above.
> >
> > We would leave the `zookeeper-shell` command for now. In the future, once
> > the
> > new tool is proven, we can consider removing the `zookeeper-shell` command.
> >
> >
> > ## Proposed changes
> >
> > Add a new command:
> > ```bash
> > bin/pulsar metadata
> > ```
> >
> > with several subcommands:
> >
> >
> > #### Get
> >
> > Examples:
> > ```bash
> > # General path get
> > $ pulsar metadata get /my-path
> >
> >
> > # Topic metadata
> > $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> > {
> > # Managed ledger metadata
> > }
> >
> > # Namespace get
> > $ pulsar metadata get namespace my-tenant/my-namespace
> > {
> > # Namespace metadata
> > }
> >
> > $ pulsar metadata get ledger 12345
> > {
> > # BK ledger metadata
> > }
> > ```
> >
> > #### Delete
> >
> > Examples:
> > ```bash
> > # General path delete
> > $ pulsar metadata delete /my-path
> >
> > # Topic metadata
> > $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> > ```
> >
> > #### Scan
> >
> > Examples:
> > ```bash
> > $ pulsar metadata scan /my-path
> > /my-path
> > /my-path/1
> > /my-path/2
> > /my-path/3
> > /my-path/3/1
> >
> > $ pulsar metadata scan --values /my-path
> > /my-path
> > {value}
> >
> > /my-path/1
> > {value}
> >
> > /my-path/2
> > {value}
> > ```
> >
> > #### Shell
> >
> > ```bash
> > $ pulsar metadata shell
> > > get topic my-tenant/my-namespace/my-topic
> > {
> > # Managed ledger metadata
> > }
> >
> > > delete topic my-tenant/my-namespace/my-topic
> >
> > > cd /my-path
> > > ls
> > 1
> > 2
> > 3
> > > delete 1 # Delete keys with relative paths
> >
> > ```
> >
> >
> > --
> > Matteo Merli
> > <mm...@apache.org>
> >
Re: [VOTE] PIP-123: Introduce Pulsar metadata CLI tool
Posted by Sijie Guo <gu...@gmail.com>.
+1
On Tue, Dec 21, 2021 at 3:59 PM Matteo Merli <mm...@apache.org> wrote:
> This is the voting thread for PIP-123. It will stay open for at least 48h.
>
> https://github.com/apache/pulsar/issues/13346
>
>
> ---------
> ## Motivation
>
> For a very long time, we have included a CLI command to start the ZooKeeper
> shell utility: `pulsar zookeeper-shell`, which is essentially a repackaging
> of the ZooKeeper tool `zkCli.sh`.
>
> This is useful in some cases to either verify the content of metadata or
> to perform cleanup and modification tasks for which there is not an
> available
> option through the Pulsar REST APIs.
>
> While it's very useful, there are some drawbacks with the `zookeeper-shell`
> as it is today:
>
> 1. This is only a ZooKeeper tool (obviously). Since we are adding more
> metadata backends, we should have a tool that works across all the
> implementations and presents a single consistent interface.
>
> 2. ZooKeeper shell is designed to be an interactive shell and it's not
> very
> good when trying to do non-interactive scriptable operations.
>
> 3. ZooKeeper is a bit clunky when using it and it requires the user to
> retype
> paths many times. The commands are not very intuitive or documented.
> It's not possible to update z-node with multi-lines content.
>
> 4. We cannot easily add functionality or improvements into ZooKeeper
> shell,
> since it belongs to a different project and the tool has been
> stagnating
> for many years.
>
> 5. In cases where the z-nodes content is binary (Protobuf) or compressed,
> there
> is no easy way to inspect the content from the ZooKeeper shell.
> Additionally, we can format and colorize JSON content to make it
> easier to
> read.
>
> 6. The paths used for metadata resources are also often using encodings
> that
> make it more difficult to construct on the shell tool.
>
> Part of what is described here is in the `pulsar-managed-ledger-admin` CLI
> tool,
> though that is a Python script that requires additional dependencies that
> are
> not typically installed, it only works with ZooKeeper, and it only targets
> accessing the managed ledger metadata.
>
> ## Goal
>
> Introduce a new Java CLI tool to access, inspect and modify metadata that
> solves
> all the issues described above.
>
> We would leave the `zookeeper-shell` command for now. In the future, once
> the
> new tool is proven, we can consider removing the `zookeeper-shell` command.
>
>
> ## Proposed changes
>
> Add a new command:
> ```bash
> bin/pulsar metadata
> ```
>
> with several subcommands:
>
>
> #### Get
>
> Examples:
> ```bash
> # General path get
> $ pulsar metadata get /my-path
>
>
> # Topic metadata
> $ pulsar metadata get topic my-tenant/my-namespace/my-topic
> {
> # Managed ledger metadata
> }
>
> # Namespace get
> $ pulsar metadata get namespace my-tenant/my-namespace
> {
> # Namespace metadata
> }
>
> $ pulsar metadata get ledger 12345
> {
> # BK ledger metadata
> }
> ```
>
> #### Delete
>
> Examples:
> ```bash
> # General path delete
> $ pulsar metadata delete /my-path
>
> # Topic metadata
> $ pulsar metadata delete topic my-tenant/my-namespace/my-topic
> ```
>
> #### Scan
>
> Examples:
> ```bash
> $ pulsar metadata scan /my-path
> /my-path
> /my-path/1
> /my-path/2
> /my-path/3
> /my-path/3/1
>
> $ pulsar metadata scan --values /my-path
> /my-path
> {value}
>
> /my-path/1
> {value}
>
> /my-path/2
> {value}
> ```
>
> #### Shell
>
> ```bash
> $ pulsar metadata shell
> > get topic my-tenant/my-namespace/my-topic
> {
> # Managed ledger metadata
> }
>
> > delete topic my-tenant/my-namespace/my-topic
>
> > cd /my-path
> > ls
> 1
> 2
> 3
> > delete 1 # Delete keys with relative paths
>
> ```
>
>
> --
> Matteo Merli
> <mm...@apache.org>
>