You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pulsar.apache.org by Nicolò Boschi <bo...@gmail.com> on 2022/06/28 08:06:01 UTC

[DISCUSS] PIP-181: Pulsar Shell

Hi all,

I opened a new PIP about Pulsar CLI tools.
Looking forward to seeing comments and suggestions.

PIP: https://github.com/apache/pulsar/issues/16250

I posted a short video that shows how the new tool will work:
https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4

## Motivation

Currently Pulsar comes with a couple of utility scripts with the goal of
managing an existing cluster, test behaviours and verify performances:
these tools are available as SH script inside the `bin` directory.
The `pulsar-admin` is the CLI tool supposed to help users and operators to
configure the system, operate over policies, install functions and much
else.

This proposal basically aims to solve two different problems:

1. `pulsar-admin` is terribly slow. Every time the script is triggered, a
new JVM process is spawned. The JVM process creation is heavy and most of
the time is spent by the JVM initialization process. A very common use case
for cluster operators is to create scripts with several commands with the
goal of initialize the cluster, initialize a specific tenant (namespaces,
topics, policies, functions..); in this case, one JVM is initialized for
each scripts leads to waste of time and resources.

2. User experience. The current design of the Pulsar CLIs can be improved.
There are a couple of aspects that may be annoying for a user and can
discourage a user to use Pulsar.
1. Poking around available commands and options in a CLI tool
(`pulsar-admin` for instance, but it's the same for `pulsar-perf` and
`pulsar-client`) is slow and hard. In order to discover commands and
options you need to use `-h` option and, since the performance issue
pointed at 1., it can be annoying and time-consuming. Autocomplete feature
could be a real game-changer in this context.
2. Different CLI tools. There are a couple of different shell scripts.
They have different goals and it's okay to keep them separated. However,
they raise a barrier for a non Pulsar expert that doesn't have a convenient
entry-point.

## Goal

Address all the issues in the previous section with a single solution.

## API Changes

A new shell script `bin/pulsar-shell` will be introduced. `bin/pulsar
shell` could be a valid alternative but it's not findable for a newbie user
since no direct file exists.

## Implementation

### Concepts
The new script `pulsar-shell` will differ from the existing for the
following reasons:

1. It's a shell. When you start it, it will wait for commands to be
executed. After the command has been executed, despite its result, the
shell session will not be destroyed and it will wait for another command.
2. Unifies all the CLI scripts. In `pulsar-shell` you'll be able to run all
the existing CLI commands. This will be done in a way that when a new
command/option is added, the pulsar-shell will be updated accordingly.
3. It comes with sophisticated autocompletion and command history to highly
improve the UX.
4. Performance. Since JVM is initiated once, it will gain on performance
thanks to the JVM warmup and internal libraries bootstraps.
5. It will accept a file or a list of commands (parameter and stdin) to
start a shell, run the commands and close the shell. We'll call it the
`non-interactive` mode and it will ease the cluster operations automations.

Note that existing tools will not be removed/changed.

### Implementation

The shell implementation will be developed in Java, using a well-known
library called [JLine 3](https://github.com/jline/jline3) for the shell
support.
There will be a new main class that will extends the existing class tools
(e.g. `PulsarAdminTool`)

#### Configuration
The configuration file taken by default will be `client.conf`, like current
CLI tools. The env setup will be exactly the same as for `pulsar-admin` and
`pulsar-client`.

#### Autocompletion
JLine3 has great support for autocompletion. The shell java class will
translate current `JCommander` tools to the JLine3 completion API. This
will ensure that all APIs will be up-to-date and covered.

#### History
JLine3 has built-in support for history, both in-memory and persisted (on
local FS). By default the history will be persisted in the user home
directory. This feature can be turned off for security concerns with a
configuration property.

#### List of tools
The proposal is to get the following tools:
- bin/pulsar-admin
- bin/pulsar-client
- bin/pulsar-perf

## Rejected Alternatives

- Create a shell mode for `pulsar-admin`. This won't be a flexible solution
because then we may need the same for other tools. Also it doesn't cover
the CLI unification part.
- Create a brand-new shell mode. Not compatible with current CLI and
another tool to maintain.
- Use another shell library. There are a couple of valid alternatives to
JLine3 but it's wide-spread and the autocompletion API is much more
flexible than others since it's not done with annotations (the shell needs
to translate current JCommander definitions from existing tools)

Nicolò Boschi

Re: [DISCUSS] PIP-181: Pulsar Shell

Posted by Nicolò Boschi <bo...@gmail.com>.

>
> Why are there 2 PIPs being discussed as  PIP-181? There's another
> discussion on dev with


Those two PIP are not related, we just had a PIP number conflict. Qiang
gently moved it to PIP-182

Nicolò Boschi


Il giorno dom 3 lug 2022 alle ore 07:50 Joe F <jo...@gmail.com> ha
scritto:

> Why are there 2 PIPs being discussed as  PIP-181? There's another
> discussion on dev with
>
> [DISCUSS] PIP-181: Provide new load balance placement strategy
> implementation for ModularLoadManagerStrategy
>
> On Sat, Jul 2, 2022 at 1:13 AM Haiting Jiang <ji...@apache.org>
> wrote:
>
> > +1, Great feature.
> >
> > Thanks,
> > Haiting
> >
> > On 2022/07/01 16:56:08 Nicolò Boschi wrote:
> > > Yes it's not a good idea to add such new features to active release
> > > branches.
> > > However, the shell will work with older cluster versions (as soon as
> the
> > > java client is compatible). Also you will be able to download the shell
> > > tarball (that just contains the minimal stuff needed to run the shell).
> > >
> > > Nicolò Boschi
> > >
> > >
> > > Il giorno ven 1 lug 2022 alle ore 18:44 tison <wa...@gmail.com>
> ha
> > > scritto:
> > >
> > > > Hi Xiaoyu,
> > > >
> > > > IIUC patch release must not include new features but only bug fixes.
> > > >
> > > > Best,
> > > > tison.
> > > >
> > > >
> > > > Anon Hxy <an...@gmail.com> 于2022年7月1日周五 23:59写道：
> > > >
> > > > > Hi Nicolò Boschi,
> > > > >
> > > > > The Pulsar Shell is really cool and I like it. And also I have a
> > > > question:
> > > > >
> > > > > >  I'd like to target this feature for 2.11
> > > > >
> > > > > Will it be possible to use Pulsar Shell in the legacy version of
> > Pulsar,
> > > > in
> > > > > other words, could we cherry-pick the PR to another branch easily?
> > > > >
> > > > > Thanks,
> > > > > Xiaoyu Hou
> > > > >
> > > > > Nicolò Boschi <bo...@gmail.com> 于2022年7月1日周五 20:40写道：
> > > > >
> > > > > > I updated the issue linking all the implementations pull
> requests.
> > > > > > If you open the issue you will be able to see 2 videos that
> better
> > > > > explain
> > > > > > how the tool works.
> > > > > > https://github.com/apache/pulsar/issues/16250
> > > > > >
> > > > > > Let me know if you have any feedbacks
> > > > > >
> > > > > > I'd like to target this feature for 2.11
> > > > > >
> > > > > > BR,
> > > > > > Nicolò Boschi
> > > > > >
> > > > > >
> > > > > > Il giorno mar 28 giu 2022 alle ore 10:06 Nicolò Boschi <
> > > > > > boschi1997@gmail.com>
> > > > > > ha scritto:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I opened a new PIP about Pulsar CLI tools.
> > > > > > > Looking forward to seeing comments and suggestions.
> > > > > > >
> > > > > > > PIP: https://github.com/apache/pulsar/issues/16250
> > > > > > >
> > > > > > > I posted a short video that shows how the new tool will work:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ## Motivation
> > > > > > >
> > > > > > > Currently Pulsar comes with a couple of utility scripts with
> the
> > goal
> > > > > of
> > > > > > > managing an existing cluster, test behaviours and verify
> > > > performances:
> > > > > > > these tools are available as SH script inside the `bin`
> > directory.
> > > > > > > The `pulsar-admin` is the CLI tool supposed to help users and
> > > > operators
> > > > > > to
> > > > > > > configure the system, operate over policies, install functions
> > and
> > > > much
> > > > > > > else.
> > > > > > >
> > > > > > > This proposal basically aims to solve two different problems:
> > > > > > >
> > > > > > > 1. `pulsar-admin` is terribly slow. Every time the script is
> > > > > triggered, a
> > > > > > > new JVM process is spawned. The JVM process creation is heavy
> and
> > > > most
> > > > > of
> > > > > > > the time is spent by the JVM initialization process. A very
> > common
> > > > use
> > > > > > case
> > > > > > > for cluster operators is to create scripts with several
> commands
> > with
> > > > > the
> > > > > > > goal of initialize the cluster, initialize a specific tenant
> > > > > (namespaces,
> > > > > > > topics, policies, functions..); in this case, one JVM is
> > initialized
> > > > > for
> > > > > > > each scripts leads to waste of time and resources.
> > > > > > >
> > > > > > > 2. User experience. The current design of the Pulsar CLIs can
> be
> > > > > > improved.
> > > > > > > There are a couple of aspects that may be annoying for a user
> > and can
> > > > > > > discourage a user to use Pulsar.
> > > > > > >     1. Poking around available commands and options in a CLI
> tool
> > > > > > > (`pulsar-admin` for instance, but it's the same for
> > `pulsar-perf` and
> > > > > > > `pulsar-client`) is slow and hard. In order to discover
> commands
> > and
> > > > > > > options you need to use `-h` option and, since the performance
> > issue
> > > > > > > pointed at 1., it can be annoying and time-consuming.
> > Autocomplete
> > > > > > feature
> > > > > > > could be a real game-changer in this context.
> > > > > > >     2. Different CLI tools. There are a couple of different
> shell
> > > > > > scripts.
> > > > > > > They have different goals and it's okay to keep them separated.
> > > > > However,
> > > > > > > they raise a barrier for a non Pulsar expert that doesn't have
> a
> > > > > > convenient
> > > > > > > entry-point.
> > > > > > >
> > > > > > > ## Goal
> > > > > > >
> > > > > > > Address all the issues in the previous section with a single
> > > > solution.
> > > > > > >
> > > > > > > ## API Changes
> > > > > > >
> > > > > > > A new shell script `bin/pulsar-shell` will be introduced.
> > `bin/pulsar
> > > > > > > shell` could be a valid alternative but it's not findable for a
> > > > newbie
> > > > > > user
> > > > > > > since no direct file exists.
> > > > > > >
> > > > > > > ## Implementation
> > > > > > >
> > > > > > > ### Concepts
> > > > > > > The new script `pulsar-shell` will differ from the existing for
> > the
> > > > > > > following reasons:
> > > > > > >
> > > > > > > 1. It's a shell. When you start it, it will wait for commands
> to
> > be
> > > > > > > executed. After the command has been executed, despite its
> > result,
> > > > the
> > > > > > > shell session will not be destroyed and it will wait for
> another
> > > > > command.
> > > > > > > 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be
> able
> > to
> > > > run
> > > > > > > all the existing CLI commands. This will be done in a way that
> > when a
> > > > > new
> > > > > > > command/option is added, the pulsar-shell will be updated
> > > > accordingly.
> > > > > > > 3. It comes with sophisticated autocompletion and command
> > history to
> > > > > > > highly improve the UX.
> > > > > > > 4. Performance. Since JVM is initiated once, it will gain on
> > > > > performance
> > > > > > > thanks to the JVM warmup and internal libraries bootstraps.
> > > > > > > 5. It will accept a file or a list of commands (parameter and
> > stdin)
> > > > to
> > > > > > > start a shell, run the commands and close the shell. We'll call
> > it
> > > > the
> > > > > > > `non-interactive` mode and it will ease the cluster operations
> > > > > > automations.
> > > > > > >
> > > > > > > Note that existing tools will not be removed/changed.
> > > > > > >
> > > > > > > ### Implementation
> > > > > > >
> > > > > > > The shell implementation will be developed in Java, using a
> > > > well-known
> > > > > > > library called [JLine 3](https://github.com/jline/jline3) for
> > the
> > > > > shell
> > > > > > > support.
> > > > > > > There will be a new main class that will extends the existing
> > class
> > > > > tools
> > > > > > > (e.g. `PulsarAdminTool`)
> > > > > > >
> > > > > > > #### Configuration
> > > > > > > The configuration file taken by default will be `client.conf`,
> > like
> > > > > > > current CLI tools. The env setup will be exactly the same as
> for
> > > > > > > `pulsar-admin` and `pulsar-client`.
> > > > > > >
> > > > > > > #### Autocompletion
> > > > > > > JLine3 has great support for autocompletion. The shell java
> class
> > > > will
> > > > > > > translate current `JCommander` tools to the JLine3 completion
> > API.
> > > > This
> > > > > > > will ensure that all APIs will be up-to-date and covered.
> > > > > > >
> > > > > > > #### History
> > > > > > > JLine3 has built-in support for history, both in-memory and
> > persisted
> > > > > (on
> > > > > > > local FS). By default the history will be persisted in the user
> > home
> > > > > > > directory. This feature can be turned off for security concerns
> > with
> > > > a
> > > > > > > configuration property.
> > > > > > >
> > > > > > > #### List of tools
> > > > > > > The proposal is to get the following tools:
> > > > > > > - bin/pulsar-admin
> > > > > > > - bin/pulsar-client
> > > > > > > - bin/pulsar-perf
> > > > > > >
> > > > > > >
> > > > > > > ## Rejected Alternatives
> > > > > > >
> > > > > > > - Create a shell mode for `pulsar-admin`. This won't be a
> > flexible
> > > > > > > solution because then we may need the same for other tools.
> Also
> > it
> > > > > > doesn't
> > > > > > > cover the CLI unification part.
> > > > > > > - Create a brand-new shell mode. Not compatible with current
> CLI
> > and
> > > > > > > another tool to maintain.
> > > > > > > - Use another shell library. There are a couple of valid
> > alternatives
> > > > > to
> > > > > > > JLine3 but it's wide-spread and the autocompletion API is much
> > more
> > > > > > > flexible than others since it's not done with annotations (the
> > shell
> > > > > > needs
> > > > > > > to translate current JCommander definitions from existing
> tools)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Nicolò Boschi
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] PIP-181: Pulsar Shell

Posted by Joe F <jo...@gmail.com>.

Why are there 2 PIPs being discussed as  PIP-181? There's another
discussion on dev with

[DISCUSS] PIP-181: Provide new load balance placement strategy
implementation for ModularLoadManagerStrategy

On Sat, Jul 2, 2022 at 1:13 AM Haiting Jiang <ji...@apache.org>
wrote:

> +1, Great feature.
>
> Thanks,
> Haiting
>
> On 2022/07/01 16:56:08 Nicolò Boschi wrote:
> > Yes it's not a good idea to add such new features to active release
> > branches.
> > However, the shell will work with older cluster versions (as soon as the
> > java client is compatible). Also you will be able to download the shell
> > tarball (that just contains the minimal stuff needed to run the shell).
> >
> > Nicolò Boschi
> >
> >
> > Il giorno ven 1 lug 2022 alle ore 18:44 tison <wa...@gmail.com> ha
> > scritto:
> >
> > > Hi Xiaoyu,
> > >
> > > IIUC patch release must not include new features but only bug fixes.
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Anon Hxy <an...@gmail.com> 于2022年7月1日周五 23:59写道：
> > >
> > > > Hi Nicolò Boschi,
> > > >
> > > > The Pulsar Shell is really cool and I like it. And also I have a
> > > question:
> > > >
> > > > >  I'd like to target this feature for 2.11
> > > >
> > > > Will it be possible to use Pulsar Shell in the legacy version of
> Pulsar,
> > > in
> > > > other words, could we cherry-pick the PR to another branch easily?
> > > >
> > > > Thanks,
> > > > Xiaoyu Hou
> > > >
> > > > Nicolò Boschi <bo...@gmail.com> 于2022年7月1日周五 20:40写道：
> > > >
> > > > > I updated the issue linking all the implementations pull requests.
> > > > > If you open the issue you will be able to see 2 videos that better
> > > > explain
> > > > > how the tool works.
> > > > > https://github.com/apache/pulsar/issues/16250
> > > > >
> > > > > Let me know if you have any feedbacks
> > > > >
> > > > > I'd like to target this feature for 2.11
> > > > >
> > > > > BR,
> > > > > Nicolò Boschi
> > > > >
> > > > >
> > > > > Il giorno mar 28 giu 2022 alle ore 10:06 Nicolò Boschi <
> > > > > boschi1997@gmail.com>
> > > > > ha scritto:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I opened a new PIP about Pulsar CLI tools.
> > > > > > Looking forward to seeing comments and suggestions.
> > > > > >
> > > > > > PIP: https://github.com/apache/pulsar/issues/16250
> > > > > >
> > > > > > I posted a short video that shows how the new tool will work:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ## Motivation
> > > > > >
> > > > > > Currently Pulsar comes with a couple of utility scripts with the
> goal
> > > > of
> > > > > > managing an existing cluster, test behaviours and verify
> > > performances:
> > > > > > these tools are available as SH script inside the `bin`
> directory.
> > > > > > The `pulsar-admin` is the CLI tool supposed to help users and
> > > operators
> > > > > to
> > > > > > configure the system, operate over policies, install functions
> and
> > > much
> > > > > > else.
> > > > > >
> > > > > > This proposal basically aims to solve two different problems:
> > > > > >
> > > > > > 1. `pulsar-admin` is terribly slow. Every time the script is
> > > > triggered, a
> > > > > > new JVM process is spawned. The JVM process creation is heavy and
> > > most
> > > > of
> > > > > > the time is spent by the JVM initialization process. A very
> common
> > > use
> > > > > case
> > > > > > for cluster operators is to create scripts with several commands
> with
> > > > the
> > > > > > goal of initialize the cluster, initialize a specific tenant
> > > > (namespaces,
> > > > > > topics, policies, functions..); in this case, one JVM is
> initialized
> > > > for
> > > > > > each scripts leads to waste of time and resources.
> > > > > >
> > > > > > 2. User experience. The current design of the Pulsar CLIs can be
> > > > > improved.
> > > > > > There are a couple of aspects that may be annoying for a user
> and can
> > > > > > discourage a user to use Pulsar.
> > > > > >     1. Poking around available commands and options in a CLI tool
> > > > > > (`pulsar-admin` for instance, but it's the same for
> `pulsar-perf` and
> > > > > > `pulsar-client`) is slow and hard. In order to discover commands
> and
> > > > > > options you need to use `-h` option and, since the performance
> issue
> > > > > > pointed at 1., it can be annoying and time-consuming.
> Autocomplete
> > > > > feature
> > > > > > could be a real game-changer in this context.
> > > > > >     2. Different CLI tools. There are a couple of different shell
> > > > > scripts.
> > > > > > They have different goals and it's okay to keep them separated.
> > > > However,
> > > > > > they raise a barrier for a non Pulsar expert that doesn't have a
> > > > > convenient
> > > > > > entry-point.
> > > > > >
> > > > > > ## Goal
> > > > > >
> > > > > > Address all the issues in the previous section with a single
> > > solution.
> > > > > >
> > > > > > ## API Changes
> > > > > >
> > > > > > A new shell script `bin/pulsar-shell` will be introduced.
> `bin/pulsar
> > > > > > shell` could be a valid alternative but it's not findable for a
> > > newbie
> > > > > user
> > > > > > since no direct file exists.
> > > > > >
> > > > > > ## Implementation
> > > > > >
> > > > > > ### Concepts
> > > > > > The new script `pulsar-shell` will differ from the existing for
> the
> > > > > > following reasons:
> > > > > >
> > > > > > 1. It's a shell. When you start it, it will wait for commands to
> be
> > > > > > executed. After the command has been executed, despite its
> result,
> > > the
> > > > > > shell session will not be destroyed and it will wait for another
> > > > command.
> > > > > > 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be able
> to
> > > run
> > > > > > all the existing CLI commands. This will be done in a way that
> when a
> > > > new
> > > > > > command/option is added, the pulsar-shell will be updated
> > > accordingly.
> > > > > > 3. It comes with sophisticated autocompletion and command
> history to
> > > > > > highly improve the UX.
> > > > > > 4. Performance. Since JVM is initiated once, it will gain on
> > > > performance
> > > > > > thanks to the JVM warmup and internal libraries bootstraps.
> > > > > > 5. It will accept a file or a list of commands (parameter and
> stdin)
> > > to
> > > > > > start a shell, run the commands and close the shell. We'll call
> it
> > > the
> > > > > > `non-interactive` mode and it will ease the cluster operations
> > > > > automations.
> > > > > >
> > > > > > Note that existing tools will not be removed/changed.
> > > > > >
> > > > > > ### Implementation
> > > > > >
> > > > > > The shell implementation will be developed in Java, using a
> > > well-known
> > > > > > library called [JLine 3](https://github.com/jline/jline3) for
> the
> > > > shell
> > > > > > support.
> > > > > > There will be a new main class that will extends the existing
> class
> > > > tools
> > > > > > (e.g. `PulsarAdminTool`)
> > > > > >
> > > > > > #### Configuration
> > > > > > The configuration file taken by default will be `client.conf`,
> like
> > > > > > current CLI tools. The env setup will be exactly the same as for
> > > > > > `pulsar-admin` and `pulsar-client`.
> > > > > >
> > > > > > #### Autocompletion
> > > > > > JLine3 has great support for autocompletion. The shell java class
> > > will
> > > > > > translate current `JCommander` tools to the JLine3 completion
> API.
> > > This
> > > > > > will ensure that all APIs will be up-to-date and covered.
> > > > > >
> > > > > > #### History
> > > > > > JLine3 has built-in support for history, both in-memory and
> persisted
> > > > (on
> > > > > > local FS). By default the history will be persisted in the user
> home
> > > > > > directory. This feature can be turned off for security concerns
> with
> > > a
> > > > > > configuration property.
> > > > > >
> > > > > > #### List of tools
> > > > > > The proposal is to get the following tools:
> > > > > > - bin/pulsar-admin
> > > > > > - bin/pulsar-client
> > > > > > - bin/pulsar-perf
> > > > > >
> > > > > >
> > > > > > ## Rejected Alternatives
> > > > > >
> > > > > > - Create a shell mode for `pulsar-admin`. This won't be a
> flexible
> > > > > > solution because then we may need the same for other tools. Also
> it
> > > > > doesn't
> > > > > > cover the CLI unification part.
> > > > > > - Create a brand-new shell mode. Not compatible with current CLI
> and
> > > > > > another tool to maintain.
> > > > > > - Use another shell library. There are a couple of valid
> alternatives
> > > > to
> > > > > > JLine3 but it's wide-spread and the autocompletion API is much
> more
> > > > > > flexible than others since it's not done with annotations (the
> shell
> > > > > needs
> > > > > > to translate current JCommander definitions from existing tools)
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Nicolò Boschi
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] PIP-181: Pulsar Shell

Posted by Haiting Jiang <ji...@apache.org>.

+1, Great feature.

Thanks,
Haiting

On 2022/07/01 16:56:08 Nicolò Boschi wrote:
> Yes it's not a good idea to add such new features to active release
> branches.
> However, the shell will work with older cluster versions (as soon as the
> java client is compatible). Also you will be able to download the shell
> tarball (that just contains the minimal stuff needed to run the shell).
> 
> Nicolò Boschi
> 
> 
> Il giorno ven 1 lug 2022 alle ore 18:44 tison <wa...@gmail.com> ha
> scritto:
> 
> > Hi Xiaoyu,
> >
> > IIUC patch release must not include new features but only bug fixes.
> >
> > Best,
> > tison.
> >
> >
> > Anon Hxy <an...@gmail.com> 于2022年7月1日周五 23:59写道：
> >
> > > Hi Nicolò Boschi,
> > >
> > > The Pulsar Shell is really cool and I like it. And also I have a
> > question:
> > >
> > > >  I'd like to target this feature for 2.11
> > >
> > > Will it be possible to use Pulsar Shell in the legacy version of Pulsar,
> > in
> > > other words, could we cherry-pick the PR to another branch easily?
> > >
> > > Thanks,
> > > Xiaoyu Hou
> > >
> > > Nicolò Boschi <bo...@gmail.com> 于2022年7月1日周五 20:40写道：
> > >
> > > > I updated the issue linking all the implementations pull requests.
> > > > If you open the issue you will be able to see 2 videos that better
> > > explain
> > > > how the tool works.
> > > > https://github.com/apache/pulsar/issues/16250
> > > >
> > > > Let me know if you have any feedbacks
> > > >
> > > > I'd like to target this feature for 2.11
> > > >
> > > > BR,
> > > > Nicolò Boschi
> > > >
> > > >
> > > > Il giorno mar 28 giu 2022 alle ore 10:06 Nicolò Boschi <
> > > > boschi1997@gmail.com>
> > > > ha scritto:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I opened a new PIP about Pulsar CLI tools.
> > > > > Looking forward to seeing comments and suggestions.
> > > > >
> > > > > PIP: https://github.com/apache/pulsar/issues/16250
> > > > >
> > > > > I posted a short video that shows how the new tool will work:
> > > > >
> > > > >
> > > >
> > >
> > https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ## Motivation
> > > > >
> > > > > Currently Pulsar comes with a couple of utility scripts with the goal
> > > of
> > > > > managing an existing cluster, test behaviours and verify
> > performances:
> > > > > these tools are available as SH script inside the `bin` directory.
> > > > > The `pulsar-admin` is the CLI tool supposed to help users and
> > operators
> > > > to
> > > > > configure the system, operate over policies, install functions and
> > much
> > > > > else.
> > > > >
> > > > > This proposal basically aims to solve two different problems:
> > > > >
> > > > > 1. `pulsar-admin` is terribly slow. Every time the script is
> > > triggered, a
> > > > > new JVM process is spawned. The JVM process creation is heavy and
> > most
> > > of
> > > > > the time is spent by the JVM initialization process. A very common
> > use
> > > > case
> > > > > for cluster operators is to create scripts with several commands with
> > > the
> > > > > goal of initialize the cluster, initialize a specific tenant
> > > (namespaces,
> > > > > topics, policies, functions..); in this case, one JVM is initialized
> > > for
> > > > > each scripts leads to waste of time and resources.
> > > > >
> > > > > 2. User experience. The current design of the Pulsar CLIs can be
> > > > improved.
> > > > > There are a couple of aspects that may be annoying for a user and can
> > > > > discourage a user to use Pulsar.
> > > > >     1. Poking around available commands and options in a CLI tool
> > > > > (`pulsar-admin` for instance, but it's the same for `pulsar-perf` and
> > > > > `pulsar-client`) is slow and hard. In order to discover commands and
> > > > > options you need to use `-h` option and, since the performance issue
> > > > > pointed at 1., it can be annoying and time-consuming. Autocomplete
> > > > feature
> > > > > could be a real game-changer in this context.
> > > > >     2. Different CLI tools. There are a couple of different shell
> > > > scripts.
> > > > > They have different goals and it's okay to keep them separated.
> > > However,
> > > > > they raise a barrier for a non Pulsar expert that doesn't have a
> > > > convenient
> > > > > entry-point.
> > > > >
> > > > > ## Goal
> > > > >
> > > > > Address all the issues in the previous section with a single
> > solution.
> > > > >
> > > > > ## API Changes
> > > > >
> > > > > A new shell script `bin/pulsar-shell` will be introduced. `bin/pulsar
> > > > > shell` could be a valid alternative but it's not findable for a
> > newbie
> > > > user
> > > > > since no direct file exists.
> > > > >
> > > > > ## Implementation
> > > > >
> > > > > ### Concepts
> > > > > The new script `pulsar-shell` will differ from the existing for the
> > > > > following reasons:
> > > > >
> > > > > 1. It's a shell. When you start it, it will wait for commands to be
> > > > > executed. After the command has been executed, despite its result,
> > the
> > > > > shell session will not be destroyed and it will wait for another
> > > command.
> > > > > 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be able to
> > run
> > > > > all the existing CLI commands. This will be done in a way that when a
> > > new
> > > > > command/option is added, the pulsar-shell will be updated
> > accordingly.
> > > > > 3. It comes with sophisticated autocompletion and command history to
> > > > > highly improve the UX.
> > > > > 4. Performance. Since JVM is initiated once, it will gain on
> > > performance
> > > > > thanks to the JVM warmup and internal libraries bootstraps.
> > > > > 5. It will accept a file or a list of commands (parameter and stdin)
> > to
> > > > > start a shell, run the commands and close the shell. We'll call it
> > the
> > > > > `non-interactive` mode and it will ease the cluster operations
> > > > automations.
> > > > >
> > > > > Note that existing tools will not be removed/changed.
> > > > >
> > > > > ### Implementation
> > > > >
> > > > > The shell implementation will be developed in Java, using a
> > well-known
> > > > > library called [JLine 3](https://github.com/jline/jline3) for the
> > > shell
> > > > > support.
> > > > > There will be a new main class that will extends the existing class
> > > tools
> > > > > (e.g. `PulsarAdminTool`)
> > > > >
> > > > > #### Configuration
> > > > > The configuration file taken by default will be `client.conf`, like
> > > > > current CLI tools. The env setup will be exactly the same as for
> > > > > `pulsar-admin` and `pulsar-client`.
> > > > >
> > > > > #### Autocompletion
> > > > > JLine3 has great support for autocompletion. The shell java class
> > will
> > > > > translate current `JCommander` tools to the JLine3 completion API.
> > This
> > > > > will ensure that all APIs will be up-to-date and covered.
> > > > >
> > > > > #### History
> > > > > JLine3 has built-in support for history, both in-memory and persisted
> > > (on
> > > > > local FS). By default the history will be persisted in the user home
> > > > > directory. This feature can be turned off for security concerns with
> > a
> > > > > configuration property.
> > > > >
> > > > > #### List of tools
> > > > > The proposal is to get the following tools:
> > > > > - bin/pulsar-admin
> > > > > - bin/pulsar-client
> > > > > - bin/pulsar-perf
> > > > >
> > > > >
> > > > > ## Rejected Alternatives
> > > > >
> > > > > - Create a shell mode for `pulsar-admin`. This won't be a flexible
> > > > > solution because then we may need the same for other tools. Also it
> > > > doesn't
> > > > > cover the CLI unification part.
> > > > > - Create a brand-new shell mode. Not compatible with current CLI and
> > > > > another tool to maintain.
> > > > > - Use another shell library. There are a couple of valid alternatives
> > > to
> > > > > JLine3 but it's wide-spread and the autocompletion API is much more
> > > > > flexible than others since it's not done with annotations (the shell
> > > > needs
> > > > > to translate current JCommander definitions from existing tools)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Nicolò Boschi
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] PIP-181: Pulsar Shell

Posted by Nicolò Boschi <bo...@gmail.com>.

Yes it's not a good idea to add such new features to active release
branches.
However, the shell will work with older cluster versions (as soon as the
java client is compatible). Also you will be able to download the shell
tarball (that just contains the minimal stuff needed to run the shell).

Nicolò Boschi


Il giorno ven 1 lug 2022 alle ore 18:44 tison <wa...@gmail.com> ha
scritto:

> Hi Xiaoyu,
>
> IIUC patch release must not include new features but only bug fixes.
>
> Best,
> tison.
>
>
> Anon Hxy <an...@gmail.com> 于2022年7月1日周五 23:59写道：
>
> > Hi Nicolò Boschi,
> >
> > The Pulsar Shell is really cool and I like it. And also I have a
> question:
> >
> > >  I'd like to target this feature for 2.11
> >
> > Will it be possible to use Pulsar Shell in the legacy version of Pulsar,
> in
> > other words, could we cherry-pick the PR to another branch easily?
> >
> > Thanks,
> > Xiaoyu Hou
> >
> > Nicolò Boschi <bo...@gmail.com> 于2022年7月1日周五 20:40写道：
> >
> > > I updated the issue linking all the implementations pull requests.
> > > If you open the issue you will be able to see 2 videos that better
> > explain
> > > how the tool works.
> > > https://github.com/apache/pulsar/issues/16250
> > >
> > > Let me know if you have any feedbacks
> > >
> > > I'd like to target this feature for 2.11
> > >
> > > BR,
> > > Nicolò Boschi
> > >
> > >
> > > Il giorno mar 28 giu 2022 alle ore 10:06 Nicolò Boschi <
> > > boschi1997@gmail.com>
> > > ha scritto:
> > >
> > > > Hi all,
> > > >
> > > > I opened a new PIP about Pulsar CLI tools.
> > > > Looking forward to seeing comments and suggestions.
> > > >
> > > > PIP: https://github.com/apache/pulsar/issues/16250
> > > >
> > > > I posted a short video that shows how the new tool will work:
> > > >
> > > >
> > >
> >
> https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4
> > > >
> > > >
> > > >
> > > >
> > > > ## Motivation
> > > >
> > > > Currently Pulsar comes with a couple of utility scripts with the goal
> > of
> > > > managing an existing cluster, test behaviours and verify
> performances:
> > > > these tools are available as SH script inside the `bin` directory.
> > > > The `pulsar-admin` is the CLI tool supposed to help users and
> operators
> > > to
> > > > configure the system, operate over policies, install functions and
> much
> > > > else.
> > > >
> > > > This proposal basically aims to solve two different problems:
> > > >
> > > > 1. `pulsar-admin` is terribly slow. Every time the script is
> > triggered, a
> > > > new JVM process is spawned. The JVM process creation is heavy and
> most
> > of
> > > > the time is spent by the JVM initialization process. A very common
> use
> > > case
> > > > for cluster operators is to create scripts with several commands with
> > the
> > > > goal of initialize the cluster, initialize a specific tenant
> > (namespaces,
> > > > topics, policies, functions..); in this case, one JVM is initialized
> > for
> > > > each scripts leads to waste of time and resources.
> > > >
> > > > 2. User experience. The current design of the Pulsar CLIs can be
> > > improved.
> > > > There are a couple of aspects that may be annoying for a user and can
> > > > discourage a user to use Pulsar.
> > > >     1. Poking around available commands and options in a CLI tool
> > > > (`pulsar-admin` for instance, but it's the same for `pulsar-perf` and
> > > > `pulsar-client`) is slow and hard. In order to discover commands and
> > > > options you need to use `-h` option and, since the performance issue
> > > > pointed at 1., it can be annoying and time-consuming. Autocomplete
> > > feature
> > > > could be a real game-changer in this context.
> > > >     2. Different CLI tools. There are a couple of different shell
> > > scripts.
> > > > They have different goals and it's okay to keep them separated.
> > However,
> > > > they raise a barrier for a non Pulsar expert that doesn't have a
> > > convenient
> > > > entry-point.
> > > >
> > > > ## Goal
> > > >
> > > > Address all the issues in the previous section with a single
> solution.
> > > >
> > > > ## API Changes
> > > >
> > > > A new shell script `bin/pulsar-shell` will be introduced. `bin/pulsar
> > > > shell` could be a valid alternative but it's not findable for a
> newbie
> > > user
> > > > since no direct file exists.
> > > >
> > > > ## Implementation
> > > >
> > > > ### Concepts
> > > > The new script `pulsar-shell` will differ from the existing for the
> > > > following reasons:
> > > >
> > > > 1. It's a shell. When you start it, it will wait for commands to be
> > > > executed. After the command has been executed, despite its result,
> the
> > > > shell session will not be destroyed and it will wait for another
> > command.
> > > > 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be able to
> run
> > > > all the existing CLI commands. This will be done in a way that when a
> > new
> > > > command/option is added, the pulsar-shell will be updated
> accordingly.
> > > > 3. It comes with sophisticated autocompletion and command history to
> > > > highly improve the UX.
> > > > 4. Performance. Since JVM is initiated once, it will gain on
> > performance
> > > > thanks to the JVM warmup and internal libraries bootstraps.
> > > > 5. It will accept a file or a list of commands (parameter and stdin)
> to
> > > > start a shell, run the commands and close the shell. We'll call it
> the
> > > > `non-interactive` mode and it will ease the cluster operations
> > > automations.
> > > >
> > > > Note that existing tools will not be removed/changed.
> > > >
> > > > ### Implementation
> > > >
> > > > The shell implementation will be developed in Java, using a
> well-known
> > > > library called [JLine 3](https://github.com/jline/jline3) for the
> > shell
> > > > support.
> > > > There will be a new main class that will extends the existing class
> > tools
> > > > (e.g. `PulsarAdminTool`)
> > > >
> > > > #### Configuration
> > > > The configuration file taken by default will be `client.conf`, like
> > > > current CLI tools. The env setup will be exactly the same as for
> > > > `pulsar-admin` and `pulsar-client`.
> > > >
> > > > #### Autocompletion
> > > > JLine3 has great support for autocompletion. The shell java class
> will
> > > > translate current `JCommander` tools to the JLine3 completion API.
> This
> > > > will ensure that all APIs will be up-to-date and covered.
> > > >
> > > > #### History
> > > > JLine3 has built-in support for history, both in-memory and persisted
> > (on
> > > > local FS). By default the history will be persisted in the user home
> > > > directory. This feature can be turned off for security concerns with
> a
> > > > configuration property.
> > > >
> > > > #### List of tools
> > > > The proposal is to get the following tools:
> > > > - bin/pulsar-admin
> > > > - bin/pulsar-client
> > > > - bin/pulsar-perf
> > > >
> > > >
> > > > ## Rejected Alternatives
> > > >
> > > > - Create a shell mode for `pulsar-admin`. This won't be a flexible
> > > > solution because then we may need the same for other tools. Also it
> > > doesn't
> > > > cover the CLI unification part.
> > > > - Create a brand-new shell mode. Not compatible with current CLI and
> > > > another tool to maintain.
> > > > - Use another shell library. There are a couple of valid alternatives
> > to
> > > > JLine3 but it's wide-spread and the autocompletion API is much more
> > > > flexible than others since it's not done with annotations (the shell
> > > needs
> > > > to translate current JCommander definitions from existing tools)
> > > >
> > > >
> > > >
> > > >
> > > > Nicolò Boschi
> > > >
> > >
> >
>

Re: [DISCUSS] PIP-181: Pulsar Shell

Posted by tison <wa...@gmail.com>.

Hi Xiaoyu,

IIUC patch release must not include new features but only bug fixes.

Best,
tison.


Anon Hxy <an...@gmail.com> 于2022年7月1日周五 23:59写道：

> Hi Nicolò Boschi,
>
> The Pulsar Shell is really cool and I like it. And also I have a question:
>
> >  I'd like to target this feature for 2.11
>
> Will it be possible to use Pulsar Shell in the legacy version of Pulsar, in
> other words, could we cherry-pick the PR to another branch easily?
>
> Thanks,
> Xiaoyu Hou
>
> Nicolò Boschi <bo...@gmail.com> 于2022年7月1日周五 20:40写道：
>
> > I updated the issue linking all the implementations pull requests.
> > If you open the issue you will be able to see 2 videos that better
> explain
> > how the tool works.
> > https://github.com/apache/pulsar/issues/16250
> >
> > Let me know if you have any feedbacks
> >
> > I'd like to target this feature for 2.11
> >
> > BR,
> > Nicolò Boschi
> >
> >
> > Il giorno mar 28 giu 2022 alle ore 10:06 Nicolò Boschi <
> > boschi1997@gmail.com>
> > ha scritto:
> >
> > > Hi all,
> > >
> > > I opened a new PIP about Pulsar CLI tools.
> > > Looking forward to seeing comments and suggestions.
> > >
> > > PIP: https://github.com/apache/pulsar/issues/16250
> > >
> > > I posted a short video that shows how the new tool will work:
> > >
> > >
> >
> https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4
> > >
> > >
> > >
> > >
> > > ## Motivation
> > >
> > > Currently Pulsar comes with a couple of utility scripts with the goal
> of
> > > managing an existing cluster, test behaviours and verify performances:
> > > these tools are available as SH script inside the `bin` directory.
> > > The `pulsar-admin` is the CLI tool supposed to help users and operators
> > to
> > > configure the system, operate over policies, install functions and much
> > > else.
> > >
> > > This proposal basically aims to solve two different problems:
> > >
> > > 1. `pulsar-admin` is terribly slow. Every time the script is
> triggered, a
> > > new JVM process is spawned. The JVM process creation is heavy and most
> of
> > > the time is spent by the JVM initialization process. A very common use
> > case
> > > for cluster operators is to create scripts with several commands with
> the
> > > goal of initialize the cluster, initialize a specific tenant
> (namespaces,
> > > topics, policies, functions..); in this case, one JVM is initialized
> for
> > > each scripts leads to waste of time and resources.
> > >
> > > 2. User experience. The current design of the Pulsar CLIs can be
> > improved.
> > > There are a couple of aspects that may be annoying for a user and can
> > > discourage a user to use Pulsar.
> > >     1. Poking around available commands and options in a CLI tool
> > > (`pulsar-admin` for instance, but it's the same for `pulsar-perf` and
> > > `pulsar-client`) is slow and hard. In order to discover commands and
> > > options you need to use `-h` option and, since the performance issue
> > > pointed at 1., it can be annoying and time-consuming. Autocomplete
> > feature
> > > could be a real game-changer in this context.
> > >     2. Different CLI tools. There are a couple of different shell
> > scripts.
> > > They have different goals and it's okay to keep them separated.
> However,
> > > they raise a barrier for a non Pulsar expert that doesn't have a
> > convenient
> > > entry-point.
> > >
> > > ## Goal
> > >
> > > Address all the issues in the previous section with a single solution.
> > >
> > > ## API Changes
> > >
> > > A new shell script `bin/pulsar-shell` will be introduced. `bin/pulsar
> > > shell` could be a valid alternative but it's not findable for a newbie
> > user
> > > since no direct file exists.
> > >
> > > ## Implementation
> > >
> > > ### Concepts
> > > The new script `pulsar-shell` will differ from the existing for the
> > > following reasons:
> > >
> > > 1. It's a shell. When you start it, it will wait for commands to be
> > > executed. After the command has been executed, despite its result, the
> > > shell session will not be destroyed and it will wait for another
> command.
> > > 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be able to run
> > > all the existing CLI commands. This will be done in a way that when a
> new
> > > command/option is added, the pulsar-shell will be updated accordingly.
> > > 3. It comes with sophisticated autocompletion and command history to
> > > highly improve the UX.
> > > 4. Performance. Since JVM is initiated once, it will gain on
> performance
> > > thanks to the JVM warmup and internal libraries bootstraps.
> > > 5. It will accept a file or a list of commands (parameter and stdin) to
> > > start a shell, run the commands and close the shell. We'll call it the
> > > `non-interactive` mode and it will ease the cluster operations
> > automations.
> > >
> > > Note that existing tools will not be removed/changed.
> > >
> > > ### Implementation
> > >
> > > The shell implementation will be developed in Java, using a well-known
> > > library called [JLine 3](https://github.com/jline/jline3) for the
> shell
> > > support.
> > > There will be a new main class that will extends the existing class
> tools
> > > (e.g. `PulsarAdminTool`)
> > >
> > > #### Configuration
> > > The configuration file taken by default will be `client.conf`, like
> > > current CLI tools. The env setup will be exactly the same as for
> > > `pulsar-admin` and `pulsar-client`.
> > >
> > > #### Autocompletion
> > > JLine3 has great support for autocompletion. The shell java class will
> > > translate current `JCommander` tools to the JLine3 completion API. This
> > > will ensure that all APIs will be up-to-date and covered.
> > >
> > > #### History
> > > JLine3 has built-in support for history, both in-memory and persisted
> (on
> > > local FS). By default the history will be persisted in the user home
> > > directory. This feature can be turned off for security concerns with a
> > > configuration property.
> > >
> > > #### List of tools
> > > The proposal is to get the following tools:
> > > - bin/pulsar-admin
> > > - bin/pulsar-client
> > > - bin/pulsar-perf
> > >
> > >
> > > ## Rejected Alternatives
> > >
> > > - Create a shell mode for `pulsar-admin`. This won't be a flexible
> > > solution because then we may need the same for other tools. Also it
> > doesn't
> > > cover the CLI unification part.
> > > - Create a brand-new shell mode. Not compatible with current CLI and
> > > another tool to maintain.
> > > - Use another shell library. There are a couple of valid alternatives
> to
> > > JLine3 but it's wide-spread and the autocompletion API is much more
> > > flexible than others since it's not done with annotations (the shell
> > needs
> > > to translate current JCommander definitions from existing tools)
> > >
> > >
> > >
> > >
> > > Nicolò Boschi
> > >
> >
>

Re: [DISCUSS] PIP-181: Pulsar Shell

Posted by Anon Hxy <an...@gmail.com>.

Hi Nicolò Boschi,

The Pulsar Shell is really cool and I like it. And also I have a question:

>  I'd like to target this feature for 2.11

Will it be possible to use Pulsar Shell in the legacy version of Pulsar, in
other words, could we cherry-pick the PR to another branch easily?

Thanks,
Xiaoyu Hou

Nicolò Boschi <bo...@gmail.com> 于2022年7月1日周五 20:40写道：

> I updated the issue linking all the implementations pull requests.
> If you open the issue you will be able to see 2 videos that better explain
> how the tool works.
> https://github.com/apache/pulsar/issues/16250
>
> Let me know if you have any feedbacks
>
> I'd like to target this feature for 2.11
>
> BR,
> Nicolò Boschi
>
>
> Il giorno mar 28 giu 2022 alle ore 10:06 Nicolò Boschi <
> boschi1997@gmail.com>
> ha scritto:
>
> > Hi all,
> >
> > I opened a new PIP about Pulsar CLI tools.
> > Looking forward to seeing comments and suggestions.
> >
> > PIP: https://github.com/apache/pulsar/issues/16250
> >
> > I posted a short video that shows how the new tool will work:
> >
> >
> https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4
> >
> >
> >
> >
> > ## Motivation
> >
> > Currently Pulsar comes with a couple of utility scripts with the goal of
> > managing an existing cluster, test behaviours and verify performances:
> > these tools are available as SH script inside the `bin` directory.
> > The `pulsar-admin` is the CLI tool supposed to help users and operators
> to
> > configure the system, operate over policies, install functions and much
> > else.
> >
> > This proposal basically aims to solve two different problems:
> >
> > 1. `pulsar-admin` is terribly slow. Every time the script is triggered, a
> > new JVM process is spawned. The JVM process creation is heavy and most of
> > the time is spent by the JVM initialization process. A very common use
> case
> > for cluster operators is to create scripts with several commands with the
> > goal of initialize the cluster, initialize a specific tenant (namespaces,
> > topics, policies, functions..); in this case, one JVM is initialized for
> > each scripts leads to waste of time and resources.
> >
> > 2. User experience. The current design of the Pulsar CLIs can be
> improved.
> > There are a couple of aspects that may be annoying for a user and can
> > discourage a user to use Pulsar.
> >     1. Poking around available commands and options in a CLI tool
> > (`pulsar-admin` for instance, but it's the same for `pulsar-perf` and
> > `pulsar-client`) is slow and hard. In order to discover commands and
> > options you need to use `-h` option and, since the performance issue
> > pointed at 1., it can be annoying and time-consuming. Autocomplete
> feature
> > could be a real game-changer in this context.
> >     2. Different CLI tools. There are a couple of different shell
> scripts.
> > They have different goals and it's okay to keep them separated. However,
> > they raise a barrier for a non Pulsar expert that doesn't have a
> convenient
> > entry-point.
> >
> > ## Goal
> >
> > Address all the issues in the previous section with a single solution.
> >
> > ## API Changes
> >
> > A new shell script `bin/pulsar-shell` will be introduced. `bin/pulsar
> > shell` could be a valid alternative but it's not findable for a newbie
> user
> > since no direct file exists.
> >
> > ## Implementation
> >
> > ### Concepts
> > The new script `pulsar-shell` will differ from the existing for the
> > following reasons:
> >
> > 1. It's a shell. When you start it, it will wait for commands to be
> > executed. After the command has been executed, despite its result, the
> > shell session will not be destroyed and it will wait for another command.
> > 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be able to run
> > all the existing CLI commands. This will be done in a way that when a new
> > command/option is added, the pulsar-shell will be updated accordingly.
> > 3. It comes with sophisticated autocompletion and command history to
> > highly improve the UX.
> > 4. Performance. Since JVM is initiated once, it will gain on performance
> > thanks to the JVM warmup and internal libraries bootstraps.
> > 5. It will accept a file or a list of commands (parameter and stdin) to
> > start a shell, run the commands and close the shell. We'll call it the
> > `non-interactive` mode and it will ease the cluster operations
> automations.
> >
> > Note that existing tools will not be removed/changed.
> >
> > ### Implementation
> >
> > The shell implementation will be developed in Java, using a well-known
> > library called [JLine 3](https://github.com/jline/jline3) for the shell
> > support.
> > There will be a new main class that will extends the existing class tools
> > (e.g. `PulsarAdminTool`)
> >
> > #### Configuration
> > The configuration file taken by default will be `client.conf`, like
> > current CLI tools. The env setup will be exactly the same as for
> > `pulsar-admin` and `pulsar-client`.
> >
> > #### Autocompletion
> > JLine3 has great support for autocompletion. The shell java class will
> > translate current `JCommander` tools to the JLine3 completion API. This
> > will ensure that all APIs will be up-to-date and covered.
> >
> > #### History
> > JLine3 has built-in support for history, both in-memory and persisted (on
> > local FS). By default the history will be persisted in the user home
> > directory. This feature can be turned off for security concerns with a
> > configuration property.
> >
> > #### List of tools
> > The proposal is to get the following tools:
> > - bin/pulsar-admin
> > - bin/pulsar-client
> > - bin/pulsar-perf
> >
> >
> > ## Rejected Alternatives
> >
> > - Create a shell mode for `pulsar-admin`. This won't be a flexible
> > solution because then we may need the same for other tools. Also it
> doesn't
> > cover the CLI unification part.
> > - Create a brand-new shell mode. Not compatible with current CLI and
> > another tool to maintain.
> > - Use another shell library. There are a couple of valid alternatives to
> > JLine3 but it's wide-spread and the autocompletion API is much more
> > flexible than others since it's not done with annotations (the shell
> needs
> > to translate current JCommander definitions from existing tools)
> >
> >
> >
> >
> > Nicolò Boschi
> >
>

Re: [DISCUSS] PIP-181: Pulsar Shell

Posted by Nicolò Boschi <bo...@gmail.com>.

I updated the issue linking all the implementations pull requests.
If you open the issue you will be able to see 2 videos that better explain
how the tool works.
https://github.com/apache/pulsar/issues/16250

Let me know if you have any feedbacks

I'd like to target this feature for 2.11

BR,
Nicolò Boschi


Il giorno mar 28 giu 2022 alle ore 10:06 Nicolò Boschi <bo...@gmail.com>
ha scritto:

> Hi all,
>
> I opened a new PIP about Pulsar CLI tools.
> Looking forward to seeing comments and suggestions.
>
> PIP: https://github.com/apache/pulsar/issues/16250
>
> I posted a short video that shows how the new tool will work:
>
> https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4
>
>
>
>
> ## Motivation
>
> Currently Pulsar comes with a couple of utility scripts with the goal of
> managing an existing cluster, test behaviours and verify performances:
> these tools are available as SH script inside the `bin` directory.
> The `pulsar-admin` is the CLI tool supposed to help users and operators to
> configure the system, operate over policies, install functions and much
> else.
>
> This proposal basically aims to solve two different problems:
>
> 1. `pulsar-admin` is terribly slow. Every time the script is triggered, a
> new JVM process is spawned. The JVM process creation is heavy and most of
> the time is spent by the JVM initialization process. A very common use case
> for cluster operators is to create scripts with several commands with the
> goal of initialize the cluster, initialize a specific tenant (namespaces,
> topics, policies, functions..); in this case, one JVM is initialized for
> each scripts leads to waste of time and resources.
>
> 2. User experience. The current design of the Pulsar CLIs can be improved.
> There are a couple of aspects that may be annoying for a user and can
> discourage a user to use Pulsar.
>     1. Poking around available commands and options in a CLI tool
> (`pulsar-admin` for instance, but it's the same for `pulsar-perf` and
> `pulsar-client`) is slow and hard. In order to discover commands and
> options you need to use `-h` option and, since the performance issue
> pointed at 1., it can be annoying and time-consuming. Autocomplete feature
> could be a real game-changer in this context.
>     2. Different CLI tools. There are a couple of different shell scripts.
> They have different goals and it's okay to keep them separated. However,
> they raise a barrier for a non Pulsar expert that doesn't have a convenient
> entry-point.
>
> ## Goal
>
> Address all the issues in the previous section with a single solution.
>
> ## API Changes
>
> A new shell script `bin/pulsar-shell` will be introduced. `bin/pulsar
> shell` could be a valid alternative but it's not findable for a newbie user
> since no direct file exists.
>
> ## Implementation
>
> ### Concepts
> The new script `pulsar-shell` will differ from the existing for the
> following reasons:
>
> 1. It's a shell. When you start it, it will wait for commands to be
> executed. After the command has been executed, despite its result, the
> shell session will not be destroyed and it will wait for another command.
> 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be able to run
> all the existing CLI commands. This will be done in a way that when a new
> command/option is added, the pulsar-shell will be updated accordingly.
> 3. It comes with sophisticated autocompletion and command history to
> highly improve the UX.
> 4. Performance. Since JVM is initiated once, it will gain on performance
> thanks to the JVM warmup and internal libraries bootstraps.
> 5. It will accept a file or a list of commands (parameter and stdin) to
> start a shell, run the commands and close the shell. We'll call it the
> `non-interactive` mode and it will ease the cluster operations automations.
>
> Note that existing tools will not be removed/changed.
>
> ### Implementation
>
> The shell implementation will be developed in Java, using a well-known
> library called [JLine 3](https://github.com/jline/jline3) for the shell
> support.
> There will be a new main class that will extends the existing class tools
> (e.g. `PulsarAdminTool`)
>
> #### Configuration
> The configuration file taken by default will be `client.conf`, like
> current CLI tools. The env setup will be exactly the same as for
> `pulsar-admin` and `pulsar-client`.
>
> #### Autocompletion
> JLine3 has great support for autocompletion. The shell java class will
> translate current `JCommander` tools to the JLine3 completion API. This
> will ensure that all APIs will be up-to-date and covered.
>
> #### History
> JLine3 has built-in support for history, both in-memory and persisted (on
> local FS). By default the history will be persisted in the user home
> directory. This feature can be turned off for security concerns with a
> configuration property.
>
> #### List of tools
> The proposal is to get the following tools:
> - bin/pulsar-admin
> - bin/pulsar-client
> - bin/pulsar-perf
>
>
> ## Rejected Alternatives
>
> - Create a shell mode for `pulsar-admin`. This won't be a flexible
> solution because then we may need the same for other tools. Also it doesn't
> cover the CLI unification part.
> - Create a brand-new shell mode. Not compatible with current CLI and
> another tool to maintain.
> - Use another shell library. There are a couple of valid alternatives to
> JLine3 but it's wide-spread and the autocompletion API is much more
> flexible than others since it's not done with annotations (the shell needs
> to translate current JCommander definitions from existing tools)
>
>
>
>
> Nicolò Boschi
>

Re: [DISCUSS] PIP-181: Pulsar Shell

Posted by Max Xu <ma...@gmail.com>.

Hi Nicolò,


This is definitely a very awesome job, thank you for bringing Pulsar Shell
to us!


Also, I was wondering if we could integrate the pulsar shell entrypoint
directly into the existing pulsar command[1]? (i.e, invoke ./bin/pulsar
shell)

So that we can reuse the current CLI instead of adding a new CLI into the
bin folder.


*Best,*

*Max Xu*

[1] https://github.com/apache/pulsar/blob/master/bin/pulsar

On Mon, Jul 11, 2022 at 6:24 PM Nicolò Boschi <bo...@gmail.com> wrote:

> Hi Asaf,
>
> Actually it's in the "Future works" of the PIP.
>
> I already implemented a PoC of it and you can see it here:
>
> https://github.com/nicoloboschi/pulsar/commit/f227be324c96daeaf89566260a63eb558357f34f
> There are still some issues with some runtime config since GraalVM requires
> to declare most of the reflection accesses you will do at runtime. And
> Pulsar does a lot of reflections.
>
> Anyway, I don't think it is a different direction but instead a future
> optimization and we should push to have it after the base Pulsar shell is
> fully committed to the master branch.
>
> BR,
> Nicolò Boschi
>
>
> Il giorno lun 11 lug 2022 alle ore 12:18 tison <wa...@gmail.com> ha
> scritto:
>
> > Hi Asaf,
> >
> > Why not compile the related package with graal and submit a report on the
> > master issue[1] :)
> >
> > AFAIK Pulsar didn't optimize for graal compilation so that you MAY meet
> > some issues, and pulsar-shell actually depends on other core module so
> that
> > it's not an orphan issue to resolve.
> >
> > Best,
> > tison.
> >
> > [1] https://github.com/apache/pulsar/issues/16250
> >
> >
> > Asaf Mesika <as...@gmail.com> 于2022年7月11日周一 18:05写道：
> >
> > > Looks awesome.
> > > I was wondering if you have thought about another direction which is
> > > supplying a GraalVM compiled binary. This should take care of load
> time.
> > So
> > > in effect, you can have a single command, which boots in microseconds,
> > and
> > > has shell auto-complete?
> > >
> > > On Mon, Jul 4, 2022 at 4:34 PM Hang Chen <ch...@apache.org> wrote:
> > >
> > > > +1 Great work!
> > > >
> > > > Thanks,
> > > > Hang
> > > >
> > > > PengHui Li <co...@gmail.com> 于2022年7月4日周一 19:36写道：
> > > > >
> > > > > + 1
> > > > >
> > > > > Penghui
> > > > > On Jul 4, 2022, 18:22 +0800, Dezhi Liu <de...@apache.org>,
> wrote:
> > > > > > +1, Great Job.
> > > > > >
> > > > > > Thanks,
> > > > > > Dezhi
> > > > > >
> > > > > > On 2022/06/28 08:06:01 Nicolò Boschi wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I opened a new PIP about Pulsar CLI tools.
> > > > > > > Looking forward to seeing comments and suggestions.
> > > > > > >
> > > > > > > PIP: https://github.com/apache/pulsar/issues/16250
> > > > > > >
> > > > > > > I posted a short video that shows how the new tool will work:
> > > > > > >
> > > >
> > >
> >
> https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ## Motivation
> > > > > > >
> > > > > > > Currently Pulsar comes with a couple of utility scripts with
> the
> > > > goal of
> > > > > > > managing an existing cluster, test behaviours and verify
> > > > performances:
> > > > > > > these tools are available as SH script inside the `bin`
> > directory.
> > > > > > > The `pulsar-admin` is the CLI tool supposed to help users and
> > > > operators to
> > > > > > > configure the system, operate over policies, install functions
> > and
> > > > much
> > > > > > > else.
> > > > > > >
> > > > > > > This proposal basically aims to solve two different problems:
> > > > > > >
> > > > > > > 1. `pulsar-admin` is terribly slow. Every time the script is
> > > > triggered, a
> > > > > > > new JVM process is spawned. The JVM process creation is heavy
> and
> > > > most of
> > > > > > > the time is spent by the JVM initialization process. A very
> > common
> > > > use case
> > > > > > > for cluster operators is to create scripts with several
> commands
> > > > with the
> > > > > > > goal of initialize the cluster, initialize a specific tenant
> > > > (namespaces,
> > > > > > > topics, policies, functions..); in this case, one JVM is
> > > initialized
> > > > for
> > > > > > > each scripts leads to waste of time and resources.
> > > > > > >
> > > > > > > 2. User experience. The current design of the Pulsar CLIs can
> be
> > > > improved.
> > > > > > > There are a couple of aspects that may be annoying for a user
> and
> > > can
> > > > > > > discourage a user to use Pulsar.
> > > > > > > 1. Poking around available commands and options in a CLI tool
> > > > > > > (`pulsar-admin` for instance, but it's the same for
> `pulsar-perf`
> > > and
> > > > > > > `pulsar-client`) is slow and hard. In order to discover
> commands
> > > and
> > > > > > > options you need to use `-h` option and, since the performance
> > > issue
> > > > > > > pointed at 1., it can be annoying and time-consuming.
> > Autocomplete
> > > > feature
> > > > > > > could be a real game-changer in this context.
> > > > > > > 2. Different CLI tools. There are a couple of different shell
> > > > scripts.
> > > > > > > They have different goals and it's okay to keep them separated.
> > > > However,
> > > > > > > they raise a barrier for a non Pulsar expert that doesn't have
> a
> > > > convenient
> > > > > > > entry-point.
> > > > > > >
> > > > > > > ## Goal
> > > > > > >
> > > > > > > Address all the issues in the previous section with a single
> > > > solution.
> > > > > > >
> > > > > > > ## API Changes
> > > > > > >
> > > > > > > A new shell script `bin/pulsar-shell` will be introduced.
> > > `bin/pulsar
> > > > > > > shell` could be a valid alternative but it's not findable for a
> > > > newbie user
> > > > > > > since no direct file exists.
> > > > > > >
> > > > > > > ## Implementation
> > > > > > >
> > > > > > > ### Concepts
> > > > > > > The new script `pulsar-shell` will differ from the existing for
> > the
> > > > > > > following reasons:
> > > > > > >
> > > > > > > 1. It's a shell. When you start it, it will wait for commands
> to
> > be
> > > > > > > executed. After the command has been executed, despite its
> > result,
> > > > the
> > > > > > > shell session will not be destroyed and it will wait for
> another
> > > > command.
> > > > > > > 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be
> able
> > to
> > > > run all
> > > > > > > the existing CLI commands. This will be done in a way that
> when a
> > > new
> > > > > > > command/option is added, the pulsar-shell will be updated
> > > > accordingly.
> > > > > > > 3. It comes with sophisticated autocompletion and command
> history
> > > to
> > > > highly
> > > > > > > improve the UX.
> > > > > > > 4. Performance. Since JVM is initiated once, it will gain on
> > > > performance
> > > > > > > thanks to the JVM warmup and internal libraries bootstraps.
> > > > > > > 5. It will accept a file or a list of commands (parameter and
> > > stdin)
> > > > to
> > > > > > > start a shell, run the commands and close the shell. We'll call
> > it
> > > > the
> > > > > > > `non-interactive` mode and it will ease the cluster operations
> > > > automations.
> > > > > > >
> > > > > > > Note that existing tools will not be removed/changed.
> > > > > > >
> > > > > > > ### Implementation
> > > > > > >
> > > > > > > The shell implementation will be developed in Java, using a
> > > > well-known
> > > > > > > library called [JLine 3](https://github.com/jline/jline3) for
> > the
> > > > shell
> > > > > > > support.
> > > > > > > There will be a new main class that will extends the existing
> > class
> > > > tools
> > > > > > > (e.g. `PulsarAdminTool`)
> > > > > > >
> > > > > > > #### Configuration
> > > > > > > The configuration file taken by default will be `client.conf`,
> > like
> > > > current
> > > > > > > CLI tools. The env setup will be exactly the same as for
> > > > `pulsar-admin` and
> > > > > > > `pulsar-client`.
> > > > > > >
> > > > > > > #### Autocompletion
> > > > > > > JLine3 has great support for autocompletion. The shell java
> class
> > > > will
> > > > > > > translate current `JCommander` tools to the JLine3 completion
> > API.
> > > > This
> > > > > > > will ensure that all APIs will be up-to-date and covered.
> > > > > > >
> > > > > > > #### History
> > > > > > > JLine3 has built-in support for history, both in-memory and
> > > > persisted (on
> > > > > > > local FS). By default the history will be persisted in the user
> > > home
> > > > > > > directory. This feature can be turned off for security concerns
> > > with
> > > > a
> > > > > > > configuration property.
> > > > > > >
> > > > > > > #### List of tools
> > > > > > > The proposal is to get the following tools:
> > > > > > > - bin/pulsar-admin
> > > > > > > - bin/pulsar-client
> > > > > > > - bin/pulsar-perf
> > > > > > >
> > > > > > >
> > > > > > > ## Rejected Alternatives
> > > > > > >
> > > > > > > - Create a shell mode for `pulsar-admin`. This won't be a
> > flexible
> > > > solution
> > > > > > > because then we may need the same for other tools. Also it
> > doesn't
> > > > cover
> > > > > > > the CLI unification part.
> > > > > > > - Create a brand-new shell mode. Not compatible with current
> CLI
> > > and
> > > > > > > another tool to maintain.
> > > > > > > - Use another shell library. There are a couple of valid
> > > > alternatives to
> > > > > > > JLine3 but it's wide-spread and the autocompletion API is much
> > more
> > > > > > > flexible than others since it's not done with annotations (the
> > > shell
> > > > needs
> > > > > > > to translate current JCommander definitions from existing
> tools)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Nicolò Boschi
> > > > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] PIP-181: Pulsar Shell

Posted by Nicolò Boschi <bo...@gmail.com>.

Hi Asaf,

Actually it's in the "Future works" of the PIP.

I already implemented a PoC of it and you can see it here:
https://github.com/nicoloboschi/pulsar/commit/f227be324c96daeaf89566260a63eb558357f34f
There are still some issues with some runtime config since GraalVM requires
to declare most of the reflection accesses you will do at runtime. And
Pulsar does a lot of reflections.

Anyway, I don't think it is a different direction but instead a future
optimization and we should push to have it after the base Pulsar shell is
fully committed to the master branch.

BR,
Nicolò Boschi


Il giorno lun 11 lug 2022 alle ore 12:18 tison <wa...@gmail.com> ha
scritto:

> Hi Asaf,
>
> Why not compile the related package with graal and submit a report on the
> master issue[1] :)
>
> AFAIK Pulsar didn't optimize for graal compilation so that you MAY meet
> some issues, and pulsar-shell actually depends on other core module so that
> it's not an orphan issue to resolve.
>
> Best,
> tison.
>
> [1] https://github.com/apache/pulsar/issues/16250
>
>
> Asaf Mesika <as...@gmail.com> 于2022年7月11日周一 18:05写道：
>
> > Looks awesome.
> > I was wondering if you have thought about another direction which is
> > supplying a GraalVM compiled binary. This should take care of load time.
> So
> > in effect, you can have a single command, which boots in microseconds,
> and
> > has shell auto-complete?
> >
> > On Mon, Jul 4, 2022 at 4:34 PM Hang Chen <ch...@apache.org> wrote:
> >
> > > +1 Great work!
> > >
> > > Thanks,
> > > Hang
> > >
> > > PengHui Li <co...@gmail.com> 于2022年7月4日周一 19:36写道：
> > > >
> > > > + 1
> > > >
> > > > Penghui
> > > > On Jul 4, 2022, 18:22 +0800, Dezhi Liu <de...@apache.org>, wrote:
> > > > > +1, Great Job.
> > > > >
> > > > > Thanks,
> > > > > Dezhi
> > > > >
> > > > > On 2022/06/28 08:06:01 Nicolò Boschi wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I opened a new PIP about Pulsar CLI tools.
> > > > > > Looking forward to seeing comments and suggestions.
> > > > > >
> > > > > > PIP: https://github.com/apache/pulsar/issues/16250
> > > > > >
> > > > > > I posted a short video that shows how the new tool will work:
> > > > > >
> > >
> >
> https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ## Motivation
> > > > > >
> > > > > > Currently Pulsar comes with a couple of utility scripts with the
> > > goal of
> > > > > > managing an existing cluster, test behaviours and verify
> > > performances:
> > > > > > these tools are available as SH script inside the `bin`
> directory.
> > > > > > The `pulsar-admin` is the CLI tool supposed to help users and
> > > operators to
> > > > > > configure the system, operate over policies, install functions
> and
> > > much
> > > > > > else.
> > > > > >
> > > > > > This proposal basically aims to solve two different problems:
> > > > > >
> > > > > > 1. `pulsar-admin` is terribly slow. Every time the script is
> > > triggered, a
> > > > > > new JVM process is spawned. The JVM process creation is heavy and
> > > most of
> > > > > > the time is spent by the JVM initialization process. A very
> common
> > > use case
> > > > > > for cluster operators is to create scripts with several commands
> > > with the
> > > > > > goal of initialize the cluster, initialize a specific tenant
> > > (namespaces,
> > > > > > topics, policies, functions..); in this case, one JVM is
> > initialized
> > > for
> > > > > > each scripts leads to waste of time and resources.
> > > > > >
> > > > > > 2. User experience. The current design of the Pulsar CLIs can be
> > > improved.
> > > > > > There are a couple of aspects that may be annoying for a user and
> > can
> > > > > > discourage a user to use Pulsar.
> > > > > > 1. Poking around available commands and options in a CLI tool
> > > > > > (`pulsar-admin` for instance, but it's the same for `pulsar-perf`
> > and
> > > > > > `pulsar-client`) is slow and hard. In order to discover commands
> > and
> > > > > > options you need to use `-h` option and, since the performance
> > issue
> > > > > > pointed at 1., it can be annoying and time-consuming.
> Autocomplete
> > > feature
> > > > > > could be a real game-changer in this context.
> > > > > > 2. Different CLI tools. There are a couple of different shell
> > > scripts.
> > > > > > They have different goals and it's okay to keep them separated.
> > > However,
> > > > > > they raise a barrier for a non Pulsar expert that doesn't have a
> > > convenient
> > > > > > entry-point.
> > > > > >
> > > > > > ## Goal
> > > > > >
> > > > > > Address all the issues in the previous section with a single
> > > solution.
> > > > > >
> > > > > > ## API Changes
> > > > > >
> > > > > > A new shell script `bin/pulsar-shell` will be introduced.
> > `bin/pulsar
> > > > > > shell` could be a valid alternative but it's not findable for a
> > > newbie user
> > > > > > since no direct file exists.
> > > > > >
> > > > > > ## Implementation
> > > > > >
> > > > > > ### Concepts
> > > > > > The new script `pulsar-shell` will differ from the existing for
> the
> > > > > > following reasons:
> > > > > >
> > > > > > 1. It's a shell. When you start it, it will wait for commands to
> be
> > > > > > executed. After the command has been executed, despite its
> result,
> > > the
> > > > > > shell session will not be destroyed and it will wait for another
> > > command.
> > > > > > 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be able
> to
> > > run all
> > > > > > the existing CLI commands. This will be done in a way that when a
> > new
> > > > > > command/option is added, the pulsar-shell will be updated
> > > accordingly.
> > > > > > 3. It comes with sophisticated autocompletion and command history
> > to
> > > highly
> > > > > > improve the UX.
> > > > > > 4. Performance. Since JVM is initiated once, it will gain on
> > > performance
> > > > > > thanks to the JVM warmup and internal libraries bootstraps.
> > > > > > 5. It will accept a file or a list of commands (parameter and
> > stdin)
> > > to
> > > > > > start a shell, run the commands and close the shell. We'll call
> it
> > > the
> > > > > > `non-interactive` mode and it will ease the cluster operations
> > > automations.
> > > > > >
> > > > > > Note that existing tools will not be removed/changed.
> > > > > >
> > > > > > ### Implementation
> > > > > >
> > > > > > The shell implementation will be developed in Java, using a
> > > well-known
> > > > > > library called [JLine 3](https://github.com/jline/jline3) for
> the
> > > shell
> > > > > > support.
> > > > > > There will be a new main class that will extends the existing
> class
> > > tools
> > > > > > (e.g. `PulsarAdminTool`)
> > > > > >
> > > > > > #### Configuration
> > > > > > The configuration file taken by default will be `client.conf`,
> like
> > > current
> > > > > > CLI tools. The env setup will be exactly the same as for
> > > `pulsar-admin` and
> > > > > > `pulsar-client`.
> > > > > >
> > > > > > #### Autocompletion
> > > > > > JLine3 has great support for autocompletion. The shell java class
> > > will
> > > > > > translate current `JCommander` tools to the JLine3 completion
> API.
> > > This
> > > > > > will ensure that all APIs will be up-to-date and covered.
> > > > > >
> > > > > > #### History
> > > > > > JLine3 has built-in support for history, both in-memory and
> > > persisted (on
> > > > > > local FS). By default the history will be persisted in the user
> > home
> > > > > > directory. This feature can be turned off for security concerns
> > with
> > > a
> > > > > > configuration property.
> > > > > >
> > > > > > #### List of tools
> > > > > > The proposal is to get the following tools:
> > > > > > - bin/pulsar-admin
> > > > > > - bin/pulsar-client
> > > > > > - bin/pulsar-perf
> > > > > >
> > > > > >
> > > > > > ## Rejected Alternatives
> > > > > >
> > > > > > - Create a shell mode for `pulsar-admin`. This won't be a
> flexible
> > > solution
> > > > > > because then we may need the same for other tools. Also it
> doesn't
> > > cover
> > > > > > the CLI unification part.
> > > > > > - Create a brand-new shell mode. Not compatible with current CLI
> > and
> > > > > > another tool to maintain.
> > > > > > - Use another shell library. There are a couple of valid
> > > alternatives to
> > > > > > JLine3 but it's wide-spread and the autocompletion API is much
> more
> > > > > > flexible than others since it's not done with annotations (the
> > shell
> > > needs
> > > > > > to translate current JCommander definitions from existing tools)
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Nicolò Boschi
> > > > > >
> > >
> >
>

Re: [DISCUSS] PIP-181: Pulsar Shell

Posted by tison <wa...@gmail.com>.

Hi Asaf,

Why not compile the related package with graal and submit a report on the
master issue[1] :)

AFAIK Pulsar didn't optimize for graal compilation so that you MAY meet
some issues, and pulsar-shell actually depends on other core module so that
it's not an orphan issue to resolve.

Best,
tison.

[1] https://github.com/apache/pulsar/issues/16250


Asaf Mesika <as...@gmail.com> 于2022年7月11日周一 18:05写道：

> Looks awesome.
> I was wondering if you have thought about another direction which is
> supplying a GraalVM compiled binary. This should take care of load time. So
> in effect, you can have a single command, which boots in microseconds, and
> has shell auto-complete?
>
> On Mon, Jul 4, 2022 at 4:34 PM Hang Chen <ch...@apache.org> wrote:
>
> > +1 Great work!
> >
> > Thanks,
> > Hang
> >
> > PengHui Li <co...@gmail.com> 于2022年7月4日周一 19:36写道：
> > >
> > > + 1
> > >
> > > Penghui
> > > On Jul 4, 2022, 18:22 +0800, Dezhi Liu <de...@apache.org>, wrote:
> > > > +1, Great Job.
> > > >
> > > > Thanks,
> > > > Dezhi
> > > >
> > > > On 2022/06/28 08:06:01 Nicolò Boschi wrote:
> > > > > Hi all,
> > > > >
> > > > > I opened a new PIP about Pulsar CLI tools.
> > > > > Looking forward to seeing comments and suggestions.
> > > > >
> > > > > PIP: https://github.com/apache/pulsar/issues/16250
> > > > >
> > > > > I posted a short video that shows how the new tool will work:
> > > > >
> >
> https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ## Motivation
> > > > >
> > > > > Currently Pulsar comes with a couple of utility scripts with the
> > goal of
> > > > > managing an existing cluster, test behaviours and verify
> > performances:
> > > > > these tools are available as SH script inside the `bin` directory.
> > > > > The `pulsar-admin` is the CLI tool supposed to help users and
> > operators to
> > > > > configure the system, operate over policies, install functions and
> > much
> > > > > else.
> > > > >
> > > > > This proposal basically aims to solve two different problems:
> > > > >
> > > > > 1. `pulsar-admin` is terribly slow. Every time the script is
> > triggered, a
> > > > > new JVM process is spawned. The JVM process creation is heavy and
> > most of
> > > > > the time is spent by the JVM initialization process. A very common
> > use case
> > > > > for cluster operators is to create scripts with several commands
> > with the
> > > > > goal of initialize the cluster, initialize a specific tenant
> > (namespaces,
> > > > > topics, policies, functions..); in this case, one JVM is
> initialized
> > for
> > > > > each scripts leads to waste of time and resources.
> > > > >
> > > > > 2. User experience. The current design of the Pulsar CLIs can be
> > improved.
> > > > > There are a couple of aspects that may be annoying for a user and
> can
> > > > > discourage a user to use Pulsar.
> > > > > 1. Poking around available commands and options in a CLI tool
> > > > > (`pulsar-admin` for instance, but it's the same for `pulsar-perf`
> and
> > > > > `pulsar-client`) is slow and hard. In order to discover commands
> and
> > > > > options you need to use `-h` option and, since the performance
> issue
> > > > > pointed at 1., it can be annoying and time-consuming. Autocomplete
> > feature
> > > > > could be a real game-changer in this context.
> > > > > 2. Different CLI tools. There are a couple of different shell
> > scripts.
> > > > > They have different goals and it's okay to keep them separated.
> > However,
> > > > > they raise a barrier for a non Pulsar expert that doesn't have a
> > convenient
> > > > > entry-point.
> > > > >
> > > > > ## Goal
> > > > >
> > > > > Address all the issues in the previous section with a single
> > solution.
> > > > >
> > > > > ## API Changes
> > > > >
> > > > > A new shell script `bin/pulsar-shell` will be introduced.
> `bin/pulsar
> > > > > shell` could be a valid alternative but it's not findable for a
> > newbie user
> > > > > since no direct file exists.
> > > > >
> > > > > ## Implementation
> > > > >
> > > > > ### Concepts
> > > > > The new script `pulsar-shell` will differ from the existing for the
> > > > > following reasons:
> > > > >
> > > > > 1. It's a shell. When you start it, it will wait for commands to be
> > > > > executed. After the command has been executed, despite its result,
> > the
> > > > > shell session will not be destroyed and it will wait for another
> > command.
> > > > > 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be able to
> > run all
> > > > > the existing CLI commands. This will be done in a way that when a
> new
> > > > > command/option is added, the pulsar-shell will be updated
> > accordingly.
> > > > > 3. It comes with sophisticated autocompletion and command history
> to
> > highly
> > > > > improve the UX.
> > > > > 4. Performance. Since JVM is initiated once, it will gain on
> > performance
> > > > > thanks to the JVM warmup and internal libraries bootstraps.
> > > > > 5. It will accept a file or a list of commands (parameter and
> stdin)
> > to
> > > > > start a shell, run the commands and close the shell. We'll call it
> > the
> > > > > `non-interactive` mode and it will ease the cluster operations
> > automations.
> > > > >
> > > > > Note that existing tools will not be removed/changed.
> > > > >
> > > > > ### Implementation
> > > > >
> > > > > The shell implementation will be developed in Java, using a
> > well-known
> > > > > library called [JLine 3](https://github.com/jline/jline3) for the
> > shell
> > > > > support.
> > > > > There will be a new main class that will extends the existing class
> > tools
> > > > > (e.g. `PulsarAdminTool`)
> > > > >
> > > > > #### Configuration
> > > > > The configuration file taken by default will be `client.conf`, like
> > current
> > > > > CLI tools. The env setup will be exactly the same as for
> > `pulsar-admin` and
> > > > > `pulsar-client`.
> > > > >
> > > > > #### Autocompletion
> > > > > JLine3 has great support for autocompletion. The shell java class
> > will
> > > > > translate current `JCommander` tools to the JLine3 completion API.
> > This
> > > > > will ensure that all APIs will be up-to-date and covered.
> > > > >
> > > > > #### History
> > > > > JLine3 has built-in support for history, both in-memory and
> > persisted (on
> > > > > local FS). By default the history will be persisted in the user
> home
> > > > > directory. This feature can be turned off for security concerns
> with
> > a
> > > > > configuration property.
> > > > >
> > > > > #### List of tools
> > > > > The proposal is to get the following tools:
> > > > > - bin/pulsar-admin
> > > > > - bin/pulsar-client
> > > > > - bin/pulsar-perf
> > > > >
> > > > >
> > > > > ## Rejected Alternatives
> > > > >
> > > > > - Create a shell mode for `pulsar-admin`. This won't be a flexible
> > solution
> > > > > because then we may need the same for other tools. Also it doesn't
> > cover
> > > > > the CLI unification part.
> > > > > - Create a brand-new shell mode. Not compatible with current CLI
> and
> > > > > another tool to maintain.
> > > > > - Use another shell library. There are a couple of valid
> > alternatives to
> > > > > JLine3 but it's wide-spread and the autocompletion API is much more
> > > > > flexible than others since it's not done with annotations (the
> shell
> > needs
> > > > > to translate current JCommander definitions from existing tools)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Nicolò Boschi
> > > > >
> >
>

Re: [DISCUSS] PIP-181: Pulsar Shell

Posted by Asaf Mesika <as...@gmail.com>.

Looks awesome.
I was wondering if you have thought about another direction which is
supplying a GraalVM compiled binary. This should take care of load time. So
in effect, you can have a single command, which boots in microseconds, and
has shell auto-complete?

On Mon, Jul 4, 2022 at 4:34 PM Hang Chen <ch...@apache.org> wrote:

> +1 Great work!
>
> Thanks,
> Hang
>
> PengHui Li <co...@gmail.com> 于2022年7月4日周一 19:36写道：
> >
> > + 1
> >
> > Penghui
> > On Jul 4, 2022, 18:22 +0800, Dezhi Liu <de...@apache.org>, wrote:
> > > +1, Great Job.
> > >
> > > Thanks,
> > > Dezhi
> > >
> > > On 2022/06/28 08:06:01 Nicolò Boschi wrote:
> > > > Hi all,
> > > >
> > > > I opened a new PIP about Pulsar CLI tools.
> > > > Looking forward to seeing comments and suggestions.
> > > >
> > > > PIP: https://github.com/apache/pulsar/issues/16250
> > > >
> > > > I posted a short video that shows how the new tool will work:
> > > >
> https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4
> > > >
> > > >
> > > >
> > > >
> > > > ## Motivation
> > > >
> > > > Currently Pulsar comes with a couple of utility scripts with the
> goal of
> > > > managing an existing cluster, test behaviours and verify
> performances:
> > > > these tools are available as SH script inside the `bin` directory.
> > > > The `pulsar-admin` is the CLI tool supposed to help users and
> operators to
> > > > configure the system, operate over policies, install functions and
> much
> > > > else.
> > > >
> > > > This proposal basically aims to solve two different problems:
> > > >
> > > > 1. `pulsar-admin` is terribly slow. Every time the script is
> triggered, a
> > > > new JVM process is spawned. The JVM process creation is heavy and
> most of
> > > > the time is spent by the JVM initialization process. A very common
> use case
> > > > for cluster operators is to create scripts with several commands
> with the
> > > > goal of initialize the cluster, initialize a specific tenant
> (namespaces,
> > > > topics, policies, functions..); in this case, one JVM is initialized
> for
> > > > each scripts leads to waste of time and resources.
> > > >
> > > > 2. User experience. The current design of the Pulsar CLIs can be
> improved.
> > > > There are a couple of aspects that may be annoying for a user and can
> > > > discourage a user to use Pulsar.
> > > > 1. Poking around available commands and options in a CLI tool
> > > > (`pulsar-admin` for instance, but it's the same for `pulsar-perf` and
> > > > `pulsar-client`) is slow and hard. In order to discover commands and
> > > > options you need to use `-h` option and, since the performance issue
> > > > pointed at 1., it can be annoying and time-consuming. Autocomplete
> feature
> > > > could be a real game-changer in this context.
> > > > 2. Different CLI tools. There are a couple of different shell
> scripts.
> > > > They have different goals and it's okay to keep them separated.
> However,
> > > > they raise a barrier for a non Pulsar expert that doesn't have a
> convenient
> > > > entry-point.
> > > >
> > > > ## Goal
> > > >
> > > > Address all the issues in the previous section with a single
> solution.
> > > >
> > > > ## API Changes
> > > >
> > > > A new shell script `bin/pulsar-shell` will be introduced. `bin/pulsar
> > > > shell` could be a valid alternative but it's not findable for a
> newbie user
> > > > since no direct file exists.
> > > >
> > > > ## Implementation
> > > >
> > > > ### Concepts
> > > > The new script `pulsar-shell` will differ from the existing for the
> > > > following reasons:
> > > >
> > > > 1. It's a shell. When you start it, it will wait for commands to be
> > > > executed. After the command has been executed, despite its result,
> the
> > > > shell session will not be destroyed and it will wait for another
> command.
> > > > 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be able to
> run all
> > > > the existing CLI commands. This will be done in a way that when a new
> > > > command/option is added, the pulsar-shell will be updated
> accordingly.
> > > > 3. It comes with sophisticated autocompletion and command history to
> highly
> > > > improve the UX.
> > > > 4. Performance. Since JVM is initiated once, it will gain on
> performance
> > > > thanks to the JVM warmup and internal libraries bootstraps.
> > > > 5. It will accept a file or a list of commands (parameter and stdin)
> to
> > > > start a shell, run the commands and close the shell. We'll call it
> the
> > > > `non-interactive` mode and it will ease the cluster operations
> automations.
> > > >
> > > > Note that existing tools will not be removed/changed.
> > > >
> > > > ### Implementation
> > > >
> > > > The shell implementation will be developed in Java, using a
> well-known
> > > > library called [JLine 3](https://github.com/jline/jline3) for the
> shell
> > > > support.
> > > > There will be a new main class that will extends the existing class
> tools
> > > > (e.g. `PulsarAdminTool`)
> > > >
> > > > #### Configuration
> > > > The configuration file taken by default will be `client.conf`, like
> current
> > > > CLI tools. The env setup will be exactly the same as for
> `pulsar-admin` and
> > > > `pulsar-client`.
> > > >
> > > > #### Autocompletion
> > > > JLine3 has great support for autocompletion. The shell java class
> will
> > > > translate current `JCommander` tools to the JLine3 completion API.
> This
> > > > will ensure that all APIs will be up-to-date and covered.
> > > >
> > > > #### History
> > > > JLine3 has built-in support for history, both in-memory and
> persisted (on
> > > > local FS). By default the history will be persisted in the user home
> > > > directory. This feature can be turned off for security concerns with
> a
> > > > configuration property.
> > > >
> > > > #### List of tools
> > > > The proposal is to get the following tools:
> > > > - bin/pulsar-admin
> > > > - bin/pulsar-client
> > > > - bin/pulsar-perf
> > > >
> > > >
> > > > ## Rejected Alternatives
> > > >
> > > > - Create a shell mode for `pulsar-admin`. This won't be a flexible
> solution
> > > > because then we may need the same for other tools. Also it doesn't
> cover
> > > > the CLI unification part.
> > > > - Create a brand-new shell mode. Not compatible with current CLI and
> > > > another tool to maintain.
> > > > - Use another shell library. There are a couple of valid
> alternatives to
> > > > JLine3 but it's wide-spread and the autocompletion API is much more
> > > > flexible than others since it's not done with annotations (the shell
> needs
> > > > to translate current JCommander definitions from existing tools)
> > > >
> > > >
> > > >
> > > >
> > > > Nicolò Boschi
> > > >
>

Re: [DISCUSS] PIP-181: Pulsar Shell

Posted by Hang Chen <ch...@apache.org>.

+1 Great work!

Thanks,
Hang

PengHui Li <co...@gmail.com> 于2022年7月4日周一 19:36写道：
>
> + 1
>
> Penghui
> On Jul 4, 2022, 18:22 +0800, Dezhi Liu <de...@apache.org>, wrote:
> > +1, Great Job.
> >
> > Thanks,
> > Dezhi
> >
> > On 2022/06/28 08:06:01 Nicolò Boschi wrote:
> > > Hi all,
> > >
> > > I opened a new PIP about Pulsar CLI tools.
> > > Looking forward to seeing comments and suggestions.
> > >
> > > PIP: https://github.com/apache/pulsar/issues/16250
> > >
> > > I posted a short video that shows how the new tool will work:
> > > https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4
> > >
> > >
> > >
> > >
> > > ## Motivation
> > >
> > > Currently Pulsar comes with a couple of utility scripts with the goal of
> > > managing an existing cluster, test behaviours and verify performances:
> > > these tools are available as SH script inside the `bin` directory.
> > > The `pulsar-admin` is the CLI tool supposed to help users and operators to
> > > configure the system, operate over policies, install functions and much
> > > else.
> > >
> > > This proposal basically aims to solve two different problems:
> > >
> > > 1. `pulsar-admin` is terribly slow. Every time the script is triggered, a
> > > new JVM process is spawned. The JVM process creation is heavy and most of
> > > the time is spent by the JVM initialization process. A very common use case
> > > for cluster operators is to create scripts with several commands with the
> > > goal of initialize the cluster, initialize a specific tenant (namespaces,
> > > topics, policies, functions..); in this case, one JVM is initialized for
> > > each scripts leads to waste of time and resources.
> > >
> > > 2. User experience. The current design of the Pulsar CLIs can be improved.
> > > There are a couple of aspects that may be annoying for a user and can
> > > discourage a user to use Pulsar.
> > > 1. Poking around available commands and options in a CLI tool
> > > (`pulsar-admin` for instance, but it's the same for `pulsar-perf` and
> > > `pulsar-client`) is slow and hard. In order to discover commands and
> > > options you need to use `-h` option and, since the performance issue
> > > pointed at 1., it can be annoying and time-consuming. Autocomplete feature
> > > could be a real game-changer in this context.
> > > 2. Different CLI tools. There are a couple of different shell scripts.
> > > They have different goals and it's okay to keep them separated. However,
> > > they raise a barrier for a non Pulsar expert that doesn't have a convenient
> > > entry-point.
> > >
> > > ## Goal
> > >
> > > Address all the issues in the previous section with a single solution.
> > >
> > > ## API Changes
> > >
> > > A new shell script `bin/pulsar-shell` will be introduced. `bin/pulsar
> > > shell` could be a valid alternative but it's not findable for a newbie user
> > > since no direct file exists.
> > >
> > > ## Implementation
> > >
> > > ### Concepts
> > > The new script `pulsar-shell` will differ from the existing for the
> > > following reasons:
> > >
> > > 1. It's a shell. When you start it, it will wait for commands to be
> > > executed. After the command has been executed, despite its result, the
> > > shell session will not be destroyed and it will wait for another command.
> > > 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be able to run all
> > > the existing CLI commands. This will be done in a way that when a new
> > > command/option is added, the pulsar-shell will be updated accordingly.
> > > 3. It comes with sophisticated autocompletion and command history to highly
> > > improve the UX.
> > > 4. Performance. Since JVM is initiated once, it will gain on performance
> > > thanks to the JVM warmup and internal libraries bootstraps.
> > > 5. It will accept a file or a list of commands (parameter and stdin) to
> > > start a shell, run the commands and close the shell. We'll call it the
> > > `non-interactive` mode and it will ease the cluster operations automations.
> > >
> > > Note that existing tools will not be removed/changed.
> > >
> > > ### Implementation
> > >
> > > The shell implementation will be developed in Java, using a well-known
> > > library called [JLine 3](https://github.com/jline/jline3) for the shell
> > > support.
> > > There will be a new main class that will extends the existing class tools
> > > (e.g. `PulsarAdminTool`)
> > >
> > > #### Configuration
> > > The configuration file taken by default will be `client.conf`, like current
> > > CLI tools. The env setup will be exactly the same as for `pulsar-admin` and
> > > `pulsar-client`.
> > >
> > > #### Autocompletion
> > > JLine3 has great support for autocompletion. The shell java class will
> > > translate current `JCommander` tools to the JLine3 completion API. This
> > > will ensure that all APIs will be up-to-date and covered.
> > >
> > > #### History
> > > JLine3 has built-in support for history, both in-memory and persisted (on
> > > local FS). By default the history will be persisted in the user home
> > > directory. This feature can be turned off for security concerns with a
> > > configuration property.
> > >
> > > #### List of tools
> > > The proposal is to get the following tools:
> > > - bin/pulsar-admin
> > > - bin/pulsar-client
> > > - bin/pulsar-perf
> > >
> > >
> > > ## Rejected Alternatives
> > >
> > > - Create a shell mode for `pulsar-admin`. This won't be a flexible solution
> > > because then we may need the same for other tools. Also it doesn't cover
> > > the CLI unification part.
> > > - Create a brand-new shell mode. Not compatible with current CLI and
> > > another tool to maintain.
> > > - Use another shell library. There are a couple of valid alternatives to
> > > JLine3 but it's wide-spread and the autocompletion API is much more
> > > flexible than others since it's not done with annotations (the shell needs
> > > to translate current JCommander definitions from existing tools)
> > >
> > >
> > >
> > >
> > > Nicolò Boschi
> > >

Re: [DISCUSS] PIP-181: Pulsar Shell

Posted by PengHui Li <co...@gmail.com>.

+ 1

Penghui
On Jul 4, 2022, 18:22 +0800, Dezhi Liu <de...@apache.org>, wrote:
> +1, Great Job.
>
> Thanks,
> Dezhi
>
> On 2022/06/28 08:06:01 Nicolò Boschi wrote:
> > Hi all,
> >
> > I opened a new PIP about Pulsar CLI tools.
> > Looking forward to seeing comments and suggestions.
> >
> > PIP: https://github.com/apache/pulsar/issues/16250
> >
> > I posted a short video that shows how the new tool will work:
> > https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4
> >
> >
> >
> >
> > ## Motivation
> >
> > Currently Pulsar comes with a couple of utility scripts with the goal of
> > managing an existing cluster, test behaviours and verify performances:
> > these tools are available as SH script inside the `bin` directory.
> > The `pulsar-admin` is the CLI tool supposed to help users and operators to
> > configure the system, operate over policies, install functions and much
> > else.
> >
> > This proposal basically aims to solve two different problems:
> >
> > 1. `pulsar-admin` is terribly slow. Every time the script is triggered, a
> > new JVM process is spawned. The JVM process creation is heavy and most of
> > the time is spent by the JVM initialization process. A very common use case
> > for cluster operators is to create scripts with several commands with the
> > goal of initialize the cluster, initialize a specific tenant (namespaces,
> > topics, policies, functions..); in this case, one JVM is initialized for
> > each scripts leads to waste of time and resources.
> >
> > 2. User experience. The current design of the Pulsar CLIs can be improved.
> > There are a couple of aspects that may be annoying for a user and can
> > discourage a user to use Pulsar.
> > 1. Poking around available commands and options in a CLI tool
> > (`pulsar-admin` for instance, but it's the same for `pulsar-perf` and
> > `pulsar-client`) is slow and hard. In order to discover commands and
> > options you need to use `-h` option and, since the performance issue
> > pointed at 1., it can be annoying and time-consuming. Autocomplete feature
> > could be a real game-changer in this context.
> > 2. Different CLI tools. There are a couple of different shell scripts.
> > They have different goals and it's okay to keep them separated. However,
> > they raise a barrier for a non Pulsar expert that doesn't have a convenient
> > entry-point.
> >
> > ## Goal
> >
> > Address all the issues in the previous section with a single solution.
> >
> > ## API Changes
> >
> > A new shell script `bin/pulsar-shell` will be introduced. `bin/pulsar
> > shell` could be a valid alternative but it's not findable for a newbie user
> > since no direct file exists.
> >
> > ## Implementation
> >
> > ### Concepts
> > The new script `pulsar-shell` will differ from the existing for the
> > following reasons:
> >
> > 1. It's a shell. When you start it, it will wait for commands to be
> > executed. After the command has been executed, despite its result, the
> > shell session will not be destroyed and it will wait for another command.
> > 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be able to run all
> > the existing CLI commands. This will be done in a way that when a new
> > command/option is added, the pulsar-shell will be updated accordingly.
> > 3. It comes with sophisticated autocompletion and command history to highly
> > improve the UX.
> > 4. Performance. Since JVM is initiated once, it will gain on performance
> > thanks to the JVM warmup and internal libraries bootstraps.
> > 5. It will accept a file or a list of commands (parameter and stdin) to
> > start a shell, run the commands and close the shell. We'll call it the
> > `non-interactive` mode and it will ease the cluster operations automations.
> >
> > Note that existing tools will not be removed/changed.
> >
> > ### Implementation
> >
> > The shell implementation will be developed in Java, using a well-known
> > library called [JLine 3](https://github.com/jline/jline3) for the shell
> > support.
> > There will be a new main class that will extends the existing class tools
> > (e.g. `PulsarAdminTool`)
> >
> > #### Configuration
> > The configuration file taken by default will be `client.conf`, like current
> > CLI tools. The env setup will be exactly the same as for `pulsar-admin` and
> > `pulsar-client`.
> >
> > #### Autocompletion
> > JLine3 has great support for autocompletion. The shell java class will
> > translate current `JCommander` tools to the JLine3 completion API. This
> > will ensure that all APIs will be up-to-date and covered.
> >
> > #### History
> > JLine3 has built-in support for history, both in-memory and persisted (on
> > local FS). By default the history will be persisted in the user home
> > directory. This feature can be turned off for security concerns with a
> > configuration property.
> >
> > #### List of tools
> > The proposal is to get the following tools:
> > - bin/pulsar-admin
> > - bin/pulsar-client
> > - bin/pulsar-perf
> >
> >
> > ## Rejected Alternatives
> >
> > - Create a shell mode for `pulsar-admin`. This won't be a flexible solution
> > because then we may need the same for other tools. Also it doesn't cover
> > the CLI unification part.
> > - Create a brand-new shell mode. Not compatible with current CLI and
> > another tool to maintain.
> > - Use another shell library. There are a couple of valid alternatives to
> > JLine3 but it's wide-spread and the autocompletion API is much more
> > flexible than others since it's not done with annotations (the shell needs
> > to translate current JCommander definitions from existing tools)
> >
> >
> >
> >
> > Nicolò Boschi
> >

Re: [DISCUSS] PIP-181: Pulsar Shell

Posted by Dezhi Liu <de...@apache.org>.

+1, Great Job.

Thanks,
Dezhi

On 2022/06/28 08:06:01 Nicolò Boschi wrote:
> Hi all,
> 
> I opened a new PIP about Pulsar CLI tools.
> Looking forward to seeing comments and suggestions.
> 
> PIP: https://github.com/apache/pulsar/issues/16250
> 
> I posted a short video that shows how the new tool will work:
> https://user-images.githubusercontent.com/23314389/176125261-35e123a1-1826-4553-b912-28d00914c0e4.mp4
> 
> 
> 
> 
> ## Motivation
> 
> Currently Pulsar comes with a couple of utility scripts with the goal of
> managing an existing cluster, test behaviours and verify performances:
> these tools are available as SH script inside the `bin` directory.
> The `pulsar-admin` is the CLI tool supposed to help users and operators to
> configure the system, operate over policies, install functions and much
> else.
> 
> This proposal basically aims to solve two different problems:
> 
> 1. `pulsar-admin` is terribly slow. Every time the script is triggered, a
> new JVM process is spawned. The JVM process creation is heavy and most of
> the time is spent by the JVM initialization process. A very common use case
> for cluster operators is to create scripts with several commands with the
> goal of initialize the cluster, initialize a specific tenant (namespaces,
> topics, policies, functions..); in this case, one JVM is initialized for
> each scripts leads to waste of time and resources.
> 
> 2. User experience. The current design of the Pulsar CLIs can be improved.
> There are a couple of aspects that may be annoying for a user and can
> discourage a user to use Pulsar.
>     1. Poking around available commands and options in a CLI tool
> (`pulsar-admin` for instance, but it's the same for `pulsar-perf` and
> `pulsar-client`) is slow and hard. In order to discover commands and
> options you need to use `-h` option and, since the performance issue
> pointed at 1., it can be annoying and time-consuming. Autocomplete feature
> could be a real game-changer in this context.
>     2. Different CLI tools. There are a couple of different shell scripts.
> They have different goals and it's okay to keep them separated. However,
> they raise a barrier for a non Pulsar expert that doesn't have a convenient
> entry-point.
> 
> ## Goal
> 
> Address all the issues in the previous section with a single solution.
> 
> ## API Changes
> 
> A new shell script `bin/pulsar-shell` will be introduced. `bin/pulsar
> shell` could be a valid alternative but it's not findable for a newbie user
> since no direct file exists.
> 
> ## Implementation
> 
> ### Concepts
> The new script `pulsar-shell` will differ from the existing for the
> following reasons:
> 
> 1. It's a shell. When you start it, it will wait for commands to be
> executed. After the command has been executed, despite its result, the
> shell session will not be destroyed and it will wait for another command.
> 2. Unifies all the CLI scripts. In `pulsar-shell` you'll be able to run all
> the existing CLI commands. This will be done in a way that when a new
> command/option is added, the pulsar-shell will be updated accordingly.
> 3. It comes with sophisticated autocompletion and command history to highly
> improve the UX.
> 4. Performance. Since JVM is initiated once, it will gain on performance
> thanks to the JVM warmup and internal libraries bootstraps.
> 5. It will accept a file or a list of commands (parameter and stdin) to
> start a shell, run the commands and close the shell. We'll call it the
> `non-interactive` mode and it will ease the cluster operations automations.
> 
> Note that existing tools will not be removed/changed.
> 
> ### Implementation
> 
> The shell implementation will be developed in Java, using a well-known
> library called [JLine 3](https://github.com/jline/jline3) for the shell
> support.
> There will be a new main class that will extends the existing class tools
> (e.g. `PulsarAdminTool`)
> 
> #### Configuration
> The configuration file taken by default will be `client.conf`, like current
> CLI tools. The env setup will be exactly the same as for `pulsar-admin` and
> `pulsar-client`.
> 
> #### Autocompletion
> JLine3 has great support for autocompletion. The shell java class will
> translate current `JCommander` tools to the JLine3 completion API. This
> will ensure that all APIs will be up-to-date and covered.
> 
> #### History
> JLine3 has built-in support for history, both in-memory and persisted (on
> local FS). By default the history will be persisted in the user home
> directory. This feature can be turned off for security concerns with a
> configuration property.
> 
> #### List of tools
> The proposal is to get the following tools:
> - bin/pulsar-admin
> - bin/pulsar-client
> - bin/pulsar-perf
> 
> 
> ## Rejected Alternatives
> 
> - Create a shell mode for `pulsar-admin`. This won't be a flexible solution
> because then we may need the same for other tools. Also it doesn't cover
> the CLI unification part.
> - Create a brand-new shell mode. Not compatible with current CLI and
> another tool to maintain.
> - Use another shell library. There are a couple of valid alternatives to
> JLine3 but it's wide-spread and the autocompletion API is much more
> flexible than others since it's not done with annotations (the shell needs
> to translate current JCommander definitions from existing tools)
> 
> 
> 
> 
> Nicolò Boschi
>