You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by Chetan Mehrotra <ch...@gmail.com> on 2018/04/26 06:05:36 UTC
Implement wskadmin in scala
Hi Team,
Currently for OpenWhisk admin operation we have tooling implemented in
couple of python scripts like wskadmin, tools/db/* etc. These script
currently talk directly to CouchDB to perform required actions.
Sometime back I discussed the option to support other databases [1]
and it was suggested to have wskadmin support various db backends.
However looking into other scripts I found some of the tool/db would
also be useful in context of other backends also.
To simplify this aspect going forward it may be better to implement
the important tooling in Scala itself as a separate sub module in core
repo. This module would produce a 'fat runnable jar' which would be
including all required dependency and can be used as a standalone cli
tool.
We used similar approach in Apache Jackrabbit Oak [2] where we produce
this single jar which consolidates all the admin tooling. This has
over the years became primary admin tooling for us.
Such an approach would have following benefits
1. Implemented in Scala and thus able to leverage existing
abstractions like ArtifactStore
2. For some of the bulk db operations it would be possible to leverage
Akka Streams to implement simpler multi threaded flows.
3. Easy to implement tests for the tooling part
4. User management operations can be done via existing ArtifactStore
feature set. So one implementation can work against multiple stores
5. No other runtime dependency i.e. specific Python version or Python
module need to be deployed. Just have JDK 1.8 and use the jar in
standalone manner. No need to even check out whole OpenWhisk repo
Key requirement for such a tooling would be to be compatible with
existing CLI argument format.
If such an approach makes sense I can work on PR to give it a try!
Chetan Mehrotra
[1] https://lists.apache.org/thread.html/921a0a6350a7ec3a2dc77564612de59104995622f8417583291f20bc@%3Cdev.openwhisk.apache.org%3E
[2] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run
Re: Implement wskadmin in scala
Posted by Markus Thoemmes <ma...@de.ibm.com>.
Hi Chetan,
Sounds like a good idea to me, especially with regards to the testability. One drawback I can see is that the user needs to have a JRE installed and configured which may or may not be a pain on remote hosts. Maybe Scala Native can work as well? (Not sure how well supported akka and friends are by Scala Native).
Other than that, my 2c towards compatibility with the existing tool: Think freely. If we can do better, we should. We can always deprecate stuff, not sure how broad wskadmin is used today.
Cheers,
Markus
Re: Implement wskadmin in scala
Posted by Carlos Santana <cs...@gmail.com>.
LGTM Chetan
-cs
On Wed, Jun 6, 2018 at 12:19 AM Chetan Mehrotra <ch...@gmail.com>
wrote:
> Thanks for the feedback. Based on it would do following changes
>
> 1. Move the src folder from core/admin to tools/admin/scala
> 2. Rename the binary output to wskadmin-next
>
> Chetan Mehrotra
>
>
> On Wed, Jun 6, 2018 at 5:33 AM, Carlos Santana <cs...@gmail.com>
> wrote:
> > +1
> >
> > Keep it in core some where under tools/admin/ maybe tools/admin/next/
> > Name it something transient like Rodric said wskadmin-next
> >
> > wskadmin can call/exec out to wskadmin-next for new features as
> transition
> > approach as next phase.
> >
> >
> >
> > On Tue, Jun 5, 2018 at 2:49 PM Rodric Rabbah <ro...@gmail.com> wrote:
> >
> >> To echo Matt:
> >>
> >> * I strongly advocate keeping the code in tools/admin, or core/tools as
> the
> >> appropriate location. This is not intended for end users but for system
> >> administrators and should have a tight coupling with the core system.
> >>
> >> * I would give the tool a transient name for now (wsk-admin,
> wskadmin-cli I
> >> do not feel strongly), and once all the functionality is subsumed,
> rename
> >> it to wskadmin.
> >>
> >> Note that the reason for the Scala implementation (in lieu of the Python
> >> implementation for wskadmin which is otherwise perfectly fine) is to
> reuse
> >> the code assets already developed the common package. These assets are
> used
> >> in the core components (controller, invoker).
> >>
> >> -r
> >>
> >>
> >> On Tue, Jun 5, 2018 at 2:27 PM, Matt Rutkowski <mr...@apache.org>
> >> wrote:
> >>
> >> > Hi Chetan,
> >> >
> >> > Took a look at your rather complete PR...
> >> >
> >> > My first thought was that we were trying to align all our client (CLI,
> >> > etc.) tooling around GoLang as it is, in theory, easier for
> developers to
> >> > contribute to and in addition had fewer Java dependencies/legal
> >> > complications for binary distribution.
> >> >
> >> > In terms of where it should "live" either follow the paradigm of wsk
> CLI
> >> > (i.e., incubator-openwhisk-admin-cli or some similar name) or include
> it
> >> > under a logical directory structure under main openwhisk repo.
> >> >
> >> > As it is Scala, and its function is tightly coupled to main OW
> (server),
> >> > IMO including it in main OW seems like the better choice at the
> moment.
> >> >
> >> > -Matt
> >> >
> >> > On 2018/06/04 06:39:59, Chetan Mehrotra <ch...@gmail.com>
> >> > wrote:
> >> > > Following up on this thread ...
> >> > >
> >> > > There is now work in progress PR #3722 [1] which implements the
> >> > > proposed tooling in Scala. Currently it supports some of the `user`
> >> > > commands like `create`, `delete` etc.
> >> > >
> >> > > There are few open questions also like
> >> > >
> >> > > 1. What should be the name of cli. Currently its named as
> wskadmin-cli
> >> > >
> >> > > 2. Where should the code live. Currently its in core/admin
> >> > >
> >> > > Please have a look and share feedback related to the approach taken
> >> > > i.e. whether its fine to pursue this as done or there are some
> >> > > concerns.
> >> > >
> >> > > Chetan Mehrotra
> >> > > [1] https://github.com/apache/incubator-openwhisk/pull/3722
> >> > >
> >> > >
> >> > > On Thu, Apr 26, 2018 at 2:46 PM, Chetan Mehrotra
> >> > > <ch...@gmail.com> wrote:
> >> > > >> (wskadmin) could become more heavy weight
> >> > > >
> >> > > > Yes thats a concern and Python dev is more light weight. I would
> >> still
> >> > > > prefer Python for ad hoc tooling required for one off tasks. But
> >> > > > anything which needs to be stable and supported properly for
> general
> >> > > > use it would be better to go for proposed approach.
> >> > > >
> >> > > >> Are you considering the totality if wskadmin or a partitioning
> and
> >> > only replacing some of partitions?
> >> > > >
> >> > > > For now the focus is on DB specific task i.e. user,limits,db. For
> >> > > > syslog I am not sure as I think its more dev tooling and can only
> >> work
> >> > > > for local setup. Other command in wskadmin on the other admin can
> be
> >> > > > used for production setups if required.
> >> > > > Chetan Mehrotra
> >> > > >
> >> > > >
> >> > > > On Thu, Apr 26, 2018 at 2:36 PM, Rodric Rabbah <ro...@gmail.com>
> >> > wrote:
> >> > > >> My initial reaction is that it (wskadmin) could become more heavy
> >> > weight (small changes becomes longer edit, compile iterations) - case
> in
> >> > point the wsk cli in Python vs Go... but weighed against the benefits
> >> > Chetan outlined with potential for a lot of shared code with the
> backend
> >> > cannot be discounted.
> >> > > >>
> >> > > >> I’m not familiar with oak-run and will take a look to educate
> >> myself.
> >> > > >>
> >> > > >> Are you considering the totality if wskadmin or a partitioning
> and
> >> > only replacing some of partitions? (I understood the former, just
> making
> >> > sure.)
> >> > > >>
> >> > > >> -r
> >> > > >>
> >> > > >>> On Apr 26, 2018, at 2:05 AM, Chetan Mehrotra <
> >> > chetan.mehrotra@gmail.com> wrote:
> >> > > >>>
> >> > > >>> Hi Team,
> >> > > >>>
> >> > > >>> Currently for OpenWhisk admin operation we have tooling
> implemented
> >> > in
> >> > > >>> couple of python scripts like wskadmin, tools/db/* etc. These
> >> script
> >> > > >>> currently talk directly to CouchDB to perform required actions.
> >> > > >>>
> >> > > >>> Sometime back I discussed the option to support other databases
> [1]
> >> > > >>> and it was suggested to have wskadmin support various db
> backends.
> >> > > >>> However looking into other scripts I found some of the tool/db
> >> would
> >> > > >>> also be useful in context of other backends also.
> >> > > >>>
> >> > > >>> To simplify this aspect going forward it may be better to
> implement
> >> > > >>> the important tooling in Scala itself as a separate sub module
> in
> >> > core
> >> > > >>> repo. This module would produce a 'fat runnable jar' which
> would be
> >> > > >>> including all required dependency and can be used as a
> standalone
> >> cli
> >> > > >>> tool.
> >> > > >>>
> >> > > >>> We used similar approach in Apache Jackrabbit Oak [2] where we
> >> > produce
> >> > > >>> this single jar which consolidates all the admin tooling. This
> has
> >> > > >>> over the years became primary admin tooling for us.
> >> > > >>>
> >> > > >>> Such an approach would have following benefits
> >> > > >>>
> >> > > >>> 1. Implemented in Scala and thus able to leverage existing
> >> > > >>> abstractions like ArtifactStore
> >> > > >>>
> >> > > >>> 2. For some of the bulk db operations it would be possible to
> >> > leverage
> >> > > >>> Akka Streams to implement simpler multi threaded flows.
> >> > > >>>
> >> > > >>> 3. Easy to implement tests for the tooling part
> >> > > >>>
> >> > > >>> 4. User management operations can be done via existing
> >> ArtifactStore
> >> > > >>> feature set. So one implementation can work against multiple
> stores
> >> > > >>>
> >> > > >>> 5. No other runtime dependency i.e. specific Python version or
> >> Python
> >> > > >>> module need to be deployed. Just have JDK 1.8 and use the jar in
> >> > > >>> standalone manner. No need to even check out whole OpenWhisk
> repo
> >> > > >>>
> >> > > >>> Key requirement for such a tooling would be to be compatible
> with
> >> > > >>> existing CLI argument format.
> >> > > >>>
> >> > > >>> If such an approach makes sense I can work on PR to give it a
> try!
> >> > > >>>
> >> > > >>> Chetan Mehrotra
> >> > > >>> [1]
> https://lists.apache.org/thread.html/921a0a6350a7ec3a2dc7756
> >> > 4612de59104995622f8417583291f20bc@%3Cdev.openwhisk.apache.org%3E
> >> > > >>> [2] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run
> >> > >
> >> >
> >>
>
Re: Implement wskadmin in scala
Posted by Chetan Mehrotra <ch...@gmail.com>.
Thanks for the feedback. Based on it would do following changes
1. Move the src folder from core/admin to tools/admin/scala
2. Rename the binary output to wskadmin-next
Chetan Mehrotra
On Wed, Jun 6, 2018 at 5:33 AM, Carlos Santana <cs...@gmail.com> wrote:
> +1
>
> Keep it in core some where under tools/admin/ maybe tools/admin/next/
> Name it something transient like Rodric said wskadmin-next
>
> wskadmin can call/exec out to wskadmin-next for new features as transition
> approach as next phase.
>
>
>
> On Tue, Jun 5, 2018 at 2:49 PM Rodric Rabbah <ro...@gmail.com> wrote:
>
>> To echo Matt:
>>
>> * I strongly advocate keeping the code in tools/admin, or core/tools as the
>> appropriate location. This is not intended for end users but for system
>> administrators and should have a tight coupling with the core system.
>>
>> * I would give the tool a transient name for now (wsk-admin, wskadmin-cli I
>> do not feel strongly), and once all the functionality is subsumed, rename
>> it to wskadmin.
>>
>> Note that the reason for the Scala implementation (in lieu of the Python
>> implementation for wskadmin which is otherwise perfectly fine) is to reuse
>> the code assets already developed the common package. These assets are used
>> in the core components (controller, invoker).
>>
>> -r
>>
>>
>> On Tue, Jun 5, 2018 at 2:27 PM, Matt Rutkowski <mr...@apache.org>
>> wrote:
>>
>> > Hi Chetan,
>> >
>> > Took a look at your rather complete PR...
>> >
>> > My first thought was that we were trying to align all our client (CLI,
>> > etc.) tooling around GoLang as it is, in theory, easier for developers to
>> > contribute to and in addition had fewer Java dependencies/legal
>> > complications for binary distribution.
>> >
>> > In terms of where it should "live" either follow the paradigm of wsk CLI
>> > (i.e., incubator-openwhisk-admin-cli or some similar name) or include it
>> > under a logical directory structure under main openwhisk repo.
>> >
>> > As it is Scala, and its function is tightly coupled to main OW (server),
>> > IMO including it in main OW seems like the better choice at the moment.
>> >
>> > -Matt
>> >
>> > On 2018/06/04 06:39:59, Chetan Mehrotra <ch...@gmail.com>
>> > wrote:
>> > > Following up on this thread ...
>> > >
>> > > There is now work in progress PR #3722 [1] which implements the
>> > > proposed tooling in Scala. Currently it supports some of the `user`
>> > > commands like `create`, `delete` etc.
>> > >
>> > > There are few open questions also like
>> > >
>> > > 1. What should be the name of cli. Currently its named as wskadmin-cli
>> > >
>> > > 2. Where should the code live. Currently its in core/admin
>> > >
>> > > Please have a look and share feedback related to the approach taken
>> > > i.e. whether its fine to pursue this as done or there are some
>> > > concerns.
>> > >
>> > > Chetan Mehrotra
>> > > [1] https://github.com/apache/incubator-openwhisk/pull/3722
>> > >
>> > >
>> > > On Thu, Apr 26, 2018 at 2:46 PM, Chetan Mehrotra
>> > > <ch...@gmail.com> wrote:
>> > > >> (wskadmin) could become more heavy weight
>> > > >
>> > > > Yes thats a concern and Python dev is more light weight. I would
>> still
>> > > > prefer Python for ad hoc tooling required for one off tasks. But
>> > > > anything which needs to be stable and supported properly for general
>> > > > use it would be better to go for proposed approach.
>> > > >
>> > > >> Are you considering the totality if wskadmin or a partitioning and
>> > only replacing some of partitions?
>> > > >
>> > > > For now the focus is on DB specific task i.e. user,limits,db. For
>> > > > syslog I am not sure as I think its more dev tooling and can only
>> work
>> > > > for local setup. Other command in wskadmin on the other admin can be
>> > > > used for production setups if required.
>> > > > Chetan Mehrotra
>> > > >
>> > > >
>> > > > On Thu, Apr 26, 2018 at 2:36 PM, Rodric Rabbah <ro...@gmail.com>
>> > wrote:
>> > > >> My initial reaction is that it (wskadmin) could become more heavy
>> > weight (small changes becomes longer edit, compile iterations) - case in
>> > point the wsk cli in Python vs Go... but weighed against the benefits
>> > Chetan outlined with potential for a lot of shared code with the backend
>> > cannot be discounted.
>> > > >>
>> > > >> I’m not familiar with oak-run and will take a look to educate
>> myself.
>> > > >>
>> > > >> Are you considering the totality if wskadmin or a partitioning and
>> > only replacing some of partitions? (I understood the former, just making
>> > sure.)
>> > > >>
>> > > >> -r
>> > > >>
>> > > >>> On Apr 26, 2018, at 2:05 AM, Chetan Mehrotra <
>> > chetan.mehrotra@gmail.com> wrote:
>> > > >>>
>> > > >>> Hi Team,
>> > > >>>
>> > > >>> Currently for OpenWhisk admin operation we have tooling implemented
>> > in
>> > > >>> couple of python scripts like wskadmin, tools/db/* etc. These
>> script
>> > > >>> currently talk directly to CouchDB to perform required actions.
>> > > >>>
>> > > >>> Sometime back I discussed the option to support other databases [1]
>> > > >>> and it was suggested to have wskadmin support various db backends.
>> > > >>> However looking into other scripts I found some of the tool/db
>> would
>> > > >>> also be useful in context of other backends also.
>> > > >>>
>> > > >>> To simplify this aspect going forward it may be better to implement
>> > > >>> the important tooling in Scala itself as a separate sub module in
>> > core
>> > > >>> repo. This module would produce a 'fat runnable jar' which would be
>> > > >>> including all required dependency and can be used as a standalone
>> cli
>> > > >>> tool.
>> > > >>>
>> > > >>> We used similar approach in Apache Jackrabbit Oak [2] where we
>> > produce
>> > > >>> this single jar which consolidates all the admin tooling. This has
>> > > >>> over the years became primary admin tooling for us.
>> > > >>>
>> > > >>> Such an approach would have following benefits
>> > > >>>
>> > > >>> 1. Implemented in Scala and thus able to leverage existing
>> > > >>> abstractions like ArtifactStore
>> > > >>>
>> > > >>> 2. For some of the bulk db operations it would be possible to
>> > leverage
>> > > >>> Akka Streams to implement simpler multi threaded flows.
>> > > >>>
>> > > >>> 3. Easy to implement tests for the tooling part
>> > > >>>
>> > > >>> 4. User management operations can be done via existing
>> ArtifactStore
>> > > >>> feature set. So one implementation can work against multiple stores
>> > > >>>
>> > > >>> 5. No other runtime dependency i.e. specific Python version or
>> Python
>> > > >>> module need to be deployed. Just have JDK 1.8 and use the jar in
>> > > >>> standalone manner. No need to even check out whole OpenWhisk repo
>> > > >>>
>> > > >>> Key requirement for such a tooling would be to be compatible with
>> > > >>> existing CLI argument format.
>> > > >>>
>> > > >>> If such an approach makes sense I can work on PR to give it a try!
>> > > >>>
>> > > >>> Chetan Mehrotra
>> > > >>> [1] https://lists.apache.org/thread.html/921a0a6350a7ec3a2dc7756
>> > 4612de59104995622f8417583291f20bc@%3Cdev.openwhisk.apache.org%3E
>> > > >>> [2] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run
>> > >
>> >
>>
Re: Implement wskadmin in scala
Posted by Carlos Santana <cs...@gmail.com>.
+1
Keep it in core some where under tools/admin/ maybe tools/admin/next/
Name it something transient like Rodric said wskadmin-next
wskadmin can call/exec out to wskadmin-next for new features as transition
approach as next phase.
On Tue, Jun 5, 2018 at 2:49 PM Rodric Rabbah <ro...@gmail.com> wrote:
> To echo Matt:
>
> * I strongly advocate keeping the code in tools/admin, or core/tools as the
> appropriate location. This is not intended for end users but for system
> administrators and should have a tight coupling with the core system.
>
> * I would give the tool a transient name for now (wsk-admin, wskadmin-cli I
> do not feel strongly), and once all the functionality is subsumed, rename
> it to wskadmin.
>
> Note that the reason for the Scala implementation (in lieu of the Python
> implementation for wskadmin which is otherwise perfectly fine) is to reuse
> the code assets already developed the common package. These assets are used
> in the core components (controller, invoker).
>
> -r
>
>
> On Tue, Jun 5, 2018 at 2:27 PM, Matt Rutkowski <mr...@apache.org>
> wrote:
>
> > Hi Chetan,
> >
> > Took a look at your rather complete PR...
> >
> > My first thought was that we were trying to align all our client (CLI,
> > etc.) tooling around GoLang as it is, in theory, easier for developers to
> > contribute to and in addition had fewer Java dependencies/legal
> > complications for binary distribution.
> >
> > In terms of where it should "live" either follow the paradigm of wsk CLI
> > (i.e., incubator-openwhisk-admin-cli or some similar name) or include it
> > under a logical directory structure under main openwhisk repo.
> >
> > As it is Scala, and its function is tightly coupled to main OW (server),
> > IMO including it in main OW seems like the better choice at the moment.
> >
> > -Matt
> >
> > On 2018/06/04 06:39:59, Chetan Mehrotra <ch...@gmail.com>
> > wrote:
> > > Following up on this thread ...
> > >
> > > There is now work in progress PR #3722 [1] which implements the
> > > proposed tooling in Scala. Currently it supports some of the `user`
> > > commands like `create`, `delete` etc.
> > >
> > > There are few open questions also like
> > >
> > > 1. What should be the name of cli. Currently its named as wskadmin-cli
> > >
> > > 2. Where should the code live. Currently its in core/admin
> > >
> > > Please have a look and share feedback related to the approach taken
> > > i.e. whether its fine to pursue this as done or there are some
> > > concerns.
> > >
> > > Chetan Mehrotra
> > > [1] https://github.com/apache/incubator-openwhisk/pull/3722
> > >
> > >
> > > On Thu, Apr 26, 2018 at 2:46 PM, Chetan Mehrotra
> > > <ch...@gmail.com> wrote:
> > > >> (wskadmin) could become more heavy weight
> > > >
> > > > Yes thats a concern and Python dev is more light weight. I would
> still
> > > > prefer Python for ad hoc tooling required for one off tasks. But
> > > > anything which needs to be stable and supported properly for general
> > > > use it would be better to go for proposed approach.
> > > >
> > > >> Are you considering the totality if wskadmin or a partitioning and
> > only replacing some of partitions?
> > > >
> > > > For now the focus is on DB specific task i.e. user,limits,db. For
> > > > syslog I am not sure as I think its more dev tooling and can only
> work
> > > > for local setup. Other command in wskadmin on the other admin can be
> > > > used for production setups if required.
> > > > Chetan Mehrotra
> > > >
> > > >
> > > > On Thu, Apr 26, 2018 at 2:36 PM, Rodric Rabbah <ro...@gmail.com>
> > wrote:
> > > >> My initial reaction is that it (wskadmin) could become more heavy
> > weight (small changes becomes longer edit, compile iterations) - case in
> > point the wsk cli in Python vs Go... but weighed against the benefits
> > Chetan outlined with potential for a lot of shared code with the backend
> > cannot be discounted.
> > > >>
> > > >> I’m not familiar with oak-run and will take a look to educate
> myself.
> > > >>
> > > >> Are you considering the totality if wskadmin or a partitioning and
> > only replacing some of partitions? (I understood the former, just making
> > sure.)
> > > >>
> > > >> -r
> > > >>
> > > >>> On Apr 26, 2018, at 2:05 AM, Chetan Mehrotra <
> > chetan.mehrotra@gmail.com> wrote:
> > > >>>
> > > >>> Hi Team,
> > > >>>
> > > >>> Currently for OpenWhisk admin operation we have tooling implemented
> > in
> > > >>> couple of python scripts like wskadmin, tools/db/* etc. These
> script
> > > >>> currently talk directly to CouchDB to perform required actions.
> > > >>>
> > > >>> Sometime back I discussed the option to support other databases [1]
> > > >>> and it was suggested to have wskadmin support various db backends.
> > > >>> However looking into other scripts I found some of the tool/db
> would
> > > >>> also be useful in context of other backends also.
> > > >>>
> > > >>> To simplify this aspect going forward it may be better to implement
> > > >>> the important tooling in Scala itself as a separate sub module in
> > core
> > > >>> repo. This module would produce a 'fat runnable jar' which would be
> > > >>> including all required dependency and can be used as a standalone
> cli
> > > >>> tool.
> > > >>>
> > > >>> We used similar approach in Apache Jackrabbit Oak [2] where we
> > produce
> > > >>> this single jar which consolidates all the admin tooling. This has
> > > >>> over the years became primary admin tooling for us.
> > > >>>
> > > >>> Such an approach would have following benefits
> > > >>>
> > > >>> 1. Implemented in Scala and thus able to leverage existing
> > > >>> abstractions like ArtifactStore
> > > >>>
> > > >>> 2. For some of the bulk db operations it would be possible to
> > leverage
> > > >>> Akka Streams to implement simpler multi threaded flows.
> > > >>>
> > > >>> 3. Easy to implement tests for the tooling part
> > > >>>
> > > >>> 4. User management operations can be done via existing
> ArtifactStore
> > > >>> feature set. So one implementation can work against multiple stores
> > > >>>
> > > >>> 5. No other runtime dependency i.e. specific Python version or
> Python
> > > >>> module need to be deployed. Just have JDK 1.8 and use the jar in
> > > >>> standalone manner. No need to even check out whole OpenWhisk repo
> > > >>>
> > > >>> Key requirement for such a tooling would be to be compatible with
> > > >>> existing CLI argument format.
> > > >>>
> > > >>> If such an approach makes sense I can work on PR to give it a try!
> > > >>>
> > > >>> Chetan Mehrotra
> > > >>> [1] https://lists.apache.org/thread.html/921a0a6350a7ec3a2dc7756
> > 4612de59104995622f8417583291f20bc@%3Cdev.openwhisk.apache.org%3E
> > > >>> [2] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run
> > >
> >
>
Re: Implement wskadmin in scala
Posted by Rodric Rabbah <ro...@gmail.com>.
To echo Matt:
* I strongly advocate keeping the code in tools/admin, or core/tools as the
appropriate location. This is not intended for end users but for system
administrators and should have a tight coupling with the core system.
* I would give the tool a transient name for now (wsk-admin, wskadmin-cli I
do not feel strongly), and once all the functionality is subsumed, rename
it to wskadmin.
Note that the reason for the Scala implementation (in lieu of the Python
implementation for wskadmin which is otherwise perfectly fine) is to reuse
the code assets already developed the common package. These assets are used
in the core components (controller, invoker).
-r
On Tue, Jun 5, 2018 at 2:27 PM, Matt Rutkowski <mr...@apache.org>
wrote:
> Hi Chetan,
>
> Took a look at your rather complete PR...
>
> My first thought was that we were trying to align all our client (CLI,
> etc.) tooling around GoLang as it is, in theory, easier for developers to
> contribute to and in addition had fewer Java dependencies/legal
> complications for binary distribution.
>
> In terms of where it should "live" either follow the paradigm of wsk CLI
> (i.e., incubator-openwhisk-admin-cli or some similar name) or include it
> under a logical directory structure under main openwhisk repo.
>
> As it is Scala, and its function is tightly coupled to main OW (server),
> IMO including it in main OW seems like the better choice at the moment.
>
> -Matt
>
> On 2018/06/04 06:39:59, Chetan Mehrotra <ch...@gmail.com>
> wrote:
> > Following up on this thread ...
> >
> > There is now work in progress PR #3722 [1] which implements the
> > proposed tooling in Scala. Currently it supports some of the `user`
> > commands like `create`, `delete` etc.
> >
> > There are few open questions also like
> >
> > 1. What should be the name of cli. Currently its named as wskadmin-cli
> >
> > 2. Where should the code live. Currently its in core/admin
> >
> > Please have a look and share feedback related to the approach taken
> > i.e. whether its fine to pursue this as done or there are some
> > concerns.
> >
> > Chetan Mehrotra
> > [1] https://github.com/apache/incubator-openwhisk/pull/3722
> >
> >
> > On Thu, Apr 26, 2018 at 2:46 PM, Chetan Mehrotra
> > <ch...@gmail.com> wrote:
> > >> (wskadmin) could become more heavy weight
> > >
> > > Yes thats a concern and Python dev is more light weight. I would still
> > > prefer Python for ad hoc tooling required for one off tasks. But
> > > anything which needs to be stable and supported properly for general
> > > use it would be better to go for proposed approach.
> > >
> > >> Are you considering the totality if wskadmin or a partitioning and
> only replacing some of partitions?
> > >
> > > For now the focus is on DB specific task i.e. user,limits,db. For
> > > syslog I am not sure as I think its more dev tooling and can only work
> > > for local setup. Other command in wskadmin on the other admin can be
> > > used for production setups if required.
> > > Chetan Mehrotra
> > >
> > >
> > > On Thu, Apr 26, 2018 at 2:36 PM, Rodric Rabbah <ro...@gmail.com>
> wrote:
> > >> My initial reaction is that it (wskadmin) could become more heavy
> weight (small changes becomes longer edit, compile iterations) - case in
> point the wsk cli in Python vs Go... but weighed against the benefits
> Chetan outlined with potential for a lot of shared code with the backend
> cannot be discounted.
> > >>
> > >> I’m not familiar with oak-run and will take a look to educate myself.
> > >>
> > >> Are you considering the totality if wskadmin or a partitioning and
> only replacing some of partitions? (I understood the former, just making
> sure.)
> > >>
> > >> -r
> > >>
> > >>> On Apr 26, 2018, at 2:05 AM, Chetan Mehrotra <
> chetan.mehrotra@gmail.com> wrote:
> > >>>
> > >>> Hi Team,
> > >>>
> > >>> Currently for OpenWhisk admin operation we have tooling implemented
> in
> > >>> couple of python scripts like wskadmin, tools/db/* etc. These script
> > >>> currently talk directly to CouchDB to perform required actions.
> > >>>
> > >>> Sometime back I discussed the option to support other databases [1]
> > >>> and it was suggested to have wskadmin support various db backends.
> > >>> However looking into other scripts I found some of the tool/db would
> > >>> also be useful in context of other backends also.
> > >>>
> > >>> To simplify this aspect going forward it may be better to implement
> > >>> the important tooling in Scala itself as a separate sub module in
> core
> > >>> repo. This module would produce a 'fat runnable jar' which would be
> > >>> including all required dependency and can be used as a standalone cli
> > >>> tool.
> > >>>
> > >>> We used similar approach in Apache Jackrabbit Oak [2] where we
> produce
> > >>> this single jar which consolidates all the admin tooling. This has
> > >>> over the years became primary admin tooling for us.
> > >>>
> > >>> Such an approach would have following benefits
> > >>>
> > >>> 1. Implemented in Scala and thus able to leverage existing
> > >>> abstractions like ArtifactStore
> > >>>
> > >>> 2. For some of the bulk db operations it would be possible to
> leverage
> > >>> Akka Streams to implement simpler multi threaded flows.
> > >>>
> > >>> 3. Easy to implement tests for the tooling part
> > >>>
> > >>> 4. User management operations can be done via existing ArtifactStore
> > >>> feature set. So one implementation can work against multiple stores
> > >>>
> > >>> 5. No other runtime dependency i.e. specific Python version or Python
> > >>> module need to be deployed. Just have JDK 1.8 and use the jar in
> > >>> standalone manner. No need to even check out whole OpenWhisk repo
> > >>>
> > >>> Key requirement for such a tooling would be to be compatible with
> > >>> existing CLI argument format.
> > >>>
> > >>> If such an approach makes sense I can work on PR to give it a try!
> > >>>
> > >>> Chetan Mehrotra
> > >>> [1] https://lists.apache.org/thread.html/921a0a6350a7ec3a2dc7756
> 4612de59104995622f8417583291f20bc@%3Cdev.openwhisk.apache.org%3E
> > >>> [2] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run
> >
>
Re: Implement wskadmin in scala
Posted by Matt Rutkowski <mr...@apache.org>.
Hi Chetan,
Took a look at your rather complete PR...
My first thought was that we were trying to align all our client (CLI, etc.) tooling around GoLang as it is, in theory, easier for developers to contribute to and in addition had fewer Java dependencies/legal complications for binary distribution.
In terms of where it should "live" either follow the paradigm of wsk CLI (i.e., incubator-openwhisk-admin-cli or some similar name) or include it under a logical directory structure under main openwhisk repo.
As it is Scala, and its function is tightly coupled to main OW (server), IMO including it in main OW seems like the better choice at the moment.
-Matt
On 2018/06/04 06:39:59, Chetan Mehrotra <ch...@gmail.com> wrote:
> Following up on this thread ...
>
> There is now work in progress PR #3722 [1] which implements the
> proposed tooling in Scala. Currently it supports some of the `user`
> commands like `create`, `delete` etc.
>
> There are few open questions also like
>
> 1. What should be the name of cli. Currently its named as wskadmin-cli
>
> 2. Where should the code live. Currently its in core/admin
>
> Please have a look and share feedback related to the approach taken
> i.e. whether its fine to pursue this as done or there are some
> concerns.
>
> Chetan Mehrotra
> [1] https://github.com/apache/incubator-openwhisk/pull/3722
>
>
> On Thu, Apr 26, 2018 at 2:46 PM, Chetan Mehrotra
> <ch...@gmail.com> wrote:
> >> (wskadmin) could become more heavy weight
> >
> > Yes thats a concern and Python dev is more light weight. I would still
> > prefer Python for ad hoc tooling required for one off tasks. But
> > anything which needs to be stable and supported properly for general
> > use it would be better to go for proposed approach.
> >
> >> Are you considering the totality if wskadmin or a partitioning and only replacing some of partitions?
> >
> > For now the focus is on DB specific task i.e. user,limits,db. For
> > syslog I am not sure as I think its more dev tooling and can only work
> > for local setup. Other command in wskadmin on the other admin can be
> > used for production setups if required.
> > Chetan Mehrotra
> >
> >
> > On Thu, Apr 26, 2018 at 2:36 PM, Rodric Rabbah <ro...@gmail.com> wrote:
> >> My initial reaction is that it (wskadmin) could become more heavy weight (small changes becomes longer edit, compile iterations) - case in point the wsk cli in Python vs Go... but weighed against the benefits Chetan outlined with potential for a lot of shared code with the backend cannot be discounted.
> >>
> >> I’m not familiar with oak-run and will take a look to educate myself.
> >>
> >> Are you considering the totality if wskadmin or a partitioning and only replacing some of partitions? (I understood the former, just making sure.)
> >>
> >> -r
> >>
> >>> On Apr 26, 2018, at 2:05 AM, Chetan Mehrotra <ch...@gmail.com> wrote:
> >>>
> >>> Hi Team,
> >>>
> >>> Currently for OpenWhisk admin operation we have tooling implemented in
> >>> couple of python scripts like wskadmin, tools/db/* etc. These script
> >>> currently talk directly to CouchDB to perform required actions.
> >>>
> >>> Sometime back I discussed the option to support other databases [1]
> >>> and it was suggested to have wskadmin support various db backends.
> >>> However looking into other scripts I found some of the tool/db would
> >>> also be useful in context of other backends also.
> >>>
> >>> To simplify this aspect going forward it may be better to implement
> >>> the important tooling in Scala itself as a separate sub module in core
> >>> repo. This module would produce a 'fat runnable jar' which would be
> >>> including all required dependency and can be used as a standalone cli
> >>> tool.
> >>>
> >>> We used similar approach in Apache Jackrabbit Oak [2] where we produce
> >>> this single jar which consolidates all the admin tooling. This has
> >>> over the years became primary admin tooling for us.
> >>>
> >>> Such an approach would have following benefits
> >>>
> >>> 1. Implemented in Scala and thus able to leverage existing
> >>> abstractions like ArtifactStore
> >>>
> >>> 2. For some of the bulk db operations it would be possible to leverage
> >>> Akka Streams to implement simpler multi threaded flows.
> >>>
> >>> 3. Easy to implement tests for the tooling part
> >>>
> >>> 4. User management operations can be done via existing ArtifactStore
> >>> feature set. So one implementation can work against multiple stores
> >>>
> >>> 5. No other runtime dependency i.e. specific Python version or Python
> >>> module need to be deployed. Just have JDK 1.8 and use the jar in
> >>> standalone manner. No need to even check out whole OpenWhisk repo
> >>>
> >>> Key requirement for such a tooling would be to be compatible with
> >>> existing CLI argument format.
> >>>
> >>> If such an approach makes sense I can work on PR to give it a try!
> >>>
> >>> Chetan Mehrotra
> >>> [1] https://lists.apache.org/thread.html/921a0a6350a7ec3a2dc77564612de59104995622f8417583291f20bc@%3Cdev.openwhisk.apache.org%3E
> >>> [2] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run
>
Re: Implement wskadmin in scala
Posted by Chetan Mehrotra <ch...@gmail.com>.
Following up on this thread ...
There is now work in progress PR #3722 [1] which implements the
proposed tooling in Scala. Currently it supports some of the `user`
commands like `create`, `delete` etc.
There are few open questions also like
1. What should be the name of cli. Currently its named as wskadmin-cli
2. Where should the code live. Currently its in core/admin
Please have a look and share feedback related to the approach taken
i.e. whether its fine to pursue this as done or there are some
concerns.
Chetan Mehrotra
[1] https://github.com/apache/incubator-openwhisk/pull/3722
On Thu, Apr 26, 2018 at 2:46 PM, Chetan Mehrotra
<ch...@gmail.com> wrote:
>> (wskadmin) could become more heavy weight
>
> Yes thats a concern and Python dev is more light weight. I would still
> prefer Python for ad hoc tooling required for one off tasks. But
> anything which needs to be stable and supported properly for general
> use it would be better to go for proposed approach.
>
>> Are you considering the totality if wskadmin or a partitioning and only replacing some of partitions?
>
> For now the focus is on DB specific task i.e. user,limits,db. For
> syslog I am not sure as I think its more dev tooling and can only work
> for local setup. Other command in wskadmin on the other admin can be
> used for production setups if required.
> Chetan Mehrotra
>
>
> On Thu, Apr 26, 2018 at 2:36 PM, Rodric Rabbah <ro...@gmail.com> wrote:
>> My initial reaction is that it (wskadmin) could become more heavy weight (small changes becomes longer edit, compile iterations) - case in point the wsk cli in Python vs Go... but weighed against the benefits Chetan outlined with potential for a lot of shared code with the backend cannot be discounted.
>>
>> I’m not familiar with oak-run and will take a look to educate myself.
>>
>> Are you considering the totality if wskadmin or a partitioning and only replacing some of partitions? (I understood the former, just making sure.)
>>
>> -r
>>
>>> On Apr 26, 2018, at 2:05 AM, Chetan Mehrotra <ch...@gmail.com> wrote:
>>>
>>> Hi Team,
>>>
>>> Currently for OpenWhisk admin operation we have tooling implemented in
>>> couple of python scripts like wskadmin, tools/db/* etc. These script
>>> currently talk directly to CouchDB to perform required actions.
>>>
>>> Sometime back I discussed the option to support other databases [1]
>>> and it was suggested to have wskadmin support various db backends.
>>> However looking into other scripts I found some of the tool/db would
>>> also be useful in context of other backends also.
>>>
>>> To simplify this aspect going forward it may be better to implement
>>> the important tooling in Scala itself as a separate sub module in core
>>> repo. This module would produce a 'fat runnable jar' which would be
>>> including all required dependency and can be used as a standalone cli
>>> tool.
>>>
>>> We used similar approach in Apache Jackrabbit Oak [2] where we produce
>>> this single jar which consolidates all the admin tooling. This has
>>> over the years became primary admin tooling for us.
>>>
>>> Such an approach would have following benefits
>>>
>>> 1. Implemented in Scala and thus able to leverage existing
>>> abstractions like ArtifactStore
>>>
>>> 2. For some of the bulk db operations it would be possible to leverage
>>> Akka Streams to implement simpler multi threaded flows.
>>>
>>> 3. Easy to implement tests for the tooling part
>>>
>>> 4. User management operations can be done via existing ArtifactStore
>>> feature set. So one implementation can work against multiple stores
>>>
>>> 5. No other runtime dependency i.e. specific Python version or Python
>>> module need to be deployed. Just have JDK 1.8 and use the jar in
>>> standalone manner. No need to even check out whole OpenWhisk repo
>>>
>>> Key requirement for such a tooling would be to be compatible with
>>> existing CLI argument format.
>>>
>>> If such an approach makes sense I can work on PR to give it a try!
>>>
>>> Chetan Mehrotra
>>> [1] https://lists.apache.org/thread.html/921a0a6350a7ec3a2dc77564612de59104995622f8417583291f20bc@%3Cdev.openwhisk.apache.org%3E
>>> [2] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run
Re: Implement wskadmin in scala
Posted by Chetan Mehrotra <ch...@gmail.com>.
> (wskadmin) could become more heavy weight
Yes thats a concern and Python dev is more light weight. I would still
prefer Python for ad hoc tooling required for one off tasks. But
anything which needs to be stable and supported properly for general
use it would be better to go for proposed approach.
> Are you considering the totality if wskadmin or a partitioning and only replacing some of partitions?
For now the focus is on DB specific task i.e. user,limits,db. For
syslog I am not sure as I think its more dev tooling and can only work
for local setup. Other command in wskadmin on the other admin can be
used for production setups if required.
Chetan Mehrotra
On Thu, Apr 26, 2018 at 2:36 PM, Rodric Rabbah <ro...@gmail.com> wrote:
> My initial reaction is that it (wskadmin) could become more heavy weight (small changes becomes longer edit, compile iterations) - case in point the wsk cli in Python vs Go... but weighed against the benefits Chetan outlined with potential for a lot of shared code with the backend cannot be discounted.
>
> I’m not familiar with oak-run and will take a look to educate myself.
>
> Are you considering the totality if wskadmin or a partitioning and only replacing some of partitions? (I understood the former, just making sure.)
>
> -r
>
>> On Apr 26, 2018, at 2:05 AM, Chetan Mehrotra <ch...@gmail.com> wrote:
>>
>> Hi Team,
>>
>> Currently for OpenWhisk admin operation we have tooling implemented in
>> couple of python scripts like wskadmin, tools/db/* etc. These script
>> currently talk directly to CouchDB to perform required actions.
>>
>> Sometime back I discussed the option to support other databases [1]
>> and it was suggested to have wskadmin support various db backends.
>> However looking into other scripts I found some of the tool/db would
>> also be useful in context of other backends also.
>>
>> To simplify this aspect going forward it may be better to implement
>> the important tooling in Scala itself as a separate sub module in core
>> repo. This module would produce a 'fat runnable jar' which would be
>> including all required dependency and can be used as a standalone cli
>> tool.
>>
>> We used similar approach in Apache Jackrabbit Oak [2] where we produce
>> this single jar which consolidates all the admin tooling. This has
>> over the years became primary admin tooling for us.
>>
>> Such an approach would have following benefits
>>
>> 1. Implemented in Scala and thus able to leverage existing
>> abstractions like ArtifactStore
>>
>> 2. For some of the bulk db operations it would be possible to leverage
>> Akka Streams to implement simpler multi threaded flows.
>>
>> 3. Easy to implement tests for the tooling part
>>
>> 4. User management operations can be done via existing ArtifactStore
>> feature set. So one implementation can work against multiple stores
>>
>> 5. No other runtime dependency i.e. specific Python version or Python
>> module need to be deployed. Just have JDK 1.8 and use the jar in
>> standalone manner. No need to even check out whole OpenWhisk repo
>>
>> Key requirement for such a tooling would be to be compatible with
>> existing CLI argument format.
>>
>> If such an approach makes sense I can work on PR to give it a try!
>>
>> Chetan Mehrotra
>> [1] https://lists.apache.org/thread.html/921a0a6350a7ec3a2dc77564612de59104995622f8417583291f20bc@%3Cdev.openwhisk.apache.org%3E
>> [2] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run
Re: Implement wskadmin in scala
Posted by Rodric Rabbah <ro...@gmail.com>.
My initial reaction is that it (wskadmin) could become more heavy weight (small changes becomes longer edit, compile iterations) - case in point the wsk cli in Python vs Go... but weighed against the benefits Chetan outlined with potential for a lot of shared code with the backend cannot be discounted.
I’m not familiar with oak-run and will take a look to educate myself.
Are you considering the totality if wskadmin or a partitioning and only replacing some of partitions? (I understood the former, just making sure.)
-r
> On Apr 26, 2018, at 2:05 AM, Chetan Mehrotra <ch...@gmail.com> wrote:
>
> Hi Team,
>
> Currently for OpenWhisk admin operation we have tooling implemented in
> couple of python scripts like wskadmin, tools/db/* etc. These script
> currently talk directly to CouchDB to perform required actions.
>
> Sometime back I discussed the option to support other databases [1]
> and it was suggested to have wskadmin support various db backends.
> However looking into other scripts I found some of the tool/db would
> also be useful in context of other backends also.
>
> To simplify this aspect going forward it may be better to implement
> the important tooling in Scala itself as a separate sub module in core
> repo. This module would produce a 'fat runnable jar' which would be
> including all required dependency and can be used as a standalone cli
> tool.
>
> We used similar approach in Apache Jackrabbit Oak [2] where we produce
> this single jar which consolidates all the admin tooling. This has
> over the years became primary admin tooling for us.
>
> Such an approach would have following benefits
>
> 1. Implemented in Scala and thus able to leverage existing
> abstractions like ArtifactStore
>
> 2. For some of the bulk db operations it would be possible to leverage
> Akka Streams to implement simpler multi threaded flows.
>
> 3. Easy to implement tests for the tooling part
>
> 4. User management operations can be done via existing ArtifactStore
> feature set. So one implementation can work against multiple stores
>
> 5. No other runtime dependency i.e. specific Python version or Python
> module need to be deployed. Just have JDK 1.8 and use the jar in
> standalone manner. No need to even check out whole OpenWhisk repo
>
> Key requirement for such a tooling would be to be compatible with
> existing CLI argument format.
>
> If such an approach makes sense I can work on PR to give it a try!
>
> Chetan Mehrotra
> [1] https://lists.apache.org/thread.html/921a0a6350a7ec3a2dc77564612de59104995622f8417583291f20bc@%3Cdev.openwhisk.apache.org%3E
> [2] https://github.com/apache/jackrabbit-oak/tree/trunk/oak-run