You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@knox.apache.org by sumit gupta <su...@apache.org> on 2017/01/30 16:38:38 UTC
[DISCUSS] Scoping KnoxShell changes for 0.12.0
Hey everyone,
The list of JIRAs for 0.12.0 have steadily increased over the last few
weeks. We have also had a lot of great activity and contributions
related to KIP-4 and KnoxShell improvements. I wanted to start a
discuss thread to tie things up a little bit for a reasonable
deliverable in this area in the 0.12.0 release.
Just to reiterate where we are:
We have had a lot of contributions that can be mapped to KIP-4 goals,
especially improvement number 4 in the list of improvements on KIP-4.
I believe Larry Mccay has a feature branch going for improvement number 2.
I have taken a stab at a Zeppelin interpreter (improvement number 3)
in a forked zeppelin repo that can be found here (the branch is
'knoxshell-interpreter'):
https://github.com/sumitg/zeppelin/tree/knoxshell-interpreter
and we have added some tests as part of KNOX-845 (improvement number 5).
Some open questions I have:
1. What are the use cases driving the Zeppelin interpreter? How is
that expected to be used and how can we make it easy to use out of
box?
2. Do we need a release module for KnoxShell? How do we want to
provide the download to users?
3. Do we need all of KIP-4 in to call this complete or is what we have
so far in the works good enough for 0.12.0?
I'm sure there are more questions to be had. I am excited by the
uptake of the client DSL library and its usefulness to end users. I
hope we can make it more useful and easier to consume in 0.12.0.
Thanks,
Sumit
Re: [DISCUSS] Scoping KnoxShell changes for 0.12.0
Posted by Sumit Gupta <su...@hortonworks.com>.
Thanks for the thoughts Larry and Sandeep.
I agree on both ideas. I do think that it would be good to be able to get
KnoxShell without having to install the gateway or having admin access.
That being said the gateway will be running somewhere and I think it is a
nice idea to be able to download it from the UI.
I¹ve added https://issues.apache.org/jira/browse/KNOX-865 to track the
release module for this.
Sumit
On 1/30/17, 1:41 PM, "larry mccay" <lm...@apache.org> wrote:
>Hi Sandeep -
>
>Interesting thoughts on the download.
>Personally, I think that we need a separate download as well.
>
>* This is more easily turned into an RPM or docker image, etc
>* The Admin UI currently requires admin credentials. It is likely to need
>to be able to support non-admins as well someday but we aren't there yet.
>While some users may be admins - not all will be. If we ever hang a test
>page off of the UI then we will likely need to support non-admins as well.
>Which will require not allowing management capabilities to be done for
>such
>users.
>* We could certainly add a link to a download for users that do have
>access
>to the Admin UI as well. In a lot of ways it makes sense. They need access
>to a gateway instance in order to use it anyway.
>
>thanks,
>
>--larry
>
>On Mon, Jan 30, 2017 at 12:56 PM, Sandeep More <mo...@gmail.com>
>wrote:
>
>> Thanks Sumit this is a great summary !
>>
>> I have thoughts on #2
>>
>> 2. Do we need a release module for KnoxShell? How do we want to
>> provide the download to users?
>>
>> SRM : Thinking out loud, I think it would be nice if KnoxShell has the
>>same
>> version as Knox and is packed in the same distribution as Knox. This
>>will
>> avoid confusion (on which KnoxShell is supported for which Knox version)
>> and additional testing (for different version of KnoxShell and Knox).
>>
>> As for the download, we could have a download link on the Admin UI (now
>> that we have a UI !!) the download could be packaged up as a zip file
>> within the WAR file. It would also be a bit convenient as no additional
>> downloads will be required.
>>
>> Best,
>> Sandeep
>>
>>
>>
>> On Mon, Jan 30, 2017 at 11:38 AM, sumit gupta <su...@apache.org> wrote:
>>
>> > Hey everyone,
>> >
>> > The list of JIRAs for 0.12.0 have steadily increased over the last few
>> > weeks. We have also had a lot of great activity and contributions
>> > related to KIP-4 and KnoxShell improvements. I wanted to start a
>> > discuss thread to tie things up a little bit for a reasonable
>> > deliverable in this area in the 0.12.0 release.
>> >
>> > Just to reiterate where we are:
>> >
>> > We have had a lot of contributions that can be mapped to KIP-4 goals,
>> > especially improvement number 4 in the list of improvements on KIP-4.
>> >
>> > I believe Larry Mccay has a feature branch going for improvement
>>number
>> 2.
>> >
>> > I have taken a stab at a Zeppelin interpreter (improvement number 3)
>> > in a forked zeppelin repo that can be found here (the branch is
>> > 'knoxshell-interpreter'):
>> >
>> > https://github.com/sumitg/zeppelin/tree/knoxshell-interpreter
>> >
>> > and we have added some tests as part of KNOX-845 (improvement number
>>5).
>> >
>> > Some open questions I have:
>> >
>> > 1. What are the use cases driving the Zeppelin interpreter? How is
>> > that expected to be used and how can we make it easy to use out of
>> > box?
>> >
>> > 2. Do we need a release module for KnoxShell? How do we want to
>> > provide the download to users?
>> >
>> > 3. Do we need all of KIP-4 in to call this complete or is what we have
>> > so far in the works good enough for 0.12.0?
>> >
>> > I'm sure there are more questions to be had. I am excited by the
>> > uptake of the client DSL library and its usefulness to end users. I
>> > hope we can make it more useful and easier to consume in 0.12.0.
>> >
>> > Thanks,
>> > Sumit
>> >
>>
Re: [DISCUSS] Scoping KnoxShell changes for 0.12.0
Posted by larry mccay <lm...@apache.org>.
Hi Sandeep -
Interesting thoughts on the download.
Personally, I think that we need a separate download as well.
* This is more easily turned into an RPM or docker image, etc
* The Admin UI currently requires admin credentials. It is likely to need
to be able to support non-admins as well someday but we aren't there yet.
While some users may be admins - not all will be. If we ever hang a test
page off of the UI then we will likely need to support non-admins as well.
Which will require not allowing management capabilities to be done for such
users.
* We could certainly add a link to a download for users that do have access
to the Admin UI as well. In a lot of ways it makes sense. They need access
to a gateway instance in order to use it anyway.
thanks,
--larry
On Mon, Jan 30, 2017 at 12:56 PM, Sandeep More <mo...@gmail.com>
wrote:
> Thanks Sumit this is a great summary !
>
> I have thoughts on #2
>
> 2. Do we need a release module for KnoxShell? How do we want to
> provide the download to users?
>
> SRM : Thinking out loud, I think it would be nice if KnoxShell has the same
> version as Knox and is packed in the same distribution as Knox. This will
> avoid confusion (on which KnoxShell is supported for which Knox version)
> and additional testing (for different version of KnoxShell and Knox).
>
> As for the download, we could have a download link on the Admin UI (now
> that we have a UI !!) the download could be packaged up as a zip file
> within the WAR file. It would also be a bit convenient as no additional
> downloads will be required.
>
> Best,
> Sandeep
>
>
>
> On Mon, Jan 30, 2017 at 11:38 AM, sumit gupta <su...@apache.org> wrote:
>
> > Hey everyone,
> >
> > The list of JIRAs for 0.12.0 have steadily increased over the last few
> > weeks. We have also had a lot of great activity and contributions
> > related to KIP-4 and KnoxShell improvements. I wanted to start a
> > discuss thread to tie things up a little bit for a reasonable
> > deliverable in this area in the 0.12.0 release.
> >
> > Just to reiterate where we are:
> >
> > We have had a lot of contributions that can be mapped to KIP-4 goals,
> > especially improvement number 4 in the list of improvements on KIP-4.
> >
> > I believe Larry Mccay has a feature branch going for improvement number
> 2.
> >
> > I have taken a stab at a Zeppelin interpreter (improvement number 3)
> > in a forked zeppelin repo that can be found here (the branch is
> > 'knoxshell-interpreter'):
> >
> > https://github.com/sumitg/zeppelin/tree/knoxshell-interpreter
> >
> > and we have added some tests as part of KNOX-845 (improvement number 5).
> >
> > Some open questions I have:
> >
> > 1. What are the use cases driving the Zeppelin interpreter? How is
> > that expected to be used and how can we make it easy to use out of
> > box?
> >
> > 2. Do we need a release module for KnoxShell? How do we want to
> > provide the download to users?
> >
> > 3. Do we need all of KIP-4 in to call this complete or is what we have
> > so far in the works good enough for 0.12.0?
> >
> > I'm sure there are more questions to be had. I am excited by the
> > uptake of the client DSL library and its usefulness to end users. I
> > hope we can make it more useful and easier to consume in 0.12.0.
> >
> > Thanks,
> > Sumit
> >
>
Re: [DISCUSS] Scoping KnoxShell changes for 0.12.0
Posted by Sandeep More <mo...@gmail.com>.
Thanks Sumit this is a great summary !
I have thoughts on #2
2. Do we need a release module for KnoxShell? How do we want to
provide the download to users?
SRM : Thinking out loud, I think it would be nice if KnoxShell has the same
version as Knox and is packed in the same distribution as Knox. This will
avoid confusion (on which KnoxShell is supported for which Knox version)
and additional testing (for different version of KnoxShell and Knox).
As for the download, we could have a download link on the Admin UI (now
that we have a UI !!) the download could be packaged up as a zip file
within the WAR file. It would also be a bit convenient as no additional
downloads will be required.
Best,
Sandeep
On Mon, Jan 30, 2017 at 11:38 AM, sumit gupta <su...@apache.org> wrote:
> Hey everyone,
>
> The list of JIRAs for 0.12.0 have steadily increased over the last few
> weeks. We have also had a lot of great activity and contributions
> related to KIP-4 and KnoxShell improvements. I wanted to start a
> discuss thread to tie things up a little bit for a reasonable
> deliverable in this area in the 0.12.0 release.
>
> Just to reiterate where we are:
>
> We have had a lot of contributions that can be mapped to KIP-4 goals,
> especially improvement number 4 in the list of improvements on KIP-4.
>
> I believe Larry Mccay has a feature branch going for improvement number 2.
>
> I have taken a stab at a Zeppelin interpreter (improvement number 3)
> in a forked zeppelin repo that can be found here (the branch is
> 'knoxshell-interpreter'):
>
> https://github.com/sumitg/zeppelin/tree/knoxshell-interpreter
>
> and we have added some tests as part of KNOX-845 (improvement number 5).
>
> Some open questions I have:
>
> 1. What are the use cases driving the Zeppelin interpreter? How is
> that expected to be used and how can we make it easy to use out of
> box?
>
> 2. Do we need a release module for KnoxShell? How do we want to
> provide the download to users?
>
> 3. Do we need all of KIP-4 in to call this complete or is what we have
> so far in the works good enough for 0.12.0?
>
> I'm sure there are more questions to be had. I am excited by the
> uptake of the client DSL library and its usefulness to end users. I
> hope we can make it more useful and easier to consume in 0.12.0.
>
> Thanks,
> Sumit
>
Re: [DISCUSS] Scoping KnoxShell changes for 0.12.0
Posted by Pierre Regazzoni <pr...@gmail.com>.
I don’t see knox shell integration as replacement for other interpreter / language supported in Zeppelin. This is merely a usability enhancement. If I have a cluster behind knox I can quickly interact with the services via DSL also might required less configuration e.g. base, hive interpreter will need ‘client’ like configuration/jar. This is particularly interesting if notebook server is running outside the cluster. So right now I don’t have any other itch to scratch :) other than that.
I am not sure how integration with visualization works in zeppelin, it might be specific to interpreter e.g. %jdbc and what data is returned but potentially could work the same I guess or could simply leverage built-in display such as table … http://zeppelin.apache.org/docs/0.6.2/displaysystem/basicdisplaysystem.html#table
As far as use case, could show some interaction with supported hadoop services with knox, could be something like:
%knox
move files to hdfs
perform hive ql to build/transform data via webhcat — assume i did not setup hive interpreter or could not
%spark
query hive data via sparkSQL
— Pierre
> On Jan 30, 2017, at 8:40 PM, larry mccay <lm...@apache.org> wrote:
>
> Hi Pierre -
>
> Yes, Sumit has already implemented an interpreter to do that.
>
> I am more interested in what you see as the value-add or the reason that someone would want to use the Knox DSL instead of existing interpreters for largely the same access.
>
> From a language perspective the DSL does have closures for async operations which are handy but I think they are available in things like scala as well.
>
> If you haven't had a particular itch in mind that the DSL scratches then maybe it is fine to say that we have the ability to use the same DSL within a notebook as we are from your desktop.
>
> We would likely want to be able to provide some sort of visualization as is generally wanted in notebooks.
>
> Can we combine visualizations that are already available in Zeppelin for other interpreters with out own?
> I've seen that there is a MD interpreter for instance - can that be used to format render JSON results in a table for instance?
>
> What would we want to use as a showcase script in Zeppelin?
>
> thanks!
>
> --larry
>
> On Mon, Jan 30, 2017 at 7:57 PM, Pierre Regazzoni <pregazzoni@gmail.com <ma...@gmail.com>> wrote:
> Hi Larry,
>
> The basic premise would be to be able to open a knox shell within the notebook as follow:
>
> %knox
> Hdfs.rm(session).file(“/path/to/file”).now()
>
> knox host, port and credentials would need to be set in the plug-in configuration.
>
> This would allow directly client interaction with the cluster and leveraging the shell api within the notebook.
>
> —Pierre
>
>
>> On Jan 30, 2017, at 9:29 AM, larry mccay <lmccay@apache.org <ma...@apache.org>> wrote:
>>
>> Hi Sumit -
>>
>> Thanks for the check point summary and DISCUSS thread.
>> The summary actually sounds like we are really making some good progress - which I knew but hadn't seen it put all together like this!
>>
>> 1. What are the use cases driving the Zeppelin interpreter? How is
>> that expected to be used and how can we make it easy to use out of
>> box?
>>
>> <ljm>
>> Excellent question. Much of what we can do with the DSL in the interpreter is available in other interpreters.
>> The DSL has async operations which are handy and a similar programming mode across all the APIs - due to the fluent code style of the DSL.
>> One of the advantages of using the DSL over other CLI type approaches is the ability to source control the scripts - zeppelin would also provide a similar way to do this with notebooks.
>>
>> @Pierre - this was a suggestion that you made to me a while ago.
>> Can you articulate the value add that you envision for it?
>> </ljm>
>>
>> 2. Do we need a release module for KnoxShell? How do we want to
>> provide the download to users?
>>
>> <ljm>
>> I believe that we do at some point and probably before we go to a 1.0 release for Knox.
>> If we could add this for 0.12.0 as an early attempt that would be great and shouldn't be that difficult.
>> </ljm>
>>
>> 3. Do we need all of KIP-4 in to call this complete or is what we have
>> so far in the works good enough for 0.12.0?
>>
>> <ljm>
>> I don't believe that 0.12.0 has to be blocked by KIP-4.
>> As with KIP-1 (LDAP Improvements), they are used as the driving usecases for the releases but can and will continue to need work and completion beyond the initial target release. Focusing this way seems to be providing a great way to bootstrap progress in specific areas that can continue to be completed and improve from release to release.
>>
>> I do want to get the #2 improvement from KIP-4 (Token service and credential collector) feature branch merged for 0.12.0.
>> I think this opens up lots of possibilities and will be great to get some early adopters.
>> </ljm>
>>
>> I'm sure there are more questions to be had. I am excited by the
>> uptake of the client DSL library and its usefulness to end users. I
>> hope we can make it more useful and easier to consume in 0.12.0.
>>
>> <ljm>
>> I am also really excited about these improvements and uptake.
>> As we move to more and more cloud deployment scenarios, this aspect of Knox is going to be more and more important.
>>
>> One thing that I would really love to have articulated are some usecases that currently require SSH access by data workers that could be done through the KnoxShell and eliminate the need for SSH. Without some of these usecases we will likely fall short by a task or two and it will be difficult to cut off SSH.
>> </ljm>
>>
>> On Mon, Jan 30, 2017 at 11:38 AM, sumit gupta <sumit@apache.org <ma...@apache.org>> wrote:
>> Hey everyone,
>>
>> The list of JIRAs for 0.12.0 have steadily increased over the last few
>> weeks. We have also had a lot of great activity and contributions
>> related to KIP-4 and KnoxShell improvements. I wanted to start a
>> discuss thread to tie things up a little bit for a reasonable
>> deliverable in this area in the 0.12.0 release.
>>
>> Just to reiterate where we are:
>>
>> We have had a lot of contributions that can be mapped to KIP-4 goals,
>> especially improvement number 4 in the list of improvements on KIP-4.
>>
>> I believe Larry Mccay has a feature branch going for improvement number 2.
>>
>> I have taken a stab at a Zeppelin interpreter (improvement number 3)
>> in a forked zeppelin repo that can be found here (the branch is
>> 'knoxshell-interpreter'):
>>
>> https://github.com/sumitg/zeppelin/tree/knoxshell-interpreter <https://github.com/sumitg/zeppelin/tree/knoxshell-interpreter>
>>
>> and we have added some tests as part of KNOX-845 (improvement number 5).
>>
>> Some open questions I have:
>>
>> 1. What are the use cases driving the Zeppelin interpreter? How is
>> that expected to be used and how can we make it easy to use out of
>> box?
>>
>> 2. Do we need a release module for KnoxShell? How do we want to
>> provide the download to users?
>>
>> 3. Do we need all of KIP-4 in to call this complete or is what we have
>> so far in the works good enough for 0.12.0?
>>
>> I'm sure there are more questions to be had. I am excited by the
>> uptake of the client DSL library and its usefulness to end users. I
>> hope we can make it more useful and easier to consume in 0.12.0.
>>
>> Thanks,
>> Sumit
>>
>
>
Re: [DISCUSS] Scoping KnoxShell changes for 0.12.0
Posted by larry mccay <lm...@apache.org>.
Hi Pierre -
Yes, Sumit has already implemented an interpreter to do that.
I am more interested in what you see as the value-add or the reason that
someone would want to use the Knox DSL instead of existing interpreters for
largely the same access.
From a language perspective the DSL does have closures for async operations
which are handy but I think they are available in things like scala as well.
If you haven't had a particular itch in mind that the DSL scratches then
maybe it is fine to say that we have the ability to use the same DSL within
a notebook as we are from your desktop.
We would likely want to be able to provide some sort of visualization as is
generally wanted in notebooks.
Can we combine visualizations that are already available in Zeppelin for
other interpreters with out own?
I've seen that there is a MD interpreter for instance - can that be used to
format render JSON results in a table for instance?
What would we want to use as a showcase script in Zeppelin?
thanks!
--larry
On Mon, Jan 30, 2017 at 7:57 PM, Pierre Regazzoni <pr...@gmail.com>
wrote:
> Hi Larry,
>
> The basic premise would be to be able to open a knox shell within the
> notebook as follow:
>
> %knox
> Hdfs.rm(session).file(“/path/to/file”).now()
>
> knox host, port and credentials would need to be set in the plug-in
> configuration.
>
> This would allow directly client interaction with the cluster and
> leveraging the shell api within the notebook.
>
> —Pierre
>
>
> On Jan 30, 2017, at 9:29 AM, larry mccay <lm...@apache.org> wrote:
>
> Hi Sumit -
>
> Thanks for the check point summary and DISCUSS thread.
> The summary actually sounds like we are really making some good progress -
> which I knew but hadn't seen it put all together like this!
>
> 1. What are the use cases driving the Zeppelin interpreter? How is
> that expected to be used and how can we make it easy to use out of
> box?
>
> <ljm>
> Excellent question. Much of what we can do with the DSL in the interpreter
> is available in other interpreters.
> The DSL has async operations which are handy and a similar programming
> mode across all the APIs - due to the fluent code style of the DSL.
> One of the advantages of using the DSL over other CLI type approaches is
> the ability to source control the scripts - zeppelin would also provide a
> similar way to do this with notebooks.
>
> @Pierre - this was a suggestion that you made to me a while ago.
> Can you articulate the value add that you envision for it?
> </ljm>
>
> 2. Do we need a release module for KnoxShell? How do we want to
> provide the download to users?
>
> <ljm>
> I believe that we do at some point and probably before we go to a 1.0
> release for Knox.
> If we could add this for 0.12.0 as an early attempt that would be great
> and shouldn't be that difficult.
> </ljm>
>
> 3. Do we need all of KIP-4 in to call this complete or is what we have
> so far in the works good enough for 0.12.0?
>
> <ljm>
> I don't believe that 0.12.0 has to be blocked by KIP-4.
> As with KIP-1 (LDAP Improvements), they are used as the driving usecases
> for the releases but can and will continue to need work and completion
> beyond the initial target release. Focusing this way seems to be providing
> a great way to bootstrap progress in specific areas that can continue to be
> completed and improve from release to release.
>
> I do want to get the #2 improvement from KIP-4 (Token service and
> credential collector) feature branch merged for 0.12.0.
> I think this opens up lots of possibilities and will be great to get some
> early adopters.
> </ljm>
>
> I'm sure there are more questions to be had. I am excited by the
> uptake of the client DSL library and its usefulness to end users. I
> hope we can make it more useful and easier to consume in 0.12.0.
>
> <ljm>
> I am also really excited about these improvements and uptake.
> As we move to more and more cloud deployment scenarios, this aspect of
> Knox is going to be more and more important.
>
> One thing that I would really love to have articulated are some usecases
> that currently require SSH access by data workers that could be done
> through the KnoxShell and eliminate the need for SSH. Without some of these
> usecases we will likely fall short by a task or two and it will be
> difficult to cut off SSH.
> </ljm>
>
> On Mon, Jan 30, 2017 at 11:38 AM, sumit gupta <su...@apache.org> wrote:
>
>> Hey everyone,
>>
>> The list of JIRAs for 0.12.0 have steadily increased over the last few
>> weeks. We have also had a lot of great activity and contributions
>> related to KIP-4 and KnoxShell improvements. I wanted to start a
>> discuss thread to tie things up a little bit for a reasonable
>> deliverable in this area in the 0.12.0 release.
>>
>> Just to reiterate where we are:
>>
>> We have had a lot of contributions that can be mapped to KIP-4 goals,
>> especially improvement number 4 in the list of improvements on KIP-4.
>>
>> I believe Larry Mccay has a feature branch going for improvement number 2.
>>
>> I have taken a stab at a Zeppelin interpreter (improvement number 3)
>> in a forked zeppelin repo that can be found here (the branch is
>> 'knoxshell-interpreter'):
>>
>> https://github.com/sumitg/zeppelin/tree/knoxshell-interpreter
>>
>> and we have added some tests as part of KNOX-845 (improvement number 5).
>>
>> Some open questions I have:
>>
>> 1. What are the use cases driving the Zeppelin interpreter? How is
>> that expected to be used and how can we make it easy to use out of
>> box?
>>
>> 2. Do we need a release module for KnoxShell? How do we want to
>> provide the download to users?
>>
>> 3. Do we need all of KIP-4 in to call this complete or is what we have
>> so far in the works good enough for 0.12.0?
>>
>> I'm sure there are more questions to be had. I am excited by the
>> uptake of the client DSL library and its usefulness to end users. I
>> hope we can make it more useful and easier to consume in 0.12.0.
>>
>> Thanks,
>> Sumit
>>
>
>
>
Re: [DISCUSS] Scoping KnoxShell changes for 0.12.0
Posted by Pierre Regazzoni <pr...@gmail.com>.
Hi Larry,
The basic premise would be to be able to open a knox shell within the notebook as follow:
%knox
Hdfs.rm(session).file(“/path/to/file”).now()
knox host, port and credentials would need to be set in the plug-in configuration.
This would allow directly client interaction with the cluster and leveraging the shell api within the notebook.
—Pierre
> On Jan 30, 2017, at 9:29 AM, larry mccay <lm...@apache.org> wrote:
>
> Hi Sumit -
>
> Thanks for the check point summary and DISCUSS thread.
> The summary actually sounds like we are really making some good progress - which I knew but hadn't seen it put all together like this!
>
> 1. What are the use cases driving the Zeppelin interpreter? How is
> that expected to be used and how can we make it easy to use out of
> box?
>
> <ljm>
> Excellent question. Much of what we can do with the DSL in the interpreter is available in other interpreters.
> The DSL has async operations which are handy and a similar programming mode across all the APIs - due to the fluent code style of the DSL.
> One of the advantages of using the DSL over other CLI type approaches is the ability to source control the scripts - zeppelin would also provide a similar way to do this with notebooks.
>
> @Pierre - this was a suggestion that you made to me a while ago.
> Can you articulate the value add that you envision for it?
> </ljm>
>
> 2. Do we need a release module for KnoxShell? How do we want to
> provide the download to users?
>
> <ljm>
> I believe that we do at some point and probably before we go to a 1.0 release for Knox.
> If we could add this for 0.12.0 as an early attempt that would be great and shouldn't be that difficult.
> </ljm>
>
> 3. Do we need all of KIP-4 in to call this complete or is what we have
> so far in the works good enough for 0.12.0?
>
> <ljm>
> I don't believe that 0.12.0 has to be blocked by KIP-4.
> As with KIP-1 (LDAP Improvements), they are used as the driving usecases for the releases but can and will continue to need work and completion beyond the initial target release. Focusing this way seems to be providing a great way to bootstrap progress in specific areas that can continue to be completed and improve from release to release.
>
> I do want to get the #2 improvement from KIP-4 (Token service and credential collector) feature branch merged for 0.12.0.
> I think this opens up lots of possibilities and will be great to get some early adopters.
> </ljm>
>
> I'm sure there are more questions to be had. I am excited by the
> uptake of the client DSL library and its usefulness to end users. I
> hope we can make it more useful and easier to consume in 0.12.0.
>
> <ljm>
> I am also really excited about these improvements and uptake.
> As we move to more and more cloud deployment scenarios, this aspect of Knox is going to be more and more important.
>
> One thing that I would really love to have articulated are some usecases that currently require SSH access by data workers that could be done through the KnoxShell and eliminate the need for SSH. Without some of these usecases we will likely fall short by a task or two and it will be difficult to cut off SSH.
> </ljm>
>
> On Mon, Jan 30, 2017 at 11:38 AM, sumit gupta <sumit@apache.org <ma...@apache.org>> wrote:
> Hey everyone,
>
> The list of JIRAs for 0.12.0 have steadily increased over the last few
> weeks. We have also had a lot of great activity and contributions
> related to KIP-4 and KnoxShell improvements. I wanted to start a
> discuss thread to tie things up a little bit for a reasonable
> deliverable in this area in the 0.12.0 release.
>
> Just to reiterate where we are:
>
> We have had a lot of contributions that can be mapped to KIP-4 goals,
> especially improvement number 4 in the list of improvements on KIP-4.
>
> I believe Larry Mccay has a feature branch going for improvement number 2.
>
> I have taken a stab at a Zeppelin interpreter (improvement number 3)
> in a forked zeppelin repo that can be found here (the branch is
> 'knoxshell-interpreter'):
>
> https://github.com/sumitg/zeppelin/tree/knoxshell-interpreter <https://github.com/sumitg/zeppelin/tree/knoxshell-interpreter>
>
> and we have added some tests as part of KNOX-845 (improvement number 5).
>
> Some open questions I have:
>
> 1. What are the use cases driving the Zeppelin interpreter? How is
> that expected to be used and how can we make it easy to use out of
> box?
>
> 2. Do we need a release module for KnoxShell? How do we want to
> provide the download to users?
>
> 3. Do we need all of KIP-4 in to call this complete or is what we have
> so far in the works good enough for 0.12.0?
>
> I'm sure there are more questions to be had. I am excited by the
> uptake of the client DSL library and its usefulness to end users. I
> hope we can make it more useful and easier to consume in 0.12.0.
>
> Thanks,
> Sumit
>
Re: [DISCUSS] Scoping KnoxShell changes for 0.12.0
Posted by larry mccay <lm...@apache.org>.
Hi Sumit -
Thanks for the check point summary and DISCUSS thread.
The summary actually sounds like we are really making some good progress -
which I knew but hadn't seen it put all together like this!
1. What are the use cases driving the Zeppelin interpreter? How is
that expected to be used and how can we make it easy to use out of
box?
<ljm>
Excellent question. Much of what we can do with the DSL in the interpreter
is available in other interpreters.
The DSL has async operations which are handy and a similar programming mode
across all the APIs - due to the fluent code style of the DSL.
One of the advantages of using the DSL over other CLI type approaches is
the ability to source control the scripts - zeppelin would also provide a
similar way to do this with notebooks.
@Pierre - this was a suggestion that you made to me a while ago.
Can you articulate the value add that you envision for it?
</ljm>
2. Do we need a release module for KnoxShell? How do we want to
provide the download to users?
<ljm>
I believe that we do at some point and probably before we go to a 1.0
release for Knox.
If we could add this for 0.12.0 as an early attempt that would be great and
shouldn't be that difficult.
</ljm>
3. Do we need all of KIP-4 in to call this complete or is what we have
so far in the works good enough for 0.12.0?
<ljm>
I don't believe that 0.12.0 has to be blocked by KIP-4.
As with KIP-1 (LDAP Improvements), they are used as the driving usecases
for the releases but can and will continue to need work and completion
beyond the initial target release. Focusing this way seems to be providing
a great way to bootstrap progress in specific areas that can continue to be
completed and improve from release to release.
I do want to get the #2 improvement from KIP-4 (Token service and
credential collector) feature branch merged for 0.12.0.
I think this opens up lots of possibilities and will be great to get some
early adopters.
</ljm>
I'm sure there are more questions to be had. I am excited by the
uptake of the client DSL library and its usefulness to end users. I
hope we can make it more useful and easier to consume in 0.12.0.
<ljm>
I am also really excited about these improvements and uptake.
As we move to more and more cloud deployment scenarios, this aspect of Knox
is going to be more and more important.
One thing that I would really love to have articulated are some usecases
that currently require SSH access by data workers that could be done
through the KnoxShell and eliminate the need for SSH. Without some of these
usecases we will likely fall short by a task or two and it will be
difficult to cut off SSH.
</ljm>
On Mon, Jan 30, 2017 at 11:38 AM, sumit gupta <su...@apache.org> wrote:
> Hey everyone,
>
> The list of JIRAs for 0.12.0 have steadily increased over the last few
> weeks. We have also had a lot of great activity and contributions
> related to KIP-4 and KnoxShell improvements. I wanted to start a
> discuss thread to tie things up a little bit for a reasonable
> deliverable in this area in the 0.12.0 release.
>
> Just to reiterate where we are:
>
> We have had a lot of contributions that can be mapped to KIP-4 goals,
> especially improvement number 4 in the list of improvements on KIP-4.
>
> I believe Larry Mccay has a feature branch going for improvement number 2.
>
> I have taken a stab at a Zeppelin interpreter (improvement number 3)
> in a forked zeppelin repo that can be found here (the branch is
> 'knoxshell-interpreter'):
>
> https://github.com/sumitg/zeppelin/tree/knoxshell-interpreter
>
> and we have added some tests as part of KNOX-845 (improvement number 5).
>
> Some open questions I have:
>
> 1. What are the use cases driving the Zeppelin interpreter? How is
> that expected to be used and how can we make it easy to use out of
> box?
>
> 2. Do we need a release module for KnoxShell? How do we want to
> provide the download to users?
>
> 3. Do we need all of KIP-4 in to call this complete or is what we have
> so far in the works good enough for 0.12.0?
>
> I'm sure there are more questions to be had. I am excited by the
> uptake of the client DSL library and its usefulness to end users. I
> hope we can make it more useful and easier to consume in 0.12.0.
>
> Thanks,
> Sumit
>