[DISCUSS] Planning for Apache Knox 1.4

[larry mccay <lm...@apache.org>]

Folks -

Out last release with end of July, I apologize for the delay in starting
the planning thread for 1.4.

We currently have a backlog of ~65 JIRAs slated for a Fix Version of 1.4.

There has been some work going on within KnoxShell to provide a general
purpose representation for tabular data. This will be leveraged for
rendering SQL query results as well as CSV files and simple processing
within KnoxShell. I will be writing up a KIP to represent the overall
vision for this work and initial set of usecases.

We also have Cloudera Manager based discovery emerging and we should target
an initial set of services to enable for CM/CDH and CDP deployments where
CM is available.

With the continued increase in cloud based deployments and Knox Gateway use
in securely accessing the exposed resources, we will concentrate on
KnoxShell as a first class environment for this access. This will likely
include an API for discovering metadata about the resources exposed through
Knox, the required authentication mechanisms, resource types and public
certs. It will also include Custom GroovyShell Commands for the KnoxShell
environment to help interact with the remote clusters and resultsets as
local in-memory tables. I will be start a KIP to try and articulate this
vision and related 1.4. usecases as well.

I propose that the CM based Service Discovery and KnoxShell access to
remote clusters be the primary themes of the Apache Knox 1.4 release.

I also propose that we target the end of November as the release date for
1.4.

Thoughts?

--larry

Re: [DISCUSS] Planning for Apache Knox 1.4

[Phil Zampino <pz...@apache.org>]

I think the proposed themes are appropriate.

CM-based service discovery is partially there already, and need only be
fleshed out for completeness and tested.
We've wanted to provide much of the Knox CLI functionality through
KnoxShell (so it could be accessed remotely) for quite some time now.

+1

On Thu, Oct 31, 2019 at 11:53 AM larry mccay <lm...@apache.org> wrote:

> Folks -
>
> Out last release with end of July, I apologize for the delay in starting
> the planning thread for 1.4.
>
> We currently have a backlog of ~65 JIRAs slated for a Fix Version of 1.4.
>
> There has been some work going on within KnoxShell to provide a general
> purpose representation for tabular data. This will be leveraged for
> rendering SQL query results as well as CSV files and simple processing
> within KnoxShell. I will be writing up a KIP to represent the overall
> vision for this work and initial set of usecases.
>
> We also have Cloudera Manager based discovery emerging and we should target
> an initial set of services to enable for CM/CDH and CDP deployments where
> CM is available.
>
> With the continued increase in cloud based deployments and Knox Gateway use
> in securely accessing the exposed resources, we will concentrate on
> KnoxShell as a first class environment for this access. This will likely
> include an API for discovering metadata about the resources exposed through
> Knox, the required authentication mechanisms, resource types and public
> certs. It will also include Custom GroovyShell Commands for the KnoxShell
> environment to help interact with the remote clusters and resultsets as
> local in-memory tables. I will be start a KIP to try and articulate this
> vision and related 1.4. usecases as well.
>
> I propose that the CM based Service Discovery and KnoxShell access to
> remote clusters be the primary themes of the Apache Knox 1.4 release.
>
> I also propose that we target the end of November as the release date for
> 1.4.
>
> Thoughts?
>
> --larry
>

Re: [DISCUSS] Planning for Apache Knox 1.4

[larry mccay <lm...@apache.org>]

The entirety of KIP-14 doesn't need to be scoped for the 1.4 release.
JIRAs do need to be filed for the work detailed there and iterative
progress will be made across multiple releases.

I think that the initial goal should be to fill the role that the KnoxLine
example represented and drove interest in a Knox SQL client.
Being able to work with the results of a SQL query is an added benefit of
this approach and we can have some basic operations for doing just that.

I don't see the overall Knox metadata endpoint making it in the 1.4 release
timeframe.
I do see some of the underlying capabilities of things like datasources
being configurable and used to interact with SQL engines rather than having
to provide connection details for each session. I have an early version of
this already working locally.


On Mon, Nov 11, 2019 at 2:38 PM Kevin Risden <kr...@apache.org> wrote:

> Based on that we are ~1/2 way through November and Thanksgiving in the US
> around the corner, I don't see much of the KnoxShell pieces getting
> integrated before end of November to make an end of November release.
> Specifically the following section:
>
> With the continued increase in cloud based deployments and Knox Gateway use
> > in securely accessing the exposed resources, we will concentrate on
> > KnoxShell as a first class environment for this access. This will likely
> > include an API for discovering metadata about the resources exposed
> through
> > Knox, the required authentication mechanisms, resource types and public
> > certs. It will also include Custom GroovyShell Commands for the KnoxShell
> > environment to help interact with the remote clusters and resultsets as
> > local in-memory tables. I will be start a KIP to try and articulate this
> > vision and related 1.4. usecases as well.
> >
>
> The KIP was just started so would be good to flesh that out more instead of
> rushing it into 1.4.0. There are a lot of moving pieces in that paragraph
> and would be good to make sure the Jiras are created and scoped
> appropriately.
>
> In addition to what was mentioned as features, there have been multiple new
> service definitions (Impala, Kudu, NiFi Registry) added as well as fixes to
> existing service definitions (Atlas, Livy, Ranger, Spark, YARN).
>
> So +1 to an end of November release, but need to make sure not trying to
> rush in new things just because a release will happen. There will be more
> releases.
>
>
> Kevin Risden
>
>
> On Fri, Nov 1, 2019 at 11:51 AM Sandeep Moré <mo...@gmail.com>
> wrote:
>
> > Thanks for starting the planning thread Larry !
> > Agree with the theme and the release date for 1.4.0.
> >
> > +1
> >
> > Best,
> > Sandeep
> >
> > On Thu, Oct 31, 2019 at 11:53 AM larry mccay <lm...@apache.org> wrote:
> >
> >> Folks -
> >>
> >> Out last release with end of July, I apologize for the delay in starting
> >> the planning thread for 1.4.
> >>
> >> We currently have a backlog of ~65 JIRAs slated for a Fix Version of
> 1.4.
> >>
> >> There has been some work going on within KnoxShell to provide a general
> >> purpose representation for tabular data. This will be leveraged for
> >> rendering SQL query results as well as CSV files and simple processing
> >> within KnoxShell. I will be writing up a KIP to represent the overall
> >> vision for this work and initial set of usecases.
> >>
> >> We also have Cloudera Manager based discovery emerging and we should
> >> target an initial set of services to enable for CM/CDH and CDP
> deployments
> >> where CM is available.
> >>
> >> With the continued increase in cloud based deployments and Knox Gateway
> >> use in securely accessing the exposed resources, we will concentrate on
> >> KnoxShell as a first class environment for this access. This will likely
> >> include an API for discovering metadata about the resources exposed
> through
> >> Knox, the required authentication mechanisms, resource types and public
> >> certs. It will also include Custom GroovyShell Commands for the
> KnoxShell
> >> environment to help interact with the remote clusters and resultsets as
> >> local in-memory tables. I will be start a KIP to try and articulate this
> >> vision and related 1.4. usecases as well.
> >>
> >> I propose that the CM based Service Discovery and KnoxShell access to
> >> remote clusters be the primary themes of the Apache Knox 1.4 release.
> >>
> >> I also propose that we target the end of November as the release date
> for
> >> 1.4.
> >>
> >> Thoughts?
> >>
> >> --larry
> >>
> >
>

Re: [DISCUSS] Planning for Apache Knox 1.4

[larry mccay <lm...@apache.org>]

The entirety of KIP-14 doesn't need to be scoped for the 1.4 release.
JIRAs do need to be filed for the work detailed there and iterative
progress will be made across multiple releases.

I think that the initial goal should be to fill the role that the KnoxLine
example represented and drove interest in a Knox SQL client.
Being able to work with the results of a SQL query is an added benefit of
this approach and we can have some basic operations for doing just that.

I don't see the overall Knox metadata endpoint making it in the 1.4 release
timeframe.
I do see some of the underlying capabilities of things like datasources
being configurable and used to interact with SQL engines rather than having
to provide connection details for each session. I have an early version of
this already working locally.


On Mon, Nov 11, 2019 at 2:38 PM Kevin Risden <kr...@apache.org> wrote:

> Based on that we are ~1/2 way through November and Thanksgiving in the US
> around the corner, I don't see much of the KnoxShell pieces getting
> integrated before end of November to make an end of November release.
> Specifically the following section:
>
> With the continued increase in cloud based deployments and Knox Gateway use
> > in securely accessing the exposed resources, we will concentrate on
> > KnoxShell as a first class environment for this access. This will likely
> > include an API for discovering metadata about the resources exposed
> through
> > Knox, the required authentication mechanisms, resource types and public
> > certs. It will also include Custom GroovyShell Commands for the KnoxShell
> > environment to help interact with the remote clusters and resultsets as
> > local in-memory tables. I will be start a KIP to try and articulate this
> > vision and related 1.4. usecases as well.
> >
>
> The KIP was just started so would be good to flesh that out more instead of
> rushing it into 1.4.0. There are a lot of moving pieces in that paragraph
> and would be good to make sure the Jiras are created and scoped
> appropriately.
>
> In addition to what was mentioned as features, there have been multiple new
> service definitions (Impala, Kudu, NiFi Registry) added as well as fixes to
> existing service definitions (Atlas, Livy, Ranger, Spark, YARN).
>
> So +1 to an end of November release, but need to make sure not trying to
> rush in new things just because a release will happen. There will be more
> releases.
>
>
> Kevin Risden
>
>
> On Fri, Nov 1, 2019 at 11:51 AM Sandeep Moré <mo...@gmail.com>
> wrote:
>
> > Thanks for starting the planning thread Larry !
> > Agree with the theme and the release date for 1.4.0.
> >
> > +1
> >
> > Best,
> > Sandeep
> >
> > On Thu, Oct 31, 2019 at 11:53 AM larry mccay <lm...@apache.org> wrote:
> >
> >> Folks -
> >>
> >> Out last release with end of July, I apologize for the delay in starting
> >> the planning thread for 1.4.
> >>
> >> We currently have a backlog of ~65 JIRAs slated for a Fix Version of
> 1.4.
> >>
> >> There has been some work going on within KnoxShell to provide a general
> >> purpose representation for tabular data. This will be leveraged for
> >> rendering SQL query results as well as CSV files and simple processing
> >> within KnoxShell. I will be writing up a KIP to represent the overall
> >> vision for this work and initial set of usecases.
> >>
> >> We also have Cloudera Manager based discovery emerging and we should
> >> target an initial set of services to enable for CM/CDH and CDP
> deployments
> >> where CM is available.
> >>
> >> With the continued increase in cloud based deployments and Knox Gateway
> >> use in securely accessing the exposed resources, we will concentrate on
> >> KnoxShell as a first class environment for this access. This will likely
> >> include an API for discovering metadata about the resources exposed
> through
> >> Knox, the required authentication mechanisms, resource types and public
> >> certs. It will also include Custom GroovyShell Commands for the
> KnoxShell
> >> environment to help interact with the remote clusters and resultsets as
> >> local in-memory tables. I will be start a KIP to try and articulate this
> >> vision and related 1.4. usecases as well.
> >>
> >> I propose that the CM based Service Discovery and KnoxShell access to
> >> remote clusters be the primary themes of the Apache Knox 1.4 release.
> >>
> >> I also propose that we target the end of November as the release date
> for
> >> 1.4.
> >>
> >> Thoughts?
> >>
> >> --larry
> >>
> >
>

Re: [DISCUSS] Planning for Apache Knox 1.4

[Kevin Risden <kr...@apache.org>]

Based on that we are ~1/2 way through November and Thanksgiving in the US
around the corner, I don't see much of the KnoxShell pieces getting
integrated before end of November to make an end of November release.
Specifically the following section:

With the continued increase in cloud based deployments and Knox Gateway use
> in securely accessing the exposed resources, we will concentrate on
> KnoxShell as a first class environment for this access. This will likely
> include an API for discovering metadata about the resources exposed through
> Knox, the required authentication mechanisms, resource types and public
> certs. It will also include Custom GroovyShell Commands for the KnoxShell
> environment to help interact with the remote clusters and resultsets as
> local in-memory tables. I will be start a KIP to try and articulate this
> vision and related 1.4. usecases as well.
>

The KIP was just started so would be good to flesh that out more instead of
rushing it into 1.4.0. There are a lot of moving pieces in that paragraph
and would be good to make sure the Jiras are created and scoped
appropriately.

In addition to what was mentioned as features, there have been multiple new
service definitions (Impala, Kudu, NiFi Registry) added as well as fixes to
existing service definitions (Atlas, Livy, Ranger, Spark, YARN).

So +1 to an end of November release, but need to make sure not trying to
rush in new things just because a release will happen. There will be more
releases.


Kevin Risden


On Fri, Nov 1, 2019 at 11:51 AM Sandeep Moré <mo...@gmail.com> wrote:

> Thanks for starting the planning thread Larry !
> Agree with the theme and the release date for 1.4.0.
>
> +1
>
> Best,
> Sandeep
>
> On Thu, Oct 31, 2019 at 11:53 AM larry mccay <lm...@apache.org> wrote:
>
>> Folks -
>>
>> Out last release with end of July, I apologize for the delay in starting
>> the planning thread for 1.4.
>>
>> We currently have a backlog of ~65 JIRAs slated for a Fix Version of 1.4.
>>
>> There has been some work going on within KnoxShell to provide a general
>> purpose representation for tabular data. This will be leveraged for
>> rendering SQL query results as well as CSV files and simple processing
>> within KnoxShell. I will be writing up a KIP to represent the overall
>> vision for this work and initial set of usecases.
>>
>> We also have Cloudera Manager based discovery emerging and we should
>> target an initial set of services to enable for CM/CDH and CDP deployments
>> where CM is available.
>>
>> With the continued increase in cloud based deployments and Knox Gateway
>> use in securely accessing the exposed resources, we will concentrate on
>> KnoxShell as a first class environment for this access. This will likely
>> include an API for discovering metadata about the resources exposed through
>> Knox, the required authentication mechanisms, resource types and public
>> certs. It will also include Custom GroovyShell Commands for the KnoxShell
>> environment to help interact with the remote clusters and resultsets as
>> local in-memory tables. I will be start a KIP to try and articulate this
>> vision and related 1.4. usecases as well.
>>
>> I propose that the CM based Service Discovery and KnoxShell access to
>> remote clusters be the primary themes of the Apache Knox 1.4 release.
>>
>> I also propose that we target the end of November as the release date for
>> 1.4.
>>
>> Thoughts?
>>
>> --larry
>>
>

Re: [DISCUSS] Planning for Apache Knox 1.4

[Kevin Risden <kr...@apache.org>]

Based on that we are ~1/2 way through November and Thanksgiving in the US
around the corner, I don't see much of the KnoxShell pieces getting
integrated before end of November to make an end of November release.
Specifically the following section:

With the continued increase in cloud based deployments and Knox Gateway use
> in securely accessing the exposed resources, we will concentrate on
> KnoxShell as a first class environment for this access. This will likely
> include an API for discovering metadata about the resources exposed through
> Knox, the required authentication mechanisms, resource types and public
> certs. It will also include Custom GroovyShell Commands for the KnoxShell
> environment to help interact with the remote clusters and resultsets as
> local in-memory tables. I will be start a KIP to try and articulate this
> vision and related 1.4. usecases as well.
>

The KIP was just started so would be good to flesh that out more instead of
rushing it into 1.4.0. There are a lot of moving pieces in that paragraph
and would be good to make sure the Jiras are created and scoped
appropriately.

In addition to what was mentioned as features, there have been multiple new
service definitions (Impala, Kudu, NiFi Registry) added as well as fixes to
existing service definitions (Atlas, Livy, Ranger, Spark, YARN).

So +1 to an end of November release, but need to make sure not trying to
rush in new things just because a release will happen. There will be more
releases.


Kevin Risden


On Fri, Nov 1, 2019 at 11:51 AM Sandeep Moré <mo...@gmail.com> wrote:

> Thanks for starting the planning thread Larry !
> Agree with the theme and the release date for 1.4.0.
>
> +1
>
> Best,
> Sandeep
>
> On Thu, Oct 31, 2019 at 11:53 AM larry mccay <lm...@apache.org> wrote:
>
>> Folks -
>>
>> Out last release with end of July, I apologize for the delay in starting
>> the planning thread for 1.4.
>>
>> We currently have a backlog of ~65 JIRAs slated for a Fix Version of 1.4.
>>
>> There has been some work going on within KnoxShell to provide a general
>> purpose representation for tabular data. This will be leveraged for
>> rendering SQL query results as well as CSV files and simple processing
>> within KnoxShell. I will be writing up a KIP to represent the overall
>> vision for this work and initial set of usecases.
>>
>> We also have Cloudera Manager based discovery emerging and we should
>> target an initial set of services to enable for CM/CDH and CDP deployments
>> where CM is available.
>>
>> With the continued increase in cloud based deployments and Knox Gateway
>> use in securely accessing the exposed resources, we will concentrate on
>> KnoxShell as a first class environment for this access. This will likely
>> include an API for discovering metadata about the resources exposed through
>> Knox, the required authentication mechanisms, resource types and public
>> certs. It will also include Custom GroovyShell Commands for the KnoxShell
>> environment to help interact with the remote clusters and resultsets as
>> local in-memory tables. I will be start a KIP to try and articulate this
>> vision and related 1.4. usecases as well.
>>
>> I propose that the CM based Service Discovery and KnoxShell access to
>> remote clusters be the primary themes of the Apache Knox 1.4 release.
>>
>> I also propose that we target the end of November as the release date for
>> 1.4.
>>
>> Thoughts?
>>
>> --larry
>>
>

Re: [DISCUSS] Planning for Apache Knox 1.4

[Sandeep Moré <mo...@gmail.com>]

Thanks for starting the planning thread Larry !
Agree with the theme and the release date for 1.4.0.

+1

Best,
Sandeep

On Thu, Oct 31, 2019 at 11:53 AM larry mccay <lm...@apache.org> wrote:

> Folks -
>
> Out last release with end of July, I apologize for the delay in starting
> the planning thread for 1.4.
>
> We currently have a backlog of ~65 JIRAs slated for a Fix Version of 1.4.
>
> There has been some work going on within KnoxShell to provide a general
> purpose representation for tabular data. This will be leveraged for
> rendering SQL query results as well as CSV files and simple processing
> within KnoxShell. I will be writing up a KIP to represent the overall
> vision for this work and initial set of usecases.
>
> We also have Cloudera Manager based discovery emerging and we should
> target an initial set of services to enable for CM/CDH and CDP deployments
> where CM is available.
>
> With the continued increase in cloud based deployments and Knox Gateway
> use in securely accessing the exposed resources, we will concentrate on
> KnoxShell as a first class environment for this access. This will likely
> include an API for discovering metadata about the resources exposed through
> Knox, the required authentication mechanisms, resource types and public
> certs. It will also include Custom GroovyShell Commands for the KnoxShell
> environment to help interact with the remote clusters and resultsets as
> local in-memory tables. I will be start a KIP to try and articulate this
> vision and related 1.4. usecases as well.
>
> I propose that the CM based Service Discovery and KnoxShell access to
> remote clusters be the primary themes of the Apache Knox 1.4 release.
>
> I also propose that we target the end of November as the release date for
> 1.4.
>
> Thoughts?
>
> --larry
>

Re: [DISCUSS] Planning for Apache Knox 1.4

[Phil Zampino <pz...@apache.org>]

I think the proposed themes are appropriate.

CM-based service discovery is partially there already, and need only be
fleshed out for completeness and tested.
We've wanted to provide much of the Knox CLI functionality through
KnoxShell (so it could be accessed remotely) for quite some time now.

+1

On Thu, Oct 31, 2019 at 11:53 AM larry mccay <lm...@apache.org> wrote:

> Folks -
>
> Out last release with end of July, I apologize for the delay in starting
> the planning thread for 1.4.
>
> We currently have a backlog of ~65 JIRAs slated for a Fix Version of 1.4.
>
> There has been some work going on within KnoxShell to provide a general
> purpose representation for tabular data. This will be leveraged for
> rendering SQL query results as well as CSV files and simple processing
> within KnoxShell. I will be writing up a KIP to represent the overall
> vision for this work and initial set of usecases.
>
> We also have Cloudera Manager based discovery emerging and we should target
> an initial set of services to enable for CM/CDH and CDP deployments where
> CM is available.
>
> With the continued increase in cloud based deployments and Knox Gateway use
> in securely accessing the exposed resources, we will concentrate on
> KnoxShell as a first class environment for this access. This will likely
> include an API for discovering metadata about the resources exposed through
> Knox, the required authentication mechanisms, resource types and public
> certs. It will also include Custom GroovyShell Commands for the KnoxShell
> environment to help interact with the remote clusters and resultsets as
> local in-memory tables. I will be start a KIP to try and articulate this
> vision and related 1.4. usecases as well.
>
> I propose that the CM based Service Discovery and KnoxShell access to
> remote clusters be the primary themes of the Apache Knox 1.4 release.
>
> I also propose that we target the end of November as the release date for
> 1.4.
>
> Thoughts?
>
> --larry
>

Re: [DISCUSS] Planning for Apache Knox 1.4

[Sandeep Moré <mo...@gmail.com>]

Thanks for starting the planning thread Larry !
Agree with the theme and the release date for 1.4.0.

+1

Best,
Sandeep

On Thu, Oct 31, 2019 at 11:53 AM larry mccay <lm...@apache.org> wrote:

> Folks -
>
> Out last release with end of July, I apologize for the delay in starting
> the planning thread for 1.4.
>
> We currently have a backlog of ~65 JIRAs slated for a Fix Version of 1.4.
>
> There has been some work going on within KnoxShell to provide a general
> purpose representation for tabular data. This will be leveraged for
> rendering SQL query results as well as CSV files and simple processing
> within KnoxShell. I will be writing up a KIP to represent the overall
> vision for this work and initial set of usecases.
>
> We also have Cloudera Manager based discovery emerging and we should
> target an initial set of services to enable for CM/CDH and CDP deployments
> where CM is available.
>
> With the continued increase in cloud based deployments and Knox Gateway
> use in securely accessing the exposed resources, we will concentrate on
> KnoxShell as a first class environment for this access. This will likely
> include an API for discovering metadata about the resources exposed through
> Knox, the required authentication mechanisms, resource types and public
> certs. It will also include Custom GroovyShell Commands for the KnoxShell
> environment to help interact with the remote clusters and resultsets as
> local in-memory tables. I will be start a KIP to try and articulate this
> vision and related 1.4. usecases as well.
>
> I propose that the CM based Service Discovery and KnoxShell access to
> remote clusters be the primary themes of the Apache Knox 1.4 release.
>
> I also propose that we target the end of November as the release date for
> 1.4.
>
> Thoughts?
>
> --larry
>