You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@knox.apache.org by larry mccay <lm...@apache.org> on 2017/08/18 16:39:53 UTC
KIP-8 Service Discovery and Topology Generation
All -
I have published an initial stab at a proposal for simplifying topology
management [1] - as I briefly mentioned on the 0.14.0 planning thread.
There are a couple TODO sections that need some investigation for Ambari
API and Zookeeper Registry API as discovery plugins.
Please give the KIP a read and let me know if it makes sense or if you
would like to take on specific pieces of this work. If you would like to
add other options or ideas, etc.
I think that this will have a huge impact as usability improvements for
management of the gateway!
thanks,
--larry
1.
https://cwiki.apache.org/confluence/display/KNOX/KIP-8+Service+Discovery+and+Topology+Generation
Re: KIP-8 Service Discovery and Topology Generation
Posted by larry mccay <lm...@apache.org>.
I agree that named provider config should suffice. HA and AclsAuth
reference services within the topology and therfore will present a
challenge for reuse across topologies.
We'll need to figure that out.
On Aug 31, 2017 9:12 AM, "Philip Zampino" <pz...@gmail.com> wrote:
> Concerning the externalized provider configuration, the idea is to have
> sets of "named" (unique ID'd) config, which can be referenced by any Knox
> topology.
> So, you could have a single provider configuration that is applied (by
> reference) to all your cluster topologies, if that made sense for your
> deployment.
> This will allow a change to a single config (externalized provider config)
> to affect all the clusters topologies that reference it; rather than what
> is required today, which is to make the same change in every affected
> cluster topology config.
> Does that answer your question?
>
> I'll defer the HA question because I'm not yet familiar enough with it.
>
> Thanks for the feedback,
> Phil
>
> On Thu, Aug 31, 2017 at 3:42 AM, Krishna Pandey <kr...@gmail.com>
> wrote:
>
> > >
> > > As a result, I'm wondering if Knox should support provider
> configuration
> > > references in topology.xml, rather than having to duplicate it across
> > > descriptors.
> >
> >
> > +1 to Phil's remark above. Adding to that as Knox provides multi-cluster
> > support, can we have external provider config cluster-wise? Or we can
> > define multiple external provider config in single "global" provider
> config
> > file and refer them with some key?
> >
> > Also, when services are deployed in HA, we specify HaProvider and
> multiple
> > Service URLs in the topology. Will it make sense to use Load balancer IP
> > address / External URL for those services in topology instead of multiple
> > service URLs and using HaProvider?
> >
> > Thanks,
> > Krishna Pandey
> >
> > On Wed, Aug 30, 2017 at 10:05 PM, larry mccay <lm...@apache.org> wrote:
> >
> > > Terrific, Phil!
> > >
> > >
> > > On Wed, Aug 30, 2017 at 11:03 AM, Philip Zampino <pz...@gmail.com>
> > > wrote:
> > >
> > > > After giving this KIP some thought, I would like to work on it. I've
> > > added
> > > > more detail to the wiki, and I'll create a JIRA for it.
> > > >
> > > > On Fri, Aug 25, 2017 at 10:03 AM, larry mccay <lm...@apache.org>
> > wrote:
> > > >
> > > > > Wonderful details, Phil!
> > > > >
> > > > >
> > > > > On Fri, Aug 25, 2017 at 9:40 AM, Philip Zampino <
> pzampino@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks! I've added the Ambari API details to the wiki.
> > > > > >
> > > > > > On Fri, Aug 25, 2017 at 8:27 AM, larry mccay <lm...@apache.org>
> > > > wrote:
> > > > > >
> > > > > > > HI Phil -
> > > > > > >
> > > > > > > Thank you for digging into this topic!
> > > > > > >
> > > > > > > I've added you as a contributor to the wiki and you should be
> > able
> > > to
> > > > > > edit
> > > > > > > the KIP now.
> > > > > > >
> > > > > > > thanks,
> > > > > > >
> > > > > > > --larry
> > > > > > >
> > > > > > > On Thu, Aug 24, 2017 at 3:45 PM, Philip Zampino <
> > > pzampino@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I've put together a quick python POC targeting Ambari as the
> > > > > discovery
> > > > > > > > source, just to prove that we can indeed get the necessary
> > > details
> > > > > via
> > > > > > > > Ambari's REST API.
> > > > > > > > It generates a proper topology.xml descriptor based on a
> simple
> > > > > > > descriptor
> > > > > > > > (I chose YAML for this POC), which has a reference to what
> I've
> > > > > called
> > > > > > a
> > > > > > > > policy descriptor (the <gateway/> portion of the topology
> > > > > descriptor).
> > > > > > > >
> > > > > > > > I am currently unable to update the KIP, but I can share the
> > REST
> > > > > APIs
> > > > > > > I've
> > > > > > > > employed if there is interest.
> > > > > > > >
> > > > > > > > As a result, I'm wondering if Knox should support provider
> > > > > > configuration
> > > > > > > > references in topology.xml, rather than having to duplicate
> it
> > > > across
> > > > > > > > descriptors.
> > > > > > > > So, instead of <gateway><provider>...</gateway> in each
> > > > > topology.xml,
> > > > > > > have
> > > > > > > > a single element that points to an external provider config
> > > (e.g.,
> > > > > > > > <policy-ref>$GATEWAY_HOME/conf/policy/my-named-provider-
> > > > > > > > config.xml</policy-ref>).
> > > > > > > > I've already externalized it for input to the POC, but I'm
> > still
> > > > > > copying
> > > > > > > > the contents into the resulting topology descriptor; I think
> it
> > > > would
> > > > > > be
> > > > > > > > better to copy only the reference.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > - Phil
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Aug 18, 2017 at 1:55 PM, larry mccay <
> > lmccay@apache.org>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Good to hear, Phil.
> > > > > > > > >
> > > > > > > > > Yes, I was looking to go back and add some of the API
> > specifics
> > > > and
> > > > > > > > > investigation details for at least Ambari and ZK.
> > > > > > > > > If others make sense to add such as Consul, etcd, etc that
> > > would
> > > > be
> > > > > > > good
> > > > > > > > as
> > > > > > > > > well and it would help to tease out some of what may be
> > needed
> > > > for
> > > > > > the
> > > > > > > > > abstraction API and config.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino <
> > > > > pzampino@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > This sounds like a good improvement from a usability
> > > > perspective.
> > > > > > > > > >
> > > > > > > > > > I agree that Knox must continue its support for direct
> > > topology
> > > > > > (XML)
> > > > > > > > > > definitions, but for those deployments where there is a
> > > service
> > > > > > > > registry
> > > > > > > > > > (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge
> of
> > > the
> > > > > > > > topology,
> > > > > > > > > > the simplified config coupled with the service discovery
> > > could
> > > > > > reduce
> > > > > > > > > > topology (especially ServiceConnectivity) errors.
> > > > > > > > > >
> > > > > > > > > > I also agree that consolidating provider configurations
> > into
> > > > > named
> > > > > > > > sets,
> > > > > > > > > > which can be referenced from topology configurations,
> will
> > > be a
> > > > > > good
> > > > > > > > > > change, especially if there are commonly grouped
> providers
> > > > > employed
> > > > > > > by
> > > > > > > > > > multiple topologies.
> > > > > > > > > >
> > > > > > > > > > Concerning the TODO sections, is your intention is that
> > they
> > > > > > contain
> > > > > > > > the
> > > > > > > > > > respective API facilities that will provide the
> information
> > > > > needed
> > > > > > to
> > > > > > > > > > populate a topology definition?
> > > > > > > > > >
> > > > > > > > > > - Phil
> > > > > > > > > >
> > > > > > > > > > On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <
> > > > lmccay@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > All -
> > > > > > > > > > >
> > > > > > > > > > > I have published an initial stab at a proposal for
> > > > simplifying
> > > > > > > > topology
> > > > > > > > > > > management [1] - as I briefly mentioned on the 0.14.0
> > > > planning
> > > > > > > > thread.
> > > > > > > > > > >
> > > > > > > > > > > There are a couple TODO sections that need some
> > > investigation
> > > > > for
> > > > > > > > > Ambari
> > > > > > > > > > > API and Zookeeper Registry API as discovery plugins.
> > > > > > > > > > >
> > > > > > > > > > > Please give the KIP a read and let me know if it makes
> > > sense
> > > > or
> > > > > > if
> > > > > > > > you
> > > > > > > > > > > would like to take on specific pieces of this work. If
> > you
> > > > > would
> > > > > > > like
> > > > > > > > > to
> > > > > > > > > > > add other options or ideas, etc.
> > > > > > > > > > >
> > > > > > > > > > > I think that this will have a huge impact as usability
> > > > > > improvements
> > > > > > > > for
> > > > > > > > > > > management of the gateway!
> > > > > > > > > > >
> > > > > > > > > > > thanks,
> > > > > > > > > > >
> > > > > > > > > > > --larry
> > > > > > > > > > >
> > > > > > > > > > > 1.
> > > > > > > > > > > https://cwiki.apache.org/
> confluence/display/KNOX/KIP-8+
> > > > > > > > > > > Service+Discovery+and+Topology+Generation
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: KIP-8 Service Discovery and Topology Generation
Posted by Philip Zampino <pz...@gmail.com>.
Concerning the externalized provider configuration, the idea is to have
sets of "named" (unique ID'd) config, which can be referenced by any Knox
topology.
So, you could have a single provider configuration that is applied (by
reference) to all your cluster topologies, if that made sense for your
deployment.
This will allow a change to a single config (externalized provider config)
to affect all the clusters topologies that reference it; rather than what
is required today, which is to make the same change in every affected
cluster topology config.
Does that answer your question?
I'll defer the HA question because I'm not yet familiar enough with it.
Thanks for the feedback,
Phil
On Thu, Aug 31, 2017 at 3:42 AM, Krishna Pandey <kr...@gmail.com>
wrote:
> >
> > As a result, I'm wondering if Knox should support provider configuration
> > references in topology.xml, rather than having to duplicate it across
> > descriptors.
>
>
> +1 to Phil's remark above. Adding to that as Knox provides multi-cluster
> support, can we have external provider config cluster-wise? Or we can
> define multiple external provider config in single "global" provider config
> file and refer them with some key?
>
> Also, when services are deployed in HA, we specify HaProvider and multiple
> Service URLs in the topology. Will it make sense to use Load balancer IP
> address / External URL for those services in topology instead of multiple
> service URLs and using HaProvider?
>
> Thanks,
> Krishna Pandey
>
> On Wed, Aug 30, 2017 at 10:05 PM, larry mccay <lm...@apache.org> wrote:
>
> > Terrific, Phil!
> >
> >
> > On Wed, Aug 30, 2017 at 11:03 AM, Philip Zampino <pz...@gmail.com>
> > wrote:
> >
> > > After giving this KIP some thought, I would like to work on it. I've
> > added
> > > more detail to the wiki, and I'll create a JIRA for it.
> > >
> > > On Fri, Aug 25, 2017 at 10:03 AM, larry mccay <lm...@apache.org>
> wrote:
> > >
> > > > Wonderful details, Phil!
> > > >
> > > >
> > > > On Fri, Aug 25, 2017 at 9:40 AM, Philip Zampino <pz...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks! I've added the Ambari API details to the wiki.
> > > > >
> > > > > On Fri, Aug 25, 2017 at 8:27 AM, larry mccay <lm...@apache.org>
> > > wrote:
> > > > >
> > > > > > HI Phil -
> > > > > >
> > > > > > Thank you for digging into this topic!
> > > > > >
> > > > > > I've added you as a contributor to the wiki and you should be
> able
> > to
> > > > > edit
> > > > > > the KIP now.
> > > > > >
> > > > > > thanks,
> > > > > >
> > > > > > --larry
> > > > > >
> > > > > > On Thu, Aug 24, 2017 at 3:45 PM, Philip Zampino <
> > pzampino@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I've put together a quick python POC targeting Ambari as the
> > > > discovery
> > > > > > > source, just to prove that we can indeed get the necessary
> > details
> > > > via
> > > > > > > Ambari's REST API.
> > > > > > > It generates a proper topology.xml descriptor based on a simple
> > > > > > descriptor
> > > > > > > (I chose YAML for this POC), which has a reference to what I've
> > > > called
> > > > > a
> > > > > > > policy descriptor (the <gateway/> portion of the topology
> > > > descriptor).
> > > > > > >
> > > > > > > I am currently unable to update the KIP, but I can share the
> REST
> > > > APIs
> > > > > > I've
> > > > > > > employed if there is interest.
> > > > > > >
> > > > > > > As a result, I'm wondering if Knox should support provider
> > > > > configuration
> > > > > > > references in topology.xml, rather than having to duplicate it
> > > across
> > > > > > > descriptors.
> > > > > > > So, instead of <gateway><provider>...</gateway> in each
> > > > topology.xml,
> > > > > > have
> > > > > > > a single element that points to an external provider config
> > (e.g.,
> > > > > > > <policy-ref>$GATEWAY_HOME/conf/policy/my-named-provider-
> > > > > > > config.xml</policy-ref>).
> > > > > > > I've already externalized it for input to the POC, but I'm
> still
> > > > > copying
> > > > > > > the contents into the resulting topology descriptor; I think it
> > > would
> > > > > be
> > > > > > > better to copy only the reference.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > - Phil
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Aug 18, 2017 at 1:55 PM, larry mccay <
> lmccay@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Good to hear, Phil.
> > > > > > > >
> > > > > > > > Yes, I was looking to go back and add some of the API
> specifics
> > > and
> > > > > > > > investigation details for at least Ambari and ZK.
> > > > > > > > If others make sense to add such as Consul, etcd, etc that
> > would
> > > be
> > > > > > good
> > > > > > > as
> > > > > > > > well and it would help to tease out some of what may be
> needed
> > > for
> > > > > the
> > > > > > > > abstraction API and config.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino <
> > > > pzampino@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > This sounds like a good improvement from a usability
> > > perspective.
> > > > > > > > >
> > > > > > > > > I agree that Knox must continue its support for direct
> > topology
> > > > > (XML)
> > > > > > > > > definitions, but for those deployments where there is a
> > service
> > > > > > > registry
> > > > > > > > > (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of
> > the
> > > > > > > topology,
> > > > > > > > > the simplified config coupled with the service discovery
> > could
> > > > > reduce
> > > > > > > > > topology (especially ServiceConnectivity) errors.
> > > > > > > > >
> > > > > > > > > I also agree that consolidating provider configurations
> into
> > > > named
> > > > > > > sets,
> > > > > > > > > which can be referenced from topology configurations, will
> > be a
> > > > > good
> > > > > > > > > change, especially if there are commonly grouped providers
> > > > employed
> > > > > > by
> > > > > > > > > multiple topologies.
> > > > > > > > >
> > > > > > > > > Concerning the TODO sections, is your intention is that
> they
> > > > > contain
> > > > > > > the
> > > > > > > > > respective API facilities that will provide the information
> > > > needed
> > > > > to
> > > > > > > > > populate a topology definition?
> > > > > > > > >
> > > > > > > > > - Phil
> > > > > > > > >
> > > > > > > > > On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <
> > > lmccay@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > All -
> > > > > > > > > >
> > > > > > > > > > I have published an initial stab at a proposal for
> > > simplifying
> > > > > > > topology
> > > > > > > > > > management [1] - as I briefly mentioned on the 0.14.0
> > > planning
> > > > > > > thread.
> > > > > > > > > >
> > > > > > > > > > There are a couple TODO sections that need some
> > investigation
> > > > for
> > > > > > > > Ambari
> > > > > > > > > > API and Zookeeper Registry API as discovery plugins.
> > > > > > > > > >
> > > > > > > > > > Please give the KIP a read and let me know if it makes
> > sense
> > > or
> > > > > if
> > > > > > > you
> > > > > > > > > > would like to take on specific pieces of this work. If
> you
> > > > would
> > > > > > like
> > > > > > > > to
> > > > > > > > > > add other options or ideas, etc.
> > > > > > > > > >
> > > > > > > > > > I think that this will have a huge impact as usability
> > > > > improvements
> > > > > > > for
> > > > > > > > > > management of the gateway!
> > > > > > > > > >
> > > > > > > > > > thanks,
> > > > > > > > > >
> > > > > > > > > > --larry
> > > > > > > > > >
> > > > > > > > > > 1.
> > > > > > > > > > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+
> > > > > > > > > > Service+Discovery+and+Topology+Generation
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: KIP-8 Service Discovery and Topology Generation
Posted by Krishna Pandey <kr...@gmail.com>.
>
> As a result, I'm wondering if Knox should support provider configuration
> references in topology.xml, rather than having to duplicate it across
> descriptors.
+1 to Phil's remark above. Adding to that as Knox provides multi-cluster
support, can we have external provider config cluster-wise? Or we can
define multiple external provider config in single "global" provider config
file and refer them with some key?
Also, when services are deployed in HA, we specify HaProvider and multiple
Service URLs in the topology. Will it make sense to use Load balancer IP
address / External URL for those services in topology instead of multiple
service URLs and using HaProvider?
Thanks,
Krishna Pandey
On Wed, Aug 30, 2017 at 10:05 PM, larry mccay <lm...@apache.org> wrote:
> Terrific, Phil!
>
>
> On Wed, Aug 30, 2017 at 11:03 AM, Philip Zampino <pz...@gmail.com>
> wrote:
>
> > After giving this KIP some thought, I would like to work on it. I've
> added
> > more detail to the wiki, and I'll create a JIRA for it.
> >
> > On Fri, Aug 25, 2017 at 10:03 AM, larry mccay <lm...@apache.org> wrote:
> >
> > > Wonderful details, Phil!
> > >
> > >
> > > On Fri, Aug 25, 2017 at 9:40 AM, Philip Zampino <pz...@gmail.com>
> > > wrote:
> > >
> > > > Thanks! I've added the Ambari API details to the wiki.
> > > >
> > > > On Fri, Aug 25, 2017 at 8:27 AM, larry mccay <lm...@apache.org>
> > wrote:
> > > >
> > > > > HI Phil -
> > > > >
> > > > > Thank you for digging into this topic!
> > > > >
> > > > > I've added you as a contributor to the wiki and you should be able
> to
> > > > edit
> > > > > the KIP now.
> > > > >
> > > > > thanks,
> > > > >
> > > > > --larry
> > > > >
> > > > > On Thu, Aug 24, 2017 at 3:45 PM, Philip Zampino <
> pzampino@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I've put together a quick python POC targeting Ambari as the
> > > discovery
> > > > > > source, just to prove that we can indeed get the necessary
> details
> > > via
> > > > > > Ambari's REST API.
> > > > > > It generates a proper topology.xml descriptor based on a simple
> > > > > descriptor
> > > > > > (I chose YAML for this POC), which has a reference to what I've
> > > called
> > > > a
> > > > > > policy descriptor (the <gateway/> portion of the topology
> > > descriptor).
> > > > > >
> > > > > > I am currently unable to update the KIP, but I can share the REST
> > > APIs
> > > > > I've
> > > > > > employed if there is interest.
> > > > > >
> > > > > > As a result, I'm wondering if Knox should support provider
> > > > configuration
> > > > > > references in topology.xml, rather than having to duplicate it
> > across
> > > > > > descriptors.
> > > > > > So, instead of <gateway><provider>...</gateway> in each
> > > topology.xml,
> > > > > have
> > > > > > a single element that points to an external provider config
> (e.g.,
> > > > > > <policy-ref>$GATEWAY_HOME/conf/policy/my-named-provider-
> > > > > > config.xml</policy-ref>).
> > > > > > I've already externalized it for input to the POC, but I'm still
> > > > copying
> > > > > > the contents into the resulting topology descriptor; I think it
> > would
> > > > be
> > > > > > better to copy only the reference.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > - Phil
> > > > > >
> > > > > >
> > > > > > On Fri, Aug 18, 2017 at 1:55 PM, larry mccay <lm...@apache.org>
> > > > wrote:
> > > > > >
> > > > > > > Good to hear, Phil.
> > > > > > >
> > > > > > > Yes, I was looking to go back and add some of the API specifics
> > and
> > > > > > > investigation details for at least Ambari and ZK.
> > > > > > > If others make sense to add such as Consul, etcd, etc that
> would
> > be
> > > > > good
> > > > > > as
> > > > > > > well and it would help to tease out some of what may be needed
> > for
> > > > the
> > > > > > > abstraction API and config.
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino <
> > > pzampino@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > This sounds like a good improvement from a usability
> > perspective.
> > > > > > > >
> > > > > > > > I agree that Knox must continue its support for direct
> topology
> > > > (XML)
> > > > > > > > definitions, but for those deployments where there is a
> service
> > > > > > registry
> > > > > > > > (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of
> the
> > > > > > topology,
> > > > > > > > the simplified config coupled with the service discovery
> could
> > > > reduce
> > > > > > > > topology (especially ServiceConnectivity) errors.
> > > > > > > >
> > > > > > > > I also agree that consolidating provider configurations into
> > > named
> > > > > > sets,
> > > > > > > > which can be referenced from topology configurations, will
> be a
> > > > good
> > > > > > > > change, especially if there are commonly grouped providers
> > > employed
> > > > > by
> > > > > > > > multiple topologies.
> > > > > > > >
> > > > > > > > Concerning the TODO sections, is your intention is that they
> > > > contain
> > > > > > the
> > > > > > > > respective API facilities that will provide the information
> > > needed
> > > > to
> > > > > > > > populate a topology definition?
> > > > > > > >
> > > > > > > > - Phil
> > > > > > > >
> > > > > > > > On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <
> > lmccay@apache.org
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > All -
> > > > > > > > >
> > > > > > > > > I have published an initial stab at a proposal for
> > simplifying
> > > > > > topology
> > > > > > > > > management [1] - as I briefly mentioned on the 0.14.0
> > planning
> > > > > > thread.
> > > > > > > > >
> > > > > > > > > There are a couple TODO sections that need some
> investigation
> > > for
> > > > > > > Ambari
> > > > > > > > > API and Zookeeper Registry API as discovery plugins.
> > > > > > > > >
> > > > > > > > > Please give the KIP a read and let me know if it makes
> sense
> > or
> > > > if
> > > > > > you
> > > > > > > > > would like to take on specific pieces of this work. If you
> > > would
> > > > > like
> > > > > > > to
> > > > > > > > > add other options or ideas, etc.
> > > > > > > > >
> > > > > > > > > I think that this will have a huge impact as usability
> > > > improvements
> > > > > > for
> > > > > > > > > management of the gateway!
> > > > > > > > >
> > > > > > > > > thanks,
> > > > > > > > >
> > > > > > > > > --larry
> > > > > > > > >
> > > > > > > > > 1.
> > > > > > > > > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+
> > > > > > > > > Service+Discovery+and+Topology+Generation
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: KIP-8 Service Discovery and Topology Generation
Posted by larry mccay <lm...@apache.org>.
Terrific, Phil!
On Wed, Aug 30, 2017 at 11:03 AM, Philip Zampino <pz...@gmail.com> wrote:
> After giving this KIP some thought, I would like to work on it. I've added
> more detail to the wiki, and I'll create a JIRA for it.
>
> On Fri, Aug 25, 2017 at 10:03 AM, larry mccay <lm...@apache.org> wrote:
>
> > Wonderful details, Phil!
> >
> >
> > On Fri, Aug 25, 2017 at 9:40 AM, Philip Zampino <pz...@gmail.com>
> > wrote:
> >
> > > Thanks! I've added the Ambari API details to the wiki.
> > >
> > > On Fri, Aug 25, 2017 at 8:27 AM, larry mccay <lm...@apache.org>
> wrote:
> > >
> > > > HI Phil -
> > > >
> > > > Thank you for digging into this topic!
> > > >
> > > > I've added you as a contributor to the wiki and you should be able to
> > > edit
> > > > the KIP now.
> > > >
> > > > thanks,
> > > >
> > > > --larry
> > > >
> > > > On Thu, Aug 24, 2017 at 3:45 PM, Philip Zampino <pz...@gmail.com>
> > > > wrote:
> > > >
> > > > > I've put together a quick python POC targeting Ambari as the
> > discovery
> > > > > source, just to prove that we can indeed get the necessary details
> > via
> > > > > Ambari's REST API.
> > > > > It generates a proper topology.xml descriptor based on a simple
> > > > descriptor
> > > > > (I chose YAML for this POC), which has a reference to what I've
> > called
> > > a
> > > > > policy descriptor (the <gateway/> portion of the topology
> > descriptor).
> > > > >
> > > > > I am currently unable to update the KIP, but I can share the REST
> > APIs
> > > > I've
> > > > > employed if there is interest.
> > > > >
> > > > > As a result, I'm wondering if Knox should support provider
> > > configuration
> > > > > references in topology.xml, rather than having to duplicate it
> across
> > > > > descriptors.
> > > > > So, instead of <gateway><provider>...</gateway> in each
> > topology.xml,
> > > > have
> > > > > a single element that points to an external provider config (e.g.,
> > > > > <policy-ref>$GATEWAY_HOME/conf/policy/my-named-provider-
> > > > > config.xml</policy-ref>).
> > > > > I've already externalized it for input to the POC, but I'm still
> > > copying
> > > > > the contents into the resulting topology descriptor; I think it
> would
> > > be
> > > > > better to copy only the reference.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > - Phil
> > > > >
> > > > >
> > > > > On Fri, Aug 18, 2017 at 1:55 PM, larry mccay <lm...@apache.org>
> > > wrote:
> > > > >
> > > > > > Good to hear, Phil.
> > > > > >
> > > > > > Yes, I was looking to go back and add some of the API specifics
> and
> > > > > > investigation details for at least Ambari and ZK.
> > > > > > If others make sense to add such as Consul, etcd, etc that would
> be
> > > > good
> > > > > as
> > > > > > well and it would help to tease out some of what may be needed
> for
> > > the
> > > > > > abstraction API and config.
> > > > > >
> > > > > >
> > > > > > On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino <
> > pzampino@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > This sounds like a good improvement from a usability
> perspective.
> > > > > > >
> > > > > > > I agree that Knox must continue its support for direct topology
> > > (XML)
> > > > > > > definitions, but for those deployments where there is a service
> > > > > registry
> > > > > > > (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of the
> > > > > topology,
> > > > > > > the simplified config coupled with the service discovery could
> > > reduce
> > > > > > > topology (especially ServiceConnectivity) errors.
> > > > > > >
> > > > > > > I also agree that consolidating provider configurations into
> > named
> > > > > sets,
> > > > > > > which can be referenced from topology configurations, will be a
> > > good
> > > > > > > change, especially if there are commonly grouped providers
> > employed
> > > > by
> > > > > > > multiple topologies.
> > > > > > >
> > > > > > > Concerning the TODO sections, is your intention is that they
> > > contain
> > > > > the
> > > > > > > respective API facilities that will provide the information
> > needed
> > > to
> > > > > > > populate a topology definition?
> > > > > > >
> > > > > > > - Phil
> > > > > > >
> > > > > > > On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <
> lmccay@apache.org
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > All -
> > > > > > > >
> > > > > > > > I have published an initial stab at a proposal for
> simplifying
> > > > > topology
> > > > > > > > management [1] - as I briefly mentioned on the 0.14.0
> planning
> > > > > thread.
> > > > > > > >
> > > > > > > > There are a couple TODO sections that need some investigation
> > for
> > > > > > Ambari
> > > > > > > > API and Zookeeper Registry API as discovery plugins.
> > > > > > > >
> > > > > > > > Please give the KIP a read and let me know if it makes sense
> or
> > > if
> > > > > you
> > > > > > > > would like to take on specific pieces of this work. If you
> > would
> > > > like
> > > > > > to
> > > > > > > > add other options or ideas, etc.
> > > > > > > >
> > > > > > > > I think that this will have a huge impact as usability
> > > improvements
> > > > > for
> > > > > > > > management of the gateway!
> > > > > > > >
> > > > > > > > thanks,
> > > > > > > >
> > > > > > > > --larry
> > > > > > > >
> > > > > > > > 1.
> > > > > > > > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+
> > > > > > > > Service+Discovery+and+Topology+Generation
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: KIP-8 Service Discovery and Topology Generation
Posted by Philip Zampino <pz...@gmail.com>.
After giving this KIP some thought, I would like to work on it. I've added
more detail to the wiki, and I'll create a JIRA for it.
On Fri, Aug 25, 2017 at 10:03 AM, larry mccay <lm...@apache.org> wrote:
> Wonderful details, Phil!
>
>
> On Fri, Aug 25, 2017 at 9:40 AM, Philip Zampino <pz...@gmail.com>
> wrote:
>
> > Thanks! I've added the Ambari API details to the wiki.
> >
> > On Fri, Aug 25, 2017 at 8:27 AM, larry mccay <lm...@apache.org> wrote:
> >
> > > HI Phil -
> > >
> > > Thank you for digging into this topic!
> > >
> > > I've added you as a contributor to the wiki and you should be able to
> > edit
> > > the KIP now.
> > >
> > > thanks,
> > >
> > > --larry
> > >
> > > On Thu, Aug 24, 2017 at 3:45 PM, Philip Zampino <pz...@gmail.com>
> > > wrote:
> > >
> > > > I've put together a quick python POC targeting Ambari as the
> discovery
> > > > source, just to prove that we can indeed get the necessary details
> via
> > > > Ambari's REST API.
> > > > It generates a proper topology.xml descriptor based on a simple
> > > descriptor
> > > > (I chose YAML for this POC), which has a reference to what I've
> called
> > a
> > > > policy descriptor (the <gateway/> portion of the topology
> descriptor).
> > > >
> > > > I am currently unable to update the KIP, but I can share the REST
> APIs
> > > I've
> > > > employed if there is interest.
> > > >
> > > > As a result, I'm wondering if Knox should support provider
> > configuration
> > > > references in topology.xml, rather than having to duplicate it across
> > > > descriptors.
> > > > So, instead of <gateway><provider>...</gateway> in each
> topology.xml,
> > > have
> > > > a single element that points to an external provider config (e.g.,
> > > > <policy-ref>$GATEWAY_HOME/conf/policy/my-named-provider-
> > > > config.xml</policy-ref>).
> > > > I've already externalized it for input to the POC, but I'm still
> > copying
> > > > the contents into the resulting topology descriptor; I think it would
> > be
> > > > better to copy only the reference.
> > > >
> > > > Thoughts?
> > > >
> > > > - Phil
> > > >
> > > >
> > > > On Fri, Aug 18, 2017 at 1:55 PM, larry mccay <lm...@apache.org>
> > wrote:
> > > >
> > > > > Good to hear, Phil.
> > > > >
> > > > > Yes, I was looking to go back and add some of the API specifics and
> > > > > investigation details for at least Ambari and ZK.
> > > > > If others make sense to add such as Consul, etcd, etc that would be
> > > good
> > > > as
> > > > > well and it would help to tease out some of what may be needed for
> > the
> > > > > abstraction API and config.
> > > > >
> > > > >
> > > > > On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino <
> pzampino@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > This sounds like a good improvement from a usability perspective.
> > > > > >
> > > > > > I agree that Knox must continue its support for direct topology
> > (XML)
> > > > > > definitions, but for those deployments where there is a service
> > > > registry
> > > > > > (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of the
> > > > topology,
> > > > > > the simplified config coupled with the service discovery could
> > reduce
> > > > > > topology (especially ServiceConnectivity) errors.
> > > > > >
> > > > > > I also agree that consolidating provider configurations into
> named
> > > > sets,
> > > > > > which can be referenced from topology configurations, will be a
> > good
> > > > > > change, especially if there are commonly grouped providers
> employed
> > > by
> > > > > > multiple topologies.
> > > > > >
> > > > > > Concerning the TODO sections, is your intention is that they
> > contain
> > > > the
> > > > > > respective API facilities that will provide the information
> needed
> > to
> > > > > > populate a topology definition?
> > > > > >
> > > > > > - Phil
> > > > > >
> > > > > > On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <lmccay@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > > > All -
> > > > > > >
> > > > > > > I have published an initial stab at a proposal for simplifying
> > > > topology
> > > > > > > management [1] - as I briefly mentioned on the 0.14.0 planning
> > > > thread.
> > > > > > >
> > > > > > > There are a couple TODO sections that need some investigation
> for
> > > > > Ambari
> > > > > > > API and Zookeeper Registry API as discovery plugins.
> > > > > > >
> > > > > > > Please give the KIP a read and let me know if it makes sense or
> > if
> > > > you
> > > > > > > would like to take on specific pieces of this work. If you
> would
> > > like
> > > > > to
> > > > > > > add other options or ideas, etc.
> > > > > > >
> > > > > > > I think that this will have a huge impact as usability
> > improvements
> > > > for
> > > > > > > management of the gateway!
> > > > > > >
> > > > > > > thanks,
> > > > > > >
> > > > > > > --larry
> > > > > > >
> > > > > > > 1.
> > > > > > > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+
> > > > > > > Service+Discovery+and+Topology+Generation
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: KIP-8 Service Discovery and Topology Generation
Posted by larry mccay <lm...@apache.org>.
Wonderful details, Phil!
On Fri, Aug 25, 2017 at 9:40 AM, Philip Zampino <pz...@gmail.com> wrote:
> Thanks! I've added the Ambari API details to the wiki.
>
> On Fri, Aug 25, 2017 at 8:27 AM, larry mccay <lm...@apache.org> wrote:
>
> > HI Phil -
> >
> > Thank you for digging into this topic!
> >
> > I've added you as a contributor to the wiki and you should be able to
> edit
> > the KIP now.
> >
> > thanks,
> >
> > --larry
> >
> > On Thu, Aug 24, 2017 at 3:45 PM, Philip Zampino <pz...@gmail.com>
> > wrote:
> >
> > > I've put together a quick python POC targeting Ambari as the discovery
> > > source, just to prove that we can indeed get the necessary details via
> > > Ambari's REST API.
> > > It generates a proper topology.xml descriptor based on a simple
> > descriptor
> > > (I chose YAML for this POC), which has a reference to what I've called
> a
> > > policy descriptor (the <gateway/> portion of the topology descriptor).
> > >
> > > I am currently unable to update the KIP, but I can share the REST APIs
> > I've
> > > employed if there is interest.
> > >
> > > As a result, I'm wondering if Knox should support provider
> configuration
> > > references in topology.xml, rather than having to duplicate it across
> > > descriptors.
> > > So, instead of <gateway><provider>...</gateway> in each topology.xml,
> > have
> > > a single element that points to an external provider config (e.g.,
> > > <policy-ref>$GATEWAY_HOME/conf/policy/my-named-provider-
> > > config.xml</policy-ref>).
> > > I've already externalized it for input to the POC, but I'm still
> copying
> > > the contents into the resulting topology descriptor; I think it would
> be
> > > better to copy only the reference.
> > >
> > > Thoughts?
> > >
> > > - Phil
> > >
> > >
> > > On Fri, Aug 18, 2017 at 1:55 PM, larry mccay <lm...@apache.org>
> wrote:
> > >
> > > > Good to hear, Phil.
> > > >
> > > > Yes, I was looking to go back and add some of the API specifics and
> > > > investigation details for at least Ambari and ZK.
> > > > If others make sense to add such as Consul, etcd, etc that would be
> > good
> > > as
> > > > well and it would help to tease out some of what may be needed for
> the
> > > > abstraction API and config.
> > > >
> > > >
> > > > On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino <pz...@gmail.com>
> > > > wrote:
> > > >
> > > > > This sounds like a good improvement from a usability perspective.
> > > > >
> > > > > I agree that Knox must continue its support for direct topology
> (XML)
> > > > > definitions, but for those deployments where there is a service
> > > registry
> > > > > (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of the
> > > topology,
> > > > > the simplified config coupled with the service discovery could
> reduce
> > > > > topology (especially ServiceConnectivity) errors.
> > > > >
> > > > > I also agree that consolidating provider configurations into named
> > > sets,
> > > > > which can be referenced from topology configurations, will be a
> good
> > > > > change, especially if there are commonly grouped providers employed
> > by
> > > > > multiple topologies.
> > > > >
> > > > > Concerning the TODO sections, is your intention is that they
> contain
> > > the
> > > > > respective API facilities that will provide the information needed
> to
> > > > > populate a topology definition?
> > > > >
> > > > > - Phil
> > > > >
> > > > > On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <lm...@apache.org>
> > > wrote:
> > > > >
> > > > > > All -
> > > > > >
> > > > > > I have published an initial stab at a proposal for simplifying
> > > topology
> > > > > > management [1] - as I briefly mentioned on the 0.14.0 planning
> > > thread.
> > > > > >
> > > > > > There are a couple TODO sections that need some investigation for
> > > > Ambari
> > > > > > API and Zookeeper Registry API as discovery plugins.
> > > > > >
> > > > > > Please give the KIP a read and let me know if it makes sense or
> if
> > > you
> > > > > > would like to take on specific pieces of this work. If you would
> > like
> > > > to
> > > > > > add other options or ideas, etc.
> > > > > >
> > > > > > I think that this will have a huge impact as usability
> improvements
> > > for
> > > > > > management of the gateway!
> > > > > >
> > > > > > thanks,
> > > > > >
> > > > > > --larry
> > > > > >
> > > > > > 1.
> > > > > > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+
> > > > > > Service+Discovery+and+Topology+Generation
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: KIP-8 Service Discovery and Topology Generation
Posted by Philip Zampino <pz...@gmail.com>.
Thanks! I've added the Ambari API details to the wiki.
On Fri, Aug 25, 2017 at 8:27 AM, larry mccay <lm...@apache.org> wrote:
> HI Phil -
>
> Thank you for digging into this topic!
>
> I've added you as a contributor to the wiki and you should be able to edit
> the KIP now.
>
> thanks,
>
> --larry
>
> On Thu, Aug 24, 2017 at 3:45 PM, Philip Zampino <pz...@gmail.com>
> wrote:
>
> > I've put together a quick python POC targeting Ambari as the discovery
> > source, just to prove that we can indeed get the necessary details via
> > Ambari's REST API.
> > It generates a proper topology.xml descriptor based on a simple
> descriptor
> > (I chose YAML for this POC), which has a reference to what I've called a
> > policy descriptor (the <gateway/> portion of the topology descriptor).
> >
> > I am currently unable to update the KIP, but I can share the REST APIs
> I've
> > employed if there is interest.
> >
> > As a result, I'm wondering if Knox should support provider configuration
> > references in topology.xml, rather than having to duplicate it across
> > descriptors.
> > So, instead of <gateway><provider>...</gateway> in each topology.xml,
> have
> > a single element that points to an external provider config (e.g.,
> > <policy-ref>$GATEWAY_HOME/conf/policy/my-named-provider-
> > config.xml</policy-ref>).
> > I've already externalized it for input to the POC, but I'm still copying
> > the contents into the resulting topology descriptor; I think it would be
> > better to copy only the reference.
> >
> > Thoughts?
> >
> > - Phil
> >
> >
> > On Fri, Aug 18, 2017 at 1:55 PM, larry mccay <lm...@apache.org> wrote:
> >
> > > Good to hear, Phil.
> > >
> > > Yes, I was looking to go back and add some of the API specifics and
> > > investigation details for at least Ambari and ZK.
> > > If others make sense to add such as Consul, etcd, etc that would be
> good
> > as
> > > well and it would help to tease out some of what may be needed for the
> > > abstraction API and config.
> > >
> > >
> > > On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino <pz...@gmail.com>
> > > wrote:
> > >
> > > > This sounds like a good improvement from a usability perspective.
> > > >
> > > > I agree that Knox must continue its support for direct topology (XML)
> > > > definitions, but for those deployments where there is a service
> > registry
> > > > (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of the
> > topology,
> > > > the simplified config coupled with the service discovery could reduce
> > > > topology (especially ServiceConnectivity) errors.
> > > >
> > > > I also agree that consolidating provider configurations into named
> > sets,
> > > > which can be referenced from topology configurations, will be a good
> > > > change, especially if there are commonly grouped providers employed
> by
> > > > multiple topologies.
> > > >
> > > > Concerning the TODO sections, is your intention is that they contain
> > the
> > > > respective API facilities that will provide the information needed to
> > > > populate a topology definition?
> > > >
> > > > - Phil
> > > >
> > > > On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <lm...@apache.org>
> > wrote:
> > > >
> > > > > All -
> > > > >
> > > > > I have published an initial stab at a proposal for simplifying
> > topology
> > > > > management [1] - as I briefly mentioned on the 0.14.0 planning
> > thread.
> > > > >
> > > > > There are a couple TODO sections that need some investigation for
> > > Ambari
> > > > > API and Zookeeper Registry API as discovery plugins.
> > > > >
> > > > > Please give the KIP a read and let me know if it makes sense or if
> > you
> > > > > would like to take on specific pieces of this work. If you would
> like
> > > to
> > > > > add other options or ideas, etc.
> > > > >
> > > > > I think that this will have a huge impact as usability improvements
> > for
> > > > > management of the gateway!
> > > > >
> > > > > thanks,
> > > > >
> > > > > --larry
> > > > >
> > > > > 1.
> > > > > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+
> > > > > Service+Discovery+and+Topology+Generation
> > > > >
> > > >
> > >
> >
>
Re: KIP-8 Service Discovery and Topology Generation
Posted by larry mccay <lm...@apache.org>.
HI Phil -
Thank you for digging into this topic!
I've added you as a contributor to the wiki and you should be able to edit
the KIP now.
thanks,
--larry
On Thu, Aug 24, 2017 at 3:45 PM, Philip Zampino <pz...@gmail.com> wrote:
> I've put together a quick python POC targeting Ambari as the discovery
> source, just to prove that we can indeed get the necessary details via
> Ambari's REST API.
> It generates a proper topology.xml descriptor based on a simple descriptor
> (I chose YAML for this POC), which has a reference to what I've called a
> policy descriptor (the <gateway/> portion of the topology descriptor).
>
> I am currently unable to update the KIP, but I can share the REST APIs I've
> employed if there is interest.
>
> As a result, I'm wondering if Knox should support provider configuration
> references in topology.xml, rather than having to duplicate it across
> descriptors.
> So, instead of <gateway><provider>...</gateway> in each topology.xml, have
> a single element that points to an external provider config (e.g.,
> <policy-ref>$GATEWAY_HOME/conf/policy/my-named-provider-
> config.xml</policy-ref>).
> I've already externalized it for input to the POC, but I'm still copying
> the contents into the resulting topology descriptor; I think it would be
> better to copy only the reference.
>
> Thoughts?
>
> - Phil
>
>
> On Fri, Aug 18, 2017 at 1:55 PM, larry mccay <lm...@apache.org> wrote:
>
> > Good to hear, Phil.
> >
> > Yes, I was looking to go back and add some of the API specifics and
> > investigation details for at least Ambari and ZK.
> > If others make sense to add such as Consul, etcd, etc that would be good
> as
> > well and it would help to tease out some of what may be needed for the
> > abstraction API and config.
> >
> >
> > On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino <pz...@gmail.com>
> > wrote:
> >
> > > This sounds like a good improvement from a usability perspective.
> > >
> > > I agree that Knox must continue its support for direct topology (XML)
> > > definitions, but for those deployments where there is a service
> registry
> > > (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of the
> topology,
> > > the simplified config coupled with the service discovery could reduce
> > > topology (especially ServiceConnectivity) errors.
> > >
> > > I also agree that consolidating provider configurations into named
> sets,
> > > which can be referenced from topology configurations, will be a good
> > > change, especially if there are commonly grouped providers employed by
> > > multiple topologies.
> > >
> > > Concerning the TODO sections, is your intention is that they contain
> the
> > > respective API facilities that will provide the information needed to
> > > populate a topology definition?
> > >
> > > - Phil
> > >
> > > On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <lm...@apache.org>
> wrote:
> > >
> > > > All -
> > > >
> > > > I have published an initial stab at a proposal for simplifying
> topology
> > > > management [1] - as I briefly mentioned on the 0.14.0 planning
> thread.
> > > >
> > > > There are a couple TODO sections that need some investigation for
> > Ambari
> > > > API and Zookeeper Registry API as discovery plugins.
> > > >
> > > > Please give the KIP a read and let me know if it makes sense or if
> you
> > > > would like to take on specific pieces of this work. If you would like
> > to
> > > > add other options or ideas, etc.
> > > >
> > > > I think that this will have a huge impact as usability improvements
> for
> > > > management of the gateway!
> > > >
> > > > thanks,
> > > >
> > > > --larry
> > > >
> > > > 1.
> > > > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+
> > > > Service+Discovery+and+Topology+Generation
> > > >
> > >
> >
>
Re: KIP-8 Service Discovery and Topology Generation
Posted by Philip Zampino <pz...@gmail.com>.
The YAML I came up with looks like this:
# Discovery source config
discovery-type : AMBARI
discovery-registry : http://c6401.ambari.apache.org:8080
# Provider config reference, the contents of which will be included in, or
referenced from, the resulting topology descriptor
provider-config-ref : ambari-cluster-policy.xml
# The cluster for which the details should be discovered
cluster: test
# The services to declare in the resulting topology descriptor, whose URLs
will be discovered (unless a value is specified)
services:
- NAMENODE
- JOBTRACKER
- WEBHDFS
- WEBHCAT
- OOZIE
- WEBHBASE
- HIVE
- RESOURCEMANAGER
- AMBARI : http://c6401.ambari.apache.org:8080
- AMBARIUI : http://c6401.ambari.apache.org:8080
The REST API features I've employed are:
To identify all the clusters known to Ambari (for executing a comprehensive
discovery of all clusters)
http://AMBARI_HOST:PORT/api/v1/clusters
To identify all the service components associated with a particular cluster
http://AMBARI_HOST:PORT/api/v1/clusters/CLUSTER_NAME/components
To determine the service components' respective hosts
http://AMBARI_HOST:PORT
/api/v1/clusters/CLUSTER_NAME/services?fields=components/host_components/HostRoles
For the contents of the active versions of the service component
configurations (hive-site, hdfs-site, etc...)
http://AMBARI_HOST:PORT
/api/v1/clusters/CLUSTER_NAME/configurations/service_config_versions?is_current=true
You can check the responses for these using curl (e.g., curl -ivku
admin:admin -H 'X-Requested-By:admin' '
http://c6401.ambari.apache.org:8080/api/v1/clusters')
From the responses to these queries, I've managed to construct the
necessary service URLs in a topology descriptor.
On Thu, Aug 24, 2017 at 5:46 PM, larry mccay <lm...@apache.org> wrote:
> I completely agree, Phil. I also think that is in line with the KIP-1
> item's intent. One thing to keep in mind is how changes to the common
> config will need to be dynamically picked up by all referencers.
>
> I will look into getting you wiki rights for updating the KIP once I am
> back at a computer.
>
> The POC sounds interesting. I'd like to see the YAML and the APIs!
>
> On Aug 24, 2017 3:45 PM, "Philip Zampino" <pz...@gmail.com> wrote:
>
> I've put together a quick python POC targeting Ambari as the discovery
> source, just to prove that we can indeed get the necessary details via
> Ambari's REST API.
> It generates a proper topology.xml descriptor based on a simple descriptor
> (I chose YAML for this POC), which has a reference to what I've called a
> policy descriptor (the <gateway/> portion of the topology descriptor).
>
> I am currently unable to update the KIP, but I can share the REST APIs I've
> employed if there is interest.
>
> As a result, I'm wondering if Knox should support provider configuration
> references in topology.xml, rather than having to duplicate it across
> descriptors.
> So, instead of <gateway><provider>...</gateway> in each topology.xml, have
> a single element that points to an external provider config (e.g.,
> <policy-ref>$GATEWAY_HOME/conf/policy/my-named-provider-
> config.xml</policy-ref>).
> I've already externalized it for input to the POC, but I'm still copying
> the contents into the resulting topology descriptor; I think it would be
> better to copy only the reference.
>
> Thoughts?
>
> - Phil
>
>
> On Fri, Aug 18, 2017 at 1:55 PM, larry mccay <lm...@apache.org> wrote:
>
> > Good to hear, Phil.
> >
> > Yes, I was looking to go back and add some of the API specifics and
> > investigation details for at least Ambari and ZK.
> > If others make sense to add such as Consul, etcd, etc that would be good
> as
> > well and it would help to tease out some of what may be needed for the
> > abstraction API and config.
> >
> >
> > On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino <pz...@gmail.com>
> > wrote:
> >
> > > This sounds like a good improvement from a usability perspective.
> > >
> > > I agree that Knox must continue its support for direct topology (XML)
> > > definitions, but for those deployments where there is a service
> registry
> > > (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of the
> topology,
> > > the simplified config coupled with the service discovery could reduce
> > > topology (especially ServiceConnectivity) errors.
> > >
> > > I also agree that consolidating provider configurations into named
> sets,
> > > which can be referenced from topology configurations, will be a good
> > > change, especially if there are commonly grouped providers employed by
> > > multiple topologies.
> > >
> > > Concerning the TODO sections, is your intention is that they contain
> the
> > > respective API facilities that will provide the information needed to
> > > populate a topology definition?
> > >
> > > - Phil
> > >
> > > On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <lm...@apache.org>
> wrote:
> > >
> > > > All -
> > > >
> > > > I have published an initial stab at a proposal for simplifying
> topology
> > > > management [1] - as I briefly mentioned on the 0.14.0 planning
> thread.
> > > >
> > > > There are a couple TODO sections that need some investigation for
> > Ambari
> > > > API and Zookeeper Registry API as discovery plugins.
> > > >
> > > > Please give the KIP a read and let me know if it makes sense or if
> you
> > > > would like to take on specific pieces of this work. If you would like
> > to
> > > > add other options or ideas, etc.
> > > >
> > > > I think that this will have a huge impact as usability improvements
> for
> > > > management of the gateway!
> > > >
> > > > thanks,
> > > >
> > > > --larry
> > > >
> > > > 1.
> > > > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+
> > > > Service+Discovery+and+Topology+Generation
> > > >
> > >
> >
>
Re: KIP-8 Service Discovery and Topology Generation
Posted by larry mccay <lm...@apache.org>.
I completely agree, Phil. I also think that is in line with the KIP-1
item's intent. One thing to keep in mind is how changes to the common
config will need to be dynamically picked up by all referencers.
I will look into getting you wiki rights for updating the KIP once I am
back at a computer.
The POC sounds interesting. I'd like to see the YAML and the APIs!
On Aug 24, 2017 3:45 PM, "Philip Zampino" <pz...@gmail.com> wrote:
I've put together a quick python POC targeting Ambari as the discovery
source, just to prove that we can indeed get the necessary details via
Ambari's REST API.
It generates a proper topology.xml descriptor based on a simple descriptor
(I chose YAML for this POC), which has a reference to what I've called a
policy descriptor (the <gateway/> portion of the topology descriptor).
I am currently unable to update the KIP, but I can share the REST APIs I've
employed if there is interest.
As a result, I'm wondering if Knox should support provider configuration
references in topology.xml, rather than having to duplicate it across
descriptors.
So, instead of <gateway><provider>...</gateway> in each topology.xml, have
a single element that points to an external provider config (e.g.,
<policy-ref>$GATEWAY_HOME/conf/policy/my-named-provider-
config.xml</policy-ref>).
I've already externalized it for input to the POC, but I'm still copying
the contents into the resulting topology descriptor; I think it would be
better to copy only the reference.
Thoughts?
- Phil
On Fri, Aug 18, 2017 at 1:55 PM, larry mccay <lm...@apache.org> wrote:
> Good to hear, Phil.
>
> Yes, I was looking to go back and add some of the API specifics and
> investigation details for at least Ambari and ZK.
> If others make sense to add such as Consul, etcd, etc that would be good
as
> well and it would help to tease out some of what may be needed for the
> abstraction API and config.
>
>
> On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino <pz...@gmail.com>
> wrote:
>
> > This sounds like a good improvement from a usability perspective.
> >
> > I agree that Knox must continue its support for direct topology (XML)
> > definitions, but for those deployments where there is a service registry
> > (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of the
topology,
> > the simplified config coupled with the service discovery could reduce
> > topology (especially ServiceConnectivity) errors.
> >
> > I also agree that consolidating provider configurations into named sets,
> > which can be referenced from topology configurations, will be a good
> > change, especially if there are commonly grouped providers employed by
> > multiple topologies.
> >
> > Concerning the TODO sections, is your intention is that they contain the
> > respective API facilities that will provide the information needed to
> > populate a topology definition?
> >
> > - Phil
> >
> > On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <lm...@apache.org> wrote:
> >
> > > All -
> > >
> > > I have published an initial stab at a proposal for simplifying
topology
> > > management [1] - as I briefly mentioned on the 0.14.0 planning thread.
> > >
> > > There are a couple TODO sections that need some investigation for
> Ambari
> > > API and Zookeeper Registry API as discovery plugins.
> > >
> > > Please give the KIP a read and let me know if it makes sense or if you
> > > would like to take on specific pieces of this work. If you would like
> to
> > > add other options or ideas, etc.
> > >
> > > I think that this will have a huge impact as usability improvements
for
> > > management of the gateway!
> > >
> > > thanks,
> > >
> > > --larry
> > >
> > > 1.
> > > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+
> > > Service+Discovery+and+Topology+Generation
> > >
> >
>
Re: KIP-8 Service Discovery and Topology Generation
Posted by Philip Zampino <pz...@gmail.com>.
I've put together a quick python POC targeting Ambari as the discovery
source, just to prove that we can indeed get the necessary details via
Ambari's REST API.
It generates a proper topology.xml descriptor based on a simple descriptor
(I chose YAML for this POC), which has a reference to what I've called a
policy descriptor (the <gateway/> portion of the topology descriptor).
I am currently unable to update the KIP, but I can share the REST APIs I've
employed if there is interest.
As a result, I'm wondering if Knox should support provider configuration
references in topology.xml, rather than having to duplicate it across
descriptors.
So, instead of <gateway><provider>...</gateway> in each topology.xml, have
a single element that points to an external provider config (e.g.,
<policy-ref>$GATEWAY_HOME/conf/policy/my-named-provider-config.xml</policy-ref>).
I've already externalized it for input to the POC, but I'm still copying
the contents into the resulting topology descriptor; I think it would be
better to copy only the reference.
Thoughts?
- Phil
On Fri, Aug 18, 2017 at 1:55 PM, larry mccay <lm...@apache.org> wrote:
> Good to hear, Phil.
>
> Yes, I was looking to go back and add some of the API specifics and
> investigation details for at least Ambari and ZK.
> If others make sense to add such as Consul, etcd, etc that would be good as
> well and it would help to tease out some of what may be needed for the
> abstraction API and config.
>
>
> On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino <pz...@gmail.com>
> wrote:
>
> > This sounds like a good improvement from a usability perspective.
> >
> > I agree that Knox must continue its support for direct topology (XML)
> > definitions, but for those deployments where there is a service registry
> > (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of the topology,
> > the simplified config coupled with the service discovery could reduce
> > topology (especially ServiceConnectivity) errors.
> >
> > I also agree that consolidating provider configurations into named sets,
> > which can be referenced from topology configurations, will be a good
> > change, especially if there are commonly grouped providers employed by
> > multiple topologies.
> >
> > Concerning the TODO sections, is your intention is that they contain the
> > respective API facilities that will provide the information needed to
> > populate a topology definition?
> >
> > - Phil
> >
> > On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <lm...@apache.org> wrote:
> >
> > > All -
> > >
> > > I have published an initial stab at a proposal for simplifying topology
> > > management [1] - as I briefly mentioned on the 0.14.0 planning thread.
> > >
> > > There are a couple TODO sections that need some investigation for
> Ambari
> > > API and Zookeeper Registry API as discovery plugins.
> > >
> > > Please give the KIP a read and let me know if it makes sense or if you
> > > would like to take on specific pieces of this work. If you would like
> to
> > > add other options or ideas, etc.
> > >
> > > I think that this will have a huge impact as usability improvements for
> > > management of the gateway!
> > >
> > > thanks,
> > >
> > > --larry
> > >
> > > 1.
> > > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+
> > > Service+Discovery+and+Topology+Generation
> > >
> >
>
Re: KIP-8 Service Discovery and Topology Generation
Posted by larry mccay <lm...@apache.org>.
Good to hear, Phil.
Yes, I was looking to go back and add some of the API specifics and
investigation details for at least Ambari and ZK.
If others make sense to add such as Consul, etcd, etc that would be good as
well and it would help to tease out some of what may be needed for the
abstraction API and config.
On Fri, Aug 18, 2017 at 1:52 PM, Philip Zampino <pz...@gmail.com> wrote:
> This sounds like a good improvement from a usability perspective.
>
> I agree that Knox must continue its support for direct topology (XML)
> definitions, but for those deployments where there is a service registry
> (e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of the topology,
> the simplified config coupled with the service discovery could reduce
> topology (especially ServiceConnectivity) errors.
>
> I also agree that consolidating provider configurations into named sets,
> which can be referenced from topology configurations, will be a good
> change, especially if there are commonly grouped providers employed by
> multiple topologies.
>
> Concerning the TODO sections, is your intention is that they contain the
> respective API facilities that will provide the information needed to
> populate a topology definition?
>
> - Phil
>
> On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <lm...@apache.org> wrote:
>
> > All -
> >
> > I have published an initial stab at a proposal for simplifying topology
> > management [1] - as I briefly mentioned on the 0.14.0 planning thread.
> >
> > There are a couple TODO sections that need some investigation for Ambari
> > API and Zookeeper Registry API as discovery plugins.
> >
> > Please give the KIP a read and let me know if it makes sense or if you
> > would like to take on specific pieces of this work. If you would like to
> > add other options or ideas, etc.
> >
> > I think that this will have a huge impact as usability improvements for
> > management of the gateway!
> >
> > thanks,
> >
> > --larry
> >
> > 1.
> > https://cwiki.apache.org/confluence/display/KNOX/KIP-8+
> > Service+Discovery+and+Topology+Generation
> >
>
Re: KIP-8 Service Discovery and Topology Generation
Posted by Philip Zampino <pz...@gmail.com>.
This sounds like a good improvement from a usability perspective.
I agree that Knox must continue its support for direct topology (XML)
definitions, but for those deployments where there is a service registry
(e.g., Ambari, Zookeeper, Consul, etc...) with knowledge of the topology,
the simplified config coupled with the service discovery could reduce
topology (especially ServiceConnectivity) errors.
I also agree that consolidating provider configurations into named sets,
which can be referenced from topology configurations, will be a good
change, especially if there are commonly grouped providers employed by
multiple topologies.
Concerning the TODO sections, is your intention is that they contain the
respective API facilities that will provide the information needed to
populate a topology definition?
- Phil
On Fri, Aug 18, 2017 at 12:39 PM, larry mccay <lm...@apache.org> wrote:
> All -
>
> I have published an initial stab at a proposal for simplifying topology
> management [1] - as I briefly mentioned on the 0.14.0 planning thread.
>
> There are a couple TODO sections that need some investigation for Ambari
> API and Zookeeper Registry API as discovery plugins.
>
> Please give the KIP a read and let me know if it makes sense or if you
> would like to take on specific pieces of this work. If you would like to
> add other options or ideas, etc.
>
> I think that this will have a huge impact as usability improvements for
> management of the gateway!
>
> thanks,
>
> --larry
>
> 1.
> https://cwiki.apache.org/confluence/display/KNOX/KIP-8+
> Service+Discovery+and+Topology+Generation
>