You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@knox.apache.org by larry mccay <lm...@apache.org> on 2017/06/26 16:11:03 UTC
[DISCUSS] Future of UI Proxying and Rewrite Rules in Apache Knox
All -
It is becoming more and more clear that UI proxying is going to continue to
be a moving target and we need to determine how to simplify the authoring
and maintenance of UI rewrite rules.
While I haven't put a lot of thought around it or even POC'd it yet, I am
thinking about the possibility of leveraging the new port-mapping feature
for UIs.
This may at least be able to eliminate the need for the "gateway/topology"
patch prefix by the fact that we dedicate specific ports to specific
topologies.
The downside of this is that it contradicts one of our early tenants of
enabling deployments to have a single host:port available to access all of
the REST API resources that they require.
We may be able to justify that UIs be on a separate port however.
Then we will also need to deal with backward compatibility issues for
deployments that are currently using the existing service definitions - we
may be able to accommodate this using the versioning built into service
defs.
Anyway, I just thought that I would start a discussion on this and see what
folks have to say.
Thoughts?
--larry
Re: [DISCUSS] Future of UI Proxying and Rewrite Rules in Apache Knox
Posted by Jeffrey Rodriguez <je...@gmail.com>.
+1 too on my side. Best of all is that it fixes current broken behavior.
On Tue, Jun 27, 2017 at 3:17 PM, Mohammad Islam <mi...@yahoo.com> wrote:
> +1 on the original proposal provided we have backward compatibility and
> easy migration path - that looks like we are already considering.
>
>
>
>
> On Tuesday, June 27, 2017 1:30 PM, Jeffrey Rodriguez <je...@gmail.com>
> wrote:
>
>
> Thanks for the clarification, I just read the KIP. Got it now :-)
> Regards,
> Jeff
>
> On Mon, Jun 26, 2017 at 10:01 AM, larry mccay <lm...@apache.org> wrote:
>
> Hi Jeff -
>
> The port-mapping is not actually on the service dispatch side but instead
> client facing.
> So, instead of requiring: https://localhost:8443/
> gateway/sandbox/webhdfs/v1/? op=LISTSTATUS
> <https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS> to
> access webhdfs through the sandbox topology we can dedicate a new port to
> listen for requests that is dedicated to the sandbox topology. This will
> allow: https://localhost:8444/ webhdfs/v1/?op=LISTSTATUS
> <https://localhost:8444/webhdfs/v1/?op=LISTSTATUS> to be used instead.
>
> You can read more about port-mapping in the KIP-6 [1]
>
> The point of this thread is to investigate the fact that the relative URLs
> that are used in UIs will automatically be going back to the same host:port
> as was used in the initial request that there is no reason to add the
> additional prefix of "gateway/sandbox" to the relative URL and how we might
> consider forcing port-mapping for UI-only topologies and simplify the
> rewrite rules for UIs.
>
> Hope that makes sense...
>
> thanks,
>
> --larry
>
>
> 1. https://cwiki.apache.org/ confluence/display/KNOX/KIP-6+
> Topology+Port+Mapping
> <https://cwiki.apache.org/confluence/display/KNOX/KIP-6+Topology+Port+Mapping>
>
> On Mon, Jun 26, 2017 at 12:46 PM, Jeffrey Rodriguez <je...@gmail.com>
> wrote:
>
> All,
> The information about hosts/ports is known to Ambari and topology
> gets that information from Ambari Stack params calls to cluster components
> variables. e.g. When we enable HA, we get all namenodes hosts, ports and
> whether SSL is on so we can setup scheme, host and port. Not only that but
> through the ports can be change.
> So there is a lot of information for UIs and REST hosts/ports and whether
> we support HA or not.
> Would the new port mapping, generate the scheme, port, HA enable/not,
> host(s)? I can see if this information would be generated by call from Knox
> stack and be part of the configuration then we would not need the topology.
> One requirement would also be that when cluster configuration would change
> the port mapping would need to be updated too. So maybe cluster services,
> WEBHDFS for example, if we set HA and have a set of namenodes, ports,
> scheme, when the service starts it would update knox data (plain file or
> json).
>
> Regards,
> Jeff Rodriguez
>
>
> On Mon, Jun 26, 2017 at 9:11 AM, larry mccay <lm...@apache.org> wrote:
>
> All -
>
> It is becoming more and more clear that UI proxying is going to continue
> to be a moving target and we need to determine how to simplify the
> authoring and maintenance of UI rewrite rules.
>
> While I haven't put a lot of thought around it or even POC'd it yet, I am
> thinking about the possibility of leveraging the new port-mapping feature
> for UIs.
>
> This may at least be able to eliminate the need for the "gateway/topology"
> patch prefix by the fact that we dedicate specific ports to specific
> topologies.
>
> The downside of this is that it contradicts one of our early tenants of
> enabling deployments to have a single host:port available to access all of
> the REST API resources that they require.
>
> We may be able to justify that UIs be on a separate port however.
>
> Then we will also need to deal with backward compatibility issues for
> deployments that are currently using the existing service definitions - we
> may be able to accommodate this using the versioning built into service
> defs.
>
> Anyway, I just thought that I would start a discussion on this and see
> what folks have to say.
>
> Thoughts?
>
> --larry
>
>
>
>
>
>
>
Re: [DISCUSS] Future of UI Proxying and Rewrite Rules in Apache
Knox
Posted by Mohammad Islam <mi...@yahoo.com>.
+1 on the original proposal provided we have backward compatibility and easy migration path - that looks like we are already considering.
On Tuesday, June 27, 2017 1:30 PM, Jeffrey Rodriguez <je...@gmail.com> wrote:
Thanks for the clarification, I just read the KIP. Got it now :-)
Regards,
Jeff
On Mon, Jun 26, 2017 at 10:01 AM, larry mccay <lm...@apache.org> wrote:
Hi Jeff -
The port-mapping is not actually on the service dispatch side but instead client facing.So, instead of requiring: https://localhost:8443/ gateway/sandbox/webhdfs/v1/? op=LISTSTATUS to access webhdfs through the sandbox topology we can dedicate a new port to listen for requests that is dedicated to the sandbox topology. This will allow: https://localhost:8444/ webhdfs/v1/?op=LISTSTATUS to be used instead.
You can read more about port-mapping in the KIP-6 [1]
The point of this thread is to investigate the fact that the relative URLs that are used in UIs will automatically be going back to the same host:port as was used in the initial request that there is no reason to add the additional prefix of "gateway/sandbox" to the relative URL and how we might consider forcing port-mapping for UI-only topologies and simplify the rewrite rules for UIs.
Hope that makes sense...
thanks,
--larry
1. https://cwiki.apache.org/ confluence/display/KNOX/KIP-6+ Topology+Port+Mapping
On Mon, Jun 26, 2017 at 12:46 PM, Jeffrey Rodriguez <je...@gmail.com> wrote:
All, The information about hosts/ports is known to Ambari and topology gets that information from Ambari Stack params calls to cluster components variables. e.g. When we enable HA, we get all namenodes hosts, ports and whether SSL is on so we can setup scheme, host and port. Not only that but through the ports can be change.So there is a lot of information for UIs and REST hosts/ports and whether we support HA or not.Would the new port mapping, generate the scheme, port, HA enable/not, host(s)? I can see if this information would be generated by call from Knox stack and be part of the configuration then we would not need the topology.One requirement would also be that when cluster configuration would change the port mapping would need to be updated too. So maybe cluster services, WEBHDFS for example, if we set HA and have a set of namenodes, ports, scheme, when the service starts it would update knox data (plain file or json).
Regards, Jeff Rodriguez
On Mon, Jun 26, 2017 at 9:11 AM, larry mccay <lm...@apache.org> wrote:
All -
It is becoming more and more clear that UI proxying is going to continue to be a moving target and we need to determine how to simplify the authoring and maintenance of UI rewrite rules.
While I haven't put a lot of thought around it or even POC'd it yet, I am thinking about the possibility of leveraging the new port-mapping feature for UIs.
This may at least be able to eliminate the need for the "gateway/topology" patch prefix by the fact that we dedicate specific ports to specific topologies.
The downside of this is that it contradicts one of our early tenants of enabling deployments to have a single host:port available to access all of the REST API resources that they require.
We may be able to justify that UIs be on a separate port however.
Then we will also need to deal with backward compatibility issues for deployments that are currently using the existing service definitions - we may be able to accommodate this using the versioning built into service defs.
Anyway, I just thought that I would start a discussion on this and see what folks have to say.
Thoughts?
--larry
Re: [DISCUSS] Future of UI Proxying and Rewrite Rules in Apache Knox
Posted by Jeffrey Rodriguez <je...@gmail.com>.
Thanks for the clarification, I just read the KIP. Got it now :-)
Regards,
Jeff
On Mon, Jun 26, 2017 at 10:01 AM, larry mccay <lm...@apache.org> wrote:
> Hi Jeff -
>
> The port-mapping is not actually on the service dispatch side but instead
> client facing.
> So, instead of requiring: https://localhost:8443/
> gateway/sandbox/webhdfs/v1/?op=LISTSTATUS to access webhdfs through the
> sandbox topology we can dedicate a new port to listen for requests that is
> dedicated to the sandbox topology. This will allow:
> https://localhost:8444/webhdfs/v1/?op=LISTSTATUS to be used instead.
>
> You can read more about port-mapping in the KIP-6 [1]
>
> The point of this thread is to investigate the fact that the relative URLs
> that are used in UIs will automatically be going back to the same host:port
> as was used in the initial request that there is no reason to add the
> additional prefix of "gateway/sandbox" to the relative URL and how we might
> consider forcing port-mapping for UI-only topologies and simplify the
> rewrite rules for UIs.
>
> Hope that makes sense...
>
> thanks,
>
> --larry
>
>
> 1. https://cwiki.apache.org/confluence/display/KNOX/KIP-6+
> Topology+Port+Mapping
>
> On Mon, Jun 26, 2017 at 12:46 PM, Jeffrey Rodriguez <je...@gmail.com>
> wrote:
>
>> All,
>> The information about hosts/ports is known to Ambari and topology
>> gets that information from Ambari Stack params calls to cluster components
>> variables. e.g. When we enable HA, we get all namenodes hosts, ports and
>> whether SSL is on so we can setup scheme, host and port. Not only that but
>> through the ports can be change.
>> So there is a lot of information for UIs and REST hosts/ports and whether
>> we support HA or not.
>> Would the new port mapping, generate the scheme, port, HA enable/not,
>> host(s)? I can see if this information would be generated by call from Knox
>> stack and be part of the configuration then we would not need the topology.
>> One requirement would also be that when cluster configuration would
>> change the port mapping would need to be updated too. So maybe cluster
>> services, WEBHDFS for example, if we set HA and have a set of namenodes,
>> ports, scheme, when the service starts it would update knox data (plain
>> file or json).
>>
>> Regards,
>> Jeff Rodriguez
>>
>>
>> On Mon, Jun 26, 2017 at 9:11 AM, larry mccay <lm...@apache.org> wrote:
>>
>>> All -
>>>
>>> It is becoming more and more clear that UI proxying is going to continue
>>> to be a moving target and we need to determine how to simplify the
>>> authoring and maintenance of UI rewrite rules.
>>>
>>> While I haven't put a lot of thought around it or even POC'd it yet, I
>>> am thinking about the possibility of leveraging the new port-mapping
>>> feature for UIs.
>>>
>>> This may at least be able to eliminate the need for the
>>> "gateway/topology" patch prefix by the fact that we dedicate specific ports
>>> to specific topologies.
>>>
>>> The downside of this is that it contradicts one of our early tenants of
>>> enabling deployments to have a single host:port available to access all of
>>> the REST API resources that they require.
>>>
>>> We may be able to justify that UIs be on a separate port however.
>>>
>>> Then we will also need to deal with backward compatibility issues for
>>> deployments that are currently using the existing service definitions - we
>>> may be able to accommodate this using the versioning built into service
>>> defs.
>>>
>>> Anyway, I just thought that I would start a discussion on this and see
>>> what folks have to say.
>>>
>>> Thoughts?
>>>
>>> --larry
>>>
>>
>>
>
Re: [DISCUSS] Future of UI Proxying and Rewrite Rules in Apache Knox
Posted by larry mccay <lm...@apache.org>.
Hi Jeff -
The port-mapping is not actually on the service dispatch side but instead
client facing.
So, instead of requiring:
https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS to access
webhdfs through the sandbox topology we can dedicate a new port to listen
for requests that is dedicated to the sandbox topology. This will allow:
https://localhost:8444/webhdfs/v1/?op=LISTSTATUS to be used instead.
You can read more about port-mapping in the KIP-6 [1]
The point of this thread is to investigate the fact that the relative URLs
that are used in UIs will automatically be going back to the same host:port
as was used in the initial request that there is no reason to add the
additional prefix of "gateway/sandbox" to the relative URL and how we might
consider forcing port-mapping for UI-only topologies and simplify the
rewrite rules for UIs.
Hope that makes sense...
thanks,
--larry
1.
https://cwiki.apache.org/confluence/display/KNOX/KIP-6+Topology+Port+Mapping
On Mon, Jun 26, 2017 at 12:46 PM, Jeffrey Rodriguez <je...@gmail.com>
wrote:
> All,
> The information about hosts/ports is known to Ambari and topology
> gets that information from Ambari Stack params calls to cluster components
> variables. e.g. When we enable HA, we get all namenodes hosts, ports and
> whether SSL is on so we can setup scheme, host and port. Not only that but
> through the ports can be change.
> So there is a lot of information for UIs and REST hosts/ports and whether
> we support HA or not.
> Would the new port mapping, generate the scheme, port, HA enable/not,
> host(s)? I can see if this information would be generated by call from Knox
> stack and be part of the configuration then we would not need the topology.
> One requirement would also be that when cluster configuration would change
> the port mapping would need to be updated too. So maybe cluster services,
> WEBHDFS for example, if we set HA and have a set of namenodes, ports,
> scheme, when the service starts it would update knox data (plain file or
> json).
>
> Regards,
> Jeff Rodriguez
>
>
> On Mon, Jun 26, 2017 at 9:11 AM, larry mccay <lm...@apache.org> wrote:
>
>> All -
>>
>> It is becoming more and more clear that UI proxying is going to continue
>> to be a moving target and we need to determine how to simplify the
>> authoring and maintenance of UI rewrite rules.
>>
>> While I haven't put a lot of thought around it or even POC'd it yet, I am
>> thinking about the possibility of leveraging the new port-mapping feature
>> for UIs.
>>
>> This may at least be able to eliminate the need for the
>> "gateway/topology" patch prefix by the fact that we dedicate specific ports
>> to specific topologies.
>>
>> The downside of this is that it contradicts one of our early tenants of
>> enabling deployments to have a single host:port available to access all of
>> the REST API resources that they require.
>>
>> We may be able to justify that UIs be on a separate port however.
>>
>> Then we will also need to deal with backward compatibility issues for
>> deployments that are currently using the existing service definitions - we
>> may be able to accommodate this using the versioning built into service
>> defs.
>>
>> Anyway, I just thought that I would start a discussion on this and see
>> what folks have to say.
>>
>> Thoughts?
>>
>> --larry
>>
>
>
Re: [DISCUSS] Future of UI Proxying and Rewrite Rules in Apache Knox
Posted by Jeffrey Rodriguez <je...@gmail.com>.
All,
The information about hosts/ports is known to Ambari and topology gets
that information from Ambari Stack params calls to cluster components
variables. e.g. When we enable HA, we get all namenodes hosts, ports and
whether SSL is on so we can setup scheme, host and port. Not only that but
through the ports can be change.
So there is a lot of information for UIs and REST hosts/ports and whether
we support HA or not.
Would the new port mapping, generate the scheme, port, HA enable/not,
host(s)? I can see if this information would be generated by call from Knox
stack and be part of the configuration then we would not need the topology.
One requirement would also be that when cluster configuration would change
the port mapping would need to be updated too. So maybe cluster services,
WEBHDFS for example, if we set HA and have a set of namenodes, ports,
scheme, when the service starts it would update knox data (plain file or
json).
Regards,
Jeff Rodriguez
On Mon, Jun 26, 2017 at 9:11 AM, larry mccay <lm...@apache.org> wrote:
> All -
>
> It is becoming more and more clear that UI proxying is going to continue
> to be a moving target and we need to determine how to simplify the
> authoring and maintenance of UI rewrite rules.
>
> While I haven't put a lot of thought around it or even POC'd it yet, I am
> thinking about the possibility of leveraging the new port-mapping feature
> for UIs.
>
> This may at least be able to eliminate the need for the "gateway/topology"
> patch prefix by the fact that we dedicate specific ports to specific
> topologies.
>
> The downside of this is that it contradicts one of our early tenants of
> enabling deployments to have a single host:port available to access all of
> the REST API resources that they require.
>
> We may be able to justify that UIs be on a separate port however.
>
> Then we will also need to deal with backward compatibility issues for
> deployments that are currently using the existing service definitions - we
> may be able to accommodate this using the versioning built into service
> defs.
>
> Anyway, I just thought that I would start a discussion on this and see
> what folks have to say.
>
> Thoughts?
>
> --larry
>