You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Sander Hoentjen <sa...@hoentjen.eu> on 2017/02/15 10:50:14 UTC
mod_remoteip and mod_http2 combined
Hey guys,
I am trying to use both mod_remoteip with ProxyProtocol and mod_http2.
It looks like mod_http2 gets handed the connection before mod_remoteip,
so things don't work as they should. ProxyProtocol should always be
handled first, since it is prepended to the stream. Any pointers to
where in the code I can look to change things around?
--
Sander
Re: mod_remoteip and mod_http2 combined
Posted by Stefan Eissing <st...@greenbytes.de>.
I can see that a flat directive namespace has its drawbacks... ;-)
> Am 01.04.2017 um 19:12 schrieb Daniel Ruggeri <DR...@primary.net>:
>
>
> On 4/1/2017 11:18 AM, Yann Ylavic wrote:
>> Hi Daniel,
>>
>> On Sat, Apr 1, 2017 at 3:56 PM, Daniel Ruggeri <DR...@primary.net> wrote:
>>> I went with the directive name
>>> RemoteIPProxyProtocolDisableHosts to align more with the fact that a
>>> single host or range can be disabled.
>> How about RemoteIPProxyProtocolExceptions since one can configure
>> either an IP or a network?
>> Host looks a bit ambiguous to me, especially with HTTP...
>>
>>
>> Regards,
>> Yann.
>
> Oh, yeah, I like that even better. I'll stage this up and will commit in
> a day or two unless we have some better suggestions.
>
> --
> Daniel Ruggeri
>
Re: mod_remoteip and mod_http2 combined
Posted by Daniel Ruggeri <DR...@primary.net>.
On 4/1/2017 11:18 AM, Yann Ylavic wrote:
> Hi Daniel,
>
> On Sat, Apr 1, 2017 at 3:56 PM, Daniel Ruggeri <DR...@primary.net> wrote:
>> I went with the directive name
>> RemoteIPProxyProtocolDisableHosts to align more with the fact that a
>> single host or range can be disabled.
> How about RemoteIPProxyProtocolExceptions since one can configure
> either an IP or a network?
> Host looks a bit ambiguous to me, especially with HTTP...
>
>
> Regards,
> Yann.
Oh, yeah, I like that even better. I'll stage this up and will commit in
a day or two unless we have some better suggestions.
--
Daniel Ruggeri
Re: mod_remoteip and mod_http2 combined
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Daniel,
On Sat, Apr 1, 2017 at 3:56 PM, Daniel Ruggeri <DR...@primary.net> wrote:
> I went with the directive name
> RemoteIPProxyProtocolDisableHosts to align more with the fact that a
> single host or range can be disabled.
How about RemoteIPProxyProtocolExceptions since one can configure
either an IP or a network?
Host looks a bit ambiguous to me, especially with HTTP...
Regards,
Yann.
Re: mod_remoteip and mod_http2 combined
Posted by Daniel Ruggeri <DR...@primary.net>.
Sorry for the top post. I've committed r1789800 which pulls out Optional
handling and adds the ability to disable based on source network. This
is more or less the code as it was donated, plus some cleanup and the
small addition to disable based on networks (overall a cleaner approach
anyway). I went with the directive name
RemoteIPProxyProtocolDisableHosts to align more with the fact that a
single host or range can be disabled. I've verified this works via
haproxy, rejects when hit directly and disables processing when coming
from a permitted network.
I'm in the process of updating the backport proposal. To be safe, I'm
removing @jim's vote given how many times the code has changed since he
reviewed and will put it back in the "active" section of STATUS.
--
Daniel Ruggeri
On 4/1/2017 8:17 AM, Daniel Ruggeri wrote:
> Agreed - as many times as I read the spec, I have no idea how I did not
> see that security advisory. It's flat-out damning to the idea of an
> "optional" mode. I'll go ahead and rip out the optional processing and
> will add your suggested idea of a list of subnets to disable parsing. I
> hope to have a patch later this morning to share. As awful as the name
> is, I'm thinking RemoteIPProxyProtocolDisableNetworks ARG1 ARG2 ARG3.
>
Re: mod_remoteip and mod_http2 combined
Posted by Daniel Ruggeri <DR...@primary.net>.
Agreed - as many times as I read the spec, I have no idea how I did not
see that security advisory. It's flat-out damning to the idea of an
"optional" mode. I'll go ahead and rip out the optional processing and
will add your suggested idea of a list of subnets to disable parsing. I
hope to have a patch later this morning to share. As awful as the name
is, I'm thinking RemoteIPProxyProtocolDisableNetworks ARG1 ARG2 ARG3.
--
Daniel Ruggeri
On 3/29/2017 4:43 PM, William A Rowe Jr wrote:
> This is the gist of my remaining objections.
>
> It would be nice if the mod_remoteip patch to PROXY protocol followed the
> security advisories of the PROXY draft security comments, and we rip out the
> 'optional' mode. The remaining objection is around the ambiguity of 'optional'
> (which can't exist) and the objection that how PROXY works as an implicit
> trust model using mod_remoteip is laughable, since the connection cannot
> be established without some PROXY protocol line interceptor yanking the
> garbage out of otherwise well-formed HTTP/1.1 - HTTP-TLS - h2c - h2 input.
>
> There is no 'untrusted PROXY header input' because that isn't part of the
> HTTP protocol and that garbage generates a 400 without an interceptor.
> No problem declaring that if we are willing to decode it, we will accept the
> input as gospel.
Re: mod_remoteip and mod_http2 combined
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Wed, Mar 29, 2017 at 4:43 PM, William A Rowe Jr <wr...@rowe-clan.net> wrote:
>
> It would be nice if the mod_remoteip patch to PROXY protocol followed the
> security advisories of the PROXY draft security comments, and we rip out the
> 'optional' mode. The remaining objection is around the ambiguity of 'optional'
> (which can't exist) and the objection that how PROXY works as an implicit
> trust model using mod_remoteip is laughable, since the connection cannot
> be established without some PROXY protocol line interceptor yanking the
> garbage out of otherwise well-formed HTTP/1.1 - HTTP-TLS - h2c - h2 input.
>
> There is no 'untrusted PROXY header input' because that isn't part of the
> HTTP protocol and that garbage generates a 400 without an interceptor.
> No problem declaring that if we are willing to decode it, we will accept the
> input as gospel.
(By this measure, I'm dropping any objection to also setting HTTPS TLS
content trust flags, although mod_ssl would typically provide much more
information about which server and client certificates had been presented
and what cipher is in use.)
Re: mod_remoteip and mod_http2 combined
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Mon, Mar 27, 2017 at 9:07 AM, Sander Hoentjen <sa...@hoentjen.eu> wrote:
>
> On 03/16/2017 10:34 AM, Sander Hoentjen wrote:
>>
>> On 03/11/2017 07:57 PM, Daniel Ruggeri wrote:
>>> Thanks, all, for the patience as I finally got back to this.
>>>
>>> On 2/24/2017 11:05 AM, Sander Hoentjen wrote:
>>>> On 02/20/2017 07:48 PM, William A Rowe Jr wrote:
>>>>> On Sat, Feb 18, 2017 at 4:25 PM, Daniel Ruggeri <dr...@apache.org> wrote:
>>>>>> On 2017-02-15 09:07 (-0600), William A Rowe Jr <wr...@rowe-clan.net> wrote:
>>>>>>> On Wed, Feb 15, 2017 at 9:02 AM, Sander Hoentjen <sa...@hoentjen.eu> wrote:
>>>>>>>> mod_remote ip has:
>>>>>>>> /* mod_proxy creates outgoing connections - we don't want those */
>>>>>>>> if (!remoteip_is_server_port(c->local_addr->port)) {
>>>>>>>> return DECLINED;
>>>>>>>> }
>>>>>>>> I am guessing something similar is needed for h2 connections?
>>>>>>> I suspect that the mod_remoteip logic is wrong, that it should be guarding
>>>>>>> against any subordinate connections and examining only explicitly configured
>>>>>>> ports / origin IPs. the PROXY protocol is not part of the HTTP protocol and
>>>>>>> incompatible with it, so the trust list logic isn't directly compatible (this is
>>>>>>> clearly explained in the PROXY pseudo-RFC.)
>>>>>>>
>>>>>> Hi, Bill. That is what the module is doing. The original authors wrote it to have a list of virtual hosts it is explicitly enabled for and explicitly disabled for. I added a third list for optional vhosts. In the pre_connection hook, it checks to see if the connection's local_addr (which should normally be the server's IP) is explicitly configured to enable PROXY handling. It then checks to see if the local port is a server port.
>>>>>>
>>>>>> Looking at the logs shared, 192.168.122.249:84 is the server IP:Port combo and is also the local IP:Port from mod_h2. If h2 sets the master of this connection, then we could skip the whole ordeal with this patch:
>>>>>>
>>>>>> Index: modules/metadata/mod_remoteip.c
>>>>>> ===================================================================
>>>>>> --- modules/metadata/mod_remoteip.c (revision 1781701)
>>>>>> +++ modules/metadata/mod_remoteip.c (working copy)
>>>>>> @@ -862,6 +862,10 @@
>>>>>> remoteip_conn_config_t *conn_conf;
>>>>>> int optional;
>>>>>>
>>>>>> + if (c->master != NULL) {
>>>>>> + return DECLINED;
>>>>>> + }
>>>>>> +
>>>>>> conf = ap_get_module_config(ap_server_conf->module_config,
>>>>>> &remoteip_module);
>>>>>>
>>>>>> .. but I don't know if that potentially means we are looking at the wrong connection.
>>>> First I'll say that with the "Optional" mode it worked, just not with On
>>>> I just tried this patch and as far as I have tested this seems to work
>>>> fine in On mode, as well as in Optional. I do see some other issue, but
>>>> that is probably in my own code, I'll try to track that down later.
>>> This is good news and about what I was expecting to happen. I will add
>>> this to the commit I've got coming that incorporates a lot of Ruediger's
>>> feedback.
>>>
>>>>> That should be close, but need to ensure c->master is initialized for
>>>>> http as well
>>>>> where there is no master/subordinate.
>>>> I am not sure what this means, how should I test this?
>>> Hi, Bill - also hoping for a bit more input. Since PROXY protocol is not
>>> tied to any particular layer 7 protocol, I don't think we'd have to
>>> verify it is initialized for HTTP - just that there is no master at all.
>>> At least, that's my understanding so I appreciate any corrections.
>> Here are my changes by the way:
>> https://github.com/AntagonistHQ/httpd/commit/2d208793b4494e73289477c231c79be9e0030a2b
>>> Sure, to clarify, the Optional use case came from a member on one of our
>>> cousin projects (Tomcat) Chris Schultz as well as my own use cases. It
>>> is useful for internally accessing the site from behind the
>>> loadbalancer. When there is a publicly addressed upstream loadbalancer
>>> (Amazon ELB or just HAProxy itself) talking to RFC1918 addressed or
>>> non-routeable backend httpd servers, it becomes impossible to enable
>>> internal communication on the RFC1918 space to the backend instances.
>>> If the goal is to monitor or probe the site and (httpd proxy) backends
>>> internally for health, this *can* be done by duplicating the virtual
>>> hosts. Depending on the complexity of the virtual hosts, what resources
>>> those virtual hosts have (proxies and whatnot) and their general size,
>>> this could result in a fairly unmanageable httpd configuration having a
>>> vhost that requires PROXY header and a second one on a different port or
>>> IP that does not.
>>> It gets even more complicated when you are aiming to do management
>>> tasks. If you have balancers configured at the vhost as intended, you
>>> can only manage those balancers from the vhost they live in. Further,
>>> you may want to view server statistics, check info about the ldap cache,
>>> etc but permit access to those things only from a trusted network in
>>> addition to the user credentials protecting it.
>>> So for those examples, aside from creating an internal HAProxy to
>>> provide the header, it's not really possible to access the site
>>> internally when "On". With "Optional" it's possible to keep the same
>>> number of vhosts and enable the various management handlers and restrict
>>> them to the RFC1918 space (plus any other authnz mechanisms). The other
>>> obvious option would be to enable those handlers to the public Internet,
>>> but that's bad juju :-)
>>>
>> Well maybe what I am doing is wrong, but I just add a port to my vhosts,
>> so I have:
>> <VirtualHost $some_ip:80 $some_ip:83 >
>> and then I also have:
>> <VirtualHost *:83>
>> RemoteIPProxyProtocol On
>> </VirtualHost>
>>
>> This way I can reach the same vhost with and without proxy protocol.
>>
> From the protocol specification
> http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt:
>
> '''The receiver MUST be configured to only receive the protocol described in this
> specification and MUST not try to guess whether the protocol header is present
> or not. This means that the protocol explicitly prevents port sharing between
> public and private access. Otherwise it would open a major security breach by
> allowing untrusted parties to spoof their connection addresses. The receiver
> SHOULD ensure proper access filtering so that only trusted proxies are allowed
> to use this protocol.'''
>
> So I don't think Optional is allowed, if you treat the specification as
> authoritative.
> A vhost reachable by multiple ip's and/or ports would be a better way to
> solve it then.
>
> Thinking more about this: Right now the patch proposes
> RemoteIPProxyProtocol inside a vhost config, but wouldn't it be better
> (since it is connection-specific) to have something like a
> ProxyProtocolListen directive? Where you say instead of:
> ------
> <VirtualHost 127.0.0.1:9001>
> RemoteIPProxyProtocol On
> </VirtualHost>
> ------
> Something like:
> ------
> ProxyProtocolListen 127.0.0.1:9001
> or
> ProxyProtocolEnable 127.0.0.1:9001
> ------
>
> IMHO this is much cleaner than within a vhost (because that has
> side-effects on other vhosts as well)
This is the gist of my remaining objections.
It would be nice if the mod_remoteip patch to PROXY protocol followed the
security advisories of the PROXY draft security comments, and we rip out the
'optional' mode. The remaining objection is around the ambiguity of 'optional'
(which can't exist) and the objection that how PROXY works as an implicit
trust model using mod_remoteip is laughable, since the connection cannot
be established without some PROXY protocol line interceptor yanking the
garbage out of otherwise well-formed HTTP/1.1 - HTTP-TLS - h2c - h2 input.
There is no 'untrusted PROXY header input' because that isn't part of the
HTTP protocol and that garbage generates a 400 without an interceptor.
No problem declaring that if we are willing to decode it, we will accept the
input as gospel.
Re: mod_remoteip and mod_http2 combined
Posted by Sander Hoentjen <sa...@hoentjen.eu>.
On 03/16/2017 10:34 AM, Sander Hoentjen wrote:
>
> On 03/11/2017 07:57 PM, Daniel Ruggeri wrote:
>> Thanks, all, for the patience as I finally got back to this.
>>
>> On 2/24/2017 11:05 AM, Sander Hoentjen wrote:
>>> On 02/20/2017 07:48 PM, William A Rowe Jr wrote:
>>>> On Sat, Feb 18, 2017 at 4:25 PM, Daniel Ruggeri <dr...@apache.org> wrote:
>>>>> On 2017-02-15 09:07 (-0600), William A Rowe Jr <wr...@rowe-clan.net> wrote:
>>>>>> On Wed, Feb 15, 2017 at 9:02 AM, Sander Hoentjen <sa...@hoentjen.eu> wrote:
>>>>>>> mod_remote ip has:
>>>>>>> /* mod_proxy creates outgoing connections - we don't want those */
>>>>>>> if (!remoteip_is_server_port(c->local_addr->port)) {
>>>>>>> return DECLINED;
>>>>>>> }
>>>>>>> I am guessing something similar is needed for h2 connections?
>>>>>> I suspect that the mod_remoteip logic is wrong, that it should be guarding
>>>>>> against any subordinate connections and examining only explicitly configured
>>>>>> ports / origin IPs. the PROXY protocol is not part of the HTTP protocol and
>>>>>> incompatible with it, so the trust list logic isn't directly compatible (this is
>>>>>> clearly explained in the PROXY pseudo-RFC.)
>>>>>>
>>>>> Hi, Bill. That is what the module is doing. The original authors wrote it to have a list of virtual hosts it is explicitly enabled for and explicitly disabled for. I added a third list for optional vhosts. In the pre_connection hook, it checks to see if the connection's local_addr (which should normally be the server's IP) is explicitly configured to enable PROXY handling. It then checks to see if the local port is a server port.
>>>>>
>>>>> Looking at the logs shared, 192.168.122.249:84 is the server IP:Port combo and is also the local IP:Port from mod_h2. If h2 sets the master of this connection, then we could skip the whole ordeal with this patch:
>>>>>
>>>>> Index: modules/metadata/mod_remoteip.c
>>>>> ===================================================================
>>>>> --- modules/metadata/mod_remoteip.c (revision 1781701)
>>>>> +++ modules/metadata/mod_remoteip.c (working copy)
>>>>> @@ -862,6 +862,10 @@
>>>>> remoteip_conn_config_t *conn_conf;
>>>>> int optional;
>>>>>
>>>>> + if (c->master != NULL) {
>>>>> + return DECLINED;
>>>>> + }
>>>>> +
>>>>> conf = ap_get_module_config(ap_server_conf->module_config,
>>>>> &remoteip_module);
>>>>>
>>>>> .. but I don't know if that potentially means we are looking at the wrong connection.
>>> First I'll say that with the "Optional" mode it worked, just not with On
>>> I just tried this patch and as far as I have tested this seems to work
>>> fine in On mode, as well as in Optional. I do see some other issue, but
>>> that is probably in my own code, I'll try to track that down later.
>> This is good news and about what I was expecting to happen. I will add
>> this to the commit I've got coming that incorporates a lot of Ruediger's
>> feedback.
>>
>>>> That should be close, but need to ensure c->master is initialized for
>>>> http as well
>>>> where there is no master/subordinate.
>>> I am not sure what this means, how should I test this?
>> Hi, Bill - also hoping for a bit more input. Since PROXY protocol is not
>> tied to any particular layer 7 protocol, I don't think we'd have to
>> verify it is initialized for HTTP - just that there is no master at all.
>> At least, that's my understanding so I appreciate any corrections.
> Here are my changes by the way:
> https://github.com/AntagonistHQ/httpd/commit/2d208793b4494e73289477c231c79be9e0030a2b
>> Sure, to clarify, the Optional use case came from a member on one of our
>> cousin projects (Tomcat) Chris Schultz as well as my own use cases. It
>> is useful for internally accessing the site from behind the
>> loadbalancer. When there is a publicly addressed upstream loadbalancer
>> (Amazon ELB or just HAProxy itself) talking to RFC1918 addressed or
>> non-routeable backend httpd servers, it becomes impossible to enable
>> internal communication on the RFC1918 space to the backend instances.
>> If the goal is to monitor or probe the site and (httpd proxy) backends
>> internally for health, this *can* be done by duplicating the virtual
>> hosts. Depending on the complexity of the virtual hosts, what resources
>> those virtual hosts have (proxies and whatnot) and their general size,
>> this could result in a fairly unmanageable httpd configuration having a
>> vhost that requires PROXY header and a second one on a different port or
>> IP that does not.
>> It gets even more complicated when you are aiming to do management
>> tasks. If you have balancers configured at the vhost as intended, you
>> can only manage those balancers from the vhost they live in. Further,
>> you may want to view server statistics, check info about the ldap cache,
>> etc but permit access to those things only from a trusted network in
>> addition to the user credentials protecting it.
>> So for those examples, aside from creating an internal HAProxy to
>> provide the header, it's not really possible to access the site
>> internally when "On". With "Optional" it's possible to keep the same
>> number of vhosts and enable the various management handlers and restrict
>> them to the RFC1918 space (plus any other authnz mechanisms). The other
>> obvious option would be to enable those handlers to the public Internet,
>> but that's bad juju :-)
>>
> Well maybe what I am doing is wrong, but I just add a port to my vhosts,
> so I have:
> <VirtualHost $some_ip:80 $some_ip:83 >
> and then I also have:
> <VirtualHost *:83>
> RemoteIPProxyProtocol On
> </VirtualHost>
>
> This way I can reach the same vhost with and without proxy protocol.
>
From the protocol specification
http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt:
'''The receiver MUST be configured to only receive the protocol described in this
specification and MUST not try to guess whether the protocol header is present
or not. This means that the protocol explicitly prevents port sharing between
public and private access. Otherwise it would open a major security breach by
allowing untrusted parties to spoof their connection addresses. The receiver
SHOULD ensure proper access filtering so that only trusted proxies are allowed
to use this protocol.'''
So I don't think Optional is allowed, if you treat the specification as
authoritative.
A vhost reachable by multiple ip's and/or ports would be a better way to
solve it then.
Thinking more about this: Right now the patch proposes
RemoteIPProxyProtocol inside a vhost config, but wouldn't it be better
(since it is connection-specific) to have something like a
ProxyProtocolListen directive? Where you say instead of:
------
<VirtualHost 127.0.0.1:9001>
RemoteIPProxyProtocol On
</VirtualHost>
------
Something like:
------
ProxyProtocolListen 127.0.0.1:9001
or
ProxyProtocolEnable 127.0.0.1:9001
------
IMHO this is much cleaner than within a vhost (because that has
side-effects on other vhosts as well)
Regards,
Sander Hoentjen
Re: mod_remoteip and mod_http2 combined
Posted by Sander Hoentjen <sa...@hoentjen.eu>.
On 03/11/2017 07:57 PM, Daniel Ruggeri wrote:
> Thanks, all, for the patience as I finally got back to this.
>
> On 2/24/2017 11:05 AM, Sander Hoentjen wrote:
>> On 02/20/2017 07:48 PM, William A Rowe Jr wrote:
>>> On Sat, Feb 18, 2017 at 4:25 PM, Daniel Ruggeri <dr...@apache.org> wrote:
>>>> On 2017-02-15 09:07 (-0600), William A Rowe Jr <wr...@rowe-clan.net> wrote:
>>>>> On Wed, Feb 15, 2017 at 9:02 AM, Sander Hoentjen <sa...@hoentjen.eu> wrote:
>>>>>> mod_remote ip has:
>>>>>> /* mod_proxy creates outgoing connections - we don't want those */
>>>>>> if (!remoteip_is_server_port(c->local_addr->port)) {
>>>>>> return DECLINED;
>>>>>> }
>>>>>> I am guessing something similar is needed for h2 connections?
>>>>> I suspect that the mod_remoteip logic is wrong, that it should be guarding
>>>>> against any subordinate connections and examining only explicitly configured
>>>>> ports / origin IPs. the PROXY protocol is not part of the HTTP protocol and
>>>>> incompatible with it, so the trust list logic isn't directly compatible (this is
>>>>> clearly explained in the PROXY pseudo-RFC.)
>>>>>
>>>> Hi, Bill. That is what the module is doing. The original authors wrote it to have a list of virtual hosts it is explicitly enabled for and explicitly disabled for. I added a third list for optional vhosts. In the pre_connection hook, it checks to see if the connection's local_addr (which should normally be the server's IP) is explicitly configured to enable PROXY handling. It then checks to see if the local port is a server port.
>>>>
>>>> Looking at the logs shared, 192.168.122.249:84 is the server IP:Port combo and is also the local IP:Port from mod_h2. If h2 sets the master of this connection, then we could skip the whole ordeal with this patch:
>>>>
>>>> Index: modules/metadata/mod_remoteip.c
>>>> ===================================================================
>>>> --- modules/metadata/mod_remoteip.c (revision 1781701)
>>>> +++ modules/metadata/mod_remoteip.c (working copy)
>>>> @@ -862,6 +862,10 @@
>>>> remoteip_conn_config_t *conn_conf;
>>>> int optional;
>>>>
>>>> + if (c->master != NULL) {
>>>> + return DECLINED;
>>>> + }
>>>> +
>>>> conf = ap_get_module_config(ap_server_conf->module_config,
>>>> &remoteip_module);
>>>>
>>>> .. but I don't know if that potentially means we are looking at the wrong connection.
>> First I'll say that with the "Optional" mode it worked, just not with On
>> I just tried this patch and as far as I have tested this seems to work
>> fine in On mode, as well as in Optional. I do see some other issue, but
>> that is probably in my own code, I'll try to track that down later.
> This is good news and about what I was expecting to happen. I will add
> this to the commit I've got coming that incorporates a lot of Ruediger's
> feedback.
>
>>> That should be close, but need to ensure c->master is initialized for
>>> http as well
>>> where there is no master/subordinate.
>> I am not sure what this means, how should I test this?
> Hi, Bill - also hoping for a bit more input. Since PROXY protocol is not
> tied to any particular layer 7 protocol, I don't think we'd have to
> verify it is initialized for HTTP - just that there is no master at all.
> At least, that's my understanding so I appreciate any corrections.
Here are my changes by the way:
https://github.com/AntagonistHQ/httpd/commit/2d208793b4494e73289477c231c79be9e0030a2b
> Sure, to clarify, the Optional use case came from a member on one of our
> cousin projects (Tomcat) Chris Schultz as well as my own use cases. It
> is useful for internally accessing the site from behind the
> loadbalancer. When there is a publicly addressed upstream loadbalancer
> (Amazon ELB or just HAProxy itself) talking to RFC1918 addressed or
> non-routeable backend httpd servers, it becomes impossible to enable
> internal communication on the RFC1918 space to the backend instances.
> If the goal is to monitor or probe the site and (httpd proxy) backends
> internally for health, this *can* be done by duplicating the virtual
> hosts. Depending on the complexity of the virtual hosts, what resources
> those virtual hosts have (proxies and whatnot) and their general size,
> this could result in a fairly unmanageable httpd configuration having a
> vhost that requires PROXY header and a second one on a different port or
> IP that does not.
> It gets even more complicated when you are aiming to do management
> tasks. If you have balancers configured at the vhost as intended, you
> can only manage those balancers from the vhost they live in. Further,
> you may want to view server statistics, check info about the ldap cache,
> etc but permit access to those things only from a trusted network in
> addition to the user credentials protecting it.
> So for those examples, aside from creating an internal HAProxy to
> provide the header, it's not really possible to access the site
> internally when "On". With "Optional" it's possible to keep the same
> number of vhosts and enable the various management handlers and restrict
> them to the RFC1918 space (plus any other authnz mechanisms). The other
> obvious option would be to enable those handlers to the public Internet,
> but that's bad juju :-)
>
Well maybe what I am doing is wrong, but I just add a port to my vhosts,
so I have:
<VirtualHost $some_ip:80 $some_ip:83 >
and then I also have:
<VirtualHost *:83>
RemoteIPProxyProtocol On
</VirtualHost>
This way I can reach the same vhost with and without proxy protocol.
--
Sander
Re: mod_remoteip and mod_http2 combined
Posted by Daniel Ruggeri <DR...@primary.net>.
Thanks, all, for the patience as I finally got back to this.
On 2/24/2017 11:05 AM, Sander Hoentjen wrote:
>
> On 02/20/2017 07:48 PM, William A Rowe Jr wrote:
>> On Sat, Feb 18, 2017 at 4:25 PM, Daniel Ruggeri <dr...@apache.org> wrote:
>>> On 2017-02-15 09:07 (-0600), William A Rowe Jr <wr...@rowe-clan.net> wrote:
>>>> On Wed, Feb 15, 2017 at 9:02 AM, Sander Hoentjen <sa...@hoentjen.eu> wrote:
>>>>> mod_remote ip has:
>>>>> /* mod_proxy creates outgoing connections - we don't want those */
>>>>> if (!remoteip_is_server_port(c->local_addr->port)) {
>>>>> return DECLINED;
>>>>> }
>>>>> I am guessing something similar is needed for h2 connections?
>>>> I suspect that the mod_remoteip logic is wrong, that it should be guarding
>>>> against any subordinate connections and examining only explicitly configured
>>>> ports / origin IPs. the PROXY protocol is not part of the HTTP protocol and
>>>> incompatible with it, so the trust list logic isn't directly compatible (this is
>>>> clearly explained in the PROXY pseudo-RFC.)
>>>>
>>> Hi, Bill. That is what the module is doing. The original authors wrote it to have a list of virtual hosts it is explicitly enabled for and explicitly disabled for. I added a third list for optional vhosts. In the pre_connection hook, it checks to see if the connection's local_addr (which should normally be the server's IP) is explicitly configured to enable PROXY handling. It then checks to see if the local port is a server port.
>>>
>>> Looking at the logs shared, 192.168.122.249:84 is the server IP:Port combo and is also the local IP:Port from mod_h2. If h2 sets the master of this connection, then we could skip the whole ordeal with this patch:
>>>
>>> Index: modules/metadata/mod_remoteip.c
>>> ===================================================================
>>> --- modules/metadata/mod_remoteip.c (revision 1781701)
>>> +++ modules/metadata/mod_remoteip.c (working copy)
>>> @@ -862,6 +862,10 @@
>>> remoteip_conn_config_t *conn_conf;
>>> int optional;
>>>
>>> + if (c->master != NULL) {
>>> + return DECLINED;
>>> + }
>>> +
>>> conf = ap_get_module_config(ap_server_conf->module_config,
>>> &remoteip_module);
>>>
>>> .. but I don't know if that potentially means we are looking at the wrong connection.
> First I'll say that with the "Optional" mode it worked, just not with On
> I just tried this patch and as far as I have tested this seems to work
> fine in On mode, as well as in Optional. I do see some other issue, but
> that is probably in my own code, I'll try to track that down later.
This is good news and about what I was expecting to happen. I will add
this to the commit I've got coming that incorporates a lot of Ruediger's
feedback.
>> That should be close, but need to ensure c->master is initialized for
>> http as well
>> where there is no master/subordinate.
> I am not sure what this means, how should I test this?
Hi, Bill - also hoping for a bit more input. Since PROXY protocol is not
tied to any particular layer 7 protocol, I don't think we'd have to
verify it is initialized for HTTP - just that there is no master at all.
At least, that's my understanding so I appreciate any corrections.
>> If the 'optional' (unwise) feature were removed, the decision to
>> inject the filter before
>> the http or h2 filter would be trivial, it would step out of the way
>> after the first-pass
>> (and perhaps not need to live on the filter stack at all - if we do a
>> fixed read against
>> the core filter - we hopefully know the number of bytes affected early
>> and can then
>> do a second read to complete the v1 vs v2 read?) --- all before we are
>> in a multiple
>> pipeline state.
I don't think that is the case because the module was also written to
support PROXY header consumption in a name-based virtual host context,
too, where ServerName abc may require it but ServerName xyz does not.
Agreed, if it were simply all or nothing (either it's enabled on this
connection or not), we could avoid some of these gymnastics but I think
that use case as well as the Optional use case requires it.
>> If we move to a conn_rec oriented one-shot nothing happens during the request
>> phase at all.
>>
>> By looking at the protocol filter stack, we should be able to glean
>> whether we are
>> talking to the core filter, or an 'unexpected' non-network filter, right?
>>
> I myself have no use-case at the moment for the "Optional" mode, maybe
> others do.
Sure, to clarify, the Optional use case came from a member on one of our
cousin projects (Tomcat) Chris Schultz as well as my own use cases. It
is useful for internally accessing the site from behind the
loadbalancer. When there is a publicly addressed upstream loadbalancer
(Amazon ELB or just HAProxy itself) talking to RFC1918 addressed or
non-routeable backend httpd servers, it becomes impossible to enable
internal communication on the RFC1918 space to the backend instances.
If the goal is to monitor or probe the site and (httpd proxy) backends
internally for health, this *can* be done by duplicating the virtual
hosts. Depending on the complexity of the virtual hosts, what resources
those virtual hosts have (proxies and whatnot) and their general size,
this could result in a fairly unmanageable httpd configuration having a
vhost that requires PROXY header and a second one on a different port or
IP that does not.
It gets even more complicated when you are aiming to do management
tasks. If you have balancers configured at the vhost as intended, you
can only manage those balancers from the vhost they live in. Further,
you may want to view server statistics, check info about the ldap cache,
etc but permit access to those things only from a trusted network in
addition to the user credentials protecting it.
So for those examples, aside from creating an internal HAProxy to
provide the header, it's not really possible to access the site
internally when "On". With "Optional" it's possible to keep the same
number of vhosts and enable the various management handlers and restrict
them to the RFC1918 space (plus any other authnz mechanisms). The other
obvious option would be to enable those handlers to the public Internet,
but that's bad juju :-)
--
Daniel Ruggeri
Re: mod_remoteip and mod_http2 combined
Posted by Sander Hoentjen <sa...@hoentjen.eu>.
On 02/20/2017 07:48 PM, William A Rowe Jr wrote:
> On Sat, Feb 18, 2017 at 4:25 PM, Daniel Ruggeri <dr...@apache.org> wrote:
>> On 2017-02-15 09:07 (-0600), William A Rowe Jr <wr...@rowe-clan.net> wrote:
>>> On Wed, Feb 15, 2017 at 9:02 AM, Sander Hoentjen <sa...@hoentjen.eu> wrote:
>>>> mod_remote ip has:
>>>> /* mod_proxy creates outgoing connections - we don't want those */
>>>> if (!remoteip_is_server_port(c->local_addr->port)) {
>>>> return DECLINED;
>>>> }
>>>> I am guessing something similar is needed for h2 connections?
>>> I suspect that the mod_remoteip logic is wrong, that it should be guarding
>>> against any subordinate connections and examining only explicitly configured
>>> ports / origin IPs. the PROXY protocol is not part of the HTTP protocol and
>>> incompatible with it, so the trust list logic isn't directly compatible (this is
>>> clearly explained in the PROXY pseudo-RFC.)
>>>
>> Hi, Bill. That is what the module is doing. The original authors wrote it to have a list of virtual hosts it is explicitly enabled for and explicitly disabled for. I added a third list for optional vhosts. In the pre_connection hook, it checks to see if the connection's local_addr (which should normally be the server's IP) is explicitly configured to enable PROXY handling. It then checks to see if the local port is a server port.
>>
>> Looking at the logs shared, 192.168.122.249:84 is the server IP:Port combo and is also the local IP:Port from mod_h2. If h2 sets the master of this connection, then we could skip the whole ordeal with this patch:
>>
>> Index: modules/metadata/mod_remoteip.c
>> ===================================================================
>> --- modules/metadata/mod_remoteip.c (revision 1781701)
>> +++ modules/metadata/mod_remoteip.c (working copy)
>> @@ -862,6 +862,10 @@
>> remoteip_conn_config_t *conn_conf;
>> int optional;
>>
>> + if (c->master != NULL) {
>> + return DECLINED;
>> + }
>> +
>> conf = ap_get_module_config(ap_server_conf->module_config,
>> &remoteip_module);
>>
>> .. but I don't know if that potentially means we are looking at the wrong connection.
First I'll say that with the "Optional" mode it worked, just not with On
I just tried this patch and as far as I have tested this seems to work
fine in On mode, as well as in Optional. I do see some other issue, but
that is probably in my own code, I'll try to track that down later.
> That should be close, but need to ensure c->master is initialized for
> http as well
> where there is no master/subordinate.
I am not sure what this means, how should I test this?
>
> If the 'optional' (unwise) feature were removed, the decision to
> inject the filter before
> the http or h2 filter would be trivial, it would step out of the way
> after the first-pass
> (and perhaps not need to live on the filter stack at all - if we do a
> fixed read against
> the core filter - we hopefully know the number of bytes affected early
> and can then
> do a second read to complete the v1 vs v2 read?) --- all before we are
> in a multiple
> pipeline state.
>
> If we move to a conn_rec oriented one-shot nothing happens during the request
> phase at all.
>
> By looking at the protocol filter stack, we should be able to glean
> whether we are
> talking to the core filter, or an 'unexpected' non-network filter, right?
>
I myself have no use-case at the moment for the "Optional" mode, maybe
others do.
Re: mod_remoteip and mod_http2 combined
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Sat, Feb 18, 2017 at 4:25 PM, Daniel Ruggeri <dr...@apache.org> wrote:
> On 2017-02-15 09:07 (-0600), William A Rowe Jr <wr...@rowe-clan.net> wrote:
>> On Wed, Feb 15, 2017 at 9:02 AM, Sander Hoentjen <sa...@hoentjen.eu> wrote:
>> >
>> > mod_remote ip has:
>> > /* mod_proxy creates outgoing connections - we don't want those */
>> > if (!remoteip_is_server_port(c->local_addr->port)) {
>> > return DECLINED;
>> > }
>> > I am guessing something similar is needed for h2 connections?
>>
>> I suspect that the mod_remoteip logic is wrong, that it should be guarding
>> against any subordinate connections and examining only explicitly configured
>> ports / origin IPs. the PROXY protocol is not part of the HTTP protocol and
>> incompatible with it, so the trust list logic isn't directly compatible (this is
>> clearly explained in the PROXY pseudo-RFC.)
>>
>
> Hi, Bill. That is what the module is doing. The original authors wrote it to have a list of virtual hosts it is explicitly enabled for and explicitly disabled for. I added a third list for optional vhosts. In the pre_connection hook, it checks to see if the connection's local_addr (which should normally be the server's IP) is explicitly configured to enable PROXY handling. It then checks to see if the local port is a server port.
>
> Looking at the logs shared, 192.168.122.249:84 is the server IP:Port combo and is also the local IP:Port from mod_h2. If h2 sets the master of this connection, then we could skip the whole ordeal with this patch:
>
> Index: modules/metadata/mod_remoteip.c
> ===================================================================
> --- modules/metadata/mod_remoteip.c (revision 1781701)
> +++ modules/metadata/mod_remoteip.c (working copy)
> @@ -862,6 +862,10 @@
> remoteip_conn_config_t *conn_conf;
> int optional;
>
> + if (c->master != NULL) {
> + return DECLINED;
> + }
> +
> conf = ap_get_module_config(ap_server_conf->module_config,
> &remoteip_module);
>
> .. but I don't know if that potentially means we are looking at the wrong connection.
That should be close, but need to ensure c->master is initialized for
http as well
where there is no master/subordinate.
If the 'optional' (unwise) feature were removed, the decision to
inject the filter before
the http or h2 filter would be trivial, it would step out of the way
after the first-pass
(and perhaps not need to live on the filter stack at all - if we do a
fixed read against
the core filter - we hopefully know the number of bytes affected early
and can then
do a second read to complete the v1 vs v2 read?) --- all before we are
in a multiple
pipeline state.
If we move to a conn_rec oriented one-shot nothing happens during the request
phase at all.
By looking at the protocol filter stack, we should be able to glean
whether we are
talking to the core filter, or an 'unexpected' non-network filter, right?
Re: mod_remoteip and mod_http2 combined
Posted by Daniel Ruggeri <dr...@apache.org>.
On 2017-02-15 09:07 (-0600), William A Rowe Jr <wr...@rowe-clan.net> wrote:
> On Wed, Feb 15, 2017 at 9:02 AM, Sander Hoentjen <sa...@hoentjen.eu> wrote:
> >
> > mod_remote ip has:
> > /* mod_proxy creates outgoing connections - we don't want those */
> > if (!remoteip_is_server_port(c->local_addr->port)) {
> > return DECLINED;
> > }
> > I am guessing something similar is needed for h2 connections?
>
> I suspect that the mod_remoteip logic is wrong, that it should be guarding
> against any subordinate connections and examining only explicitly configured
> ports / origin IPs. the PROXY protocol is not part of the HTTP protocol and
> incompatible with it, so the trust list logic isn't directly compatible (this is
> clearly explained in the PROXY pseudo-RFC.)
>
Hi, Bill. That is what the module is doing. The original authors wrote it to have a list of virtual hosts it is explicitly enabled for and explicitly disabled for. I added a third list for optional vhosts. In the pre_connection hook, it checks to see if the connection's local_addr (which should normally be the server's IP) is explicitly configured to enable PROXY handling. It then checks to see if the local port is a server port.
Looking at the logs shared, 192.168.122.249:84 is the server IP:Port combo and is also the local IP:Port from mod_h2. If h2 sets the master of this connection, then we could skip the whole ordeal with this patch:
Index: modules/metadata/mod_remoteip.c
===================================================================
--- modules/metadata/mod_remoteip.c (revision 1781701)
+++ modules/metadata/mod_remoteip.c (working copy)
@@ -862,6 +862,10 @@
remoteip_conn_config_t *conn_conf;
int optional;
+ if (c->master != NULL) {
+ return DECLINED;
+ }
+
conf = ap_get_module_config(ap_server_conf->module_config,
&remoteip_module);
.. but I don't know if that potentially means we are looking at the wrong connection.
Sander, would it be possible to try this out?
Re: mod_remoteip and mod_http2 combined
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Wed, Feb 15, 2017 at 9:02 AM, Sander Hoentjen <sa...@hoentjen.eu> wrote:
>
> mod_remote ip has:
> /* mod_proxy creates outgoing connections - we don't want those */
> if (!remoteip_is_server_port(c->local_addr->port)) {
> return DECLINED;
> }
> I am guessing something similar is needed for h2 connections?
I suspect that the mod_remoteip logic is wrong, that it should be guarding
against any subordinate connections and examining only explicitly configured
ports / origin IPs. the PROXY protocol is not part of the HTTP protocol and
incompatible with it, so the trust list logic isn't directly compatible (this is
clearly explained in the PROXY pseudo-RFC.)
Re: mod_remoteip and mod_http2 combined
Posted by Sander Hoentjen <sa...@hoentjen.eu>.
On 02/15/2017 12:19 PM, Jordan Gigov wrote:
> On 15 February 2017 at 12:50, Sander Hoentjen <sa...@hoentjen.eu> wrote:
>> Hey guys,
>>
>> I am trying to use both mod_remoteip with ProxyProtocol and mod_http2.
>> It looks like mod_http2 gets handed the connection before mod_remoteip,
>> so things don't work as they should. ProxyProtocol should always be
>> handled first, since it is prepended to the stream. Any pointers to
>> where in the code I can look to change things around?
>>
>> --
>> Sander
> Try modules/http2/h2_h2.c line 550 add "mod_remoteip.c" after "mod_ssl.c".
>
> This reminds me since remoteip is being updated, maybe it should also
> specify some before and after mods to avoid.
It seems I was wrong, the issue is that for some reason
remoteip_hook_pre_connection is called twice for a h2 connection:
[Wed Feb 15 14:32:08.048968 2017] [remoteip:debug] [pid 8521]
mod_remoteip.c(1015): [client 192.168.122.1:35136] AH03503:
RemoteIPProxyProtocol: enabled on connection to 192.168.122.249:84
[Wed Feb 15 14:32:08.049082 2017] [remoteip:debug] [pid 8521]
mod_remoteip.c(1406): [client 192.168.122.1:35136] AH03511:
RemoteIPProxyProtocol: received valid PROXY header: 192.168.122.1:35136
[Wed Feb 15 14:32:08.049554 2017] [http2:debug] [pid 8521]
h2_session.c(994): [client 192.168.122.1:35136] AH03200: h2_session(4)
created, max_streams=100, stream_mem=32768, workers_limit=6,
workers_max=1, push_diary(type=1,N=256)
[Wed Feb 15 14:32:08.049579 2017] [http2:debug] [pid 8521]
h2_session.c(1088): [client 192.168.122.1:35136] AH03201: h2_session(4):
start, INITIAL_WINDOW_SIZE=65535, MAX_CONCURRENT_STREAMS=100
[Wed Feb 15 14:32:08.049589 2017] [http2:debug] [pid 8521]
h2_session.c(2067): [client 192.168.122.1:35136] AH03079: h2_session(4):
started on localhost.localdomain:84
[Wed Feb 15 14:32:08.049596 2017] [http2:debug] [pid 8521]
h2_session.c(1721): [client 192.168.122.1:35136] AH03078: h2_session(4):
transit [INIT] -- init --> [BUSY]
[Wed Feb 15 14:32:08.049616 2017] [http2:debug] [pid 8521]
h2_session.c(441): [client 192.168.122.1:35136] AH03066: h2_session(4):
recv FRAME[SETTINGS[length=0, stream=0]], frames=0/0 (r/s)
[Wed Feb 15 14:32:08.049628 2017] [http2:debug] [pid 8521]
h2_stream.c(189): [client 192.168.122.1:35136] AH03082: h2_stream(4-1):
opened
[Wed Feb 15 14:32:08.049655 2017] [http2:debug] [pid 8521]
h2_session.c(441): [client 192.168.122.1:35136] AH03066: h2_session(4):
recv FRAME[HEADERS[length=31, hend=1, stream=1, eos=1]], frames=1/0 (r/s)
[Wed Feb 15 14:32:08.049706 2017] [http2:debug] [pid 8521]
h2_session.c(677): [client 192.168.122.1:35136] AH03068: h2_session(4):
sent FRAME[SETTINGS[length=6, stream=0]], frames=2/0 (r/s)
[Wed Feb 15 14:32:08.049768 2017] [remoteip:debug] [pid 8521]
mod_remoteip.c(1015): [client 192.168.122.1:35136] AH03503:
RemoteIPProxyProtocol: enabled on connection to 192.168.122.249:84
[Wed Feb 15 14:32:08.049810 2017] [remoteip:warn] [pid 8521] [client
192.168.122.1:35136] AH03496: RemoteIPProxyProtocol data is missing, but
required! Aborting request.
[Wed Feb 15 14:32:08.049840 2017] [http2:debug] [pid 8521]
h2_request.c(271): [client 192.168.122.1:35136] AH03367: h2_request:
access_status=400, request_create failed
mod_remote ip has:
/* mod_proxy creates outgoing connections - we don't want those */
if (!remoteip_is_server_port(c->local_addr->port)) {
return DECLINED;
}
I am guessing something similar is needed for h2 connections?
--
Sander
Re: mod_remoteip and mod_http2 combined
Posted by Jordan Gigov <co...@gmail.com>.
Try modules/http2/h2_h2.c line 550 add "mod_remoteip.c" after "mod_ssl.c".
This reminds me since remoteip is being updated, maybe it should also
specify some before and after mods to avoid.
On 15 February 2017 at 12:50, Sander Hoentjen <sa...@hoentjen.eu> wrote:
> Hey guys,
>
> I am trying to use both mod_remoteip with ProxyProtocol and mod_http2.
> It looks like mod_http2 gets handed the connection before mod_remoteip,
> so things don't work as they should. ProxyProtocol should always be
> handled first, since it is prepended to the stream. Any pointers to
> where in the code I can look to change things around?
>
> --
> Sander