You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by "Mikhail T." <mi...@aldan.algebra.com> on 2013/08/02 03:55:52 UTC
Resolved (sort of): Struggling with AuthMerging
30.07.2013 19:18, I wrote:
>
> authorization result of Require all granted: granted
> authorization result of <RequireAny>: granted
> authorization result of AuthMerging Any: granted
> authorization result of Require all granted: granted
> authorization result of <RequireAny>: granted
> authorization result of AuthMerging Any: granted
> authorization result of Require tiv ipaddress: denied (no
> authenticated user yet)
> authorization result of Require tiv expiration: denied (no
> authenticated user yet)
> authorization result of <RequireAll>: denied (no authenticated
> user yet)
> authorization result of <RequireAny>: denied (no authenticated
> user yet)
>
After instrumenting mod_authz_core.c to provide a little more
information about the actual request_rec being processed, I got the
following output, which explains, what's happening.
The problem -- and I do think, it is a bug -- is that the entire
authorization is invoked not just for the request, but for the "internal
subrequests" too. So, in my scenario, when I asked for /tiv/, the authz
core made the following checks (color-coded to match the above-quoted
log-entries):
1. Check location /tiv/ -- granted, no problem.
2. Check location /tiv/index.php -- granted as well, so far so good.
3. We use mod_actions to hand-off processing of php-files to php-fpm,
so Apache also checks location */php-fpm/*tiv/index.php...
Because there is no explicit sublocation defined for /php-fpm/, the
rules for the Location / are invoked. Which leads to our tiv being
queried -- denying the request...
Items 1 and 2 explain, why I saw the "Require all granted" rule
processed twice in the log. Item 3 explain, why the rules of the top
Location got invoked at all...
I got things to work by changing the sub-location configuration to read
simply:
<LocationMatch ^(/php-fpm)?/tiv/>
Require all granted
DirectoryIndex index.php
</LocationMatch>
There is no need for AuthMerging even, after all. But I struggle to
understand, why the same HTTP-hit results in multiple processing of the
same authorization rules (some of which may, actually, be heavy...). Is
this really normal and expected behavior?
01.08.2013 21:05, Ben Reser wrote:
> If the resulting response is AUTHZ_DENIED_NO_USER then processing continues.
Is that so that if any of the subsequent children of the same RequireAll
say AUTHZ_DENIED, the server will not even bother figuring out the user?
Ok, makes sense, thank you. Turns out, this is not related to the main
problem, after all.
-mi
Re: Resolved (sort of): Struggling with AuthMerging
Posted by Stefan Fritsch <sf...@sfritsch.de>.
Am Freitag, 2. August 2013, 23:05:09 schrieb Ben Reser:
> If all of your authz/authn providers are using the CONF flag and
> you're getting duplicated authz processing for subrequests that have
> the same conf applied to them then it's possible there's a bug
> here. I haven't ever specifically looked at this code so I can't
> tell you how it works, all I can do is tell you what the
> documentation says it does. Unfortunately I have other priorities
> right now so I don't really have the time to go down that rabbit
> hole and find out if there is a bug that's causing extra authz
> calls.
>
> Any attempt to optimize when authn/authz happens on subrequests is
> going to have to be done very carefully since you really can't make
> any assumptions about what should be allowed. I realize that in
> your specific case that seems absurd but the httpd core code
> supports use cases that are very different from yours.
The AP_AUTH_INTERNAL_PER_CONF / AP_AUTH_INTERNAL_PER_URI switch has a
design flaw in that modules cannot decide which flag to use
dynamically, depending on the modules' configuration. Instead, if any
module using AP_AUTH_INTERNAL_PER_URI is loaded, httpd will call the
auth hooks for every URI, even if the respective module is not enabled
by configuration. It would be much better if that flag was a runtime
setting that could be switched on only if needed, on a per vhost
basis.
AFAICS, the only module shipped with httpd that uses
AP_AUTH_INTERNAL_PER_URI is mod_log_debug, where I have assumed that
the ability to inject trace messages for every subrequest is more
important than performance. Not sure if that was a good idea.
Do you have mod_log_debug loaded? If yes, try if things change if you
don't load it.
Re: Resolved (sort of): Struggling with AuthMerging
Posted by Ben Reser <be...@reser.org>.
On Sat, Aug 3, 2013 at 11:34 AM, Mikhail T. <mi...@aldan.algebra.com> wrote:
> Point is, it is erring. I asked Ben for possible use-cases and his two
> examples were modules, which use the authorization rules to generate
> different content depending on the result. Rather than to decide, whether to
> authorize the request at all.
I think you're jumping to massive conclusions about my ability to
answer questions. While I've been an httpd user for a relatively long
time I've only had httpd commit access for all of a week. I wasn't
involved in writing the auth code that exists and I may not have
provided you the best examples. Just because I haven't provided
better examples doesn't mean they exist. I gave you the examples that
came to mind.
First and foremost the primary concern of security code is always
correctly implementing the security model, not running as fast as
possible. If you can make it run faster without breaking the model,
then I'm sure your patch would be accepted.
For what it's worth I can't replicate your issue. I tried replicating
just the duplicate authz check against the directory and then against
the DirectoryIndex.
I end up getting exactly one authz process:
[Sat Aug 03 11:48:43.170081 2013] [authz_core:debug] [pid 48038:tid
4316540928] mod_authz_core.c(845): [client 127.0.0.1:63874] AH01626:
authorization result of Require all granted: granted
[Sat Aug 03 11:48:43.170096 2013] [authz_core:debug] [pid 48038:tid
4316540928] mod_authz_core.c(845): [client 127.0.0.1:63874] AH01626:
authorization result of <RequireAny>: granted
[Sat Aug 03 11:48:43.170102 2013] [core:trace3] [pid 48038:tid
4316540928] request.c(238): [client 127.0.0.1:63874] request
authorized without authentication by access_checker_ex hook: /
Can you produce a minimal example configuration that reproduces the
issue. Let's avoid the PHP scripts for now.
> The situation would've made some sense, if I could configure things
> separately. For example:
>
> # Lock-out attempts to invoke php-fpm directly:
> <Location /php-fpm>
> Require none granted
> </Location>
>
> # Allow any PHP script under DocumentRoot to be executed otherwise:
> <LocationMatch \.php$>
> Require all granted
> </LocationMatch>
>
> But I can't -- all requests for foo.php would go through both of the
> above...
Have you actually tried this? Because it does work. The only problem
you'll run into is that the main request for the /php-fpm/ URI will
fail before the DirectoryIndex applies. For instance I have something
that looks like this:
<Directory />
Require all denied
</Directory>
<Directory "/Users/breser/httpd-2.4.6-root/htdocs">
DirectoryIndex index.html
Require all granted
</Directory>
<Location />
Require all denied
</Location>
<LocationMatch \.html$>
Require all granted
</LocationMatch>
I can hit /index.html but not index.txt. I can't get / and the
directory index doesn't work, but I can easily fix that by doing
changing my LocationMatch to:
(^/$)|(\.html$)
If I change DirectoryIndex to index.txt it'll break because I've
disallowed that. Note that you've actually just shown an example of
why sub-requests should check security.
Side note: It'd be Require all denied to block all access.
Re: Resolved (sort of): Struggling with AuthMerging
Posted by "Mikhail T." <mi...@aldan.algebra.com>.
03.08.2013 15:19, Eric Covener ???????(??):
> I didn't interpret his response that way. Those are modules that will
> create subrequests/internal redirects to new URIs that could have
> separate authz applied to them from the original URI -- you can't
> assume the server is any less interested in performing authz on them.
Ben's examples -- given in
CADkdwvRme0QObKdQVCjF+_h7St+CG8zDHhpnLXjup2V=KpQazQ@mail.gmail.com
-- were mod_autoindex and mod_dav_svn. Both -- as far as I understood --
used additional authz-checks when generating the /body/ of the response.
Not to decide, whether to authorize the request itself, but to decide,
what exactly to send back after the authorization already succeeded.
> The server can't tell the difference between that and
> your mod_actions internal redirect to a new URI -- they need to be
> checked.
Then, perhaps, there should be a way for me to tell the server, that
such a decision can be made for a particular Location (or Directory, or
vhost).
Yours,
-mi
Re: Resolved (sort of): Struggling with AuthMerging
Posted by Eric Covener <co...@gmail.com>.
On Sat, Aug 3, 2013 at 2:34 PM, Mikhail T. <mi...@aldan.algebra.com> wrote:
> 03.08.2013 14:14, Eric Covener wrote:
>
> I don't agree re: necessity. As Ben said, httpd only knows that /tiv
> (where you tried to punch a hole) and the target of your Action
> directive have different per-directory configurations, so
> authorization is checked on the subrequest. It's erring on the side
> of running authz checks, and I don't disagree that it could be
> enhanced/optimized.
>
> Point is, it is erring. I asked Ben for possible use-cases and his two
> examples were modules, which use the authorization rules to generate
> different content depending on the result. Rather than to decide, whether to
> authorize the request at all.
I didn't interpret his response that way. Those are modules that will
create subrequests/internal redirects to new URIs that could have
separate authz applied to them from the original URI -- you can't
assume the server is any less interested in performing authz on them.
Consider something as basic as (per-directory) mod_rewrite or
mod_include. The server can't tell the difference between that and
your mod_actions internal redirect to a new URI -- they need to be
checked.
Re: Resolved (sort of): Struggling with AuthMerging
Posted by "Mikhail T." <mi...@aldan.algebra.com>.
03.08.2013 14:14, Eric Covener wrote:
> I don't agree re: necessity. As Ben said, httpd only knows that /tiv
> (where you tried to punch a hole) and the target of your Action
> directive have different per-directory configurations, so
> authorization is checked on the subrequest. It's erring on the side
> of running authz checks, and I don't disagree that it could be
> enhanced/optimized.
Point is, it is /erring/. I asked Ben for possible use-cases and his two
examples were modules, which use the authorization rules to generate
different content depending on the result. Rather than to decide,
whether to authorize the request at all.
The situation would've made some sense, if I could configure things
separately. For example:
# Lock-out attempts to invoke php-fpm directly:
<Location /php-fpm>
Require none granted
</Location>
# Allow any PHP script under DocumentRoot to be executed otherwise:
<LocationMatch \.php$>
Require all granted
</LocationMatch>
But I can't -- all requests for foo.php would go through /both/ of the
above...
-mi
Re: Resolved (sort of): Struggling with AuthMerging
Posted by Eric Covener <co...@gmail.com>.
On Sat, Aug 3, 2013 at 1:41 PM, Mikhail T. <mi...@aldan.algebra.com> wrote:
> 03.08.2013 02:05, Ben Reser wrote:
>
> You don't seriously expect the auth system to know all of those intricacies?
>
> Let me take a step back here. What I found about my particular situation is
> -- using your own term -- absurd:
>
> The current behavior is not documented.
> The current behavior is not even known: neither you, nor anybody else on
> this list of httpd developers (!) were able to recognize it, when I first
> asked, nor explain, what's happening. Asking elsewhere proved fruitless too.
> The current behavior suffers from an obvious performance penalty -- wasting
> CPU-cycles rerunning authz rules multiple times on each hit.
>
> You agree with that -- the only mitigation offered was: a) it used to be
> even worse in 2.2; b) fixing it "would have to be done very carefully".
>
> Need anything more be said or written for a consensus, that things do need
> fixing? And, as per point 3. above, not only on the documentation side... At
> the very least, I'd say, there should be a way to turn it off per subconfig
> (Location, Directory, or vhost). I don't know, how to do it. But it seems
> rather obvious, that it needs to be done.
I don't agree re: necessity. As Ben said, httpd only knows that /tiv
(where you tried to punch a hole) and the target of your Action
directive have different per-directory configurations, so
authorization is checked on the subrequest. It's erring on the side
of running authz checks, and I don't disagree that it could be
enhanced/optimized.
Re: Resolved (sort of): Struggling with AuthMerging
Posted by "Mikhail T." <mi...@aldan.algebra.com>.
03.08.2013 02:05, Ben Reser wrote:
> You don't seriously expect the auth system to know all of those intricacies?
Let me take a step back here. What I found about my particular situation
is -- using your own term -- absurd:
1. The current behavior is not documented.
2. The current behavior is not even known: neither you, nor anybody
else on this list of httpd developers (!) were able to recognize it,
when I first asked, nor explain, what's happening. Asking elsewhere
<http://stackoverflow.com/questions/17974096/apache-2-4-how-to-close-entire-site-except-one-subdirectory>
proved fruitless too.
3. The current behavior suffers from an obvious performance penalty --
wasting CPU-cycles rerunning authz rules multiple times on each hit.
You agree with that -- the only mitigation offered was: a) it used to be
even worse in 2.2; b) fixing it "would have to be done very carefully".
Need anything more be said or written for a consensus, that things do
need fixing? And, as per point 3. above, not only on the documentation
side... At the very least, I'd say, there should be a way to turn it off
per subconfig (Location, Directory, or vhost). I don't know, how to do
it. But it seems rather obvious, that it needs to be done.
03.08.2013 13:15, Stefan Fritsch wrote:
> Do you have mod_log_debug loaded? If yes, try if things change if you don't load it.
No, there is no mod_log_debug loaded here. And my own -- mod_authnz_tiv
-- does not use the define either:
ap_register_auth_provider(p, AUTHN_PROVIDER_GROUP, "tiv",
AUTHN_PROVIDER_VERSION, &provider,
AP_AUTH_INTERNAL_PER_CONF);
Thanks. Yours,
-mi
Re: Resolved (sort of): Struggling with AuthMerging
Posted by Ben Reser <be...@reser.org>.
On Fri, Aug 2, 2013 at 8:24 PM, Mikhail T. <mi...@aldan.algebra.com> wrote:
> The modules in your examples deliberately use the authz mechanism to
> generate different output based on the results. But what is doing it in the
> case I describe -- where the generated content is exactly the same?
Obviously nothing is doing a subrequest to validate the auth, they're
doing sub-requests to fulfill the original request. httpd is doing
the auth because it's the right thing to do from a security
perspective. The generated content is not the same. The main request
has no generated content it's essentially a redirect, it just happens
to be handled internally.
> Why would anyone want to provide distinct authorization rules to /foo/ and
> /foo/index.html, for example? What is a possible use-case?
They are different paths they may have different auth rules. httpd
can't assume that it'll be okay. Keep reading and I'll explain why.
> Likewise, why would anyone configure mod_fastcgi to hand off processing of
> PHP-files to FPM:
>
> FastCGIExternalServer /usr/sbin/php-fpm -socket /var/run/fpm/main.sock
> AddHandler php-fastcgi .php
> Action php-fastcgi /php-fpm
> ScriptAlias /php-fpm /usr/sbin/php-fpm
>
> but need the /php-fpm/foo/ to have authorization rules different from those
> of /foo/?
You don't seriously expect the auth system to know all of those
intricacies? It has no idea why the subrequest is being made it just
knows there's a request to process. Even if it did know why the
subrequest is being made the variety of authz systems doesn't allow
you to make assumptions like the above.
In 2.4.x there's the new provider architecture that at least lets you
optimize away subrequests that have the same conf. But the modules
have to say it's ok because if the modules are handling path based
matching on their own then even httpd has no idea if /foo and
/foo/index.html have the same configuration. mod_authz_svn is an
example of such a module that could not use the CONF flag (but at this
time it isn't using the new provider system and is still using the old
hooks system). given that there are authz modules like this you can't
even make the argument that we can ever assume that authz isn't
necessary
If all of your authz/authn providers are using the CONF flag and
you're getting duplicated authz processing for subrequests that have
the same conf applied to them then it's possible there's a bug here.
I haven't ever specifically looked at this code so I can't tell you
how it works, all I can do is tell you what the documentation says it
does. Unfortunately I have other priorities right now so I don't
really have the time to go down that rabbit hole and find out if there
is a bug that's causing extra authz calls.
Any attempt to optimize when authn/authz happens on subrequests is
going to have to be done very carefully since you really can't make
any assumptions about what should be allowed. I realize that in your
specific case that seems absurd but the httpd core code supports use
cases that are very different from yours.
Re: Resolved (sort of): Struggling with AuthMerging
Posted by "Mikhail T." <mi...@aldan.algebra.com>.
02.08.2013 20:17, Ben Reser написав(ла):
> mod_autoindex automatically provides a directory listing of files
> under a path. However, by default it doesn't display any paths that
> you don't have access to, e.g. .htaccess. It does this by issuing
> subrequests for those other paths so that authz can run on them.
> (This behavior could be changed with IndexOptions ShowForbidden).
>
> mod_dav_svn. Numerous commands in SVN impact other paths than the URI
> (e.g. `svn list` which is similar to the autoindex case above,
> commiting a copy or move which touch two paths one of which is on in
> the URI but rather in the headers).
The modules in your examples /deliberately/ use the authz mechanism to
generate different output based on the results. But what is doing it in
the case I describe -- where the generated content is exactly the same?
Why would anyone want to provide distinct authorization rules to /foo/
and /foo/index.html, for example? What is a possible use-case?
Likewise, why would anyone configure mod_fastcgi to hand off processing
of PHP-files to FPM:
FastCGIExternalServer /usr/sbin/php-fpm -socket /var/run/fpm/main.sock
AddHandler php-fastcgi .php
Action php-fastcgi /php-fpm
ScriptAlias /php-fpm /usr/sbin/php-fpm
but need the /php-fpm/foo/ to have authorization rules different from
those of /foo/?
-mi
Re: Resolved (sort of): Struggling with AuthMerging
Posted by Ben Reser <be...@reser.org>.
On Thu, Aug 1, 2013 at 7:54 PM, Mikhail T. <mi...@aldan.algebra.com> wrote:
> 01.08.2013 22:47, Ben Reser написав(ла):
>> That's not a bug at all. In some cases it may be necessary for
>> authorization to run for sub-requests.
>
> Could you give an example or two? Thanks,
Sure.
mod_autoindex automatically provides a directory listing of files
under a path. However, by default it doesn't display any paths that
you don't have access to, e.g. .htaccess. It does this by issuing
subrequests for those other paths so that authz can run on them.
(This behavior could be changed with IndexOptions ShowForbidden).
mod_dav_svn. Numerous commands in SVN impact other paths than the URI
(e.g. `svn list` which is similar to the autoindex case above,
commiting a copy or move which touch two paths one of which is on in
the URI but rather in the headers). mod_dav_svn issues sub requests
for these other paths to find out if the actions are allowed on them.
Because of the performance impact of this mod_dav_svn has the
SVNPathAuthz directive to change this behavior. One of the possible
values is short_circuit which causes mod_dav_svn to talk directly to
mod_authz_svn rather than doing a subrequest. You might ask why don't
we just do this by default? The reason being is if there are other
authorization limits placed from other modules then they would no
longer respected.
Re: Resolved (sort of): Struggling with AuthMerging
Posted by "Mikhail T." <mi...@aldan.algebra.com>.
01.08.2013 22:47, Ben Reser ???????(??):
>> > <LocationMatch ^(/php-fpm)?/tiv/>
>> > Require all granted
>> > DirectoryIndex index.php
>> > </LocationMatch>
> I'm guessing you're using AP_AUTH_INTERNAL_PER_CONF which should avoid
> the 3rd call with the above configuration.
Even without my module at all -- using purely Apache's own -- the above
configuration results in a browser's request for /tiv/ being checked
three times now:
1. /tiv/
2. /tiv/index.php
3. /php-fpm/tiv/index.php
This does not seem right... Even if it may be necessary in some cases,
there's got to be a way to turn the behavior off...
> That's not a bug at all. In some cases it may be necessary for
> authorization to run for sub-requests.
Could you give an example or two? Thanks,
-mi
Re: Resolved (sort of): Struggling with AuthMerging
Posted by Ben Reser <be...@reser.org>.
On Thu, Aug 1, 2013 at 6:55 PM, Mikhail T. <mi...@aldan.algebra.com> wrote:
> The problem -- and I do think, it is a bug -- is that the entire
> authorization is invoked not just for the request, but for the "internal
> subrequests" too. So, in my scenario, when I asked for /tiv/, the authz core
> made the following checks (color-coded to match the above-quoted
> log-entries):
>
> Check location /tiv/ -- granted, no problem.
> Check location /tiv/index.php -- granted as well, so far so good.
> We use mod_actions to hand-off processing of php-files to php-fpm, so Apache
> also checks location /php-fpm/tiv/index.php...
> Because there is no explicit sublocation defined for /php-fpm/, the rules
> for the Location / are invoked. Which leads to our tiv being queried --
> denying the request...
That's not a bug at all. In some cases it may be necessary for
authorization to run for sub-requests.
If you're using the ap_register_auth_provider() system then you have
some control over that by using AP_AUTH_INTERNAL_PER_CONF.
> I got things to work by changing the sub-location configuration to read
> simply:
>
> <LocationMatch ^(/php-fpm)?/tiv/>
> Require all granted
> DirectoryIndex index.php
> </LocationMatch>
I'm guessing you're using AP_AUTH_INTERNAL_PER_CONF which should avoid
the 3rd call with the above configuration.
> There is no need for AuthMerging even, after all. But I struggle to
> understand, why the same HTTP-hit results in multiple processing of the same
> authorization rules (some of which may, actually, be heavy...). Is this
> really normal and expected behavior?
Yes. In fact there are cases where subrequests are explicitly used to
find out if a path is authorized. e.g. the -F and -U tests for
RewriteCond:
http://httpd.apache.org/docs/current/mod/mod_rewrite.html#rewritecond
In fact it used to be worse in 2.2.x because AP_AUTH_INTERNAL_PER_URI
was the old behavior.
See this following text from
http://httpd.apache.org/docs/2.4/developer/new_api_2_4.html
[[[
When possible, registering all access control hooks (including
authentication and authorization hooks) using
AP_AUTH_INTERNAL_PER_CONF is recommended. If all modules' access
control hooks are registered with this flag, then whenever the server
handles an internal sub-request that matches the same set of access
control configuration directives as the initial request (which is the
common case), it can avoid invoking the access control hooks another
time.
If your module requires the old behavior and must perform access
control checks on every sub-request with a different URI from the
initial request, even if that URI matches the same set of access
control configuration directives, then use AP_AUTH_INTERNAL_PER_URI.
]]]
> 01.08.2013 21:05, Ben Reser wrote:
> If the resulting response is AUTHZ_DENIED_NO_USER then processing continues.
>
>
> Is that so that if any of the subsequent children of the same RequireAll say
> AUTHZ_DENIED, the server will not even bother figuring out the user? Ok,
> makes sense, thank you. Turns out, this is not related to the main problem,
> after all.
Right that is how it would behave.