Not-a-bug: .asis handler isn't driven

["William A. Rowe, Jr." <wr...@covalent.net>]

It isn't a problem, 1.3 mime/negotation is subtely broken, so of course
this works in 1.3.

OK ... let's start from page one.

Two files in a directory:

index.html.en    1590 bytes
index.html.fr    1632 bytes

if the user's Accept-Language header is simply en, and they request
index.html, they will get the .en variant.  Accept-Language of fr
will return the french variant.  Accept-Language of anything else
returns Not Acceptable, and no Accept-Language will return the
index.html.en variant if the LanguagePriority is set with en and fr
both present and occuring in that order.

Now, Admin copies index.html.fr to index.html.temp.  Starts editing,
deletes two paragraphs...

index.html.temp  1039 bytes

User requests index.html with AcceptLanguage en or fr.  They aught
to get the right file.  User has another Accept-Language.  They MAY
get index.html.temp, because the others failed, and .temp inserts no
no language identification to reject.  They provide no Accept-Language,
and they _will_ get index.html.temp, based on size alone.

The sequence of files in the directory becomes significant.  The last
file in the readdir will either win or kill multiviews parsing if it's 
an unrecognized extension.  Filesize becomes significant, absurdly so.
The old behavior is bogus, Users who received bogus autoindex listings 
often had an unrecognized file at the _end_ of the readdir, mostly 
because they updated their index.html.** entries for the docroot from 
a new version of Apache, but they didn't update the list of languages 
in httpd.conf.

I've identified the patches to apply, and they should resolve this bug 
in 1.3 as well.  Let's get back to 'usual' behavior.  We also accepted
the handler in 1.3 multiviews and carried this into 2.0.  I noticed 
this when I added the Add{Input|Output}Filter processing.

The asis-handler .asis doesn't contribute to the negotiation.  If the 
user doesn't ask for .asis, it shouldn't be served.  This would become
much more pronounced when the user starts associating Filters with
file extensions.

Asking multiviews to select by handler (or to serve unrecognized 
extensions) is as bogus as similar IIS behaviors.  It isn't predictable.
Why would you expect .asis to be 'magically' served when the user doesn't
ask for the extension, while you expect .bak to 'magically' not be
served.  

If it's an element of the Accept-* negotiation, it needs to be
multiview-matched.  If it isn't, it has no business in there.

If you disagree with the above sentence, let's narrow the argument to
that statement.

Bill

Re: Apache 1.3 :: Not-a-bug: .asis handler

["William A. Rowe, Jr." <wr...@covalent.net>]

From: "Justin Erenkrantz" <je...@ebuilt.com>
Sent: Tuesday, October 02, 2001 1:04 PM


> On Tue, Oct 02, 2001 at 12:07:46PM -0500, William A. Rowe, Jr. wrote:
> > It's bitten several folks.  It's been written up on bugtraq as an outstanding
> > case.  There are a dozen variations of cause and effect.  But whatever, you are 
> > the RM, and nobody can veto a release.
> 
> Pardon my ignorance, but why can't someone veto a release if they feel
> it is the wrong time to do so?  -- justin

http://dev.apache.org/guidelines.html


[-1 Votes]
No. On issues where consensus is required, this vote counts as a veto. 
An action item requiring consensus approval must receive at least 
3 binding +1 votes and no vetos. An action item requiring majority 
approval must receive at least 3 binding +1 votes and more +1 votes 
than -1 votes (i.e., a majority with a minimum quorum of three positive 
votes). All other action items are considered to have lazy approval until 
someone votes -1, after which point they are decided by either consensus 
or a majority vote, depending upon the type of action item. 

Release Testing
Majority approval is required before the tarball can be publically released. 

So releasing a version of Apache is by Majority, not by Consensus.

And it is designed that way --- no one individual can stand in the way of
moving forward with changes.  The only reason my changes thus far are in
the 1.3.21 plan is that they have been on the table for some time, and the 
RM concurred with including them (even by lazy concensus, nobody objected
when they were introduced into STATUS.)

Re: Apache 1.3 :: Not-a-bug: .asis handler

[Ben Hyde <bh...@pobox.com>]

Justin Erenkrantz wrote:
 > Pardon my ignorance, but why can't someone veto a release if they feel
 > it is the wrong time to do so?  -- justin

We have a mess of rules/traditions that act to prevent
obstructionism and push people toward constructive
argument and constructive actions.  Releasing is always
a matter of tearing the damn thing out of the hands of
the evil demon "one more really really important thing".

 - ben

Re: Apache 1.3 :: Not-a-bug: .asis handler

[Justin Erenkrantz <je...@ebuilt.com>]

On Tue, Oct 02, 2001 at 12:07:46PM -0500, William A. Rowe, Jr. wrote:
> It's bitten several folks.  It's been written up on bugtraq as an outstanding
> case.  There are a dozen variations of cause and effect.  But whatever, you are 
> the RM, and nobody can veto a release.

Pardon my ignorance, but why can't someone veto a release if they feel
it is the wrong time to do so?  -- justin

Re: Apache 1.3 :: Not-a-bug: .asis handler

["William A. Rowe, Jr." <wr...@covalent.net>]

From: "Bill Stoddard" <bi...@wstoddard.com>
Sent: Monday, October 01, 2001 3:03 PM


> This is looking more and more like config funkiness as opposed to a serious 
> code defect.  Unless I see something to change my mind, I am not inclined to 
> attempt to fix this in the 1.3 code base.

Whatever.  I advertised the fix, I couldn't care less if anyone backports it.

It's bitten several folks.  It's been written up on bugtraq as an outstanding
case.  There are a dozen variations of cause and effect.  But whatever, you are 
the RM, and nobody can veto a release.

The problem is that each of the modules, mime, negotation, and autoindex, as
well as the default handler have all contributed to this thorny problem.
Yes - you can call it config error.  But I'd argue that not being able to
see the whole picture doesn't make this a config error; our inability to
unlock the 'secrets' of our modules interaction means that the admin has no
hope of ever understanding this.

We advertised mod_negotation as a real module, not 'experimental'.  If our
own modules interact impolitely, we should re-relegate it to experimental.

BTW - I agree with keeping the 'found' match logic for matching AddHandler
extensions in 1.3 - to retain backwards brokenness and to meet the "prinicpal 
of least surprize" test.  That's not my opinion on 2.0, but that's another
thread.  So "index.html.asis" should continue to match "index.html" for 1.3.

Bill

Re: Not-a-bug: .asis handler isn't driven

[Bill Stoddard <bi...@wstoddard.com>]

> Still working on recreating this...
>
> > > The sub_req for index.html.temp is destroyed because
> > > sub_req->content_type == NULL, which looks correct to me.
> >
> > Try index.html.en.temp - which may have been what I was thinking of.
>
> .temp is the last extension inspected by mod_mime. If temp is not in any of the tables
> (content-type, charset, content-language, content-encoding) and is not a special
handler,
> the sub_req->content_type is set to NULL which excludes the file from being negotiated.
>
> > Assure that it becomes the last file in the directory listing.
> >
> > The specific bug folks observed was that .lang.charset would be served,
> > even though charset wasn't negotiated.
>
> Ditto. If charset is the last extension and it is not configured, the file will not be
> considered for negotiation.
>

index.html.yadda.Big5 will not be considered for negotiation because, as you have pointed
out, the content type will be lost (reset to NULL) by mod_mime during the processing of
the unknown 'yadda' extension. A NULL content_type prevents the file from being added to
the list of negotiable files. There is one exception...

AFAIK, the only way this file can be added to the list of negotiable files (for use by
mod_negotiate) is if the directory has a ForceType in effect. ForceType will cause
mod_mime to force set the content_type to the configured value.

This is looking more and more like config funkiness as opposed to a serious code defect.
Unless I see something to change my mind, I am not inclined to attempt to fix this in the
1.3 code base.

Bill

Re: Not-a-bug: .asis handler isn't driven

[Bill Stoddard <bi...@wstoddard.com>]

Still working on recreating this...

> > The sub_req for index.html.temp is destroyed because
> > sub_req->content_type == NULL, which looks correct to me.
>
> Try index.html.en.temp - which may have been what I was thinking of.

.temp is the last extension inspected by mod_mime. If temp is not in any of the tables
(content-type, charset, content-language, content-encoding) and is not a special handler,
the sub_req->content_type is set to NULL which excludes the file from being negotiated.

> Assure that it becomes the last file in the directory listing.
>
> The specific bug folks observed was that .lang.charset would be served,
> even though charset wasn't negotiated.

Ditto. If charset is the last extension and it is not configured, the file will not be
considered for negotiation.

>
> How isn't content_type set, am I thinking of .temp.html?

This will cause content_type to be set and the file will be considered for negotiation,
regardless of whether the previous extensions are known or unknown. I have not managed to
get index.html.en.temp.html (or similar) to serve up but they -are- placed in the list of
negotiable files. So this looks to be a possible failure scenario even though I have not
been able to reproduce it.

Any of this ring any bells?

Bill

Re: Not-a-bug: .asis handler isn't driven

["William A. Rowe, Jr." <wr...@covalent.net>]

From: "Bill Stoddard" <bi...@wstoddard.com>
Sent: Monday, October 01, 2001 11:35 AM


> 
> The sub_req for index.html.temp is destroyed because 
> sub_req->content_type == NULL, which looks correct to me.

Try index.html.en.temp - which may have been what I was thinking of.
Assure that it becomes the last file in the directory listing.

The specific bug folks observed was that .lang.charset would be served,
even though charset wasn't negotiated.

How isn't content_type set, am I thinking of .temp.html?  I acknowledge
that this is _very_ tricky to reproduce.  I went *30* rounds with one
reporter, trying to duplicate their conditions.  I would succeed on and
off, the issues interact with one another to produce unexpected results.

In any case, this first example is the bug I've pointed you at.
The second case that Greg Ames is complaining of is the question,
do we take .asis (or any associated handler) and treat as-negotated.

Re: Not-a-bug: .asis handler isn't driven

[Bill Stoddard <bi...@wstoddard.com>]

Testing in 1.3...

> It isn't a problem, 1.3 mime/negotation is subtely broken, so of course
> this works in 1.3.
>
> OK ... let's start from page one.
>
> Two files in a directory:
>
> index.html.en    1590 bytes
> index.html.fr    1632 bytes
>
> if the user's Accept-Language header is simply en, and they request
> index.html, they will get the .en variant.
Yep.

> Accept-Language of fr
> will return the french variant.
Yep.

> Accept-Language of anything else
> returns Not Acceptable,
Yep.

> and no Accept-Language will return the
> index.html.en variant if the LanguagePriority is set with en and fr
> both present and occuring in that order.
Yep.

>
> Now, Admin copies index.html.fr to index.html.temp.  Starts editing,
> deletes two paragraphs...
>
> index.html.temp  1039 bytes

Done.

>
> User requests index.html with AcceptLanguage en or fr.  They aught
> to get the right file.
Yep.

> User has another Accept-Language.  They MAY
> get index.html.temp, because the others failed, and .temp inserts no
> no language identification to reject.
Set IE to request Basque [eu] and received 406 Not Acceptable which is correct.

> They provide no Accept-Language,
> and they _will_ get index.html.temp, based on size alone.

Nope, I don't get index.html.temp. I get index.html.en. In fact, if I attach a debugger
and watch what mod_negotiation is doing, it does not even allow index.html.temp to be
considered for negotiation because the content_type is NULL. Here is the code...

    while ((dir_entry = readdir(dirp))) {
        request_rec *sub_req;
        /* Do we have a match? */
        if (strncmp(dir_entry->d_name, filp, prefix_len)) {
            continue;
        }
        if (dir_entry->d_name[prefix_len] != '.') {
            continue;
        }
        /* Yep.  See if it's something which we have access to, and
         * which has a known type and encoding (as opposed to something
         * which we'll be slapping default_type on later).
         */

        sub_req = ap_sub_req_lookup_file(dir_entry->d_name, r);

        /* If it has a handler, we'll pretend it's a CGI script,
         * since that's a good indication of the sort of thing it
         * might be doing.
         */
        if (sub_req->handler && !sub_req->content_type) {
            sub_req->content_type = CGI_MAGIC_TYPE;
        }

        /* HTTP_FORBIDDEN is returned, e.g., if the path length limit was exceeded */
        /* HTTP_OK does NOT necessarily mean that the file is really readable! */
        if (sub_req->status == HTTP_OK)
            forbidden.all = 0;
        else if (sub_req->status == HTTP_FORBIDDEN)
            forbidden.any = 1;

        if (sub_req->status != HTTP_OK || !sub_req->content_type) {
            ap_destroy_sub_req(sub_req);
            continue;
        }

The sub_req for index.html.temp is destroyed because sub_req->content_type == NULL, which
looks correct to me.

Bill

Re: Not-a-bug: .asis handler isn't driven

[Joshua Slive <sl...@commerce.ubc.ca>]

On Mon, 1 Oct 2001, William A. Rowe, Jr. wrote:
> If it's an element of the Accept-* negotiation, it needs to be
> multiview-matched.  If it isn't, it has no business in there.
>
> If you disagree with the above sentence, let's narrow the argument to
> that statement.
>

I've only read this thread very quickly (just catching up on mail),
but let me reiterate a point that I believe has been mentioned briefly:

Multiviews has two purposes in 1.3: It can be used for negotiated
documents, or it can be used to hide implementation details.

The first part is obvious.  To be specific about the second part,
the idea is:
1. File extensions are a convenient way to store/configure
meta-information about resources.
2. The URLs do not need to contain this meta-information, and in fact
often SHOULD NOT contain it.

For example, the URL may be
http://example.com/info
This may start off as info.html, but later an ssi document is substituted
so it becomes info.shtml, and later the url is changed so info.asis is
substituted.  The client doesn't care about all this garbage, since it can
just request /info and get the correct thing.

Yes, it is bogus that a selection can be made based on length.  It is a
configuration error to put have two different files which may be selected
by multi-views but are not "negotiable" with the client.  Either an error
could be returned, or we could just document that the selection is
arbitrary in that case.

Joshua.