You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Marc Girod <gi...@shire.ntc.nokia.com> on 2003/01/02 10:37:31 UTC

[users@httpd] Re: patch to avoid loops in htdig indexing

Hello,

I have still the same problem as last November now with 1.3.27, i.e.
htdif when indexing my web site will loop forever when processing
pages generated by mod_autoindex, because these will follow soft
links.

I offered a patch to fix the problem by modifying the hard-coded
preamble emitted by the module, so that it would contain a "robots"
meta-tag telling htdig to ignore such generated pages.

I asked for comments and got some:

>>>>> "JS" == Joshua Slive <jo...@slive.ca> writes:

JS> Some easier ways to deal with this problem:

JS> 1. Use an Alias or a Redirect rather than a symlink.

JS> 2. Put the exclusions in /robots.txt

JS> 3. Use IndexOptions SuppressHTMLPreamble in combination with HeaderName
JS> to change the html code before the index:

1) is an html only solution, and thus inapplicable.
3) would require a special file per directory, and thus unmaintainable
2) would require a new crontab job for generating the robots.txt file
   based on "find -type l" and would be a significant penalty.

What about making the emitted preamble configurable?
Would this be an acceptable fix?

-- 
Marc Girod        P.O. Box 323        Voice:  +358-71 80 25581
Nokia NBI         00045 NOKIA Group   Mobile: +358-50 38 78415
Takomo 1 / 4c27   Finland             Fax:    +358-71 80 61604



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


[users@httpd] Re: HeaderName

Posted by Marc Girod <gi...@shire.ntc.nokia.com>.
>>>>> "MG" == Marc Girod <gi...@shire.ntc.nokia.com> writes:

MG> OK. Not for some time. I got lazy and decided that it was enough to
MG> get a fixed title and header for all the pages.

Whereas my HeaderName setting seems to work as expected for
interactive uses, the indexing with htdig seems to by-pass it
completely, and goes back to looping forever.

I did remove the /var/lib/htdig directory before statring the
indexation.

-- 
Marc Girod        P.O. Box 323        Voice:  +358-71 80 25581
Nokia NBI         00045 NOKIA Group   Mobile: +358-50 38 78415
Takomo 1 / 4c27   Finland             Fax:    +358-71 80 61604


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


[users@httpd] Re: HeaderName (was: patch to avoid loops in htdig indexing)

Posted by Marc Girod <gi...@shire.ntc.nokia.com>.
>>>>> "MG" == Marc Girod <gi...@shire.ntc.nokia.com> writes:

MG> I.e. I may again ask for help.

OK. Not for some time. I got lazy and decided that it was enough to
get a fixed title and header for all the pages.

- The '+' syntax in "IndexOptions +SuppressHTMLPreamble" didn't work
- I had to comment away the default: HeaderName HEADER before the
  global /autoindex-header started to apply.
- I *had* to use MultiViews Option

-- 
Marc Girod        P.O. Box 323        Voice:  +358-71 80 25581
Nokia NBI         00045 NOKIA Group   Mobile: +358-50 38 78415
Takomo 1 / 4c27   Finland             Fax:    +358-71 80 61604


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


[users@httpd] Re: HeaderName (was: patch to avoid loops in htdig indexing)

Posted by Marc Girod <gi...@shire.ntc.nokia.com>.
Thanks again, and thanks to John.

>>>>> "JS" == Joshua Slive <jo...@slive.ca> writes:

JS> In recent Apache versions, this is not the case. You must
JS> explictly name HeaderName Header.html

Not yet in 1.3.27 as it seems, but this is valuable information for me
for the future.

JS> or you must turn on Multiviews in the directory's "Options".

This is what I was missing. I am sorry but I failed to understand this
from the docs (mod_autoindex.html#headername)

Now, I got my first local header included, after <body>, which mean
that I'll have to use the SupressHTMLPreamble you mentioned before, as
well as a CGI (or server-side include... I try to grasp the paragraph
on Content negociation) to produce the expected title from a global
setting.

I.e. I may again ask for help.

-- 
Marc Girod        P.O. Box 323        Voice:  +358-71 80 25581
Nokia NBI         00045 NOKIA Group   Mobile: +358-50 38 78415
Takomo 1 / 4c27   Finland             Fax:    +358-71 80 61604


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] HeaderName (was: patch to avoid loops in htdig indexing)

Posted by Joshua Slive <jo...@slive.ca>.
On Tue, 7 Jan 2003, Marc Girod wrote:
>
> I understand that the default and global "HeaderName HEADER" should
> actually find a local HEADER.html (or HEADER) file and insert it. It
> doesn't in my case.

In recent Apache versions, this is not the case.  You must explictly name
HeaderName Header.html
or you must turn on Multiviews in the directory's "Options".

Joshua.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] HeaderName (was: patch to avoid loops in htdig indexing)

Posted by "John P. Dodge" <do...@cruciate.ca.boeing.com>.
Did you try:

HeaderName HEADER.html

See:
http://httpd.apache.org/docs/mod/mod_autoindex.html#headername

I believe that the action you describe is only applicable to pre v1.3.6
versions and that in post v1.3.6 versions the <filename> argument is a
relative URI.


On 7 Jan 2003, Marc Girod wrote:

> I am still unable to get the HeaderName directive to work as I
> understand it should.
>
> I was able to comment away the FancyIndexing, and to have this apply
> by restarting my httpd service, but I still fail to insert my header
> lines into the emitted preamble.
>
> I understand that the default and global "HeaderName HEADER" should
> actually find a local HEADER.html (or HEADER) file and insert it. It
> doesn't in my case. Could somebody give me an example of a suitable
> contents? Or tell me what is wrong with:
>
> <!-- foo bar -->
> <p>Foo bar
>
> Thanks.
>
> --
> Marc Girod        P.O. Box 323        Voice:  +358-71 80 25581
> Nokia NBI         00045 NOKIA Group   Mobile: +358-50 38 78415
> Takomo 1 / 4c27   Finland             Fax:    +358-71 80 61604
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>    "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>

----------------------------------------
"Mon aƩroglisseur est plein d'anguilles"
John P. Dodge
Boeing Shared Services


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


[users@httpd] HeaderName (was: patch to avoid loops in htdig indexing)

Posted by Marc Girod <gi...@shire.ntc.nokia.com>.
I am still unable to get the HeaderName directive to work as I
understand it should.

I was able to comment away the FancyIndexing, and to have this apply
by restarting my httpd service, but I still fail to insert my header
lines into the emitted preamble.

I understand that the default and global "HeaderName HEADER" should
actually find a local HEADER.html (or HEADER) file and insert it. It
doesn't in my case. Could somebody give me an example of a suitable
contents? Or tell me what is wrong with:

<!-- foo bar -->
<p>Foo bar

Thanks.

-- 
Marc Girod        P.O. Box 323        Voice:  +358-71 80 25581
Nokia NBI         00045 NOKIA Group   Mobile: +358-50 38 78415
Takomo 1 / 4c27   Finland             Fax:    +358-71 80 61604


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


[users@httpd] Re: patch to avoid loops in htdig indexing

Posted by Marc Girod <gi...@shire.ntc.nokia.com>.
>>>>> "MG" == Marc Girod <gi...@shire.ntc.nokia.com> writes:

MG> What do I do wrong?

I would add that within the same <IfModule ...> </IfModule> brackets,
I have the(default?):

    IndexOptions FancyIndexing NameWidth=*

IndexOptions.

The two files I set as headers (global and local) are both world
readable. They end in a new line.

-- 
Marc Girod        P.O. Box 323        Voice:  +358-71 80 25581
Nokia NBI         00045 NOKIA Group   Mobile: +358-50 38 78415
Takomo 1 / 4c27   Finland             Fax:    +358-71 80 61604



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


[users@httpd] Re: patch to avoid loops in htdig indexing

Posted by Marc Girod <gi...@shire.ntc.nokia.com>.
>>>>> "JS" == Joshua Slive <jo...@slive.ca> writes:

JS> No, this is not correct. HeaderName can take an absolute-url-path,
JS> as in Headername /globalheader.html

Thanks for correcting me.

I tried again, but failed so far.
I have (by default?) in httpd.conf:

<IfModule mod_autoindex.c>
[...]
    ReadmeName README
    HeaderName HEADER
[...]
    IndexIgnore .??* *~ *# HEADER* README* RCS CVS *,v *,t
[...]
</IfModule>

I added:

    HeaderName /autoindex-header.html

Under my DirectoryRoot, I added an autoindex-header.html file with a
comment as contents:

<!-- foo bar -->


Actually, in a directory without an index, I added a HEADER file with
a different contents.

Now, looking at the source of the generated Index, I can find neither
of them.
What do I do wrong?

-- 
Marc Girod        P.O. Box 323        Voice:  +358-71 80 25581
Nokia NBI         00045 NOKIA Group   Mobile: +358-50 38 78415
Takomo 1 / 4c27   Finland             Fax:    +358-71 80 61604



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Re: patch to avoid loops in htdig indexing

Posted by Joshua Slive <jo...@slive.ca>.
On Thu, 2 Jan 2003, Marc Girod wrote:
> I have still the same problem as last November now with 1.3.27, i.e.
> htdif when indexing my web site will loop forever when processing
> pages generated by mod_autoindex, because these will follow soft
> links.
>
> I offered a patch to fix the problem by modifying the hard-coded
> preamble emitted by the module, so that it would contain a "robots"
> meta-tag telling htdig to ignore such generated pages.
>
> JS> 3. Use IndexOptions SuppressHTMLPreamble in combination with HeaderName
> JS> to change the html code before the index:
>
> 3) would require a special file per directory, and thus unmaintainable

No, this is not correct.  HeaderName can take an absolute-url-path, as in
Headername /globalheader.html

Joshua.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org