You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Jeremy Howard <jh...@fastmail.fm> on 2000/05/17 15:20:12 UTC

Guide search engine (was Re: multiple copies of a module)

Stas Bekman wrote:
> Hold on, at this very moment a few mod_perl fellas are working on having a
> good search engine for the guide. Just give it some more time, I'm trying
> to bring the best so it'll take a while...
> 
I'm glad you brought this up again. Since I mentioned I'd be happy to host such a thing, and asked for suggestions, I've got a total of one (from Stas--thanks!). That suggestion was to use ht://dig <http://www.htdig.org/>.

Has anyone got a search engine up and running that they're happy with? Stas has made the good point that it needs to be able to hilight found words, since the pages are quite large. If anyone has a chance to do a bit of research about (free) search engines, I'd really appreciate it if you could let me know what you find out. It'd be nice publicity if it was mod_perl based, I guess, but it doesn't really matter.

My only concern is that it seems a little odd to keep this just to the Guide. Wouldn't it be useful for the rest of perl.apache.org? I wouldn't have thought it's much extra work to add a drop-down box to search specific areas of the sight (the Guide being one)...

If there's a good reason to have the Guide's search engine separate to the rest of perl.apache.org, should it have a separate domain (modperlguide.org?, guide.perl.apache.org?)?


--
  Jeremy Howard
  jeremy@fastmail.fm

Re: Guide search engine (was Re: multiple copies of a module)

Posted by "Keith G. Murphy" <ke...@mindspring.com>.
Jeremy Howard wrote:
> 
> I'm glad you brought this up again. Since I mentioned I'd be happy to host such a thing, and asked for suggestions, I've got a total of one (from Stas--thanks!). That suggestion was to use ht://dig <http://www.htdig.org/>.
> 
> Has anyone got a search engine up and running that they're happy with? Stas has made the good point that it needs to be able to hilight found words, since the pages are quite large. If anyone has a chance to do a bit of research about (free) search engines, I'd really appreciate it if you could let me know what you find out. It'd be nice publicity if it was mod_perl based, I guess, but it doesn't really matter.
> 
> 
I know this is absolute anathema, considering you guys are developers,
but...

Have you looked at www.atomz.com, at least as a temporary solution?  (A
free service for sites with fewer than 500 pages).  Basically, the
search brings up their page, but you can customize it to look just like
one of yours.  It truly is fast (as hell) and flexible, and it does
highlight found words.  Even does soundalikes in the absence of other
matches.  The result page will show their logo, though, but it's rather
unobtrusive.

(The biggest drawback, as a long-term solution, is that if you change
the look of your pages, you have one more maintenance chore to do, in
that you have to go over to atomz.com and change your result page there
as well).

O'Reilly uses it, if that helps!  :-)

Try this:

http://search.atomz.com/search/?sp-a=0002078e-sp00000000&sp-q=cgi&sp-k=Books

(Looks for O'Reilly books pages containing 'cgi').

Yeah, I know, I'd rather roll my own, too, given time...

Re: Guide search engine (was Re: multiple copies of a module)

Posted by Jay J <3p...@iname.com>.
Jeremy Howard wrote:
>
> I'm glad you brought this up again. Since I mentioned I'd be happy to
> host such a thing, and asked for suggestions, I've got
> a total of one (from Stas--thanks!). That suggestion was to use
> ht://dig <http://www.htdig.org/>.
>
> Has anyone got a search engine up and running that they're happy with?
> Stas has made the good point that it needs to be able
> to hilight found words, since the pages are quite large. If anyone has
> a chance to do a bit of research about (free) search
> engines, I'd really appreciate it if you could let me know what you
> find out. It'd be nice publicity if it was mod_perl based,
> I guess, but it doesn't really matter.

I'm happy with ht://dig, I use it mainly for looking up docs I've
squirreled away in /manual. (instead of grep)

It's been a while since I've been to htdig.org but I did grab a tarball
recently, so I'm fairly confident there isn't* an existing mod_perl
wrapper -- but maybe there should be.

There are a number of perl scripts in the distribution, and I thought*
there was a plain Perl wrapper, but I could be mistaken.

I think a mod_perl frontend/wrapper could work well, that is, htsearch
is about 900K+ and takes a moment to fire up (on my box anyway) -- how
much worse could it be?

OTOH, one could* (conceivably) get crazy and access the DB's directly
and maybe XS any needed portions of htsearch (ambitious :-). However,
this still leaves htdig, htfuzzy, htmerge, etc .. to handle the
indexing.

As far as highlighting, I have a piece of code I'm using -- we could use
it as a starting point. Downside is it uses $` $' (it can probably be
tweeked to avoid this), but it handles the critical stuff like skipping
keywords within href's/tags, etc.

RE: Matt Sergeant -- Perhaps highlighting is overrated, but it usually
doesn't hurt. I too have a proprietary search facility, and a inverted
indexing prototype (stores packed doc-id integers in MySQL, for example)
-- but a great deal of work has gone into ht://dig ......


> My only concern is that it seems a little odd to keep this just to the
> Guide. Wouldn't it be useful for the rest of
> perl.apache.org? I wouldn't have thought it's much extra work to add a
> drop-down box to search specific areas of the sight
> (the Guide being one)...

I'd have to agree there.


> If there's a good reason to have the Guide's search engine separate to
> the rest of perl.apache.org, should it have a
> separate domain (modperlguide.org?, guide.perl.apache.org?)?
> 
> --
> Jeremy Howard
> jeremy@fastmail.fm

ht://dig allows for the param 'restrict' => /to_this_directory .. which
might be useful for seperating things.

Count me in, whatever we choose.

-Jay J

# use Text::Wrapper;

Re: Guide search engine (was Re: multiple copies of a module)

Posted by Matt Sergeant <ma...@sergeant.org>.
BTW: Your email client is broken and not wrapping words.

On Wed, 17 May 2000, Jeremy Howard wrote:

> Stas Bekman wrote:
> > Hold on, at this very moment a few mod_perl fellas are working on having a
> > good search engine for the guide. Just give it some more time, I'm trying
> > to bring the best so it'll take a while...

> I'm glad you brought this up again. Since I mentioned I'd be happy to
> host such a thing, and asked for suggestions, I've got a total of one
> (from Stas--thanks!). That suggestion was to use ht://dig
> <http://www.htdig.org/>.

While htdig is a reasonable engine, Stas's idea is this needs to be "guide
specific". Meaning what I'm not sure, but I'm assuming it means to pick
out only certain words to index...

> Has anyone got a search engine up and running that they're happy with?

I just wrote a very simple SQL based engine - so I would say I'm happy
with that. It's fast and it's all in perl. I could very simply rip out the
search parts of the code for someone to play with if they wanted to.

> Stas has made the good point that it needs to be able to hilight found
> words, since the pages are quite large. If anyone has a chance to do a
> bit of research about (free) search engines, I'd really appreciate it
> if you could let me know what you find out. It'd be nice publicity if
> it was mod_perl based, I guess, but it doesn't really matter.

I think word highlighting is overrated. It's only necessary in this case
because the guide is so damn huge now. The size problem could be
eliminated by making the guide split itself up into smaller sections. My
proposal would be to do that by converting the guide to docbookXML and use
AxKit to display the resulting docbook pages. The AxKit docbook
stylesheets are nice and friendly, and written in Perl, not some obscure
XML stylesheet language. And after all that, it would make converting the
guide to a format O'Reilly likes to publish (i.e. docbook), trivial.

> My only concern is that it seems a little odd to keep this just to the
> Guide. Wouldn't it be useful for the rest of perl.apache.org? I
> wouldn't have thought it's much extra work to add a drop-down box to
> search specific areas of the sight (the Guide being one)...

perl.apache.org already has a search engine.

> If there's a good reason to have the Guide's search engine separate to
> the rest of perl.apache.org, should it have a separate domain
> (modperlguide.org?, guide.perl.apache.org?)?

guide.modperl.org ?

-- 
<Matt/>

Fastnet Software Ltd. High Performance Web Specialists
Providing mod_perl, XML, Sybase and Oracle solutions
Email for training and consultancy availability.
http://sergeant.org http://xml.sergeant.org