You are viewing a plain text version of this content. The canonical link for it is here.
Posted to docs-dev@perl.apache.org by Bill Moseley <mo...@hank.org> on 2002/03/07 04:42:00 UTC
Searching perl
Hi,
First, a note on the Javascript error. I did a File->Save to disk, and
then read that into NS4.0 and the Javascript worked. The difference is the
style sheet is not loaded when reading the page off disk. I wonder if it's
just the CSS or if the stylesheet is making it hard to find the correct
form name.
I've been testing out using swish-e's "buzzword" feature to see if that can
help search perl code. Normally, something like $| is not indexed, but if
that's defined as a buzzword then swish will index that. Of course, swish
has to find that string, and how it does that is by tokenizing the text
into white-space separated words, then removing leading and trailing
characters. These are defined with the IgnoreFirstChar and IgnoreLastChar
settings.
For example:
Buzzwords mod_perl c++ $_ $+ $| $_ @$ @_ $ENV $SIG
IgnoreFirstChar ({[]}):,
IgnoreLastChar ({[]}):,.
So it will find $| (or any of the other ones defined) if they are separated
by white space (and optionally those characters).
It's not perfect, but it gets us closer. Clearly, it won't catch all
situations. You can try it at the URL I posted earlier. It will be
interesting to use grep to compare -- and that might help to improve indexing.
Stas, you recommended creating two indexes. But I've been having a hard
time finding a set of characters that could be used to separate the parts
of the perl code that someone might want to search. Plus that will
complicate searching. So I think using these "buzzwords" will be probably
our best bet.
Normally, (without buzzwords) swish has a list of characters called
"WordCharacters" which defines the allowable characters in a word.
A more typical setup might be to define WordCharacters as A-Z and, say, the
underscore. And then set IgnoreFirstChar and IgnoreLastChar also to the
underscore. Then swish indexes: "mod_perl", "__mod_perl", and "mod_perl__"
all as "mod_perl".
This makes it "somewhat" easy to do term highlighting since you can build a
regular expression:
@words = split /^$ignoref([$wc]+?)$ignorel$/i, $text;
and end up with an array of all the text, separated into words that swish
indexes and all the other text. That makes processing easy.
But throw in buzzwords and it's not that simple, since the text has to be
spilt by whitespace, stripped of the the IgnoreFirstChar and IgnoreLastChar
settings, checked for buzzwords, then split by wordcharacters.
Why am I telling you this. The first reason is that the term highlighting
(which I spent most of the day writing) is really slow. It has to process
each word one by one, and the code is kind of ugly. It's not simple regex
search and replace, which would be a lot faster, but less accurate and
would highlight phrases. I kid of think it's important that the
highlighting is reasonably accurate.
The second reasons is maybe if I'm lucky someone will tell me how to make
it faster! I've asked on Perlmonks, but I've not had great luck there (I
think it's a more complicated problem than can be answered in a short
lookover by someone. Someday I'll write it in C, but I'm not sure that
would be that much faster, either.
Anyway, think that will get us by for a while for searching perl code?
--
Bill Moseley
mailto:moseley@hank.org
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-dev-unsubscribe@perl.apache.org
For additional commands, e-mail: docs-dev-help@perl.apache.org
Re: Searching perl
Posted by allan <la...@inet.uni2.dk>.
Bill Moseley wrote:
> First, a note on the Javascript error. I did a File->Save to disk, and
> then read that into NS4.0 and the Javascript worked. The difference is the
> style sheet is not loaded when reading the page off disk. I wonder if it's
> just the CSS or if the stylesheet is making it hard to find the correct
> form name.
hi bill,
for "fun" please do try and load the stylesheet in ns4 and
then do:
...
div.rightbox {
/* position: absolute;*/
...
}
does js work (apart from the screwed positioning)?
./allan
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-dev-unsubscribe@perl.apache.org
For additional commands, e-mail: docs-dev-help@perl.apache.org
Re: Searching perl
Posted by Stas Bekman <st...@stason.org>.
so your opinion is that normal index with unaltered words definition,
plus customized buzzwords, plus 'close' words rank better is the best
solution?
> In that previous thread we also discussed searching two indexes at the same
> time, but that doesn't work well since duplicates are returned -- and since
> swish returns the results a page at a time you can't remove duplicates very
> easily (without asking swish to return all results for every search and
> then removing duplicates).
oops, didn't think about this one. The same problem as doing JOIN on
tables with paging. ;(
> $r->no_cache is easy. You just search for the phrase "$r->no_cache" (with
> quotes). The standard index even finds "$self->{r}->no_cache" with that
> search.
>
> Same with modules names. I actually like that method (using quotes) best,
> as it's most flexible. You can search for Registry and find
> Apache::Registry, and you can search for "Apache::Registry" and find just
> that.
it's just very unintuitive for us Perl users, Apache::Registry is a
perfect word to me :).
But if we teach users to do it that way I guess we can live with that.
i.e. we need a good/simple guide on using the search.
Thanks Bill!
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://ticketmaster.com http://apacheweek.com
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-dev-unsubscribe@perl.apache.org
For additional commands, e-mail: docs-dev-help@perl.apache.org
Re: Searching perl
Posted by Bill Moseley <mo...@hank.org>.
At 12:05 PM 03/07/02 +0800, Stas Bekman wrote:
>Hmm, you mean the thread we had before and the outcome were useless?
>I thought we have agreed on putting most of the chars into the
>WordCharacters variable.
I tested a bunch of settings and had mixed results. In some cases I'd gain
being able to search some types of perl code successfully, but then some
normal text was not searchable as expected.
In that previous thread we also discussed searching two indexes at the same
time, but that doesn't work well since duplicates are returned -- and since
swish returns the results a page at a time you can't remove duplicates very
easily (without asking swish to return all results for every search and
then removing duplicates).
>I'm absolutely against trying to come with a
>list of buzzwords, since it's going to take so much work and you still
>won't cover all the required "special" words. If we can search for
>Apache::Registry and $! because they are "words", that's all we need.
>This is what I've suggested to add to WordCharacters: >-$%@:*[]{}|&
You can try those characters on my machine. (I should move it over to my
fastest machine for better highlighting speed...) There's three index
selections now, one uses the standard settings, one adds the above
characters to the settings, and one is the standard plus a few buzzwords:
Buzzwords mod_perl c++ $_ $+ $| $_ @$ @_ $ENV $SIG
IgnoreFirstChar ({[]}):,
IgnoreLastChar ({[]}):,.
I think the standard index works best, in general. But as you might notice
and expect, some things work better in one index while others work better
in a different index.
>Actually it would be cool if swish-e was able to accept char sequences
>as valid word segments. For example we don't want > or - to be counted
>as parts of the word, but we do want -> (think $r->no_cache)
$r->no_cache is easy. You just search for the phrase "$r->no_cache" (with
quotes). The standard index even finds "$self->{r}->no_cache" with that
search.
Same with modules names. I actually like that method (using quotes) best,
as it's most flexible. You can search for Registry and find
Apache::Registry, and you can search for "Apache::Registry" and find just
that.
The advantage of using buzzwords is you can search for things like $|, but
still use a standard index which is more flexible. It won't find all
occurrences of $|, but it will find some.
Try some searches. Here's some examples:
Search Results
standard index with >-$%@:*[]{}|& with buzzword mod_perl
mod_perl 912 878 866
"mod_perl" 904 852 866
So the difference between 904 and 852 is that the ">-$%@:*[]{}|&" is
causing some of the "mod_perl" words to be indexed differently (probably
mod_perl:) and thus not found.
$env works differently:
standard index with >-$%@:*[]{}|& with buzzword mod_perl
env 42 15 42
$env 42 1 1
The standard index may give (a lot of) false hits, but that's better than
missing ones, in some cases.
standard index with >-$%@:*[]{}|& with buzzword mod_perl
$ENV{MOD_PERL} 32 8 22
"$ENV{MOD_PERL}" 9 8 9
That one shows why it would be nice to adjust ranking on how close words
are together in a document. Searching $ENV{MOD_PERL} is really searching
three words, so it would be helpful if the docs where those words are close
together ranked highest -- so of those 32 hits, the top 9 in rank would be
the real ones.
>Also you were talking about highlighting. Are you talking about
>highlighting for the snippets presented on the hits page? Or are you
>working on highlighting in the real 'full' text?
Highlighting for the snippets. I feel that search results are suppose to
help find the page you are looking for, not specific words on the final
document. When I use google's cached pages I find the highlighted words
awkward and in the way.
Besides, our search results take you to the section of the document anyway,
so I see no reason for highlighting the final doc. Besides, we don't want
dynamic content generation -- (but it would make dealing with netscape
easier ;). Hey, I've got some javascript that will highlight terms on a
page ;)
Sorry for rambling on so long.
--
Bill Moseley
mailto:moseley@hank.org
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-dev-unsubscribe@perl.apache.org
For additional commands, e-mail: docs-dev-help@perl.apache.org
Re: Searching perl
Posted by Stas Bekman <st...@stason.org>.
Bill Moseley wrote:
> Hi,
>
> First, a note on the Javascript error. I did a File->Save to disk, and
> then read that into NS4.0 and the Javascript worked. The difference is the
> style sheet is not loaded when reading the page off disk. I wonder if it's
> just the CSS or if the stylesheet is making it hard to find the correct
> form name.
>
> I've been testing out using swish-e's "buzzword" feature to see if that can
> help search perl code. Normally, something like $| is not indexed, but if
> that's defined as a buzzword then swish will index that. Of course, swish
> has to find that string, and how it does that is by tokenizing the text
> into white-space separated words, then removing leading and trailing
> characters. These are defined with the IgnoreFirstChar and IgnoreLastChar
> settings.
>
> For example:
>
> Buzzwords mod_perl c++ $_ $+ $| $_ @$ @_ $ENV $SIG
> IgnoreFirstChar ({[]}):,
> IgnoreLastChar ({[]}):,.
>
> So it will find $| (or any of the other ones defined) if they are separated
> by white space (and optionally those characters).
>
> It's not perfect, but it gets us closer. Clearly, it won't catch all
> situations. You can try it at the URL I posted earlier. It will be
> interesting to use grep to compare -- and that might help to improve indexing.
>
> Stas, you recommended creating two indexes. But I've been having a hard
> time finding a set of characters that could be used to separate the parts
> of the perl code that someone might want to search. Plus that will
> complicate searching. So I think using these "buzzwords" will be probably
> our best bet.
>
> Normally, (without buzzwords) swish has a list of characters called
> "WordCharacters" which defines the allowable characters in a word.
>
> A more typical setup might be to define WordCharacters as A-Z and, say, the
> underscore. And then set IgnoreFirstChar and IgnoreLastChar also to the
> underscore. Then swish indexes: "mod_perl", "__mod_perl", and "mod_perl__"
> all as "mod_perl".
>
> This makes it "somewhat" easy to do term highlighting since you can build a
> regular expression:
>
> @words = split /^$ignoref([$wc]+?)$ignorel$/i, $text;
>
> and end up with an array of all the text, separated into words that swish
> indexes and all the other text. That makes processing easy.
>
> But throw in buzzwords and it's not that simple, since the text has to be
> spilt by whitespace, stripped of the the IgnoreFirstChar and IgnoreLastChar
> settings, checked for buzzwords, then split by wordcharacters.
>
> Why am I telling you this. The first reason is that the term highlighting
> (which I spent most of the day writing) is really slow. It has to process
> each word one by one, and the code is kind of ugly. It's not simple regex
> search and replace, which would be a lot faster, but less accurate and
> would highlight phrases. I kid of think it's important that the
> highlighting is reasonably accurate.
>
> The second reasons is maybe if I'm lucky someone will tell me how to make
> it faster! I've asked on Perlmonks, but I've not had great luck there (I
> think it's a more complicated problem than can be answered in a short
> lookover by someone. Someday I'll write it in C, but I'm not sure that
> would be that much faster, either.
>
> Anyway, think that will get us by for a while for searching perl code?
Hmm, you mean the thread we had before and the outcome were useless?
I thought we have agreed on putting most of the chars into the
WordCharacters variable. I'm absolutely against trying to come with a
list of buzzwords, since it's going to take so much work and you still
won't cover all the required "special" words. If we can search for
Apache::Registry and $! because they are "words", that's all we need.
This is what I've suggested to add to WordCharacters: >-$%@:*[]{}|&
Actually it would be cool if swish-e was able to accept char sequences
as valid word segments. For example we don't want > or - to be counted
as parts of the word, but we do want -> (think $r->no_cache)
Also you were talking about highlighting. Are you talking about
highlighting for the snippets presented on the hits page? Or are you
working on highlighting in the real 'full' text?
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://ticketmaster.com http://apacheweek.com
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-dev-unsubscribe@perl.apache.org
For additional commands, e-mail: docs-dev-help@perl.apache.org
Re: Searching perl
Posted by Stas Bekman <st...@stason.org>.
Bill Moseley wrote:
> To follow up to myself, still can't search things like $SIG{ALRM} very
> well, even with Buzzwords (because "ALRM" messes it up). I to do it really
> right we need a perl tokenizer, but that may be trying too hard (and too
> trying, as well ;).
with an extended definition of the word you should be able to, no?
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://ticketmaster.com http://apacheweek.com
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-dev-unsubscribe@perl.apache.org
For additional commands, e-mail: docs-dev-help@perl.apache.org
Re: Searching perl
Posted by Bill Moseley <mo...@hank.org>.
To follow up to myself, still can't search things like $SIG{ALRM} very
well, even with Buzzwords (because "ALRM" messes it up). I to do it really
right we need a perl tokenizer, but that may be trying too hard (and too
trying, as well ;).
--
Bill Moseley
mailto:moseley@hank.org
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-dev-unsubscribe@perl.apache.org
For additional commands, e-mail: docs-dev-help@perl.apache.org