You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucy.apache.org by we...@ctosonline.org on 2009/06/13 22:08:39 UTC
Bus error with QueryParser and Perl class
I�m trying to use QueryParser with WildCardQuery/RegexpTermQuery. I
have a subclass of QueryParser that overrides the make_term_query
method to return a regexp query (a Perl subclass of Query). When I
search for some words, it works fine, but with others it crashes (bus
error). The attached script crashes during the search, but if I
uncomment the �use KSx::Search::WildCardQuery� it crashes while
parsing the query. If I change the search term from �trog� to
�philaret�, it works.
The index I�m using can be found at http://synodinresistance.org/tmp/index.zip
.
BTW, I�m using rev. 4669.
Re: New CPAN release of RegexpTermQuery et al. (Re: Bus error with QueryParser and Perl class)
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Wed, Jun 17, 2009 at 02:15:42PM -0700, webmasters@ctosonline.org wrote:
> >The uploaded file
> >
> > KSx-Search-WildCardQuery-0.05.tar.gz
> >
> >has entered CPAN as
Great, thanks. :)
I've removed the compat code in r4826 and r4827. Nice to get that in before
the upcoming dev release.
Marvin Humphrey
New CPAN release of RegexpTermQuery et al. (Re: Bus error with QueryParser and Perl class)
Posted by we...@ctosonline.org.
On Jun 16, 2009, at 8:09 PM, Marvin Humphrey wrote:
> Nevertheless, I had already concluded that deep-sixing Tally was the
> best
> course of action. I've left some compatibility code in there for
> now, but
> when you have the opportunity, I would appreciate it if you would
> apply the
> patch below (also attached, and should survive via the CC) and
> release a new
> version to CPAN. Once that's done, I can remove the compatibility
> code.
Begin forwarded message:
> From: PAUSE <up...@pause.perl.org>
> Date: June 17, 2009 2:04:38 PM PDT
> To: "Father Chrysostomos" <we...@ctosonline.org>
> Subject: CPAN Upload: S/SP/SPROUT/KSx-Search-WildCardQuery-0.05.tar.gz
> Reply-To: cpan-uploads@perl.org
>
> The uploaded file
>
> KSx-Search-WildCardQuery-0.05.tar.gz
>
> has entered CPAN as
>
> file: $CPAN/authors/id/S/SP/SPROUT/KSx-Search-
> WildCardQuery-0.05.tar.gz
> size: 6901 bytes
> md5: 53daf1a978eb7e5bacf0e795684d2534
>
> No action is required on your part
> Request entered by: SPROUT (Father Chrysostomos)
> Request entered on: Wed, 17 Jun 2009 21:03:46 GMT
> Request completed: Wed, 17 Jun 2009 21:04:38 GMT
>
> Thanks,
> --
> paused, v1047
Re: Bus error with QueryParser and Perl class
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Tue, Jun 16, 2009 at 01:23:24PM -0700, webmasters@ctosonline.org wrote:
>
> On Jun 14, 2009, at 8:25 PM, Marvin Humphrey wrote:
>
> >On Sun, Jun 14, 2009 at 06:42:20PM -0700, webmasters@ctosonline.org
> >wrote:
> >
> >>Strangely, make_termquery still works (rev. 4798) but make_term_query
> >>causes a bus error. Having both present also causes the crash. (Try
> >>the same script with that change.)
> >
> >I presume that this is without applying the patch to RegexpTermQuery? If
> >so, then I get the behavior you describe.
>
> I have applied the patch. But, since it was inline and not attached to
> your message, it got a bit scrambled and I had to apply it by hand.
> Maybe I made a mistake. The new version is on CPAN already. Could you
> test it? I still get the bus error both with 5.8.8 and 5.10.0.
This turns out to have been an error in ORMatcher.c. It was 100% reproducible
once I got everything squared, so I can only conclude that I botched the last
go-round.
The bug, which was failed loop termination resulting in a NULL-pointer
dereference, has been fixed by r4813. WildCardQuery 0.04 works with that
revision.
Nevertheless, I had already concluded that deep-sixing Tally was the best
course of action. I've left some compatibility code in there for now, but
when you have the opportunity, I would appreciate it if you would apply the
patch below (also attached, and should survive via the CC) and release a new
version to CPAN. Once that's done, I can remove the compatibility code.
Marvin Humphrey
diff -ur KSx-Search-WildCardQuery-0.04/lib/KSx/Search/RegexpTermQuery.pm KSx-Search-WildCardQuery-0.04-revised/lib/KSx/Search/RegexpTermQuery.pm
--- KSx-Search-WildCardQuery-0.04/lib/KSx/Search/RegexpTermQuery.pm 2009-06-14 18:39:35.000000000 -0700
+++ KSx-Search-WildCardQuery-0.04-revised/lib/KSx/Search/RegexpTermQuery.pm 2009-06-16 19:52:52.000000000 -0700
@@ -65,6 +65,9 @@
my($pack, %args) = @_;
my $searcher = $args{searchable};
+ my $reader = $searcher->get_reader;
+ my $lex_reader = $reader->fetch("KinoSearch::Index::LexiconReader");
+ my $post_reader = $reader->fetch("KinoSearch::Index::PostingsReader");
# Retrieve the correct Similarity for the Query's field.
my $sim = $args{similarity} =
@@ -77,8 +80,7 @@
# Get a lexicon and find our place therein
my( $re, $prefix ) = ($re{$parent}, $prefix{$parent});
ref $re eq 'Regexp' or $re = qr/$re/; # avoid repetitive recompilation
- my $reader = $searcher->get_reader;
- my $lexcn = $reader->lexicon( field => $field{$parent} );
+ my $lexcn = $lex_reader->lexicon( field => $field{$parent} );
$lexcn->seek(defined $prefix ? $prefix : '');
# iterate through it, stopping at terms that match
@@ -103,8 +105,7 @@
# the doc freq has to be 2, since the re matches two docs. The doc
# freqs of the individual terms are 1 and 2, so we can’t add or
# average them.
-# push @plists, my $plist = $reader->posting_list(
- my $plist = $reader->posting_list(
+ my $plist = $post_reader->posting_list(
term => $term,
field => $field{$parent},
);
@@ -214,10 +215,8 @@
package KSx::Search::RegexpTermScorer;
use base 'KinoSearch::Search::Matcher';
-use KinoSearch::Search::Tally;
-
use Hash::Util::FieldHash::Compat 'fieldhashes';
-fieldhashes\my( %doc_nums, %pos, %wv, %sim, %compiler, %tally );
+fieldhashes\my( %doc_nums, %pos, %wv, %sim, %compiler );
sub new {
my ($class, %args) = @_;
@@ -234,7 +233,6 @@
$pos{$self} = -1;
$wv {$self} = $compiler->get_value;
$compiler{$self} = $compiler;
- $tally{$self} = KinoSearch::Search::Tally->new;
$self
}
@@ -253,20 +251,13 @@
return $pos < scalar @$doc_nums ? $$doc_nums[$pos] : 0;
}
-sub tally {
+sub score {
my $self = shift;
my $pos = $pos{$self};
my $doc_nums = $doc_nums{$self};
- return unless $pos < scalar @$doc_nums;
-
- (my $tally = $tally{$self})
- ->set_score(
- $wv{$self} * $sim{$self}->tf(
- $tfs{$compiler{$self}}{$$doc_nums[$pos]}
- )
- );
-
- $tally;
+ return $wv{$self} * $sim{$self}->tf(
+ $tfs{$compiler{$self}}{$$doc_nums[$pos]}
+ );
}
Re: Bus error with QueryParser and Perl class
Posted by we...@ctosonline.org.
On Jun 14, 2009, at 8:25 PM, Marvin Humphrey wrote:
> On Sun, Jun 14, 2009 at 06:42:20PM -0700, webmasters@ctosonline.org
> wrote:
>
>> Strangely, make_termquery still works (rev. 4798) but make_term_query
>> causes a bus error. Having both present also causes the crash. (Try
>> the same script with that change.)
>
> I presume that this is without applying the patch to
> RegexpTermQuery? If so,
> then I get the behavior you describe.
I have applied the patch. But, since it was inline and not attached to
your message, it got a bit scrambled and I had to apply it by hand.
Maybe I made a mistake. The new version is on CPAN already. Could you
test it? I still get the bus error both with 5.8.8 and 5.10.0.
Father Chrysostomos
Re: Bus error with QueryParser and Perl class
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Sun, Jun 14, 2009 at 06:42:20PM -0700, webmasters@ctosonline.org wrote:
> > FYI, I just today changed QueryParser's factory method names from e.g.
> > "make_termquery" to "make_term_query". (: Seems like the right move since
> > you'd accidentally used the new spelling in your post. :) You'll need to
> > change those in your QueryParser subclasses.
>
> Strangely, make_termquery still works (rev. 4798) but make_term_query
> causes a bus error. Having both present also causes the crash. (Try
> the same script with that change.)
I presume that this is without applying the patch to RegexpTermQuery? If so,
then I get the behavior you describe.
But "make_termquery still works" isn't quite right. What's happening is that
you've installed a method that KS doesn't care about -- make_termquery() --
but haven't overridden the one that it *does* care about, make_term_query().
So the stock make_term_query() is returning a plain old TermQuery which
compiles down to a TermScorer. TermScorer doesn't have the segfaulting Tally
issue, so the script finishes cleanly -- but that's just hiding the problem.
Unfortunately, unlike most of the changes that go into KS, this one induces
silent failure. It's possible to make it catastrophic failure, by installing
stub methods into QueryParser, detecting if they've been overridden and
throwing an error, but for esoteric reasons that's a lot of work... and I
think that the number of people who are overriding QueryParser's factory
methods is small at this point, since they're only public in svn trunk.
> > Also, Scorer has been replaced by Matcher and Compiler_Make_Scorer() has
> > been replaced by Compiler_Make_Matcher(). I'd put some compatibility
> > stubs into KS trunk specifically so that the WildCardQuery 0.03 distro
> > would still pass its tests, but it turns out that you're going to need to
> > release a new version anyway. Diff below.
>
> Now you’re writing my code for me! That’s nice. Thank you. :-)
There may be another patch coming. I'm not sure what to do about tally(), but
today I'm leaning towards junking it and going back to score().
I'd wanted to run the tally() experiment in KS so that we'd know how it worked
for Lucy, but it's just that -- an experiment -- and it won't pay off until I
write some code which exploits it. (I doubt that anyone else would take
advantage of it until I demonstrate its usefulness.) There are a lot of other
experiments for me to run, though, and score() is a known working design.
Marvin Humphrey
Re: Bus error with QueryParser and Perl class
Posted by we...@ctosonline.org.
On Jun 13, 2009, at 9:54 PM, Marvin Humphrey wrote:
> On Sat, Jun 13, 2009 at 01:08:39PM -0700, webmasters@ctosonline.org
> wrote:
> It never occurred to me that someone would return a new Tally()
> on every scoring loop.
And I don’t know what (or whether) I was thinking when I wrote that.
> FYI, I just today changed QueryParser's factory method names from e.g.
> "make_termquery" to "make_term_query". (: Seems like the right move
> since
> you'd accidentally used the new spelling in your post. :) You'll
> need to
> change those in your QueryParser subclasses.
Strangely, make_termquery still works (rev. 4798) but make_term_query
causes a bus error. Having both present also causes the crash. (Try
the same script with that change.)
>
> Also, Scorer has been replaced by Matcher and Compiler_Make_Scorer()
> has been
> replaced by Compiler_Make_Matcher(). I'd put some compatibility
> stubs into KS
> trunk specifically so that the WildCardQuery 0.03 distro would still
> pass its
> tests, but it turns out that you're going to need to release a new
> version
> anyway. Diff below.
Now you’re writing my code for me! That’s nice. Thank you. :-)
Father Chrysostomos
Re: Bus error with QueryParser and Perl class
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Sat, Jun 13, 2009 at 01:08:39PM -0700, webmasters@ctosonline.org wrote:
> I’m trying to use QueryParser with WildCardQuery/RegexpTermQuery. I
> have a subclass of QueryParser that overrides the make_term_query
> method to return a regexp query (a Perl subclass of Query). When I
> search for some words, it works fine, but with others it crashes (bus
> error).
> The attached script crashes during the search,
Crap. :(
This is a refcounting error, and while it's easy to fix in your module, a
general fix is going to be costly.
The immediate cause is that you're passing in a new Tally object each time,
rather than keeping a reference around. What you should do instead is cache a
Tally object at object construction and just mod its score. That's faster,
because there's less object creation/destruction -- and in this case, it will
stop the segfaulting.
The root of the problem is that you're passing in the last refcount for that
Tally object -- but the callback for Matcher_Tally() doesn't know that it needs
to take ownership of that refcount right away. Perl "mortalizes" that scalar,
and once that happens, it becomes the walking dead -- it has an SvREFCNT of 0,
but destruction is getting delayed for a little bit so that the caller can
decide what it wants to do with it. Unfortunately, the callback code isn't
claiming it in time, DESTROY is getting called, and when the calling code tries
to access members within the Tally struct, it's too late -- that memory has
already been freed.
The most "correct" fix is to have Matcher_Tally() return an "incremented
Tally*" rather than a "Tally*". That causes the autogenerated callback code to
invoke INCREF before the Perl callback closes, ensuring that the object will
live on afterwards.
But then everything that calls Matcher_Tally() will have to take ownership of
a refcount, and Tally() is an inner loop method so we don't want to be
constantly incrementing and decrementing refcounts. :(
This was an unanticipated use case because the code in RegexpTermQuery::tally()
is wasteful. It never occurred to me that someone would return a new Tally()
on every scoring loop. But explaining why you need to avoid doing that
requires explaining an excriciatingly complex memory managment issue; we might
get away with it this time, but it's just not acceptable to guard against this
class of error with documentation rather than code.
For Lucene people reading along wondering what on earth a "Tally" is... it's a
struct that conveys scoring information. Tally has a float "score" member
which is used by the default scoring system.
There's a small cost involved with dereferencing "tally->score" instead of
using "score" directly, naturally. [Cue Mike McCandless frowning in
disapproval.] However, because Tally is a struct, it's extensible -- so it
can hold information about e.g. what positions matched. That, in turn, makes
it possible to e.g. implement the proximity boosting algorithm that the Page
and Brin described 11 years ago in their seminal paper.
It's ticked me off for a long time that we throw positional information from
the query string away; replacing Scorer.score() with Matcher_Tally() was
supposed to open the door to exploiting that information and improving
relevance -- for once. One easy fix to the problem exposed by WildCardQuery
would be to completely zap Tally() and go back to the tried and true Score() --
but that means that we stay mired a decade behind in our exploitation of
well-known IR retrieval algorithms. I don't want to do that, because there are
all kinds of things we can do to optimize for speed, but very few opportunities
for improving IR precision. Crap Crap Crap.
> but if I uncomment the ‘use KSx::Search::WildCardQuery’ it crashes while
> parsing the query.
KS was trying to deserialize your Schema, which involves deserializing your
Analyzers. It didn't find KSx::Analysis::StripAccents, so the deserializer
returned the wrong kind of object (a Hash). The class conflict wasn't
detected by PolyAnalyzer's autogenerated Load() method, so a segfault happened
later when KS tried to call an Analyzer method on a Hash.
Fixed by r4791 and r4792 -- now we get a sensible class-not-loaded error
message.
> BTW, I’m using rev. 4669.
FYI, I just today changed QueryParser's factory method names from e.g.
"make_termquery" to "make_term_query". (: Seems like the right move since
you'd accidentally used the new spelling in your post. :) You'll need to
change those in your QueryParser subclasses.
Also, Scorer has been replaced by Matcher and Compiler_Make_Scorer() has been
replaced by Compiler_Make_Matcher(). I'd put some compatibility stubs into KS
trunk specifically so that the WildCardQuery 0.03 distro would still pass its
tests, but it turns out that you're going to need to release a new version
anyway. Diff below.
Marvin Humphrey
--- KSx-Search-WildCardQuery-0.03-old/lib/KSx/Search/RegexpTermQuery.pm 2009-05-30 12:16:18.000000000 -0700
+++ KSx-Search-WildCardQuery-0.03/lib/KSx/Search/RegexpTermQuery.pm 2009-06-13 21:30:33.000000000 -0700
@@ -169,13 +169,13 @@
= $raw_impact{$self} * $idf{$self} * $query_norm_factor;
}
-sub make_scorer {
- my ( $self, $reader ) = @_;
+sub make_matcher {
+ my $self = shift;
return KSx::Search::RegexpTermScorer->new(
# posting_lists => $plists{$self},
- similarity => $self->get_similarity,
- weight => $self,
+ @_,
+ compiler => $self,
);
}
@@ -212,26 +212,29 @@
package KSx::Search::RegexpTermScorer;
-use base 'KinoSearch::Search::Scorer';
+use base 'KinoSearch::Search::Matcher';
use KinoSearch::Search::Tally;
use Hash::Util::FieldHash::Compat 'fieldhashes';
-fieldhashes\my( %doc_nums, %pos, %wv, %sim, %weight );
+fieldhashes\my( %doc_nums, %pos, %wv, %sim, %compiler, %tally );
sub new {
my ($class, %args) = @_;
# my $plists = delete $args{posting_lists};
- my $weight = delete $args{weight};
- my $self = $class->SUPER::new(%args);
- $sim{$self} = $args{similarity};
+ my $compiler = delete $args{compiler};
+ my $reader = delete $args{reader};
+ my $need_score = delete $args{need_score};
+ my $self = $class->SUPER::new(%args);
+ $sim{$self} = $compiler->get_similarity;
- my $tfs = $tfs{$weight};
+ my $tfs = $tfs{$compiler};
$doc_nums{$self} = [ sort { $a <=> $b } keys %$tfs ];
$pos{$self} = -1;
- $wv {$self} = $weight->get_value;
- $weight{$self} = $weight;
+ $wv {$self} = $compiler->get_value;
+ $compiler{$self} = $compiler;
+ $tally{$self} = KinoSearch::Search::Tally->new;
$self
}
@@ -256,10 +259,10 @@
my $doc_nums = $doc_nums{$self};
return unless $pos < scalar @$doc_nums;
- (my $tally = KinoSearch::Search::Tally->new)
+ (my $tally = $tally{$self})
->set_score(
$wv{$self} * $sim{$self}->tf(
- $tfs{$weight{$self}}{$$doc_nums[$pos]}
+ $tfs{$compiler{$self}}{$$doc_nums[$pos]}
)
);
Re: Bus error with QueryParser and Perl class
Posted by we...@ctosonline.org.
On Jun 13, 2009, at 1:08 PM, webmasters@ctosonline.org wrote:
> The attached script
The lucy-dev list stripped the attachment. I�m sending it again,
cc�ing to the KS list, which doesn�t mind attachments.