Controlling Hits

[Otis Gospodnetic <ot...@yahoo.com>]

Hi,

Could we make Hits non-final, or at least expose something in Hits to control the number of Documents it reads from disk?
I'm looking at:
  getMoreDocs(50); // retrieve 100 initially
...
  int n = min * 2;    // double # retrieved
(Why * 2?  To "read and cache ahead"?)

... and wondering how I could control those two numbers.  If reading Documents out of the index is the thing to minimize, then by changing/decreasing that 50 and maybe eliminating that * 2, I would hit my disk a little less, no?  Maybe something as simple as this will do:

  public static void fetchDocs(int)

Or maybe the answer is: Use the search method that returns TopDocs if you want more control...?

Thanks,
Otis




---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org

Re: Controlling Hits

[Chris Hostetter <ho...@fucit.org>]

: In fact, I fail to see a situation where "Hits"'s concept of "random access"
: to the results (you can ask for result #30 and then #70) even makes sense.
: In all search applications I'm familar with, at the time you call search(),
: you already know how many results you want to display - and you don't need
: someone to guess for you that you need 50 results, and if that's not enough
: then you need 100 results, and then 200, and so on.

Hits attempts to do the best it can for a simple clients without forcing
the client to use an API that requires knowing exactly how it will get
used in advance of executing hte search -- this can be practical in a
simple MVC framework where the Contorller may not know exactly how the
View wants to display things; if the Controller is forced to tell the
Searcher how many results it wants and at what offset, then that's one
more place that information has to be used (instead of in just hte View)

as for the random access: imagine you are outputing a multi column listing
of N results per page, using an HTML table: you display # 1, then # N/2+1,
then a row boundry, then # 2, then # N/2+2, etc... likewise imagine you
want to prefetch some data baout each item on your "page" so you iterate
over all of them once to get their image path (or something) then you
iterate over all of them again to display the results.

...there are lots of possible usecases, and that's what Hits is all about:
making it trivially easy for people do do just about anything with a
really simplistic API -- without concern for performance.   once
performance becomes your concern, then you start reconsidering your API,
and have to understand more about the Lucene internals to make the right
choice for your application.

I hate the Hits class with a passion -- almost as much as I hate the
RangeQuery class -- i have yet to read/hear a question asked for which i
thought "Hits" was the right answer ... but i don't think we should ever
deprecate it unless it's because we've aded an even *simpler* API ...
Like QueryParser, StandardAnalyzer, IndexMOdifier, and the IndexSearcher
constructor that takes a "String" arguement, Hits provides a super low
barrier to entry that lets people start writting Lucene apps without
needing to understand the complexities of the underlying Lucene APIs



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org

Re: Controlling Hits

[Nadav Har'El <ny...@math.technion.ac.il>]

On Fri, Nov 24, 2006, Otis Gospodnetic wrote about "Controlling Hits":
> Hi,
> 
> Could we make Hits non-final, or at least expose something in Hits to control the number of Documents it reads from disk?
>...
> Or maybe the answer is: Use the search method that returns TopDocs if you want more control...?

In an application I was writing, I was facing similar issues: "Hits" was fine
for a short Demo in Lucene, but when it came to a real application, it didn't
give me enough control: it reran the search too many times when you wanted
to see, e.g, the 20th result page, and wouldn't allow me adding a HitCollector
which I needed. I started by modifying Hits (which wasn't just final - much
of its functionality was private), but then realized: there's simply no
reason to use Hits! IndexSearcher.search() which returns TopDocs already
gives you full control, and frankly isn't that much harder to use.

In fact, I fail to see a situation where "Hits"'s concept of "random access"
to the results (you can ask for result #30 and then #70) even makes sense.
In all search applications I'm familar with, at the time you call search(),
you already know how many results you want to display - and you don't need
someone to guess for you that you need 50 results, and if that's not enough
then you need 100 results, and then 200, and so on.
And since this concept of "random access" is what differenciates Hits from
TopDocs, perhaps we don't need Hits at all?

So, how about deprecating Hits altogether, and recommending the TopDocs
alternatives instead?

-- 
Nadav Har'El                        |       Sunday, Nov 26 2006, 5 Kislev 5767
IBM Haifa Research Lab              |-----------------------------------------
                                    |God created the world out of nothing, but
http://nadav.harel.org.il           |the nothingness still shows through.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org

Re: Controlling Hits

[Marvin Humphrey <ma...@rectangular.com>]

On Nov 24, 2006, at 10:38 PM, Otis Gospodnetic wrote:

> Could we make Hits non-final, or at least expose something in Hits  
> to control the number of Documents it reads from disk?
> I'm looking at:
>   getMoreDocs(50); // retrieve 100 initially
> ...
>   int n = min * 2;    // double # retrieved
> (Why * 2?  To "read and cache ahead"?)
>
> ... and wondering how I could control those two numbers.  If  
> reading Documents out of the index is the thing to minimize, then  
> by changing/decreasing that 50 and maybe eliminating that * 2, I  
> would hit my disk a little less, no?  Maybe something as simple as  
> this will do:
>
>   public static void fetchDocs(int)

FWIW, in KinoSearch, this is the interface:

   my $hits = $searcher->search( query => $query );
   $hits->seek( $offset, $num_wanted );
   while ( my $hit = $hits->fetch_hit_hashref ) {
        print "$hit->{title}\n";
   }

The actual scoring doesn't happen until seek() gets called.  That  
way, it's possible to know exactly how many hits need to be  
collected, by adding $offset and $num_wanted.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/



---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org