Result based sorting for KWIC?

[Christian Wittern <cw...@gmail.com>]

Hi there,

I am investigating using solr for a project that requires presentation of
search results in a KWIC display, sorted according to either the string
following the matches or the (reverse) of the characters previous to the
matches.  Can this be done with Solr?  How would I go about implement this?

Any ideas appreciated,

Christian

-- 

 Christian Wittern
 Institute for Research in Humanities, Kyoto University
 47 Higashiogura-cho, Kitashirakawa, Sakyo-ku, Kyoto 606-8265, JAPAN

Re: Result based sorting for KWIC?

[Christian Wittern <cw...@gmail.com>]

Chris Hostetter wrote:
>
> That still doesn't really answer a fairly fundemental question i've been 
> trying to understand: *why* would having the results in that order be much 
> more useful to for the users? 
>   
Well, there are several reasons: One is that it allows users to easily 
spot related entries, for example quotations of a text appearing within 
another document.  Another reason is that it allows to easily detect 
linguistic patterns.

Of course, this is not the only sorting to be offered, but the one I am 
currently struggling with and trying to evaluate whether Solr would be 
of help here.
> what are you going to do if the term input more then once in a single document?
>   
The KWIC representation is generated for every hit, so if there are 5 
matches in a doc, you get five hits.

> SOlr can sort your results on any indexed, single value, field - but for 
> something like this you'd need to write your own plugin to do the sorting.  
> Note that your plugin would basically need to do the same thing you 
> currently do on the client, the only real speed performance gain would be 
> in reducing the amount of data sent over the wire.
>   
Indeed.  Except that Solr might be able to use mature, efficient and 
well-debugged code to do that, which I can't say about my client code.  
Well, not knowing anything about the internals used in Solr (or Lucene 
for that matter), I just assumed that this in some sense parallels the 
way a ranking value is calculated for a search term and then the results 
are sorted by relevance.

But I think I have enough information now to decide how to proceed.

Christian

-- 

 Christian Wittern 
 Institute for Research in Humanities, Kyoto University
 47 Higashiogura-cho, Kitashirakawa, Sakyo-ku, Kyoto 606-8265, JAPAN

Re: Result based sorting for KWIC?

[Chris Hostetter <ho...@fucit.org>]

: If you go to http://tkb.mydns.jp:8899/exist/rest/db/new/tkb.xq you will see
: what I currently have.  Just click search to search for the example, or maybe
: delete the last character so that you get more results (this is not released

I don't read Japanese so i'm really not sure what I'm looking at.  I see 
hte search term highlighted, but I don't know enough about the langauge to 
really understand what is special about hte order of the results...

: yet, so don't be surprised if it breaks...). You will see the search term
: highlighted in the middle, context is available from the blue arrow to the
: right.  The display would be much more useful for the users, if this could be
: sorted on the characters following the hit (ignoring punctuation).  Another
: option would be to sort on the characters previous to the hit.  But in this
: case, the sorting has to be reversed, so that if I have:
:  ABCD<hit>FGHI
: the sort-key would be constructed as DCBA for this case.

That still doesn't really answer a fairly fundemental question i've been 
trying to understand: *why* would having the results in that order be much 
more useful to for the users? 

what are you going to do if the term input more then once in a single document?

: would be very slow, so I am looking for other ways.   Erik also said that down
: the road there might be a sort function that could be called, which is what I
: would need here. 

SOlr can sort your results on any indexed, single value, field - but for 
something like this you'd need to write your own plugin to do the sorting.  
Note that your plugin would basically need to do the same thing you 
currently do on the client, the only real speed performance gain would be 
in reducing the amount of data sent over the wire.


-Hoss

Re: Result based sorting for KWIC?

[Christian Wittern <cw...@gmail.com>]

Chris Hostetter wrote:
> 1) if you've got full text search, why would you even want KWIC?  
>   
Well, KWIC is a way to present the full text search results so that they 
can be easily read. 
> 2) your description of how you'd want the results ordered is extrmely 
> confusing to me ... can you give a simple concrete example of some 
> documents / queries / result-doclists that you would want to see?
>   
If you go to http://tkb.mydns.jp:8899/exist/rest/db/new/tkb.xq you will 
see what I currently have.  Just click search to search for the example, 
or maybe delete the last character so that you get more results (this is 
not released yet, so don't be surprised if it breaks...). 
You will see the search term highlighted in the middle, context is 
available from the blue arrow to the right.  The display would be much 
more useful for the users, if this could be sorted on the characters 
following the hit (ignoring punctuation).  Another option would be to 
sort on the characters previous to the hit.  But in this case, the 
sorting has to be reversed, so that if I have:
  ABCD<hit>FGHI
the sort-key would be constructed as DCBA for this case.

I know that this can be done by post-processing the results on the 
client (which is what Erik suggested offline), but if I get thousands of 
hits, that would be very slow, so I am looking for other ways.   Erik 
also said that down the road there might be a sort function that could 
be called, which is what I would need here. 

Cheers,

Christian

-- 
 Christian Wittern 
 Institute for Research in Humanities, Kyoto University
 47 Higashiogura-cho, Kitashirakawa, Sakyo-ku, Kyoto 606-8265, JAPAN

Re: Result based sorting for KWIC?

[Chris Hostetter <ho...@fucit.org>]

: I am investigating using solr for a project that requires presentation of
: search results in a KWIC display, sorted according to either the string
: following the matches or the (reverse) of the characters previous to the
: matches.  Can this be done with Solr?  How would I go about implement this?

1) if you've got full text search, why would you even want KWIC?  

2) your description of how you'd want the results ordered is extrmely 
confusing to me ... can you give a simple concrete example of some 
documents / queries / result-doclists that you would want to see?



-Hoss