You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Marcel Bruch <br...@cs.tu-darmstadt.de> on 2010/06/11 15:35:07 UTC
Using Lucene with a rather simplistic scoring system?
Hi!
We are working on an experimental code-search engine that helps users to
find example code snippets based on what a developer already typed
inside her editor. Our "homemade search engine" produces some cool
results but its performance is somehow limited :-) Thus, we are
evaluating whether Lucene can solve our performance issues. However, we
are not familiar with Lucene and I wonder if some of you could help me
to learn whether Lucene fits our problem well. Thanks in advance for
your comments.
The situation is as follows. For each source code file we extract some
code properties like which types are used inside the code, which methods
are overridden or which methods are called inside a method body etc. For
each source code file we get a JSON structure similar to this:
{
"class" : my.ExampleClass
"extends" : the.SuperClass
"overrides" :
- the.SuperClass.method1()
- the.SuperClass.method2()
"used types":
- a.Type1
- a.Type2
- ...
"used methods":
- a.Type1.method32()
- a.Type1.method23()
- ...
<few more things>
}
The scoring function we use is rather simplistic. Given a query (which
looks somehow identical to the document above) we determine for each
feature (i.e. "used methods", "used types", "overrides" etc.) a simple
matching strategy: the percentage of overlap between each query-document
feature and db-document feature. Then we simply multiply each
feature-score f_i with an individual feature-weight w_i and sum it all
up into one overall score.
My questions are: Is it meaningful to use Lucene here in this setup- or
put different - can I implement that scoring scheme with Lucene easily?
How would such a solution look like? By just subclassing Scorer?
Many thanks in advance for advice
All the best,
Marcel
Re: Using Lucene with a rather simplistic scoring system?
Posted by Erick Erickson <er...@gmail.com>.
I think CustomScoreQuery might be what you're looking for....
HTH
Erick
On Fri, Jun 11, 2010 at 12:12 PM, Marcel Bruch <br...@cs.tu-darmstadt.de>wrote:
> Hi Ian,
>
> Am 11.06.2010 15:51, schrieb Ian Lea:
>
> Others can comment on how to customize scoring, but I wonder if
>> lucene's default scoring might do the job as is.
>>
>>
> I agree. It would probably work out of the box with lucene's default
> scoring. Actually, I'm getting my feet wet with lucene using just the
> WhitespaceAnalyser and the schema you proposed. However, we want to
> experiment with different scoring functions, features and feature weights -
> thus, need full control over the scoring...
>
> [...]
>
> Sounds like an interesting project.
>>
>>
> I think it is :-) It's tightly integrated into Eclipse where (i) it grabs
> your code of your current editor on demand, (ii) automatically creates and
> submits a query from it, and (iii) displays the best code examples directly
> aside your editor in a separate view. As soon as scoring works with Lucene
> you can test the first version in a week or two.
>
> However, can I implement this scoring function with Lucene:
> score(d,q) = \sum_{i \in I} w_i * f_i(d,q)
>
> Thanks,
> Marcel
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Re: Using Lucene with a rather simplistic scoring system?
Posted by Marcel Bruch <br...@cs.tu-darmstadt.de>.
Hi all.
to close my thread:
My requirement was to build a simple scoring system that basically
reuses Lucene's index infrastructure but not its advanced scoring
system, i.e., I had to replace the query and scorer infrastructure with
my own implementation.
In detail I had to come up with my own versions of:
* ExamplesSearchQuery extends Query
* ExamplesSearchWeight extends Weight
* ExamplesScorer extends Scorer
The usage of this API looks similar to standard Lucene usage:
ExamplesSearchQuery query = LuceneQueryUtil.toCodeSearchQuery(request);
IndexSearcher searcher = new IndexSearcher(luceneIndexReader);
TopDocs search = searcher.search(query, 15);
...
To accomplish my scoring function, the scorer delegates to several
subscorers similar to the BooleanScorer and sums up the sub-scores to
build the final score.
Since yesterday, the first prototype based on Lucene is available for
download and a blog-post showing the Eclipse integration in action is
available here:
http://code-recommenders.blogspot.com/2010/07/why-is-google-codesearch-not-google-for.html
- and I would be glad to take your comments on the post :-)
Thanks again for your help on Lucene,
Marcel
On 11.06.2010 18:12, wrote Marcel Bruch:
> [...]
>> Sounds like an interesting project.
> I think it is :-) It's tightly integrated into Eclipse where (i) it
> grabs your code of your current editor on demand, (ii) automatically
> creates and submits a query from it, and (iii) displays the best code
> examples directly aside your editor in a separate view. As soon as
> scoring works with Lucene you can test the first version in a week or
> two.
>
> However, can I implement this scoring function with Lucene:
> score(d,q) = \sum_{i \in I} w_i * f_i(d,q)
Re: Using Lucene with a rather simplistic scoring system?
Posted by Marcel Bruch <br...@cs.tu-darmstadt.de>.
Hi Ian,
Am 11.06.2010 15:51, schrieb Ian Lea:
> Others can comment on how to customize scoring, but I wonder if
> lucene's default scoring might do the job as is.
>
I agree. It would probably work out of the box with lucene's default
scoring. Actually, I'm getting my feet wet with lucene using just the
WhitespaceAnalyser and the schema you proposed. However, we want to
experiment with different scoring functions, features and feature
weights - thus, need full control over the scoring...
[...]
> Sounds like an interesting project.
>
I think it is :-) It's tightly integrated into Eclipse where (i) it
grabs your code of your current editor on demand, (ii) automatically
creates and submits a query from it, and (iii) displays the best code
examples directly aside your editor in a separate view. As soon as
scoring works with Lucene you can test the first version in a week or two.
However, can I implement this scoring function with Lucene:
score(d,q) = \sum_{i \in I} w_i * f_i(d,q)
Thanks,
Marcel
Re: Using Lucene with a rather simplistic scoring system?
Posted by Ian Lea <ia...@gmail.com>.
Others can comment on how to customize scoring, but I wonder if
lucene's default scoring might do the job as is.
If you've got a document in the index (simple translation from your JSON)
class: my.ExampleClass
extends: the.SuperClass
overrides: the.SuperClass.method1() the.SuperClass.method2()
used types: a.Type1 a.Type2
used methods: a.Type1.method32() a.Type1.method23()
then sample queries
"extends: the.SuperClass"
"extends: the.SuperClass overrides: the.SuperClass.method1()"
would both match, but the second one should score higher because it
matches more terms. The weighting could be done by boosts e.g. if you
care more about overrides
"extends: the.SuperClass overrides: the.SuperClass.method1()^2"
Whatever you do you'll need to play with analyzers if need to keep the
dots and brackets and case-sensitivity. And will need to make sure
you've got the right must/should/and/or logic in place.
Good luck. Sounds like an interesting project.
--
Ian.
On Fri, Jun 11, 2010 at 2:35 PM, Marcel Bruch <br...@cs.tu-darmstadt.de> wrote:
> Hi!
>
> We are working on an experimental code-search engine that helps users to
> find example code snippets based on what a developer already typed inside
> her editor. Our “homemade search engine” produces some cool results but its
> performance is somehow limited :-) Thus, we are evaluating whether Lucene
> can solve our performance issues. However, we are not familiar with Lucene
> and I wonder if some of you could help me to learn whether Lucene fits our
> problem well. Thanks in advance for your comments.
>
> The situation is as follows. For each source code file we extract some code
> properties like which types are used inside the code, which methods are
> overridden or which methods are called inside a method body etc. For each
> source code file we get a JSON structure similar to this:
> {
> “class” : my.ExampleClass
> “extends” : the.SuperClass
> “overrides” :
> - the.SuperClass.method1()
> - the.SuperClass.method2()
> “used types”:
> - a.Type1
> - a.Type2
> - ...
> “used methods”:
> - a.Type1.method32()
> - a.Type1.method23()
> - ...
> <few more things>
> }
> The scoring function we use is rather simplistic. Given a query (which looks
> somehow identical to the document above) we determine for each feature (i.e.
> “used methods”, “used types”, “overrides” etc.) a simple matching strategy:
> the percentage of overlap between each query-document feature and
> db-document feature. Then we simply multiply each feature-score f_i with an
> individual feature-weight w_i and sum it all up into one overall score.
>
> My questions are: Is it meaningful to use Lucene here in this setup- or put
> different - can I implement that scoring scheme with Lucene easily? How
> would such a solution look like? By just subclassing Scorer?
>
> Many thanks in advance for advice
>
> All the best,
> Marcel
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org