You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Peyman Faratin <pe...@gmail.com> on 2015/07/14 09:06:56 UTC
FunctionQuery of FloatFieldSource (Lucene 5.0)
Hi
I am having problems accessing float values in a lucene 5.0 index via the
functionquery.
My setup is as follows
Indexing time
------------------
Document doc = new Document();
FieldType f = new FieldType();
f.setStored(false);
f.setNumericType(NumericType.FLOAT);
f.setDocValuesType(DocValuesType.NUMERIC);
f.setNumericPrecisionStep(4);
f.setIndexOptions(IndexOptions.DOCS);
for(Entry<Integer, Float> component:vector.entrySet()) {
String w = component.getKey().toString();
Float score = component.getValue();
doc.add(new FloatField(w, score, f));
}
writer.addDocument(doc);
At end of indexing I do
writer.forceMerge(1);
writer.close();
Search Time
------------------
for(Entry<Integer,Float> vector:vectors.entrySet())
{
w = vector.getKey().toString();
Float score = (Float) vector.getValue();
Query tq= NumericRangeQuery.newFloatRange(w, 0.0f, 100.0f, true, true
);
FunctionQuery fq = new FunctionQuery( new FloatFieldSource(w) );
CustomScoreQuery customQ = new My_CustomScorerQuery(tq, fq,score);
TopDocs topDocs = indexSearcher.search(customQ,10000);
}
where My_CustomScorerQuery() is defined as follows:
public class My_CustomScorerQuery extends CustomScoreQuery{
public My_CustomScorerQuery(Query mainQuery,FunctionQuery valSrcQuery,Float
mainQueryScore) {
super(mainQuery,valSrcQuery);
this.mainQueryScore = mainQueryScore;
}
public CustomScoreProvider getCustomScoreProvider(LeafReaderContext r) {
return new My_CustomScorer(r);
}
private class My_CustomScorer extends CustomScoreProvider{
public My_CustomScorer(LeafReaderContext context) {
super(context);
}
public float customScore(int doc,float subQueryScore, float valSrcScore) {
System.out.println("\thit lucene docID: "+doc+
"\n\tquery score: "+mainQueryScore+
"\n\tsubQueryScore: "+subQueryScore+
"\n\tvalSrcScore: "+valSrcScore);
return (float) (mainQueryScore * valSrcScore);
}
}
}
The problem I am seeing is that the `valSrcScore` is always 0, and
sometimes disappears if I change the "setNumericPrecisionStep" above 4. I
am indexing the following 2 docs
Map<Integer, Float> doc1 = new LinkedHashMap<Integer, Float>();
doc1.put(12,0.5f);
doc1.put(18,0.4f);
doc1.put(10,0.1f);
indexer.indexVector("doc1", doc1);
Map<Integer, Float> doc2 = new LinkedHashMap<Integer, Float>();
doc2.put(10,0.9f);
doc2.put(1,0.8f);
doc2.put(9,0.2f);
doc2.put(2,0.1f);
and testing with the following query:
Map<Integer, Float> query = new LinkedHashMap<Integer, Float>();
query.put(10,0.8f);
query.put(9,0.6f);
query.put(2,0.01f);
So field `10` in the query should have the following total scores for the
two documents in the index
score(query,doc0) = 0.8*0.1
score(query,doc1) = 0.8*0.9
but I only see
score(query,doc0) = 0.8*0.0
score(query,doc1) = 0.8*0.0
i.e. FloatFieldSource is always returning 0. If I subclass
"FloatFieldSource" then accessing
NumericDocValues arr = DocValues.getNumeric(readerContext.reader(), field);
tells me "NumericDocValues of doc0: 0" which _seems_ to suggest indexing
does not contain the docvalues? I can see the docs fine in Luke. There is a
subtle nuance (related to the way I am indexing the fields --> some fields
in a doc are "not present" and some are).
Any pointers would be much appreciated
Peyman
Re: FunctionQuery of FloatFieldSource (Lucene 5.0)
Posted by Upayavira <uv...@odoko.co.uk>.
You should ask this question on the java-user@lucene.apache.org mailing
list - you will likely find more people there who can help you.
Also note, I don't think that you can use doc values on triefields.
Upayavira
On Tue, Jul 14, 2015, at 08:06 AM, Peyman Faratin wrote:
> Hi
>
> I am having problems accessing float values in a lucene 5.0 index via
> the functionquery.
>
> My setup is as follows
>
> Indexing time
> ------------------
> Documentdoc = newDocument();
> FieldTypef = newFieldType();
> f.setStored(false);
> f.setNumericType(NumericType.FLOAT);
> f.setDocValuesType(DocValuesType.NUMERIC);
> f.setNumericPrecisionStep(4);
> f.setIndexOptions(IndexOptions.DOCS);
>
> for(Entry<Integer, Float> component:vector.entrySet()) {
> String w� = component.getKey().toString();
> Float score = component.getValue();
>
> doc.add(new FloatField(w, score, f));
> }
> writer.addDocument(doc);
> At end of indexing I do
>
> writer.forceMerge(1);
>
> writer.close();
>
> Search Time
> ------------------
> for(Entry<Integer,Float> vector:vectors.entrySet())
> {
> w�= vector.getKey().toString();
> Float score = (Float) vector.getValue();
> Query �tq= NumericRangeQuery.newFloatRange(w, 0.0f, 100.0f,
> true, true);
> FunctionQuery �fq = newFunctionQuery( newFloatFieldSource(w) );
> CustomScoreQuery �customQ = new�My_CustomScorerQuery(tq, fq,score);
> TopDocs topDocs = indexSearcher.search(customQ,10000);
> }
> where My_CustomScorerQuery() is defined as follows:
> public class�My_CustomScorerQuery�extends CustomScoreQuery{
> public�My_CustomScorerQuery(Query mainQuery,FunctionQuery
> valSrcQuery,Float mainQueryScore) {
> super(mainQuery,valSrcQuery);
> this.mainQueryScore = mainQueryScore;
> }
> public CustomScoreProvider
> getCustomScoreProvider(LeafReaderContext r) {
> return new�My_CustomScorer(r);
> }
> private class�My_CustomScorer�extends CustomScoreProvider{
> public�My_CustomScorer(LeafReaderContext context) {
> super(context);
> }
> public float customScore(int doc,float subQueryScore, float
> valSrcScore) {
> System.out.println("\thit lucene docID: "+doc+
> "\n\tquery score: "+mainQueryScore+
> "\n\tsubQueryScore: "+subQueryScore+
> "\n\tvalSrcScore: "+valSrcScore);
>
> return (float) (mainQueryScore * valSrcScore);
> }
> }
> }
> The problem I am seeing is that the `valSrcScore` is always 0, and
> sometimes disappears if I change the "setNumericPrecisionStep" above
> 4. I am indexing the following 2 docs
> Map<Integer, Float> doc1 = newLinkedHashMap<Integer, Float>();
> doc1.put(12,0.5f);
> doc1.put(18,0.4f);
> doc1.put(10,0.1f);
>
> indexer.indexVector("doc1", doc1);
>
> Map<Integer, Float> doc2 = newLinkedHashMap<Integer, Float>();
> doc2.put(10,0.9f);
> doc2.put(1,0.8f);
> doc2.put(9,0.2f);
>
> doc2.put(2,0.1f);
>
> and testing with the following query:
> Map<Integer, Float> query = newLinkedHashMap<Integer, Float>();
> query.put(10,0.8f);
> query.put(9,0.6f);
>
> query.put(2,0.01f);
>
> So field `10` in the query should have the following total scores for
> the two documents in the index
> score(query,doc0) = 0.8*0.1
> score(query,doc1) = 0.8*0.9
>
> but I only see
> score(query,doc0) = 0.8*0.0
> score(query,doc1) = 0.8*0.0
>
> i.e. FloatFieldSource is always returning 0. If I subclass
> "FloatFieldSource" then accessing
>
> NumericDocValues arr =
> DocValues.getNumeric(readerContext.reader(), field);
>
> tells me "NumericDocValues of doc0: 0" �which _seems_ to suggest
> indexing does not contain the docvalues? I can see the docs fine in
> Luke. There is a subtle nuance (related to the way I am indexing the
> fields --> some fields in a doc are "not present" and some are).
>
> Any pointers would be much appreciated
>
> Peyman
>
>
>
>