You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "michael.boom" <my...@yahoo.com> on 2014/01/13 09:27:46 UTC
Simple payloads example not working
Hi,
I'm trying to test payloads in Solr
Using solr 4.6.0 and the example configuration, i posted 3 docs to solr:
<add>
<doc>
<field name="id">1</field>
<field name="title">Doc one</field>
<field name="payloads">testone|100 testtwo|30 testthree|5</field>
<field name="text">I testone, you testtwo, they testthree</field>
</doc>
<doc>
<field name="id">2</field>
<field name="title">Doc two</field>
<field name="payloads">testone|30 testtwo|200 testthree|5</field>
<field name="text">I testone, you testtwo, they testthree</field>
</doc>
<doc>
<field name="id">3</field>
<field name="title">Doc three</field>
<field name="payloads">testone|5 testtwo|100 testthree|300</field>
<field name="text">I testone, you testtwo, they testthree</field>
</doc>
</add>
Then in the Admin UI i queryed:
http://localhost:8983/solr/collection1/select?q=text%3Atestone&wt=json&indent=true
The result was:
{
"responseHeader":{
"status":0,
"QTime":0,
"params":{
"indent":"true",
"q":"text:testone",
"wt":"json"}},
"response":{"numFound":3,"start":0,"docs":[
{
"id":"2",
"title":["Doc two"],
"payloads":"testone|30 testtwo|200 testthree|5",
"_version_":1457102453306556416},
{
"id":"3",
"title":["Doc three"],
"payloads":"testone|5 testtwo|100 testthree|300",
"_version_":1457102453657829376},
{
"id":"1",
"title":["Doc one"],
"payloads":"testone|100 testtwo|30 testthree|5",
"_version_":1457102486106013696}]
}}
So, doc two has the biggest score although i gave doc one the biggest
payload for term "testone".
I am missing something here or is there a bug?
Thanks!
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Simple payloads example not working
Posted by "michael.boom" <my...@yahoo.com>.
Thanks Eric,
I did create a custom query parser, which seems to work just fine. My only
problem now is the one above, with all docs having the same score for some
reason.
See below the query parserL
import org.apache.commons.lang.StringUtils;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.BooleanClause.Occur;
import org.apache.lucene.search.payloads.AveragePayloadFunction;
import org.apache.lucene.search.payloads.PayloadTermQuery;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.search.QParser;
import org.apache.solr.search.QParserPlugin;
import org.apache.solr.search.SyntaxError;
/**
* Parser plugin to parse payload queries.
*/
public class PayloadQParserPlugin extends QParserPlugin {
@Override
public QParser createParser(String qstr, SolrParams localParams,
SolrParams params, SolrQueryRequest req) {
return new PayloadQParser(qstr, localParams, params, req);
}
@Override
public void init(NamedList args) {
}
}
class PayloadQParser extends QParser {
public PayloadQParser(String qstr, SolrParams localParams, SolrParams
params,
SolrQueryRequest req) {
super(qstr, localParams, params, req);
}
@Override
public Query parse() throws SyntaxError {
BooleanQuery q = new BooleanQuery();
String[] nvps = StringUtils.split(qstr, " ");
for (int i = 0; i < nvps.length; i++) {
String[] nv = StringUtils.split(nvps[i], ":");
if (nv[0].startsWith("+")) {
q.add(new PayloadTermQuery(new Term(nv[0].substring(1), nv[1]),
new AveragePayloadFunction(), false), Occur.MUST);
} else {
q.add(new PayloadTermQuery(new Term(nv[0], nv[1]),
new AveragePayloadFunction(), false), Occur.SHOULD);
}
}
return q;
}
}
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998p4111050.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Simple payloads example not working
Posted by Erik Hatcher <er...@gmail.com>.
There’s also the PayloadTermQueryParser here <https://issues.apache.org/jira/browse/SOLR-1485> (which is surely a bit out of date as well, but maybe a bit simpler starting point).
Erik
On Jan 13, 2014, at 12:30 AM, michael.boom <my...@yahoo.com> wrote:
> Actually, i just checked the debugQuery output: they all have the same score:
> "explain": {
> "1": "\n0.24276763 = (MATCH) weight(text:testone in 0)
> [DefaultSimilarity], result of:\n 0.24276763 = fieldWeight in 0, product
> of:\n 1.0 = tf(freq=1.0), with freq of:\n 1.0 = termFreq=1.0\n
> 0.7768564 = idf(docFreq=4, maxDocs=4)\n 0.3125 = fieldNorm(doc=0)\n",
> "2": "\n0.24276763 = (MATCH) weight(text:testone in 1)
> [DefaultSimilarity], result of:\n 0.24276763 = fieldWeight in 1, product
> of:\n 1.0 = tf(freq=1.0), with freq of:\n 1.0 = termFreq=1.0\n
> 0.7768564 = idf(docFreq=4, maxDocs=4)\n 0.3125 = fieldNorm(doc=1)\n",
> "3": "\n0.24276763 = (MATCH) weight(text:testone in 2)
> [DefaultSimilarity], result of:\n 0.24276763 = fieldWeight in 2, product
> of:\n 1.0 = tf(freq=1.0), with freq of:\n 1.0 = termFreq=1.0\n
> 0.7768564 = idf(docFreq=4, maxDocs=4)\n 0.3125 = fieldNorm(doc=2)\n"
> },
>
> No payload seems to be considered in score calculation - do i need to use a
> special query handler?
>
>
>
> -----
> Thanks,
> Michael
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998p4110999.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Simple payloads example not working
Posted by "michael.boom" <my...@yahoo.com>.
Hi Ahmet,
Yes, I did, also tried various scenarios with the same outcome. I used the
stock example, with minimum customization ( custom similarity and query
parser ).
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998p4111324.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Simple payloads example not working
Posted by Ahmet Arslan <io...@yahoo.com>.
Hi Michael
Did you re-index after you register your custom similarity?
Ahmet
On Tuesday, January 14, 2014 4:36 PM, michael.boom <my...@yahoo.com> wrote:
Hi Markus,
Do you have any example/tutorials of your payloads in custom filter
implementation ?
I really want to get payloads working, in any way.
Thanks!
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998p4111244.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Simple payloads example not working
Posted by "michael.boom" <my...@yahoo.com>.
Hi Markus,
Do you have any example/tutorials of your payloads in custom filter
implementation ?
I really want to get payloads working, in any way.
Thanks!
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998p4111244.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Simple payloads example not working
Posted by "michael.boom" <my...@yahoo.com>.
Yes, it's float:
<filter class="solr.DelimitedPayloadTokenFilterFactory" encoder="float"
delimiter="|"/>
The scenario is simple to replicate - default solr-4.6.0 example, with a
custom Similarity class (the one above) and a custom queryparser (again,
listed above).
I posted the docs in XML format (docs also listed above) using
exampledocs/post.sh utility.
Indeed it looks weird, and can't explain it.
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998p4111219.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Simple payloads example not working
Posted by Markus Jelsma <ma...@openindex.io>.
Strange, is it really floats you are inserting as payload? We use payloads too but write them via PayloadAttribute in custom token filters as float.
-----Original message-----
> From:michael.boom <my...@yahoo.com>
> Sent: Tuesday 14th January 2014 11:59
> To: solr-user@lucene.apache.org
> Subject: RE: Simple payloads example not working
>
> Investigating, it looks that the payload.bytes property is where the problem
> is.
> payload.toString() outputs corrects values, but .bytes property seems to
> behave a little weird:
> public class CustomSimilarity extends DefaultSimilarity {
>
> @Override
> public float scorePayload(int doc, int start, int end, BytesRef payload)
> {
> if (payload != null) {
> Float pscore = PayloadHelper.decodeFloat(payload.bytes);
> System.out.println("payload : " + payload.toString() + ",
> payload bytes: " + payload.bytes.toString() + ", decoded value is " +
> pscore);
> return pscore;
> }
> return 1.0f;
> }
> }
>
> outputs on query:
> http://localhost:8983/solr/collection1/pds-search?q=payloads:testone&wt=json&indent=true&debugQuery=true
>
> payload : [41 26 66 66], payload bytes: [B@149c678, decoded value is 10.4
> payload : [41 f0 0 0], payload bytes: [B@149c678, decoded value is 10.4
> payload : [42 4a cc cd], payload bytes: [B@149c678, decoded value is 10.4
> payload : [42 c6 0 0], payload bytes: [B@149c678, decoded value is 10.4
> payload : [41 26 66 66], payload bytes: [B@850fb7, decoded value is 10.4
> payload : [41 f0 0 0], payload bytes: [B@1cad357, decoded value is 10.4
> payload : [42 4a cc cd], payload bytes: [B@f922cf, decoded value is 10.4
> payload : [42 c6 0 0], payload bytes: [B@5c4dc4, decoded value is 10.4
>
>
> Something doesn't seem right here. Any idea why this behaviour?
> Is anyone using payloads using Solr 4.6.0 ?
>
>
>
>
> -----
> Thanks,
> Michael
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998p4111214.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
RE: Simple payloads example not working
Posted by "michael.boom" <my...@yahoo.com>.
Investigating, it looks that the payload.bytes property is where the problem
is.
payload.toString() outputs corrects values, but .bytes property seems to
behave a little weird:
public class CustomSimilarity extends DefaultSimilarity {
@Override
public float scorePayload(int doc, int start, int end, BytesRef payload)
{
if (payload != null) {
Float pscore = PayloadHelper.decodeFloat(payload.bytes);
System.out.println("payload : " + payload.toString() + ",
payload bytes: " + payload.bytes.toString() + ", decoded value is " +
pscore);
return pscore;
}
return 1.0f;
}
}
outputs on query:
http://localhost:8983/solr/collection1/pds-search?q=payloads:testone&wt=json&indent=true&debugQuery=true
payload : [41 26 66 66], payload bytes: [B@149c678, decoded value is 10.4
payload : [41 f0 0 0], payload bytes: [B@149c678, decoded value is 10.4
payload : [42 4a cc cd], payload bytes: [B@149c678, decoded value is 10.4
payload : [42 c6 0 0], payload bytes: [B@149c678, decoded value is 10.4
payload : [41 26 66 66], payload bytes: [B@850fb7, decoded value is 10.4
payload : [41 f0 0 0], payload bytes: [B@1cad357, decoded value is 10.4
payload : [42 4a cc cd], payload bytes: [B@f922cf, decoded value is 10.4
payload : [42 c6 0 0], payload bytes: [B@5c4dc4, decoded value is 10.4
Something doesn't seem right here. Any idea why this behaviour?
Is anyone using payloads using Solr 4.6.0 ?
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998p4111214.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Simple payloads example not working
Posted by "michael.boom" <my...@yahoo.com>.
Correction:
I observed a pattern, the returned score is the same for all docs and equals
with the payload of the term in the first doc:
http://localhost:8983/solr/collection1/pds-search?q=payloads:testone&wt=json&indent=true&debugQuery=true
--->
"explain":{
"1":"\n15.4 = (MATCH) btq(includeSpanScore=false), result of:\n 15.4
= AveragePayloadFunction.docScore()\n",
"2":"\n15.4 = (MATCH) btq(includeSpanScore=false), result of:\n 15.4
= AveragePayloadFunction.docScore()\n",
"3":"\n15.4 = (MATCH) btq(includeSpanScore=false), result of:\n 15.4
= AveragePayloadFunction.docScore()\n",
"4":"\n15.4 = (MATCH) btq(includeSpanScore=false), result of:\n 15.4
= AveragePayloadFunction.docScore()\n"},
http://localhost:8983/solr/collection1/pds-search?q=payloads:testthree&wt=json&indent=true&debugQuery=true
"explain":{
"1":"\n5.0 = (MATCH) btq(includeSpanScore=false), result of:\n 5.0 =
AveragePayloadFunction.docScore()\n",
"2":"\n5.0 = (MATCH) btq(includeSpanScore=false), result of:\n 5.0 =
AveragePayloadFunction.docScore()\n",
"3":"\n5.0 = (MATCH) btq(includeSpanScore=false), result of:\n 5.0 =
AveragePayloadFunction.docScore()\n",
"4":"\n5.0 = (MATCH) btq(includeSpanScore=false), result of:\n 5.0 =
AveragePayloadFunction.docScore()\n"},
Any clue why is this happening?
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998p4111060.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Simple payloads example not working
Posted by "michael.boom" <my...@yahoo.com>.
Thanks, that indeed fixed the problem.
Now i've created a custom Similarity class and used it in schema.xml.
Problem is now that for all docs the calculated payload score is the same:
public class CustomSolrSimilarity extends DefaultSimilarity {
@Override
public float scorePayload(int doc, int start, int end, BytesRef payload)
{
if (payload != null) {
Float pscore = PayloadHelper.decodeFloat(payload.bytes);
System.out.println("payload is: " + payload.toString() + " with
score: " + Float.toString(pscore));
return pscore;
}
return 1.0f;
}
}
Output log:
payload is: [41 26 66 66] with score: 10.4
payload is: [41 f0 0 0] with score: 10.4
payload is: [42 4a cc cd] with score: 10.4
payload is: [42 c6 0 0] with score: 10.4
payload is: [41 26 66 66] with score: 10.4
payload is: [41 f0 0 0] with score: 10.4
payload is: [42 4a cc cd] with score: 10.4
payload is: [42 c6 0 0] with score: 10.4
Any idea why is it always the same ?
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998p4111045.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Simple payloads example not working
Posted by Markus Jelsma <ma...@openindex.io>.
Check the bytes property:
http://lucene.apache.org/core/4_0_0/core/org/apache/lucene/util/BytesRef.html#bytes
@Override
public float scorePayload(int doc, int start, int end, BytesRef payload) {
if (payload != null) {
return PayloadHelper.decodeFloat(payload.bytes);
}
return 1.0f;
}
-----Original message-----
> From:michael.boom <my...@yahoo.com>
> Sent: Monday 13th January 2014 14:49
> To: solr-user@lucene.apache.org
> Subject: Re: Simple payloads example not working
>
> Thanks iorixxx,
>
> Actually I've just tried it and I hit a small wall, the tutorial looks not
> to be up to date with the codebase.
> When implementing my custom similarity class i should be using
> PayloadHelper, but following happens:
>
> in PayloadHelper:
> public static final float decodeFloat(byte [] bytes, int offset)
>
> in DefaultSimilarity:
> public float scorePayload(int doc, int start, int end, BytesRef payload)
>
> So it's BytesRef vs byte[].
> How should i proceed in this scenario?
>
>
>
> -----
> Thanks,
> Michael
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998p4111040.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Simple payloads example not working
Posted by "michael.boom" <my...@yahoo.com>.
Thanks iorixxx,
Actually I've just tried it and I hit a small wall, the tutorial looks not
to be up to date with the codebase.
When implementing my custom similarity class i should be using
PayloadHelper, but following happens:
in PayloadHelper:
public static final float decodeFloat(byte [] bytes, int offset)
in DefaultSimilarity:
public float scorePayload(int doc, int start, int end, BytesRef payload)
So it's BytesRef vs byte[].
How should i proceed in this scenario?
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998p4111040.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Simple payloads example not working
Posted by Ahmet Arslan <io...@yahoo.com>.
Hi Michael,
To make payloads to be considered in score calculation you need two more things:
1) A custom similarity
2) A query parser that produces Payload*Query family.
This blog post can be a good start point.
http://digitalpebble.blogspot.com/2010/08/using-payloads-with-dismaxqparser-in.html
On Monday, January 13, 2014 10:37 AM, michael.boom <my...@yahoo.com> wrote:
Actually, i just checked the debugQuery output: they all have the same score:
"explain": {
"1": "\n0.24276763 = (MATCH) weight(text:testone in 0)
[DefaultSimilarity], result of:\n 0.24276763 = fieldWeight in 0, product
of:\n 1.0 = tf(freq=1.0), with freq of:\n 1.0 = termFreq=1.0\n
0.7768564 = idf(docFreq=4, maxDocs=4)\n 0.3125 = fieldNorm(doc=0)\n",
"2": "\n0.24276763 = (MATCH) weight(text:testone in 1)
[DefaultSimilarity], result of:\n 0.24276763 = fieldWeight in 1, product
of:\n 1.0 = tf(freq=1.0), with freq of:\n 1.0 = termFreq=1.0\n
0.7768564 = idf(docFreq=4, maxDocs=4)\n 0.3125 = fieldNorm(doc=1)\n",
"3": "\n0.24276763 = (MATCH) weight(text:testone in 2)
[DefaultSimilarity], result of:\n 0.24276763 = fieldWeight in 2, product
of:\n 1.0 = tf(freq=1.0), with freq of:\n 1.0 = termFreq=1.0\n
0.7768564 = idf(docFreq=4, maxDocs=4)\n 0.3125 = fieldNorm(doc=2)\n"
},
No payload seems to be considered in score calculation - do i need to use a
special query handler?
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998p4110999.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Simple payloads example not working
Posted by "michael.boom" <my...@yahoo.com>.
Actually, i just checked the debugQuery output: they all have the same score:
"explain": {
"1": "\n0.24276763 = (MATCH) weight(text:testone in 0)
[DefaultSimilarity], result of:\n 0.24276763 = fieldWeight in 0, product
of:\n 1.0 = tf(freq=1.0), with freq of:\n 1.0 = termFreq=1.0\n
0.7768564 = idf(docFreq=4, maxDocs=4)\n 0.3125 = fieldNorm(doc=0)\n",
"2": "\n0.24276763 = (MATCH) weight(text:testone in 1)
[DefaultSimilarity], result of:\n 0.24276763 = fieldWeight in 1, product
of:\n 1.0 = tf(freq=1.0), with freq of:\n 1.0 = termFreq=1.0\n
0.7768564 = idf(docFreq=4, maxDocs=4)\n 0.3125 = fieldNorm(doc=1)\n",
"3": "\n0.24276763 = (MATCH) weight(text:testone in 2)
[DefaultSimilarity], result of:\n 0.24276763 = fieldWeight in 2, product
of:\n 1.0 = tf(freq=1.0), with freq of:\n 1.0 = termFreq=1.0\n
0.7768564 = idf(docFreq=4, maxDocs=4)\n 0.3125 = fieldNorm(doc=2)\n"
},
No payload seems to be considered in score calculation - do i need to use a
special query handler?
-----
Thanks,
Michael
--
View this message in context: http://lucene.472066.n3.nabble.com/Simple-payloads-example-not-working-tp4110998p4110999.html
Sent from the Solr - User mailing list archive at Nabble.com.