You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Shawn Heisey <so...@elyograg.org> on 2010/04/12 19:55:42 UTC
dismax and date boosts
I am trying to boost relevancy based on a date field with dismax, and
I've included the requestHandler config below. The post_date field in
my database is simple UNIX time, seconds since epoch. It's in a MySQL
bigint field, so I've stored it as a tlong in Solr. This filed is
required by our application in integer format.
You can see a boost function commented out below that I tried to use,
which should multiply the post_date by 1000 to match the Solr way of
milliseconds since epoch, then use a reciprocal example similar to one I
found on the Solr Wiki. When I uncomment it, I get an error message:
Problem accessing /solr/ncmain/select. Reason:
undefined field product(post_date
Lance Norskog mentioned that boost functions are slow, and suggested
that I use something like the following query, which I am willing to
try, but I first have to get my "seconds since epoch" into a format that
Solr can use. So, I need two things that I cannot figure out how to
do. One is schema.xml (or DIH, maybe?) config to convert the date
format into an additional field, the other is exactly what to put in the
dismax requestHandler to automatically include the boost.
[NOW-6MONTHS TO NOW]^5.0 ,
[NOW-1YEARS TO NOW-6MONTHS]^3.0
[NOW-2YEARS TO NOW-1YEARS]^2.0
[* TO NOW-2YEARS]^1.0
My current dismax requestHandler, some of which is still a work in progress:
<requestHandler name="dismax" class="solr.SearchHandler">
<lst name="defaults">
<str name="defType">dismax</str>
<bool name="partialResults">true</bool>
<int name="timeAllowed">6000</int>
<str name="echoParams">explicit</str>
<int name="rows">50</int>
<str
name="shards">idxinc:8983/solr/live,idxst0-b:8983/solr/live,idxst1-b:8983/solr/live,idxst2-b:8983/solr/live,idxst3-b:8983/solr/live,idxst4-b:8983/solr/live,idxst5-b:8983/solr/live</str>
<float name="tie">0.1</float>
<str name="qf">doc_id^5 text_preview^1.25 ft_text keywords^0.75
ft_dname^1.25</str>
<str name="pf">doc_id^5 text_preview^1.25 ft_text keywords^0.75
ft_dname^1.25</str>
<!--
<str name="bf">recip(ms(NOW,product(post_date,1000)),3.17e-11,1,1)</str>
-->
<str name="fl">score,*</str>
<str name="mm">0<-1 4<-50%</str>
<str name="q.alt">*:*</str>
</lst>
</requestHandler>
Re: dismax and date boosts
Posted by Shawn Heisey <so...@elyograg.org>.
I have finally figured out how to turn this off in Thunderbird 3:
Go to Tools, Options, Display, and turn off "Display emoticons as
graphics".
On 4/12/2010 12:04 PM, Shawn Heisey wrote:
> On 4/12/2010 11:55 AM, Shawn Heisey wrote:
>> [NOW-6MONTHS TO NOW]^5.0 ,
>> [NOW-1YEARS TO NOW-6MONTHS]^3.0
>> [NOW-2YEARS TO NOW-1YEARS]^2.0
>> [* TO NOW-2YEARS]^1.0
>
> And here we have the perfect example of something I mentioned a while
> ago - my Thunderbird (v3.0.4 on Win7) turning Solr boost syntax into
> pretty exponents. Attached a cropped screenshot. If anyone has any
> idea how to turn this off, I would appreciate knowing how.
>
> Shawn
>
Re: dismax and date boosts
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/12/2010 11:55 AM, Shawn Heisey wrote:
> [NOW-6MONTHS TO NOW]^5.0 ,
> [NOW-1YEARS TO NOW-6MONTHS]^3.0
> [NOW-2YEARS TO NOW-1YEARS]^2.0
> [* TO NOW-2YEARS]^1.0
And here we have the perfect example of something I mentioned a while
ago - my Thunderbird (v3.0.4 on Win7) turning Solr boost syntax into
pretty exponents. Attached a cropped screenshot. If anyone has any
idea how to turn this off, I would appreciate knowing how.
Shawn
Re: dismax and date boosts
Posted by Shawn Heisey <so...@elyograg.org>.
I found what I believe is a better option even if the multiplication
would work - FROM_UNIXTIME. That returns the same kind of output as you
get from an actual database date field.
On 4/20/2010 12:07 PM, Shawn Heisey wrote:
> So, if I have my database multiply my value by 1000, I can put that
> directly into a tdate field and it'll work as expected?
>
> If that's the case, I think I might be able to modify my query from
> "SELECT *" to "SELECT *,post_date*1000 as pdate" and add the pdate
> field to the schema as type tdate. Sound right?
>
>
> On 4/17/2010 3:00 PM, Lance Norskog wrote:
>> No, a copyField will not do the xlation from (seconds from epoch) to
>> (milliseconds from 1/1/1970). You should be able to do this with a
>> combination of functions in your database SELECT call. The major DBs
>> all have a wealth of functions that xform between numbers and dates.
>>
>> The DIH is smart about types: if the DB select returns a value as a
>> database date type, it becomes a java.sql.Date which the DIH then
>> makes a Solr 'tdate'.
>>
>> In MYSQL there's a bunch of time/date/number/string xformers:
>>
>> http://dev.mysql.com/tech-resources/articles/4.1/time.html
>>
>> It's easier to do this in the database, than in the DIH.
>
Re: dismax and date boosts
Posted by Shawn Heisey <so...@elyograg.org>.
So, if I have my database multiply my value by 1000, I can put that
directly into a tdate field and it'll work as expected?
If that's the case, I think I might be able to modify my query from
"SELECT *" to "SELECT *,post_date*1000 as pdate" and add the pdate field
to the schema as type tdate. Sound right?
On 4/17/2010 3:00 PM, Lance Norskog wrote:
> No, a copyField will not do the xlation from (seconds from epoch) to
> (milliseconds from 1/1/1970). You should be able to do this with a
> combination of functions in your database SELECT call. The major DBs
> all have a wealth of functions that xform between numbers and dates.
>
> The DIH is smart about types: if the DB select returns a value as a
> database date type, it becomes a java.sql.Date which the DIH then
> makes a Solr 'tdate'.
>
> In MYSQL there's a bunch of time/date/number/string xformers:
>
> http://dev.mysql.com/tech-resources/articles/4.1/time.html
>
> It's easier to do this in the database, than in the DIH.
>
Re: dismax and date boosts
Posted by Lance Norskog <go...@gmail.com>.
No, a copyField will not do the xlation from (seconds from epoch) to
(milliseconds from 1/1/1970). You should be able to do this with a
combination of functions in your database SELECT call. The major DBs
all have a wealth of functions that xform between numbers and dates.
The DIH is smart about types: if the DB select returns a value as a
database date type, it becomes a java.sql.Date which the DIH then
makes a Solr 'tdate'.
In MYSQL there's a bunch of time/date/number/string xformers:
http://dev.mysql.com/tech-resources/articles/4.1/time.html
It's easier to do this in the database, than in the DIH.
On 4/15/10, Shawn Heisey <so...@elyograg.org> wrote:
> It's stored in the database as a bigint, seconds since epoch, not a
> date. I need to store the integer value in Solr for the application to
> consume (done as a tlong), but I want to create a second copy that's
> tdate. That's the part I don't know how to do. If I just do a
> copyField in the schema, will that do it, or will I need to write some
> magic somewhere?
>
> Shawn
>
>
> On 4/14/2010 9:46 PM, Lance Norskog wrote:
>> Wait- it's in a database? That's much simpler.
>>
>> The select should supply the date as an SQL Date or Timestamp type.
>> The DIH will get this directly and store it as a Solr tdate. You
>> should not have to do any arithmetic or formatting of date strings.
>>
>> This may need a few layers of SQL functions.
>>
>> On 4/14/10, Shawn Heisey<so...@elyograg.org> wrote:
>>
>>> On 4/14/2010 8:12 AM, Shawn Heisey wrote:
>>>
>>>> On 4/12/2010 9:29 PM, Lance Norskog wrote:
>>>>
>>>>> During indexing: the basic Solr XmlUpdateHandler does not have a
>>>>> facility for this. In the DataImportHandler you can add Javascript
>>>>> that takes your 'seconds since epoch', adds the delta between your
>>>>> epoch and 1/1/1970, and multiplies by 1000.
>>>>>
>>>> I had hoped there was a way to tell schema.xml to do the math, but
>>>> this sounds like there isn't. If you could point me at some examples
>>>> of how to put javascript into data-config.xml, then I could probably
>>>> work out how to do the javascript that I would need. I might be able
>>>> to alter my SQL query in the DIH config to multiply the value by 1000,
>>>> but that puts all the load on the SQL server instead of utilizing the
>>>> multiple CPUs involved in my shards. I'm pretty sure that the epoch
>>>> on my dates is also 1970/01/01.
>>>>
>>>> What I'd truly like is a way to convert it directly to tdate, so all
>>>> of Solr's date capabilities become available. Is that nontrivial, or
>>>> is that perhaps what you've been already describing?
>>>>
>>>> Thanks for all the help you've given so far. I've made a lot of
>>>> progress.
>>>>
>>>>
>>>
>>>
>>
>>
>
>
--
Lance Norskog
goksron@gmail.com
Re: dismax and date boosts
Posted by Shawn Heisey <so...@elyograg.org>.
It's stored in the database as a bigint, seconds since epoch, not a
date. I need to store the integer value in Solr for the application to
consume (done as a tlong), but I want to create a second copy that's
tdate. That's the part I don't know how to do. If I just do a
copyField in the schema, will that do it, or will I need to write some
magic somewhere?
Shawn
On 4/14/2010 9:46 PM, Lance Norskog wrote:
> Wait- it's in a database? That's much simpler.
>
> The select should supply the date as an SQL Date or Timestamp type.
> The DIH will get this directly and store it as a Solr tdate. You
> should not have to do any arithmetic or formatting of date strings.
>
> This may need a few layers of SQL functions.
>
> On 4/14/10, Shawn Heisey<so...@elyograg.org> wrote:
>
>> On 4/14/2010 8:12 AM, Shawn Heisey wrote:
>>
>>> On 4/12/2010 9:29 PM, Lance Norskog wrote:
>>>
>>>> During indexing: the basic Solr XmlUpdateHandler does not have a
>>>> facility for this. In the DataImportHandler you can add Javascript
>>>> that takes your 'seconds since epoch', adds the delta between your
>>>> epoch and 1/1/1970, and multiplies by 1000.
>>>>
>>> I had hoped there was a way to tell schema.xml to do the math, but
>>> this sounds like there isn't. If you could point me at some examples
>>> of how to put javascript into data-config.xml, then I could probably
>>> work out how to do the javascript that I would need. I might be able
>>> to alter my SQL query in the DIH config to multiply the value by 1000,
>>> but that puts all the load on the SQL server instead of utilizing the
>>> multiple CPUs involved in my shards. I'm pretty sure that the epoch
>>> on my dates is also 1970/01/01.
>>>
>>> What I'd truly like is a way to convert it directly to tdate, so all
>>> of Solr's date capabilities become available. Is that nontrivial, or
>>> is that perhaps what you've been already describing?
>>>
>>> Thanks for all the help you've given so far. I've made a lot of
>>> progress.
>>>
>>>
>>
>>
>
>
Re: dismax and date boosts
Posted by Lance Norskog <go...@gmail.com>.
Wait- it's in a database? That's much simpler.
The select should supply the date as an SQL Date or Timestamp type.
The DIH will get this directly and store it as a Solr tdate. You
should not have to do any arithmetic or formatting of date strings.
This may need a few layers of SQL functions.
On 4/14/10, Shawn Heisey <so...@elyograg.org> wrote:
> On 4/14/2010 8:12 AM, Shawn Heisey wrote:
>> On 4/12/2010 9:29 PM, Lance Norskog wrote:
>>> During indexing: the basic Solr XmlUpdateHandler does not have a
>>> facility for this. In the DataImportHandler you can add Javascript
>>> that takes your 'seconds since epoch', adds the delta between your
>>> epoch and 1/1/1970, and multiplies by 1000.
>>
>> I had hoped there was a way to tell schema.xml to do the math, but
>> this sounds like there isn't. If you could point me at some examples
>> of how to put javascript into data-config.xml, then I could probably
>> work out how to do the javascript that I would need. I might be able
>> to alter my SQL query in the DIH config to multiply the value by 1000,
>> but that puts all the load on the SQL server instead of utilizing the
>> multiple CPUs involved in my shards. I'm pretty sure that the epoch
>> on my dates is also 1970/01/01.
>>
>> What I'd truly like is a way to convert it directly to tdate, so all
>> of Solr's date capabilities become available. Is that nontrivial, or
>> is that perhaps what you've been already describing?
>>
>> Thanks for all the help you've given so far. I've made a lot of
>> progress.
>>
>
>
--
Lance Norskog
goksron@gmail.com
Re: dismax and date boosts
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/14/2010 8:12 AM, Shawn Heisey wrote:
> On 4/12/2010 9:29 PM, Lance Norskog wrote:
>> During indexing: the basic Solr XmlUpdateHandler does not have a
>> facility for this. In the DataImportHandler you can add Javascript
>> that takes your 'seconds since epoch', adds the delta between your
>> epoch and 1/1/1970, and multiplies by 1000.
>
> I had hoped there was a way to tell schema.xml to do the math, but
> this sounds like there isn't. If you could point me at some examples
> of how to put javascript into data-config.xml, then I could probably
> work out how to do the javascript that I would need. I might be able
> to alter my SQL query in the DIH config to multiply the value by 1000,
> but that puts all the load on the SQL server instead of utilizing the
> multiple CPUs involved in my shards. I'm pretty sure that the epoch
> on my dates is also 1970/01/01.
>
> What I'd truly like is a way to convert it directly to tdate, so all
> of Solr's date capabilities become available. Is that nontrivial, or
> is that perhaps what you've been already describing?
>
> Thanks for all the help you've given so far. I've made a lot of
> progress.
>
Re: dismax and date boosts
Posted by Lance Norskog <go...@gmail.com>.
So, you have a user-visible field that is in 'seconds since epoch'.
You would like to index this, and the Solr date features are handy.
And, you do not have an application prepping the data to index, nor do
you have a UI that can receive Solr dates and turn them back into
'seconds since epoch'.
During indexing: the basic Solr XmlUpdateHandler does not have a
facility for this. In the DataImportHandler you can add Javascript
that takes your 'seconds since epoch', adds the delta between your
epoch and 1/1/1970, and multiplies by 1000.
During query: there are several programmable output writers, including
velocity, ruby, and xslt (in decreasing order of useability), and you
can preprocess the output with one of these. You can even create
completely different formats for the output.
On Mon, Apr 12, 2010 at 10:55 AM, Shawn Heisey <so...@elyograg.org> wrote:
> I am trying to boost relevancy based on a date field with dismax, and I've
> included the requestHandler config below. The post_date field in my
> database is simple UNIX time, seconds since epoch. It's in a MySQL bigint
> field, so I've stored it as a tlong in Solr. This filed is required by our
> application in integer format.
>
> You can see a boost function commented out below that I tried to use, which
> should multiply the post_date by 1000 to match the Solr way of milliseconds
> since epoch, then use a reciprocal example similar to one I found on the
> Solr Wiki. When I uncomment it, I get an error message:
>
> Problem accessing /solr/ncmain/select. Reason:
> undefined field product(post_date
>
> Lance Norskog mentioned that boost functions are slow, and suggested that I
> use something like the following query, which I am willing to try, but I
> first have to get my "seconds since epoch" into a format that Solr can use.
> So, I need two things that I cannot figure out how to do. One is
> schema.xml (or DIH, maybe?) config to convert the date format into an
> additional field, the other is exactly what to put in the dismax
> requestHandler to automatically include the boost.
>
> [NOW-6MONTHS TO NOW]^5.0 ,
> [NOW-1YEARS TO NOW-6MONTHS]^3.0
> [NOW-2YEARS TO NOW-1YEARS]^2.0
> [* TO NOW-2YEARS]^1.0
>
> My current dismax requestHandler, some of which is still a work in progress:
>
> <requestHandler name="dismax" class="solr.SearchHandler">
> <lst name="defaults">
> <str name="defType">dismax</str>
> <bool name="partialResults">true</bool>
> <int name="timeAllowed">6000</int>
> <str name="echoParams">explicit</str>
> <int name="rows">50</int>
> <str
> name="shards">idxinc:8983/solr/live,idxst0-b:8983/solr/live,idxst1-b:8983/solr/live,idxst2-b:8983/solr/live,idxst3-b:8983/solr/live,idxst4-b:8983/solr/live,idxst5-b:8983/solr/live</str>
> <float name="tie">0.1</float>
> <str name="qf">doc_id^5 text_preview^1.25 ft_text keywords^0.75
> ft_dname^1.25</str>
> <str name="pf">doc_id^5 text_preview^1.25 ft_text keywords^0.75
> ft_dname^1.25</str>
> <!--
> <str name="bf">recip(ms(NOW,product(post_date,1000)),3.17e-11,1,1)</str>
> -->
> <str name="fl">score,*</str>
> <str name="mm">0<-1 4<-50%</str>
> <str name="q.alt">*:*</str>
> </lst>
> </requestHandler>
>
>
--
Lance Norskog
goksron@gmail.com