Handling missing date fields in a date-oriented function query

[Chris Harris <ry...@gmail.com>]

I'm using function queries to boost more recent documents, using
something like the

    recip(ms(NOW,mydatefield),3.16e-11,1,1)

approach described on the wiki:
http://wiki.apache.org/solr/FunctionQuery#Date_Boosting

What I'd like to do is figure out the best way to tweak how documents
with missing date fields are handled.

For reference, what the above expression does with docs with missing
mydatefield values is to treat those docs as if they were dated at the
epoch start. (This is because the default value for a missing field,
including a date field, is 0.)

What I want to try, in contrast, is to treat docs with missing dates
as if they were dated NOW. Can anyone suggest a satisfying way to do
this? Here are some options that won't work:

Option 1. Use map

The most obvious way to do this would be to wrap the reference to
mydatefield inside a map, like this:

    recip(ms(NOW,map(mydatefield,0,0,ms(NOW)),3.16e-11,1,1))

However, this throws an exception because map is hard-coded to take
float constants, rather than arbitrary subqueries.

Option 2. Use min

The second most obvious way is to replace ms(NOW,mydatefield) with
this expression:

    min(ms(NOW),ms(NOW,mydatefield)

(Docs with mydatefield will have a very large ms(NOW,mydatefield) term.)

However there is no min function, so that won't work.

Option 2a. Fake min with max and multiplication

Since min(a,b) == -1*max(-1*a, -1*b), you could rewrite the previous
expression using this more complicated logic and it would work. But
that's ugly.

Also, it would crash anyway. It looks like max currently requires one
of its arguments to be a float constant, and neither of our args would
be a float constant.

Re: Handling missing date fields in a date-oriented function query

[Yonik Seeley <yo...@lucidimagination.com>]

On Fri, Apr 16, 2010 at 4:42 PM, Chris Harris <ry...@gmail.com> wrote:
> The docs dated exactly *at* the epoch, though, are trouble, because I
> can't tell those docs apart from the undated docs in my function
> query.

Neither can Solr currently... it's a Lucene FieldCache limitation.
The other thing we can't do because of this limitation is
sortMissingFirst, sortMissingLast like we can do with string based
fields.  I'm hopeful we'll be able to get this added to Lucene, and
that will enable us to truly deprecate the string-based numeric
fields.  We'll also be able to add true defaults in function queries
for documents without a value.

-Yonik
Apache Lucene Eurocon 2010
18-21 May 2010 | Prague

Re: Handling missing date fields in a date-oriented function query

[Chris Harris <ry...@gmail.com>]

I still like this approach, but I've discovered one wrinkle, which is
that I have dates in my dataset dated at the epoch (i.e. midnight Jan
1, 1970), as well as before the epoch (e.g. midnight Jan 1, 1950).

The docs dated *before* the epoch so far don't seem to be a problem;
they end up having a negative numeric value, but that seems workable.

The docs dated exactly *at* the epoch, though, are trouble, because I
can't tell those docs apart from the undated docs in my function
query. (Both end up with numeric value of 0 in the function query
code. So missing date is the same as midnight Jan 1, 1970.)

So far, in my case, the best bet seems to be changing the time
component of my dates. So rather than rounding dates to the nearest
midnight (e.g. 1970-01-01T00:00:00Z), I could round them to the
nearest, say, 1AM (e.g. 1970-01-01T01:00:00Z), with the goal of making
sure that none of my legitimate date field values will evaluate to
numeric value 0. Since I don't show the time component of dates to my
users, I don't think this would cause any real trouble. It feels
slightly unclean, though.

On Thu, Apr 8, 2010 at 1:05 PM, Chris Harris <ry...@gmail.com> wrote:
> If anyone is curious, I've created a patch that creates a variant of
> map that can be used in the way indicated below. See
> http://issues.apache.org/jira/browse/SOLR-1871
>
> On Wed, Apr 7, 2010 at 3:41 PM, Chris Harris <ry...@gmail.com> wrote:
>
>> Option 1. Use map
>>
>> The most obvious way to do this would be to wrap the reference to
>> mydatefield inside a map, like this:
>>
>>    recip(ms(NOW,map(mydatefield,0,0,ms(NOW)),3.16e-11,1,1))
>>
>> However, this throws an exception because map is hard-coded to take
>> float constants, rather than arbitrary subqueries.
>

Re: Handling missing date fields in a date-oriented function query

[Chris Harris <ry...@gmail.com>]

If anyone is curious, I've created a patch that creates a variant of
map that can be used in the way indicated below. See
http://issues.apache.org/jira/browse/SOLR-1871

On Wed, Apr 7, 2010 at 3:41 PM, Chris Harris <ry...@gmail.com> wrote:

> Option 1. Use map
>
> The most obvious way to do this would be to wrap the reference to
> mydatefield inside a map, like this:
>
>    recip(ms(NOW,map(mydatefield,0,0,ms(NOW)),3.16e-11,1,1))
>
> However, this throws an exception because map is hard-coded to take
> float constants, rather than arbitrary subqueries.

Re: Handling missing date fields in a date-oriented function query

[Chris Harris <ry...@gmail.com>]

On Wed, Apr 7, 2010 at 7:10 PM, Lance Norskog <go...@gmail.com> wrote:
>> Since min(a,b) == -1*max(-1*a, -1*b), you could rewrite the previous
>> expression using this more complicated logic and it would work. But
>> that's ugly.
>>
>> Also, it would crash anyway. It looks like max currently requires one
>> of its arguments to be a float constant, and neither of our args would
>> be a float constant.
>
> Ouch. There should be a min() function. The identity function will be
> slower. But, the 'parse a float' code should take an integer format
> string and parse it.

I should have been more specific about why max (and thus why a
hypothetical min) is inadequate for expressing something like

    min(ms(NOW),ms(NOW,mydatefield))

The problem is that while

    max(popularity,4)

works, putting anything more complicated as a second argument to max
fails. For example, these all fail:

    max(4,popularity)
    max(price,popularity)
    max(5,ms(NOW))

The stack trace in each case is along these lines:

java.lang.NumberFormatException: empty String
	at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:994)
	at java.lang.Float.parseFloat(Float.java:422)
	at org.apache.solr.search.QueryParsing$StrParser.getFloat(QueryParsing.java:553)
	at org.apache.solr.search.FunctionQParser.parseFloat(FunctionQParser.java:85)
	at org.apache.solr.search.ValueSourceParser$5.parse(ValueSourceParser.java:102)
	at org.apache.solr.search.FunctionQParser.parseValueSource(FunctionQParser.java:223)
	at org.apache.solr.search.FunctionQParser.parseValueSource(FunctionQParser.java:151)
	at org.apache.solr.search.FunctionQParser.parse(FunctionQParser.java:40)
	at org.apache.solr.search.BoostQParserPlugin$1.parse(BoostQParserPlugin.java:58)
	at org.apache.solr.search.QParser.getQuery(QParser.java:131)
	at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:89)

This behavior is sort of documented at

    http://wiki.apache.org/solr/FunctionQuery#max

which says that "max(x,c) returns the max of another function __and a
constant__".

I assume the decision to support only constants was made with
performance in mind.

>
> On Wed, Apr 7, 2010 at 3:41 PM, Chris Harris <ry...@gmail.com> wrote:
>> I'm using function queries to boost more recent documents, using
>> something like the
>>
>>    recip(ms(NOW,mydatefield),3.16e-11,1,1)
>>
>> approach described on the wiki:
>> http://wiki.apache.org/solr/FunctionQuery#Date_Boosting
>>
>> What I'd like to do is figure out the best way to tweak how documents
>> with missing date fields are handled.
>>
>> For reference, what the above expression does with docs with missing
>> mydatefield values is to treat those docs as if they were dated at the
>> epoch start. (This is because the default value for a missing field,
>> including a date field, is 0.)
>>
>> What I want to try, in contrast, is to treat docs with missing dates
>> as if they were dated NOW. Can anyone suggest a satisfying way to do
>> this? Here are some options that won't work:
>>
>> Option 1. Use map
>>
>> The most obvious way to do this would be to wrap the reference to
>> mydatefield inside a map, like this:
>>
>>    recip(ms(NOW,map(mydatefield,0,0,ms(NOW)),3.16e-11,1,1))
>>
>> However, this throws an exception because map is hard-coded to take
>> float constants, rather than arbitrary subqueries.
>>
>> Option 2. Use min
>>
>> The second most obvious way is to replace ms(NOW,mydatefield) with
>> this expression:
>>
>>    min(ms(NOW),ms(NOW,mydatefield)
>>
>> (Docs with mydatefield will have a very large ms(NOW,mydatefield) term.)
>>
>> However there is no min function, so that won't work.
>>
>> Option 2a. Fake min with max and multiplication
>>
>> Since min(a,b) == -1*max(-1*a, -1*b), you could rewrite the previous
>> expression using this more complicated logic and it would work. But
>> that's ugly.
>>
>> Also, it would crash anyway. It looks like max currently requires one
>> of its arguments to be a float constant, and neither of our args would
>> be a float constant.
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>

Re: Handling missing date fields in a date-oriented function query

[Lance Norskog <go...@gmail.com>]

> Since min(a,b) == -1*max(-1*a, -1*b), you could rewrite the previous
> expression using this more complicated logic and it would work. But
> that's ugly.
>
> Also, it would crash anyway. It looks like max currently requires one
> of its arguments to be a float constant, and neither of our args would
> be a float constant.

Ouch. There should be a min() function. The identity function will be
slower. But, the 'parse a float' code should take an integer format
string and parse it.

On Wed, Apr 7, 2010 at 3:41 PM, Chris Harris <ry...@gmail.com> wrote:
> I'm using function queries to boost more recent documents, using
> something like the
>
>    recip(ms(NOW,mydatefield),3.16e-11,1,1)
>
> approach described on the wiki:
> http://wiki.apache.org/solr/FunctionQuery#Date_Boosting
>
> What I'd like to do is figure out the best way to tweak how documents
> with missing date fields are handled.
>
> For reference, what the above expression does with docs with missing
> mydatefield values is to treat those docs as if they were dated at the
> epoch start. (This is because the default value for a missing field,
> including a date field, is 0.)
>
> What I want to try, in contrast, is to treat docs with missing dates
> as if they were dated NOW. Can anyone suggest a satisfying way to do
> this? Here are some options that won't work:
>
> Option 1. Use map
>
> The most obvious way to do this would be to wrap the reference to
> mydatefield inside a map, like this:
>
>    recip(ms(NOW,map(mydatefield,0,0,ms(NOW)),3.16e-11,1,1))
>
> However, this throws an exception because map is hard-coded to take
> float constants, rather than arbitrary subqueries.
>
> Option 2. Use min
>
> The second most obvious way is to replace ms(NOW,mydatefield) with
> this expression:
>
>    min(ms(NOW),ms(NOW,mydatefield)
>
> (Docs with mydatefield will have a very large ms(NOW,mydatefield) term.)
>
> However there is no min function, so that won't work.
>
> Option 2a. Fake min with max and multiplication
>
> Since min(a,b) == -1*max(-1*a, -1*b), you could rewrite the previous
> expression using this more complicated logic and it would work. But
> that's ugly.
>
> Also, it would crash anyway. It looks like max currently requires one
> of its arguments to be a float constant, and neither of our args would
> be a float constant.
>



-- 
Lance Norskog
goksron@gmail.com