Date Faceting and Double Counting

[Stephen Duncan Jr <st...@gmail.com>]

If we do date faceting and start at 2009-01-01T00:00:00Z, end at
2009-01-03T00:00:00Z, with a gap of +1DAY, then documents that occur at
exactly 2009-01-02T00:00:00Z will be included in both the returned counts
(2009-01-01T00:00:00Z and 2009-01-02T00:00:00Z).  At the moment, this is
quite bad for us, as we only index the day-level, so all of our documents
are exactly on the line between each facet-range.

Because we know our data is indexed as being exactly at midnight each day, I
think we can simply always start from 1 second prior and get the results we
want (start=2008-12-31T23:59:59Z, end=2009-01-02T23:59:59Z), but I think
this problem would affect everyone, even if usually more subtly (instead of
all documents being counted twice, only a few on the fencepost between
ranges).

Is this a known behavior people are happy with, or should I file an issue
asking for ranges in date-facets to be constructed to subtract one second
from the end of each range (so that the effective range queries for my case
would be: [2009-01-01T00:00:00Z TO 2009-01-01T23:59:59Z] &
[2009-01-02T00:00:00Z TO 2009-01-02T23:59:59Z])?

Alternatively, is there some other suggested way of using the date faceting
to avoid this problem?

-- 
Stephen Duncan Jr
www.stephenduncanjr.com

Re: Date Faceting and Double Counting

[Avlesh Singh <av...@gmail.com>]

I don't think this behavior needs to be fixed. It is justified for the data
you have indexed.
"date minus 1 second" should definitely work for you.

Cheers
Avlesh

On Mon, Aug 31, 2009 at 11:37 PM, Stephen Duncan Jr <
stephen.duncan@gmail.com> wrote:

> If we do date faceting and start at 2009-01-01T00:00:00Z, end at
> 2009-01-03T00:00:00Z, with a gap of +1DAY, then documents that occur at
> exactly 2009-01-02T00:00:00Z will be included in both the returned counts
> (2009-01-01T00:00:00Z and 2009-01-02T00:00:00Z).  At the moment, this is
> quite bad for us, as we only index the day-level, so all of our documents
> are exactly on the line between each facet-range.
>
> Because we know our data is indexed as being exactly at midnight each day,
> I
> think we can simply always start from 1 second prior and get the results we
> want (start=2008-12-31T23:59:59Z, end=2009-01-02T23:59:59Z), but I think
> this problem would affect everyone, even if usually more subtly (instead of
> all documents being counted twice, only a few on the fencepost between
> ranges).
>
> Is this a known behavior people are happy with, or should I file an issue
> asking for ranges in date-facets to be constructed to subtract one second
> from the end of each range (so that the effective range queries for my case
> would be: [2009-01-01T00:00:00Z TO 2009-01-01T23:59:59Z] &
> [2009-01-02T00:00:00Z TO 2009-01-02T23:59:59Z])?
>
> Alternatively, is there some other suggested way of using the date faceting
> to avoid this problem?
>
> --
> Stephen Duncan Jr
> www.stephenduncanjr.com
>

Re: Date Faceting and Double Counting

[Chris Hostetter <ho...@fucit.org>]

: Is this a known behavior people are happy with, or should I file an issue
: asking for ranges in date-facets to be constructed to subtract one second
: from the end of each range (so that the effective range queries for my case

It's a known anoyance, but not something that seems to anoy people enough 
that there have been any patches to improve the situation.  typically the 
off by 1 millisecond approach you describe works for people.


-Hoss

Re: Date Faceting and Double Counting

[Chris Hostetter <ho...@fucit.org>]

: What are wild-card range searches?

i'm pretty sure we was just refering to open ended range searchers, like 
the example he asked about...

: > What does this mean?
: >
: >      {* TO *}
: 
: Same thing as [* TO *] - not worth trying to make it different IMO.

...right, that's something the SolrQueryParser already supports.  the 
inclusive/exclusive syntax is ignored if it's open ended.




-Hoss

Re: Date Faceting and Double Counting

[Yonik Seeley <yo...@lucidimagination.com>]

On Fri, Sep 11, 2009 at 3:59 PM, Lance Norskog <go...@gmail.com> wrote:
> Thanks! I had to find this in the Lucene query parser syntax- it is
> not mentioned anywhere in the Solr wiki. You are right [a TO z} and {a
> TO z] are obvious improvements and solve the bucket-search problem the
> right way. But this collides with wild-card range searches.

What are wild-card range searches?

> What does this mean?
>
>      {* TO *}

Same thing as [* TO *] - not worth trying to make it different IMO.

-Yonik
http://www.lucidimagination.com

Re: Date Faceting and Double Counting

[Lance Norskog <go...@gmail.com>]

Thanks! I had to find this in the Lucene query parser syntax- it is
not mentioned anywhere in the Solr wiki. You are right [a TO z} and {a
TO z] are obvious improvements and solve the bucket-search problem the
right way. But this collides with wild-card range searches.

What does this mean?

      {* TO *}


On Fri, Sep 11, 2009 at 10:02 AM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : datefield:[X TO* Y] for X to Y-0.00000000...1
> :
> : This would be backwards-compatible. {} are used for other things and lexing
>
> You lost me there ... {} aren't used for "other things" in the query
> parser -- they're used for range queries that are exclusive of their end
> points.  datefield:{X TO Y} is already legal syntax, i'm just saying it
> would be nice if datefield:[X TO Y} and datefield:{X TO Y] were legal as
> well.
>
>
> -Hoss
>
>



-- 
Lance Norskog
goksron@gmail.com

Re: Date Faceting and Double Counting

[Chris Hostetter <ho...@fucit.org>]

: datefield:[X TO* Y] for X to Y-0.00000000...1
: 
: This would be backwards-compatible. {} are used for other things and lexing

You lost me there ... {} aren't used for "other things" in the query 
parser -- they're used for range queries that are exclusive of their end 
points.  datefield:{X TO Y} is already legal syntax, i'm just saying it 
would be nice if datefield:[X TO Y} and datefield:{X TO Y] were legal as 
well.


-Hoss

Re: Date Faceting and Double Counting

[Lance Norskog <go...@gmail.com>]

datefield:[X TO* Y] for X to Y-0.00000000...1

This would be backwards-compatible. {} are used for other things and lexing
is a dying art. Using a * causes mistakes to trigger wildcard syntaxes,
which will fail loudly.

On Tue, Sep 8, 2009 at 5:20 PM, Chris Hostetter <ho...@fucit.org>wrote:

>
> : I ran into that problem as well but the solution was provided to me by
> : this very list :) See
> : http://www.nabble.com/Range-queries-td24057317.html It's not the
> : cleanest solution, but as long as you know what you're doing it's not
> : that bad.
>
> Hmmm... yeah, that's a total hack.  one of these days we really need to
> fix the lucene query parser grammer so inclusive/exclusive can be
> different for hte upper/lower bounds...
>
>        datefield:[NOW/DAY TO NOW/DAY+1DAY}
>
>
> -Hoss
>
>


-- 
Lance Norskog
goksron@gmail.com

Re: Date Faceting and Double Counting

[Chris Hostetter <ho...@fucit.org>]

: I ran into that problem as well but the solution was provided to me by
: this very list :) See
: http://www.nabble.com/Range-queries-td24057317.html It's not the
: cleanest solution, but as long as you know what you're doing it's not
: that bad.

Hmmm... yeah, that's a total hack.  one of these days we really need to 
fix the lucene query parser grammer so inclusive/exclusive can be 
different for hte upper/lower bounds...

	datefield:[NOW/DAY TO NOW/DAY+1DAY}


-Hoss

Re: Date Faceting and Double Counting

[gwk <gi...@eyefi.nl>]

Chris Hostetter wrote:
> : When I added numerical faceting to my checkout of solr (solr-1240) I basically
> : copied date faceting and modified it to work with numbers instead of dates.
> : With numbers I got a lot of doulbe-counted values as well. So to fix my
> : problem I added an extra parameter to number faceting where you can specify if
> : either end of each range should be inclusive or exclusive. I just ported it
>
> gwk:
>
> 1) would you mind opening a Jira issue for your date faceting improvements 
> as well (email attachments tend to get lost, and there are legal headaches 
> with committing them that Jira solves by asking you explicitly if you 
> license them to the ASF)
>   
Sure, I've added it to Jira https://issues.apache.org/jira/browse/SOLR-1402.
> 2) i haven't looked t your patch, but one of the reasons i never 
> implemented an option like this with date faceting is that the query 
> parser doesn't have any way of letting you write a query that is inclusive 
> on one end, and exclusive on the other end -- so you might get accurate 
> facet counts for range A-B and B-C (inclusive of the lower, exclusive of 
> hte upp), but if you try to filter by one of those ranges, your counts 
> will be off.  did you find a nice solution for this?
>
>
>   
I ran into that problem as well but the solution was provided to me by
this very list :) See
http://www.nabble.com/Range-queries-td24057317.html It's not the
cleanest solution, but as long as you know what you're doing it's not
that bad.

The reason I created 1240 was exactly because my counts were off, with
date faceting exact matches are a rarity, or at least you can make them
to be one. But since with numbers (in my case, prices) being off by 1
cent is not acceptable I needed this exclusivity. The only real reason
for all of this was the geek-candy the price slider on our website, the
counts are sent via ajax and the range slider can simply sum the counts
for the selected range to get the exact count for that range without
having to query Solr for more data.

Regards,

gwk

Re: Date Faceting and Double Counting

[Chris Hostetter <ho...@fucit.org>]

: When I added numerical faceting to my checkout of solr (solr-1240) I basically
: copied date faceting and modified it to work with numbers instead of dates.
: With numbers I got a lot of doulbe-counted values as well. So to fix my
: problem I added an extra parameter to number faceting where you can specify if
: either end of each range should be inclusive or exclusive. I just ported it

gwk:

1) would you mind opening a Jira issue for your date faceting improvements 
as well (email attachments tend to get lost, and there are legal headaches 
with committing them that Jira solves by asking you explicitly if you 
license them to the ASF)

2) i haven't looked t your patch, but one of the reasons i never 
implemented an option like this with date faceting is that the query 
parser doesn't have any way of letting you write a query that is inclusive 
on one end, and exclusive on the other end -- so you might get accurate 
facet counts for range A-B and B-C (inclusive of the lower, exclusive of 
hte upp), but if you try to filter by one of those ranges, your counts 
will be off.  did you find a nice solution for this?




-Hoss

Re: Date Faceting and Double Counting

[gwk <gi...@eyefi.nl>]

Hi Stephen,

When I added numerical faceting to my checkout of solr (solr-1240) I 
basically copied date faceting and modified it to work with numbers 
instead of dates. With numbers I got a lot of doulbe-counted values as 
well. So to fix my problem I added an extra parameter to number faceting 
where you can specify if either end of each range should be inclusive or 
exclusive. I just ported it back to date faceting (disclaimer, 
completely untested) and it should be attached to my post.

The following parameter is added: facet.date.exclusive
valid values for the parameter are: start, end, both and neither

To maintain compatibility with solr without the patch the default is 
neither. I hope the meaning of the values are self-explanatory.

Regards,

gwk

Stephen Duncan Jr wrote:
> If we do date faceting and start at 2009-01-01T00:00:00Z, end at
> 2009-01-03T00:00:00Z, with a gap of +1DAY, then documents that occur at
> exactly 2009-01-02T00:00:00Z will be included in both the returned counts
> (2009-01-01T00:00:00Z and 2009-01-02T00:00:00Z).  At the moment, this is
> quite bad for us, as we only index the day-level, so all of our documents
> are exactly on the line between each facet-range.
>
> Because we know our data is indexed as being exactly at midnight each day, I
> think we can simply always start from 1 second prior and get the results we
> want (start=2008-12-31T23:59:59Z, end=2009-01-02T23:59:59Z), but I think
> this problem would affect everyone, even if usually more subtly (instead of
> all documents being counted twice, only a few on the fencepost between
> ranges).
>
> Is this a known behavior people are happy with, or should I file an issue
> asking for ranges in date-facets to be constructed to subtract one second
> from the end of each range (so that the effective range queries for my case
> would be: [2009-01-01T00:00:00Z TO 2009-01-01T23:59:59Z] &
> [2009-01-02T00:00:00Z TO 2009-01-02T23:59:59Z])?
>
> Alternatively, is there some other suggested way of using the date faceting
> to avoid this problem?
>
>