You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Charlie Jackson <Ch...@cision.com> on 2009/09/24 18:49:59 UTC
Sorting/paging problem
I've run into a strange issue with my Solr installation. I'm running
queries that are sorting by a DateField field but from time to time, I'm
seeing individual records very much out of order. What's more, they
appear on multiple pages of my result set. Let me give an example.
Starting with a basic query, I sort on the date that the document was
added to the index and see these rows on the first page (I'm just
showing the date field here):
<doc><date name="indexed_date">2009-09-23T19:24:47.419Z</date></doc>
<doc><date name="indexed_date">2009-09-23T19:25:03.229Z</date></doc>
<doc><date name="indexed_date">2009-09-23T19:25:03.400Z</date></doc>
<doc><date name="indexed_date">2009-09-23T19:25:19.951</date></doc>
<doc><date name="indexed_date">2009-09-23T20:10:07.919Z</date></doc>
Note how the last document's date jumps a bit. Not necessarily a
problem, but the next page looks this:
<doc><date name="indexed_date">2009-09-23T19:26:16.022Z</date></doc>
<doc><date name="indexed_date">2009-09-23T19:26:32.547Z</date></doc>
<doc><date name="indexed_date">2009-09-23T19:27:45.470Z</date></doc>
<doc><date name="indexed_date">2009-09-23T19:27:45.592Z</date></doc>
<doc><date name="indexed_date">2009-09-23T20:10:07.919Z</date></doc>
So, not only is the date sorting wrong, but the exact same document
shows up on the next page, also still out of date order. I've seen the
same document show up in 4-5 pages in some cases. It's always the last
record on the page, too. If I change the page size, the problem seems to
disappear for a while, but then starts up again later. Also, running the
same query/queries later on doesn't show the same behavior.
Could it be some sort of page boundary issue with the cache? Has anyone
else run into a problem like this? I'm using the Sept 22 nightly build.
- Charlie
Re: Sorting/paging problem
Posted by Lance Norskog <go...@gmail.com>.
Which version of Java are you using?
Please try the standard tricks:
Do a fresh checkout of the Solr trunk.
Do 'ant clean dist' and use the newly built war & latest lucene libraries.
Try changing the JVM startup parameters which control how incremental
compilation works: -server and others. Also try changing the garbage
collection algorithms.
On Thu, Sep 24, 2009 at 9:49 AM, Charlie Jackson
<Ch...@cision.com> wrote:
> I've run into a strange issue with my Solr installation. I'm running
> queries that are sorting by a DateField field but from time to time, I'm
> seeing individual records very much out of order. What's more, they
> appear on multiple pages of my result set. Let me give an example.
> Starting with a basic query, I sort on the date that the document was
> added to the index and see these rows on the first page (I'm just
> showing the date field here):
>
>
>
> <doc><date name="indexed_date">2009-09-23T19:24:47.419Z</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T19:25:03.229Z</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T19:25:03.400Z</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T19:25:19.951</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T20:10:07.919Z</date></doc>
>
>
>
> Note how the last document's date jumps a bit. Not necessarily a
> problem, but the next page looks this:
>
>
>
> <doc><date name="indexed_date">2009-09-23T19:26:16.022Z</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T19:26:32.547Z</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T19:27:45.470Z</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T19:27:45.592Z</date></doc>
>
> <doc><date name="indexed_date">2009-09-23T20:10:07.919Z</date></doc>
>
>
>
> So, not only is the date sorting wrong, but the exact same document
> shows up on the next page, also still out of date order. I've seen the
> same document show up in 4-5 pages in some cases. It's always the last
> record on the page, too. If I change the page size, the problem seems to
> disappear for a while, but then starts up again later. Also, running the
> same query/queries later on doesn't show the same behavior.
>
>
>
> Could it be some sort of page boundary issue with the cache? Has anyone
> else run into a problem like this? I'm using the Sept 22 nightly build.
>
>
>
> - Charlie
>
>
--
Lance Norskog
goksron@gmail.com
RE: Sorting/paging problem
Posted by Charlie Jackson <Ch...@cision.com>.
Oops, the missing trailing Z was probably just a cut and paste error.
It might be tough to come up with a case that can reproduce it -- it's a
sticky issue. I'll post it if I can, though.
-----Original Message-----
From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
Sent: Tuesday, September 29, 2009 6:08 PM
To: solr-user@lucene.apache.org
Subject: Re: Sorting/paging problem
: <doc><date name="indexed_date">2009-09-23T19:25:03.400Z</date></doc>
:
: <doc><date name="indexed_date">2009-09-23T19:25:19.951</date></doc>
:
: <doc><date name="indexed_date">2009-09-23T20:10:07.919Z</date></doc>
is that a cut/paste error, or did you really get a date back from Solr
w/o the trailing "Z" ?!?!?!
...
: So, not only is the date sorting wrong, but the exact same document
: shows up on the next page, also still out of date order. I've seen the
: same document show up in 4-5 pages in some cases. It's always the last
: record on the page, too. If I change the page size, the problem seems
to
that is really freaking weird. can you reproduce this in a simple
example? maybe an index that's small enough (and doesn't contain
confidential information) that you could zip up and post online?
-Hoss
Re: Sorting/paging problem
Posted by Chris Hostetter <ho...@fucit.org>.
: <doc><date name="indexed_date">2009-09-23T19:25:03.400Z</date></doc>
:
: <doc><date name="indexed_date">2009-09-23T19:25:19.951</date></doc>
:
: <doc><date name="indexed_date">2009-09-23T20:10:07.919Z</date></doc>
is that a cut/paste error, or did you really get a date back from Solr
w/o the trailing "Z" ?!?!?!
...
: So, not only is the date sorting wrong, but the exact same document
: shows up on the next page, also still out of date order. I've seen the
: same document show up in 4-5 pages in some cases. It's always the last
: record on the page, too. If I change the page size, the problem seems to
that is really freaking weird. can you reproduce this in a simple
example? maybe an index that's small enough (and doesn't contain
confidential information) that you could zip up and post online?
-Hoss