unstable results on refresh

[Giovanni Bricconi <gi...@banzai.it>]

Hello

I have a procedure that sends small data changes during the day to a
solrcloud cluster, version 4.8

The cluster is made of three nodes, and three shards, each node contains
two shards

The procedure has been running for days; I don't know when but at some
point one of the cores has gone out of synch and so repeating the same
query has began to show small differences.

The core graph was not useful, everything seemed active.

I have solved the problem reindexing all, because the collection is quite
small, but is there a way to fix this problem? Suppose I can figure out
which core returns different results, is there a command to force that core
to refetch the whole index from its master?

Thanks

Giovanni

Re: unstable results on refresh

[Ramzi Alqrainy <ra...@gmail.com>]

Can you please provide us the exception when the shard goes out of sync ?
Please monitor the logs. 



--
View this message in context: http://lucene.472066.n3.nabble.com/unstable-results-on-refresh-tp4164913p4165002.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: unstable results on refresh

[Giovanni Bricconi <gi...@banzai.it>]

I noticed the problem looking at a group query, the groups returned where
sorted on the score field of the first result, and then showed to the user.
Repeating the same query I noticed that the order of two group started
switching

Thank you, I will look for the thread you said

2014-10-20 22:07 GMT+02:00 Alexandre Rafalovitch <ar...@gmail.com>:

> What are the differences on. The document count or things like facets?
> This could be important.
>
> Also, I think there was a similar thread on the mailing list a week or
> two ago, might be worth looking for it.
>
> Regards,
>    Alex.
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
>
> On 20 October 2014 04:49, Giovanni Bricconi <gi...@banzai.it>
> wrote:
> > Hello
> >
> > I have a procedure that sends small data changes during the day to a
> > solrcloud cluster, version 4.8
> >
> > The cluster is made of three nodes, and three shards, each node contains
> > two shards
> >
> > The procedure has been running for days; I don't know when but at some
> > point one of the cores has gone out of synch and so repeating the same
> > query has began to show small differences.
> >
> > The core graph was not useful, everything seemed active.
> >
> > I have solved the problem reindexing all, because the collection is quite
> > small, but is there a way to fix this problem? Suppose I can figure out
> > which core returns different results, is there a command to force that
> core
> > to refetch the whole index from its master?
> >
> > Thanks
> >
> > Giovanni
>

Re: unstable results on refresh

[Alexandre Rafalovitch <ar...@gmail.com>]

What are the differences on. The document count or things like facets?
This could be important.

Also, I think there was a similar thread on the mailing list a week or
two ago, might be worth looking for it.

Regards,
   Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853


On 20 October 2014 04:49, Giovanni Bricconi <gi...@banzai.it> wrote:
> Hello
>
> I have a procedure that sends small data changes during the day to a
> solrcloud cluster, version 4.8
>
> The cluster is made of three nodes, and three shards, each node contains
> two shards
>
> The procedure has been running for days; I don't know when but at some
> point one of the cores has gone out of synch and so repeating the same
> query has began to show small differences.
>
> The core graph was not useful, everything seemed active.
>
> I have solved the problem reindexing all, because the collection is quite
> small, but is there a way to fix this problem? Suppose I can figure out
> which core returns different results, is there a command to force that core
> to refetch the whole index from its master?
>
> Thanks
>
> Giovanni

Re: unstable results on refresh

[Shawn Heisey <ap...@elyograg.org>]

On 10/23/2014 2:44 AM, Giovanni Bricconi wrote:
> My user interface shows some boxes to describe results categories. After
> half a day of small updates and delete I noticed with various queries that
> the boxes started swapping while browsing.
> For sure I relied too much in getting the same results on each call, now
> I'm keeping the categories order in request parameters to avoid the blink
> effect while browsing.
> 
> The optimize process is really slow, and I can't use it. Since I have many
> other parameters that should be carried along the request to make sure that
> the navigation is consistent, I would like to understand if is there a
> setup that can limit the idf change and keep it low enough
> 
> I tried with
> 
> <indexConfig>
> 
> <mergeFactor>5</mergeFactor>
> 
> </indexConfig>
> In solrconfig but this morning /solr/admin/cores?action=STATUS still
> reports a number of segments above ten for all cores of the shard. (I'm
> sure I have reloaded each core after changing the value)
> 
> Now I'm trying with expungeDeletes called from solrj, but still I don't see
> the segment count decrease

It's completely normal to have more segments than the mergeFactor.
Think about this scenario with a mergeFactor of 5:

You index five segments.  They get merged to one segment.  Let's say
that this happens a total of four times, so you've indexed a total of 20
segments and merging has reduced that to four larger segments.  Let's
say that you now index four more segments.  You'll be completely stable
with eight segments.  If you index another one, that will result in a
fifth larger segment.  This sets conditions up just right for another
merge -- to one even larger segment.  This represents three "levels" of
merging, and there can be even more levels, each of which can have four
segments and remain stable.  Starting at the last state I described, if
you then indexed 24 more segments, you'd have a stable index with a
total of nine segments - four of them would be "normal" sized, four of
them would be about five times normal size, and the first one would be
about 25 times normal size.

The Solr default for the merge policy in all recent versions is
TieredMergePolicy, and this can make things slightly more complicated
than I've described, because it can merge *any* segments, not just those
indexed sequentially, and I believe that it can delay merging until the
right number of segments with suitable characteristics appear.

I've got merge settings equivalent to a mergeFactor of 35, but I
regularly see the segment count approach 100, and there's absolutely
nothing wrong with my merging.

If I understand it correctly, expungeDeletes will not decrease the
segment count.  It will simply rewrite segments that have deleted
documents so there are none.  I'm not 100% sure that I know exactly what
expungeDeletes does, though.

Thanks,
Shawn

Re: unstable results on refresh

[Giovanni Bricconi <gi...@banzai.it>]

My user interface shows some boxes to describe results categories. After
half a day of small updates and delete I noticed with various queries that
the boxes started swapping while browsing.
For sure I relied too much in getting the same results on each call, now
I'm keeping the categories order in request parameters to avoid the blink
effect while browsing.

The optimize process is really slow, and I can't use it. Since I have many
other parameters that should be carried along the request to make sure that
the navigation is consistent, I would like to understand if is there a
setup that can limit the idf change and keep it low enough

I tried with

<indexConfig>

<mergeFactor>5</mergeFactor>

</indexConfig>
In solrconfig but this morning /solr/admin/cores?action=STATUS still
reports a number of segments above ten for all cores of the shard. (I'm
sure I have reloaded each core after changing the value)

Now I'm trying with expungeDeletes called from solrj, but still I don't see
the segment count decrease

UpdateRequest commitRequest = new UpdateRequest();

  commitRequest.setAction

//(action, waitFlush, waitSearcher, maxSegments, softCommit, expungeDeletes)

   ( ACTION.COMMIT, true, true, 10, false, true);

  commitRequest.process(solrServer);



2014-10-22 15:48 GMT+02:00 Erick Erickson <er...@gmail.com>:

> I would rather ask whether such small differences matter enough to
> do this. Is this something users will _ever_ notice? Optimization
> is quite a heavyweight operation, and is generally not recommended
> on indexes that change often, and 5 minutes is certainly below
> the recommendation for optimizing.
>
> There is/has been work done on "distributed IDF", but I don't quite
> know the current status that should address this (I think).
>
> But other than in a test setup, is it worth the effort?
>
> Best,
> Erick
>
> On Wed, Oct 22, 2014 at 3:54 AM, Giovanni Bricconi
> <gi...@banzai.it> wrote:
> > I have made some small patch to the application to make this problem less
> > visible, and I'm trying to perform the optimize once per hour, yesterday
> it
> > took 5 minutes to perform it, this morning 15 minutes. Today I will
> collect
> > some statistics but the publication process sends documents every 5
> > minutes, and I think the optimize is taking too much time.
> >
> > I have no default mergeFactor configured for this collection, do you
> think
> > that setting it to a small value could improve the situation? If I have
> > understood well having to merge segments will keep similar stats on all
> > nodes. It's ok to have the indexing process a little bit slower.
> >
> >
> > 2014-10-21 18:44 GMT+02:00 Erick Erickson <er...@gmail.com>:
> >
> >> Giovanni:
> >>
> >> To see how this happens, consider a shard with a leader and two
> >> followers. Assume your autocommit interval is 60 seconds on each.
> >>
> >> This interval can expire at slightly different "wall clock" times.
> >> Even if the servers started perfectly in synch, they can get slightly
> >> out of sync. So, you index a bunch of docs and these replicas close
> >> the current segment and re-open a new segment with slightly different
> >> contents.
> >>
> >> Now docs come in that replace older docs. The tf/idf statistics
> >> _include_ deleted document data (which is purged on optimize). Given
> >> that doc X an be in different segments (or, more accurately, segments
> >> that get merged at different times on different machines), replica 1
> >> may have slightly different stats than replica 2, thus computing
> >> slightly different scores.
> >>
> >> Optimizing purges all data related to deleted documents, so it all
> >> regularizes itself on optimize.
> >>
> >> Best,
> >> Erick
> >>
> >> On Tue, Oct 21, 2014 at 11:08 AM, Giovanni Bricconi
> >> <gi...@banzai.it> wrote:
> >> > I noticed again the problem, now I was able to collect some data. in
> my
> >> > paste http://pastebin.com/nVwf327c you can see the result of the same
> >> query
> >> > issued twice, the 2nd and 3rd group are swapped.
> >> >
> >> > I pasted also the clusterstate and the core state for each core.
> >> >
> >> > The logs did'n show any problem related to indexing, only some
> malformed
> >> > query.
> >> >
> >> > After doing an optimize the problem disappeared.
> >> >
> >> > So, is the problem related to documents that where deleted from the
> >> index?
> >> >
> >> > The optimization took 5 minutes to complete
> >> >
> >> > 2014-10-21 11:41 GMT+02:00 Giovanni Bricconi <
> >> giovanni.bricconi@banzai.it>:
> >> >
> >> >> Nice!
> >> >> I will monitor the index and try this if the problem comes back.
> >> >> Actually the problem was due to small differences in score, so I
> think
> >> the
> >> >> problem has the same origin
> >> >>
> >> >> 2014-10-21 8:10 GMT+02:00 lboutros <bo...@gmail.com>:
> >> >>
> >> >>> Hi Giovanni,
> >> >>>
> >> >>> we had this problem as well.
> >> >>> The cause was that the different nodes have slightly different idf
> >> values.
> >> >>>
> >> >>> We solved this problem by doing an optimize operation which really
> >> remove
> >> >>> suppressed data.
> >> >>>
> >> >>> Ludovic.
> >> >>>
> >> >>>
> >> >>>
> >> >>> -----
> >> >>> Jouve
> >> >>> France.
> >> >>> --
> >> >>> View this message in context:
> >> >>>
> >>
> http://lucene.472066.n3.nabble.com/unstable-results-on-refresh-tp4164913p4165086.html
> >> >>> Sent from the Solr - User mailing list archive at Nabble.com.
> >> >>>
> >> >>
> >> >>
> >>
>

Re: unstable results on refresh

[Erick Erickson <er...@gmail.com>]

I would rather ask whether such small differences matter enough to
do this. Is this something users will _ever_ notice? Optimization
is quite a heavyweight operation, and is generally not recommended
on indexes that change often, and 5 minutes is certainly below
the recommendation for optimizing.

There is/has been work done on "distributed IDF", but I don't quite
know the current status that should address this (I think).

But other than in a test setup, is it worth the effort?

Best,
Erick

On Wed, Oct 22, 2014 at 3:54 AM, Giovanni Bricconi
<gi...@banzai.it> wrote:
> I have made some small patch to the application to make this problem less
> visible, and I'm trying to perform the optimize once per hour, yesterday it
> took 5 minutes to perform it, this morning 15 minutes. Today I will collect
> some statistics but the publication process sends documents every 5
> minutes, and I think the optimize is taking too much time.
>
> I have no default mergeFactor configured for this collection, do you think
> that setting it to a small value could improve the situation? If I have
> understood well having to merge segments will keep similar stats on all
> nodes. It's ok to have the indexing process a little bit slower.
>
>
> 2014-10-21 18:44 GMT+02:00 Erick Erickson <er...@gmail.com>:
>
>> Giovanni:
>>
>> To see how this happens, consider a shard with a leader and two
>> followers. Assume your autocommit interval is 60 seconds on each.
>>
>> This interval can expire at slightly different "wall clock" times.
>> Even if the servers started perfectly in synch, they can get slightly
>> out of sync. So, you index a bunch of docs and these replicas close
>> the current segment and re-open a new segment with slightly different
>> contents.
>>
>> Now docs come in that replace older docs. The tf/idf statistics
>> _include_ deleted document data (which is purged on optimize). Given
>> that doc X an be in different segments (or, more accurately, segments
>> that get merged at different times on different machines), replica 1
>> may have slightly different stats than replica 2, thus computing
>> slightly different scores.
>>
>> Optimizing purges all data related to deleted documents, so it all
>> regularizes itself on optimize.
>>
>> Best,
>> Erick
>>
>> On Tue, Oct 21, 2014 at 11:08 AM, Giovanni Bricconi
>> <gi...@banzai.it> wrote:
>> > I noticed again the problem, now I was able to collect some data. in my
>> > paste http://pastebin.com/nVwf327c you can see the result of the same
>> query
>> > issued twice, the 2nd and 3rd group are swapped.
>> >
>> > I pasted also the clusterstate and the core state for each core.
>> >
>> > The logs did'n show any problem related to indexing, only some malformed
>> > query.
>> >
>> > After doing an optimize the problem disappeared.
>> >
>> > So, is the problem related to documents that where deleted from the
>> index?
>> >
>> > The optimization took 5 minutes to complete
>> >
>> > 2014-10-21 11:41 GMT+02:00 Giovanni Bricconi <
>> giovanni.bricconi@banzai.it>:
>> >
>> >> Nice!
>> >> I will monitor the index and try this if the problem comes back.
>> >> Actually the problem was due to small differences in score, so I think
>> the
>> >> problem has the same origin
>> >>
>> >> 2014-10-21 8:10 GMT+02:00 lboutros <bo...@gmail.com>:
>> >>
>> >>> Hi Giovanni,
>> >>>
>> >>> we had this problem as well.
>> >>> The cause was that the different nodes have slightly different idf
>> values.
>> >>>
>> >>> We solved this problem by doing an optimize operation which really
>> remove
>> >>> suppressed data.
>> >>>
>> >>> Ludovic.
>> >>>
>> >>>
>> >>>
>> >>> -----
>> >>> Jouve
>> >>> France.
>> >>> --
>> >>> View this message in context:
>> >>>
>> http://lucene.472066.n3.nabble.com/unstable-results-on-refresh-tp4164913p4165086.html
>> >>> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>>
>> >>
>> >>
>>

Re: unstable results on refresh

[Giovanni Bricconi <gi...@banzai.it>]

I have made some small patch to the application to make this problem less
visible, and I'm trying to perform the optimize once per hour, yesterday it
took 5 minutes to perform it, this morning 15 minutes. Today I will collect
some statistics but the publication process sends documents every 5
minutes, and I think the optimize is taking too much time.

I have no default mergeFactor configured for this collection, do you think
that setting it to a small value could improve the situation? If I have
understood well having to merge segments will keep similar stats on all
nodes. It's ok to have the indexing process a little bit slower.


2014-10-21 18:44 GMT+02:00 Erick Erickson <er...@gmail.com>:

> Giovanni:
>
> To see how this happens, consider a shard with a leader and two
> followers. Assume your autocommit interval is 60 seconds on each.
>
> This interval can expire at slightly different "wall clock" times.
> Even if the servers started perfectly in synch, they can get slightly
> out of sync. So, you index a bunch of docs and these replicas close
> the current segment and re-open a new segment with slightly different
> contents.
>
> Now docs come in that replace older docs. The tf/idf statistics
> _include_ deleted document data (which is purged on optimize). Given
> that doc X an be in different segments (or, more accurately, segments
> that get merged at different times on different machines), replica 1
> may have slightly different stats than replica 2, thus computing
> slightly different scores.
>
> Optimizing purges all data related to deleted documents, so it all
> regularizes itself on optimize.
>
> Best,
> Erick
>
> On Tue, Oct 21, 2014 at 11:08 AM, Giovanni Bricconi
> <gi...@banzai.it> wrote:
> > I noticed again the problem, now I was able to collect some data. in my
> > paste http://pastebin.com/nVwf327c you can see the result of the same
> query
> > issued twice, the 2nd and 3rd group are swapped.
> >
> > I pasted also the clusterstate and the core state for each core.
> >
> > The logs did'n show any problem related to indexing, only some malformed
> > query.
> >
> > After doing an optimize the problem disappeared.
> >
> > So, is the problem related to documents that where deleted from the
> index?
> >
> > The optimization took 5 minutes to complete
> >
> > 2014-10-21 11:41 GMT+02:00 Giovanni Bricconi <
> giovanni.bricconi@banzai.it>:
> >
> >> Nice!
> >> I will monitor the index and try this if the problem comes back.
> >> Actually the problem was due to small differences in score, so I think
> the
> >> problem has the same origin
> >>
> >> 2014-10-21 8:10 GMT+02:00 lboutros <bo...@gmail.com>:
> >>
> >>> Hi Giovanni,
> >>>
> >>> we had this problem as well.
> >>> The cause was that the different nodes have slightly different idf
> values.
> >>>
> >>> We solved this problem by doing an optimize operation which really
> remove
> >>> suppressed data.
> >>>
> >>> Ludovic.
> >>>
> >>>
> >>>
> >>> -----
> >>> Jouve
> >>> France.
> >>> --
> >>> View this message in context:
> >>>
> http://lucene.472066.n3.nabble.com/unstable-results-on-refresh-tp4164913p4165086.html
> >>> Sent from the Solr - User mailing list archive at Nabble.com.
> >>>
> >>
> >>
>

Re: unstable results on refresh

[Erick Erickson <er...@gmail.com>]

Giovanni:

To see how this happens, consider a shard with a leader and two
followers. Assume your autocommit interval is 60 seconds on each.

This interval can expire at slightly different "wall clock" times.
Even if the servers started perfectly in synch, they can get slightly
out of sync. So, you index a bunch of docs and these replicas close
the current segment and re-open a new segment with slightly different
contents.

Now docs come in that replace older docs. The tf/idf statistics
_include_ deleted document data (which is purged on optimize). Given
that doc X an be in different segments (or, more accurately, segments
that get merged at different times on different machines), replica 1
may have slightly different stats than replica 2, thus computing
slightly different scores.

Optimizing purges all data related to deleted documents, so it all
regularizes itself on optimize.

Best,
Erick

On Tue, Oct 21, 2014 at 11:08 AM, Giovanni Bricconi
<gi...@banzai.it> wrote:
> I noticed again the problem, now I was able to collect some data. in my
> paste http://pastebin.com/nVwf327c you can see the result of the same query
> issued twice, the 2nd and 3rd group are swapped.
>
> I pasted also the clusterstate and the core state for each core.
>
> The logs did'n show any problem related to indexing, only some malformed
> query.
>
> After doing an optimize the problem disappeared.
>
> So, is the problem related to documents that where deleted from the index?
>
> The optimization took 5 minutes to complete
>
> 2014-10-21 11:41 GMT+02:00 Giovanni Bricconi <gi...@banzai.it>:
>
>> Nice!
>> I will monitor the index and try this if the problem comes back.
>> Actually the problem was due to small differences in score, so I think the
>> problem has the same origin
>>
>> 2014-10-21 8:10 GMT+02:00 lboutros <bo...@gmail.com>:
>>
>>> Hi Giovanni,
>>>
>>> we had this problem as well.
>>> The cause was that the different nodes have slightly different idf values.
>>>
>>> We solved this problem by doing an optimize operation which really remove
>>> suppressed data.
>>>
>>> Ludovic.
>>>
>>>
>>>
>>> -----
>>> Jouve
>>> France.
>>> --
>>> View this message in context:
>>> http://lucene.472066.n3.nabble.com/unstable-results-on-refresh-tp4164913p4165086.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>
>>

Re: unstable results on refresh

[Giovanni Bricconi <gi...@banzai.it>]

I noticed again the problem, now I was able to collect some data. in my
paste http://pastebin.com/nVwf327c you can see the result of the same query
issued twice, the 2nd and 3rd group are swapped.

I pasted also the clusterstate and the core state for each core.

The logs did'n show any problem related to indexing, only some malformed
query.

After doing an optimize the problem disappeared.

So, is the problem related to documents that where deleted from the index?

The optimization took 5 minutes to complete

2014-10-21 11:41 GMT+02:00 Giovanni Bricconi <gi...@banzai.it>:

> Nice!
> I will monitor the index and try this if the problem comes back.
> Actually the problem was due to small differences in score, so I think the
> problem has the same origin
>
> 2014-10-21 8:10 GMT+02:00 lboutros <bo...@gmail.com>:
>
>> Hi Giovanni,
>>
>> we had this problem as well.
>> The cause was that the different nodes have slightly different idf values.
>>
>> We solved this problem by doing an optimize operation which really remove
>> suppressed data.
>>
>> Ludovic.
>>
>>
>>
>> -----
>> Jouve
>> France.
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/unstable-results-on-refresh-tp4164913p4165086.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>

Re: unstable results on refresh

[Giovanni Bricconi <gi...@banzai.it>]

Nice!
I will monitor the index and try this if the problem comes back.
Actually the problem was due to small differences in score, so I think the
problem has the same origin

2014-10-21 8:10 GMT+02:00 lboutros <bo...@gmail.com>:

> Hi Giovanni,
>
> we had this problem as well.
> The cause was that the different nodes have slightly different idf values.
>
> We solved this problem by doing an optimize operation which really remove
> suppressed data.
>
> Ludovic.
>
>
>
> -----
> Jouve
> France.
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/unstable-results-on-refresh-tp4164913p4165086.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: unstable results on refresh

[lboutros <bo...@gmail.com>]

Hi Giovanni,

we had this problem as well.
The cause was that the different nodes have slightly different idf values.

We solved this problem by doing an optimize operation which really remove
suppressed data.

Ludovic.



-----
Jouve
France.
--
View this message in context: http://lucene.472066.n3.nabble.com/unstable-results-on-refresh-tp4164913p4165086.html
Sent from the Solr - User mailing list archive at Nabble.com.