You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by dprasadx <dp...@gmail.com> on 2012/01/31 16:21:50 UTC
Solrj commit affecting documents count
Hi, I am using solrj server to commit few changes in the data into the master
index through a java program. It works OK unless we do not do a full-import.
But when I do a full-import (say for 800 records), and if I perform a solrj
commit in between the full-import indexing, I see a commit happens to the
index at that time, and my total documents in the index gets affected.
For example while indexing 800 odd records, if solrj commit happens in the
middle of indexing, I see the documents count in the master has changed to
400... and once the full-import indexing is completed, I get back all the
800 records.
Is there a way to commit the changes into the master without affecting the
count while the indexing is going on ? Please help.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solrj-commit-affecting-documents-count-tp3703646p3703646.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solrj commit affecting documents count
Posted by Erick Erickson <er...@gmail.com>.
I really don't understand this. It seems that not doing a full import
and just re-submitting the changed documents is something
you should consider.
Or just don't commit. Or consider using two cores, the idea here
is that you have your "live" core that serves requests, and
indexing to your new core, then swapping the cores after the
indexing is complete.
Best
Erick
On Tue, Jan 31, 2012 at 11:54 AM, dprasadx <dp...@gmail.com> wrote:
> I need to index our data 10 times a day due to frequent data changes.
> We have placed a mechanism where the data entered by the user in the front
> end is submitted into the solr index directly through solrj server. So, if
> the solrj commit occurs during the middle of indexing, I lose all the
> records in the index and the index contains only the documents during that
> point of commit. Once the indexing is completed, I get back all the records.
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solrj-commit-affecting-documents-count-tp3703646p3703972.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solrj commit affecting documents count
Posted by dprasadx <dp...@gmail.com>.
I need to index our data 10 times a day due to frequent data changes.
We have placed a mechanism where the data entered by the user in the front
end is submitted into the solr index directly through solrj server. So, if
the solrj commit occurs during the middle of indexing, I lose all the
records in the index and the index contains only the documents during that
point of commit. Once the indexing is completed, I get back all the records.
--
View this message in context: http://lucene.472066.n3.nabble.com/Solrj-commit-affecting-documents-count-tp3703646p3703972.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solrj commit affecting documents count
Posted by Ahmet Arslan <io...@yahoo.com>.
Not sure if this helps but full-import deletes whole index using *:* query in the beginning of import. Can you can disable this behavior by using &clean=false&command=full-import
--- On Tue, 1/31/12, Andre Bois-Crettez <an...@kelkoo.com> wrote:
> From: Andre Bois-Crettez <an...@kelkoo.com>
> Subject: Re: Solrj commit affecting documents count
> To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
> Date: Tuesday, January 31, 2012, 6:35 PM
> Why do you commit in the middle of a
> full import then, if you don't have
> to ?
>
> dprasadx wrote:
> > Hi, I am using solrj server to commit few changes in
> the data into the master
> > index through a java program. It works OK unless we do
> not do a full-import.
> > But when I do a full-import (say for 800 records), and
> if I perform a solrj
> > commit in between the full-import indexing, I see a
> commit happens to the
> > index at that time, and my total documents in the index
> gets affected.
> > For example while indexing 800 odd records, if solrj
> commit happens in the
> > middle of indexing, I see the documents count in the
> master has changed to
> > 400... and once the full-import indexing is completed,
> I get back all the
> > 800 records.
> >
> > Is there a way to commit the changes into the master
> without affecting the
> > count while the indexing is going on ? Please help.
> >
> > --
> > View this message in context: http://lucene.472066.n3.nabble.com/Solrj-commit-affecting-documents-count-tp3703646p3703646.html
> > Sent from the Solr - User mailing list archive at
> Nabble.com.
> >
> >
>
> --
> André Bois-Crettez
>
> Search technology, Kelkoo
> http://www.kelkoo.com/
>
>
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de € 4.168.964,30
> Siège social : 8, rue du Sentier 75002 Paris
> 425 093 069 RCS Paris
>
> Ce message et les pièces jointes sont confidentiels et
> établis à l'attention exclusive de leurs destinataires. Si
> vous n'êtes pas le destinataire de ce message, merci de le
> détruire et d'en avertir l'expéditeur.
>
Re: Solrj commit affecting documents count
Posted by Andre Bois-Crettez <an...@kelkoo.com>.
Why do you commit in the middle of a full import then, if you don't have
to ?
dprasadx wrote:
> Hi, I am using solrj server to commit few changes in the data into the master
> index through a java program. It works OK unless we do not do a full-import.
> But when I do a full-import (say for 800 records), and if I perform a solrj
> commit in between the full-import indexing, I see a commit happens to the
> index at that time, and my total documents in the index gets affected.
> For example while indexing 800 odd records, if solrj commit happens in the
> middle of indexing, I see the documents count in the master has changed to
> 400... and once the full-import indexing is completed, I get back all the
> 800 records.
>
> Is there a way to commit the changes into the master without affecting the
> count while the indexing is going on ? Please help.
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solrj-commit-affecting-documents-count-tp3703646p3703646.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
André Bois-Crettez
Search technology, Kelkoo
http://www.kelkoo.com/
Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris
Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur.