You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Ranganathan, Sharmila" <sr...@library.rochester.edu> on 2009/10/19 18:19:29 UTC
Solr commits before documents are added
Hi,
My application indexes huge number of documents(like in millions). Below
is the snapshot of my code where I add all documents to Solr, and then
at last issue commit command. I use Solrj. I find that last few
documents are not committed to Solr. Is this because adding documents
to Solr took longer time and it reached commit command even before it
finished adding documents? Is there are way to ensure that solr waits
for all documents to be added and then commits? Please advise me how to
solve this issue.
For loop
solrServer.add(doc); // Add document to Solr
End for loop
solrServer.commit(); // Commit to Solr
Thanks,
Sharmila
RE: Solr commits before documents are added
Posted by SharmilaR <sr...@library.rochester.edu>.
I am not using master slave setup. To give more information, I use
EmbeddedSolrServer.
- Sharmila
Feak, Todd wrote:
>
> Any chance you are indexing to a Master, then synching to a Slave and you
> aren't seeing those last 20 on the Slave?
>
> There is an issue with synching between Master and Slave that we've
> experienced. If the last commit is very small (20 sounds possible!) it can
> occur in the same clock second on that machine. The Master will see the
> commit and its index will show the data fine. However, the Slave cannot
> see the second commit on the same clock second, so it will be missing the
> last 20 due to sync between the two.
>
> It's an edge case, but we ran into it recently.
>
> -Todd
>
> -----Original Message-----
> From: SharmilaR [mailto:sranganathan@library.rochester.edu]
> Sent: Monday, October 19, 2009 1:07 PM
> To: solr-user@lucene.apache.org
> Subject: RE: Solr commits before documents are added
>
>
> Solr version is 1.3
> I am indexing total of 1.4 million documents. Yes, I
> commit(waitFlush="true"
> waitSearcher="true") every 100k documents and then one at the end.
> I have a counter next to addDoc(SolrDocument) statement to keep track of
> number of documents added. When I query Solr after commit, the total
> number
> of documents returned does not match the number of documents added. This
> happens only when I index millions of documents and not when I index like
> 500 documents. In this case, I know its the last 20 documents which are
> not
> committed because each document has a field 'RECORD_ID' which is assigned
> sequential number(in java code). When I query Solr using Solr admin
> interface, the documents with last 20 RECORD_ID are missing.(example the
> last id is 999,980 instead of 1,000,000)
>
> - Sharmila
>
>
> Feak, Todd wrote:
>>
>> A few questions to help the troubleshooting.
>>
>> Solr version #?
>>
>> Is there just 1 commit through Solrj for the millions of documents?
>>
>> Or do you do it on a regular interval (every 100k documents for example)
>> and then one at the end to be sure?
>>
>> How are you observing that the last few didn't make it in? Are you
>> looking
>> at a slave or master?
>>
>> -Todd
>>
>>
> -----Original Message-----
> From: Ranganathan, Sharmila [mailto:sranganathan@library.rochester.edu]
> Sent: Monday, October 19, 2009 9:19 AM
> To: solr-user@lucene.apache.org
> Subject: Solr commits before documents are added
>
> Hi,
>
> My application indexes huge number of documents(like in millions). Below
> is the snapshot of my code where I add all documents to Solr, and then
> at last issue commit command. I use Solrj. I find that last few
> documents are not committed to Solr. Is this because adding documents
> to Solr took longer time and it reached commit command even before it
> finished adding documents? Is there are way to ensure that solr waits
> for all documents to be added and then commits? Please advise me how to
> solve this issue.
>
>
>
> For loop
>
> solrServer.add(doc); // Add document to Solr
>
> End for loop
>
> solrServer.commit(); // Commit to Solr
>
>
>
>
>
> Thanks,
>
> Sharmila
>
>
>
>
>
> --
> View this message in context:
> http://www.nabble.com/Solr-commits-before-documents-are-added-tp25961191p25964770.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
>
--
View this message in context: http://www.nabble.com/Solr-commits-before-documents-are-added-tp25961191p25978477.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Solr commits before documents are added
Posted by "Feak, Todd" <To...@smss.sony.com>.
Any chance you are indexing to a Master, then synching to a Slave and you aren't seeing those last 20 on the Slave?
There is an issue with synching between Master and Slave that we've experienced. If the last commit is very small (20 sounds possible!) it can occur in the same clock second on that machine. The Master will see the commit and its index will show the data fine. However, the Slave cannot see the second commit on the same clock second, so it will be missing the last 20 due to sync between the two.
It's an edge case, but we ran into it recently.
-Todd
-----Original Message-----
From: SharmilaR [mailto:sranganathan@library.rochester.edu]
Sent: Monday, October 19, 2009 1:07 PM
To: solr-user@lucene.apache.org
Subject: RE: Solr commits before documents are added
Solr version is 1.3
I am indexing total of 1.4 million documents. Yes, I commit(waitFlush="true"
waitSearcher="true") every 100k documents and then one at the end.
I have a counter next to addDoc(SolrDocument) statement to keep track of
number of documents added. When I query Solr after commit, the total number
of documents returned does not match the number of documents added. This
happens only when I index millions of documents and not when I index like
500 documents. In this case, I know its the last 20 documents which are not
committed because each document has a field 'RECORD_ID' which is assigned
sequential number(in java code). When I query Solr using Solr admin
interface, the documents with last 20 RECORD_ID are missing.(example the
last id is 999,980 instead of 1,000,000)
- Sharmila
Feak, Todd wrote:
>
> A few questions to help the troubleshooting.
>
> Solr version #?
>
> Is there just 1 commit through Solrj for the millions of documents?
>
> Or do you do it on a regular interval (every 100k documents for example)
> and then one at the end to be sure?
>
> How are you observing that the last few didn't make it in? Are you looking
> at a slave or master?
>
> -Todd
>
>
-----Original Message-----
From: Ranganathan, Sharmila [mailto:sranganathan@library.rochester.edu]
Sent: Monday, October 19, 2009 9:19 AM
To: solr-user@lucene.apache.org
Subject: Solr commits before documents are added
Hi,
My application indexes huge number of documents(like in millions). Below
is the snapshot of my code where I add all documents to Solr, and then
at last issue commit command. I use Solrj. I find that last few
documents are not committed to Solr. Is this because adding documents
to Solr took longer time and it reached commit command even before it
finished adding documents? Is there are way to ensure that solr waits
for all documents to be added and then commits? Please advise me how to
solve this issue.
For loop
solrServer.add(doc); // Add document to Solr
End for loop
solrServer.commit(); // Commit to Solr
Thanks,
Sharmila
--
View this message in context: http://www.nabble.com/Solr-commits-before-documents-are-added-tp25961191p25964770.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr commits before documents are added
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Mon, Oct 19, 2009 at 7:39 PM, Lance Norskog <go...@gmail.com> wrote:
> commit(waitFlush="true", waitSearcher="true") waits for the entire
> operation and when it finishes, all 1 million documents should be
> searchable.
That waits for the commit to complete, but not any adds that may be
happening in parallel (that's pretty much impossible).
If the client uses multiple threads to do adds, it's currently the
responsibility of the client to wait for all of the pending adds to
complete before calling commit.
-Yonik
http://www.lucidimagination.com
> Please try this same test with Solr 1.4 and post your results. To make
> it easier, here is the first release candidate:
>
> http://people.apache.org/~gsingers/solr/1.4.0-RC/
>
>
> On Mon, Oct 19, 2009 at 1:06 PM, SharmilaR
> <sr...@library.rochester.edu> wrote:
>>
>> Solr version is 1.3
>> I am indexing total of 1.4 million documents. Yes, I commit(waitFlush="true"
>> waitSearcher="true") every 100k documents and then one at the end.
>> I have a counter next to addDoc(SolrDocument) statement to keep track of
>> number of documents added. When I query Solr after commit, the total number
>> of documents returned does not match the number of documents added. This
>> happens only when I index millions of documents and not when I index like
>> 500 documents. In this case, I know its the last 20 documents which are not
>> committed because each document has a field 'RECORD_ID' which is assigned
>> sequential number(in java code). When I query Solr using Solr admin
>> interface, the documents with last 20 RECORD_ID are missing.(example the
>> last id is 999,980 instead of 1,000,000)
>>
>> - Sharmila
>>
>>
>> Feak, Todd wrote:
>>>
>>> A few questions to help the troubleshooting.
>>>
>>> Solr version #?
>>>
>>> Is there just 1 commit through Solrj for the millions of documents?
>>>
>>> Or do you do it on a regular interval (every 100k documents for example)
>>> and then one at the end to be sure?
>>>
>>> How are you observing that the last few didn't make it in? Are you looking
>>> at a slave or master?
>>>
>>> -Todd
>>>
>>>
>> -----Original Message-----
>> From: Ranganathan, Sharmila [mailto:sranganathan@library.rochester.edu]
>> Sent: Monday, October 19, 2009 9:19 AM
>> To: solr-user@lucene.apache.org
>> Subject: Solr commits before documents are added
>>
>> Hi,
>>
>> My application indexes huge number of documents(like in millions). Below
>> is the snapshot of my code where I add all documents to Solr, and then
>> at last issue commit command. I use Solrj. I find that last few
>> documents are not committed to Solr. Is this because adding documents
>> to Solr took longer time and it reached commit command even before it
>> finished adding documents? Is there are way to ensure that solr waits
>> for all documents to be added and then commits? Please advise me how to
>> solve this issue.
>>
>>
>>
>> For loop
>>
>> solrServer.add(doc); // Add document to Solr
>>
>> End for loop
>>
>> solrServer.commit(); // Commit to Solr
>>
>>
>>
>>
>>
>> Thanks,
>>
>> Sharmila
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://www.nabble.com/Solr-commits-before-documents-are-added-tp25961191p25964770.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: Solr commits before documents are added
Posted by Lance Norskog <go...@gmail.com>.
commit(waitFlush="true", waitSearcher="true") waits for the entire
operation and when it finishes, all 1 million documents should be
searchable.
Please try this same test with Solr 1.4 and post your results. To make
it easier, here is the first release candidate:
http://people.apache.org/~gsingers/solr/1.4.0-RC/
On Mon, Oct 19, 2009 at 1:06 PM, SharmilaR
<sr...@library.rochester.edu> wrote:
>
> Solr version is 1.3
> I am indexing total of 1.4 million documents. Yes, I commit(waitFlush="true"
> waitSearcher="true") every 100k documents and then one at the end.
> I have a counter next to addDoc(SolrDocument) statement to keep track of
> number of documents added. When I query Solr after commit, the total number
> of documents returned does not match the number of documents added. This
> happens only when I index millions of documents and not when I index like
> 500 documents. In this case, I know its the last 20 documents which are not
> committed because each document has a field 'RECORD_ID' which is assigned
> sequential number(in java code). When I query Solr using Solr admin
> interface, the documents with last 20 RECORD_ID are missing.(example the
> last id is 999,980 instead of 1,000,000)
>
> - Sharmila
>
>
> Feak, Todd wrote:
>>
>> A few questions to help the troubleshooting.
>>
>> Solr version #?
>>
>> Is there just 1 commit through Solrj for the millions of documents?
>>
>> Or do you do it on a regular interval (every 100k documents for example)
>> and then one at the end to be sure?
>>
>> How are you observing that the last few didn't make it in? Are you looking
>> at a slave or master?
>>
>> -Todd
>>
>>
> -----Original Message-----
> From: Ranganathan, Sharmila [mailto:sranganathan@library.rochester.edu]
> Sent: Monday, October 19, 2009 9:19 AM
> To: solr-user@lucene.apache.org
> Subject: Solr commits before documents are added
>
> Hi,
>
> My application indexes huge number of documents(like in millions). Below
> is the snapshot of my code where I add all documents to Solr, and then
> at last issue commit command. I use Solrj. I find that last few
> documents are not committed to Solr. Is this because adding documents
> to Solr took longer time and it reached commit command even before it
> finished adding documents? Is there are way to ensure that solr waits
> for all documents to be added and then commits? Please advise me how to
> solve this issue.
>
>
>
> For loop
>
> solrServer.add(doc); // Add document to Solr
>
> End for loop
>
> solrServer.commit(); // Commit to Solr
>
>
>
>
>
> Thanks,
>
> Sharmila
>
>
>
>
>
> --
> View this message in context: http://www.nabble.com/Solr-commits-before-documents-are-added-tp25961191p25964770.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Lance Norskog
goksron@gmail.com
RE: Solr commits before documents are added
Posted by SharmilaR <sr...@library.rochester.edu>.
Solr version is 1.3
I am indexing total of 1.4 million documents. Yes, I commit(waitFlush="true"
waitSearcher="true") every 100k documents and then one at the end.
I have a counter next to addDoc(SolrDocument) statement to keep track of
number of documents added. When I query Solr after commit, the total number
of documents returned does not match the number of documents added. This
happens only when I index millions of documents and not when I index like
500 documents. In this case, I know its the last 20 documents which are not
committed because each document has a field 'RECORD_ID' which is assigned
sequential number(in java code). When I query Solr using Solr admin
interface, the documents with last 20 RECORD_ID are missing.(example the
last id is 999,980 instead of 1,000,000)
- Sharmila
Feak, Todd wrote:
>
> A few questions to help the troubleshooting.
>
> Solr version #?
>
> Is there just 1 commit through Solrj for the millions of documents?
>
> Or do you do it on a regular interval (every 100k documents for example)
> and then one at the end to be sure?
>
> How are you observing that the last few didn't make it in? Are you looking
> at a slave or master?
>
> -Todd
>
>
-----Original Message-----
From: Ranganathan, Sharmila [mailto:sranganathan@library.rochester.edu]
Sent: Monday, October 19, 2009 9:19 AM
To: solr-user@lucene.apache.org
Subject: Solr commits before documents are added
Hi,
My application indexes huge number of documents(like in millions). Below
is the snapshot of my code where I add all documents to Solr, and then
at last issue commit command. I use Solrj. I find that last few
documents are not committed to Solr. Is this because adding documents
to Solr took longer time and it reached commit command even before it
finished adding documents? Is there are way to ensure that solr waits
for all documents to be added and then commits? Please advise me how to
solve this issue.
For loop
solrServer.add(doc); // Add document to Solr
End for loop
solrServer.commit(); // Commit to Solr
Thanks,
Sharmila
--
View this message in context: http://www.nabble.com/Solr-commits-before-documents-are-added-tp25961191p25964770.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Solr commits before documents are added
Posted by "Feak, Todd" <To...@smss.sony.com>.
A few questions to help the troubleshooting.
Solr version #?
Is there just 1 commit through Solrj for the millions of documents?
Or do you do it on a regular interval (every 100k documents for example) and then one at the end to be sure?
How are you observing that the last few didn't make it in? Are you looking at a slave or master?
-Todd
-----Original Message-----
From: Ranganathan, Sharmila [mailto:sranganathan@library.rochester.edu]
Sent: Monday, October 19, 2009 9:19 AM
To: solr-user@lucene.apache.org
Subject: Solr commits before documents are added
Hi,
My application indexes huge number of documents(like in millions). Below
is the snapshot of my code where I add all documents to Solr, and then
at last issue commit command. I use Solrj. I find that last few
documents are not committed to Solr. Is this because adding documents
to Solr took longer time and it reached commit command even before it
finished adding documents? Is there are way to ensure that solr waits
for all documents to be added and then commits? Please advise me how to
solve this issue.
For loop
solrServer.add(doc); // Add document to Solr
End for loop
solrServer.commit(); // Commit to Solr
Thanks,
Sharmila