Solr Indexing with huge number of documents

[bsargurunathan <bs...@gmail.com>]

Hi Everyone,

In Solr Indexing, I needs to index millions of millions records in single
time from xml file.
While I am doing the indexing, I am constructing the xml file and passing to
the solr.
But right now I am controlling the record count and based on the record
count the xml will create and it will index into solr.
But the requirement is the indexing should be happen continues, if I did not
control the the count I will get the Socket exception.

So what is the exact solution for that? How can I overcome it?


Thanks,
Guru 



--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Indexing-with-huge-number-of-documents-tp4029094.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Indexing with huge number of documents

[Gora Mohanty <go...@mimirtech.com>]

On 26 December 2012 15:00, bsargurunathan <bs...@gmail.com> wrote:
> Hi Everyone,
>
> In Solr Indexing, I needs to index millions of millions records in single
> time from xml file.
> While I am doing the indexing, I am constructing the xml file and passing to
> the solr.
> But right now I am controlling the record count and based on the record
> count the xml will create and it will index into solr.
> But the requirement is the indexing should be happen continues, if I did not
> control the the count I will get the Socket exception.
>
> So what is the exact solution for that? How can I overcome it?

Sorry, but your question is not very clear. Maybe you
could rephrase it?

Yes, it is a good idea to break up a large number of
records into multiple XML files. You will need to
build an indexing layer that does this, and also
recovers gracefully from network errors. This should
not be too difficult.

Regards,
Gora

P.S. Are you really serious about millions of millions,
       i.e., of the order of 10^12, records? If so, you
       will need to worry about many more things in
       order to ensure scalability.