EmbeddedSolrServer and BinaryRequestWriter

[Phil Hagelberg <ph...@hagelb.org>]

I'm trying to reduce memory usage when indexing, and I see that using
the binary format may be a good way to do this. Unfortunately I can't
see a way to do this using the EmbeddedSolrServer since only the
CommonsHttpSolrServer has a setRequestWriter method. If I'm running out
of memory constructing XML request documents, does that mean I just have
to switch away from the EmbeddedSolrServer?

I understand I can stream requests if I'm just indexing files already on
disk, but I'm constructing them on the fly, and I run out of memory
constructing the XML document to submit to solr, not in actual indexing,
so it seems writing the document to disk would run into the same problems.

thanks,
Phil

Re: EmbeddedSolrServer and BinaryRequestWriter

[Otis Gospodnetic <ot...@yahoo.com>]

Hi,

Running out of memory because of XML document when indexing documents sounds very weird/suspicious.
Are you running out of memory on the server side?
Are you indexing super large batches?
How big is your JVM heap?
How big is your ramBufferSizeMB?


Otis 
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch



----- Original Message ----
> From: Phil Hagelberg <ph...@hagelb.org>
> To: solr-user@lucene.apache.org
> Sent: Thu, January 14, 2010 5:57:31 PM
> Subject: EmbeddedSolrServer and BinaryRequestWriter
> 
> 
> I'm trying to reduce memory usage when indexing, and I see that using
> the binary format may be a good way to do this. Unfortunately I can't
> see a way to do this using the EmbeddedSolrServer since only the
> CommonsHttpSolrServer has a setRequestWriter method. If I'm running out
> of memory constructing XML request documents, does that mean I just have
> to switch away from the EmbeddedSolrServer?
> 
> I understand I can stream requests if I'm just indexing files already on
> disk, but I'm constructing them on the fly, and I run out of memory
> constructing the XML document to submit to solr, not in actual indexing,
> so it seems writing the document to disk would run into the same problems.
> 
> thanks,
> Phil