ConcurrentUpdateSolrServer flush on size of documents rather than queue size

[Mike Hugo <mi...@piragua.com>]

Does anyone know if a version of ConcurrentUpdateSolrServer exists that
would use the size in memory of the queue to decide when to send documents
to the solr server?

For example, if I set up a ConcurrentUpdateSolrServer with 4 threads and a
batch size of 200 that works if my documents are small.  But if I am
building up documents that have a lot of text, I have run into an
OutOfMemory exception in my process that builds the docs.  The document
sizes are variable.

What I'd like to be able to do is submit documents to the solr sever when
the size of the queue reaches (or is greater than) 200MB or something like
that, so rather than specifying the number of document to put in the queue,
I'd specify the size in MB to build up before submitting.

Does something like this exist already?

Thanks,

Mike

Re: ConcurrentUpdateSolrServer flush on size of documents rather than queue size

[Otis Gospodnetic <ot...@gmail.com>]

H Mike,

Doesn't exist as far as I know, but would be a nice contribution.

Otis
--
Solr & ElasticSearch Support
http://sematext.com/





On Fri, Mar 1, 2013 at 11:30 AM, Mike Hugo <mi...@piragua.com> wrote:

> Does anyone know if a version of ConcurrentUpdateSolrServer exists that
> would use the size in memory of the queue to decide when to send documents
> to the solr server?
>
> For example, if I set up a ConcurrentUpdateSolrServer with 4 threads and a
> batch size of 200 that works if my documents are small.  But if I am
> building up documents that have a lot of text, I have run into an
> OutOfMemory exception in my process that builds the docs.  The document
> sizes are variable.
>
> What I'd like to be able to do is submit documents to the solr sever when
> the size of the queue reaches (or is greater than) 200MB or something like
> that, so rather than specifying the number of document to put in the queue,
> I'd specify the size in MB to build up before submitting.
>
> Does something like this exist already?
>
> Thanks,
>
> Mike
>