suggestions for DIH batchSize

[Joel Nylund <jn...@yahoo.com>]

Hi,

it looks like from looking at the code the default is 500, is the  
recommended setting for this?

Has anyone notice any significant performance/memory tradeoffs by  
making this much bigger?

thanks
Joel

Re: suggestions for DIH batchSize

[Marc Sturlese <ma...@gmail.com>]

If you want to retrieve a huge volume of rows you will end up with an
OutOfMemoryException due to the jdbc driver. Setting batchSize to -1 in your
data-config.xml (that internally will set it to Integer.MIN_VALUE) will make
the query to be executed in streaming, avoiding the memory exception.

Joel Nylund wrote:
> 
> Hi,
> 
> it looks like from looking at the code the default is 500, is the  
> recommended setting for this?
> 
> Has anyone notice any significant performance/memory tradeoffs by  
> making this much bigger?
> 
> thanks
> Joel
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/suggestions-for-DIH-batchSize-tp26894539p26897636.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: suggestions for DIH batchSize

[Noble Paul നോബിള്‍ नोब्ळ् <no...@corp.aol.com>]

A bigger batchSize results in increased memory usage.  I guess
performance should be slightly better with bigger values but not
verified.

On Wed, Dec 23, 2009 at 2:51 AM, Joel Nylund <jn...@yahoo.com> wrote:
> Hi,
>
> it looks like from looking at the code the default is 500, is the
> recommended setting for this?
>
> Has anyone notice any significant performance/memory tradeoffs by making
> this much bigger?
>
> thanks
> Joel
>
>



-- 
-----------------------------------------------------
Noble Paul | Systems Architect| AOL | http://aol.com