You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Jianbin Dai <dj...@yahoo.com> on 2009/06/21 03:23:53 UTC

Use DIH with large xml file

Hi,

I have about 50GB of data to be indexed each day using DIH. Some of the files are as large as 6GB. I set the JVM Xmx to be 3GB, but the DIH crashes on those big files. Is there any way to handle it?

Thanks.

JB

Re: Use DIH with large xml file

Posted by Erik Hatcher <er...@ehatchersolutions.com>.

How are you configuring DIH to read those files?  It is likely that  
you'll need at least as much RAM to the JVM as the largest file you're  
processing, though that depends entirely on how the file is being  
processed.

	Erik

On Jun 20, 2009, at 9:23 PM, Jianbin Dai wrote:

>
> Hi,
>
> I have about 50GB of data to be indexed each day using DIH. Some of  
> the files are as large as 6GB. I set the JVM Xmx to be 3GB, but the  
> DIH crashes on those big files. Is there any way to handle it?
>
> Thanks.
>
> JB
>
>
>