You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Yogesh patel <yo...@gmail.com> on 2014/05/13 07:58:22 UTC

Lucene: Index Writer to write in multiple file instead make one heavy file

HI

I am using lucene 3.0.1. I am writing many documents with lucene
Indexwriter. But Indexwriter add all documents into file which becomes more
than 4GB in my case. so can i distribute files or partitioned ?

-- 




*Regards,Yogesh Patel*

Re: Lucene: Index Writer to write in multiple file instead make one heavy file

Posted by Michael McCandless <lu...@mikemccandless.com>.
Just don't call optimize...

In theory, you could make a custom Directory impl that would split a
single large file (from Lucene's standpoint) into multiple OS files,
but this ... would be a lot of work.  It's simpler to just not
optimize.

Mike McCandless

http://blog.mikemccandless.com


On Wed, May 14, 2014 at 10:09 AM, Yogesh patel
<yo...@gmail.com> wrote:
> In my case, it creates CFS(Compound File) As 10 GB. Can we split that file
> while optimize or writing index.
>
> Thanks
>
>
> On Wed, May 14, 2014 at 7:38 PM, Yogesh patel
> <yo...@gmail.com>wrote:
>
>> Thanks for reply!!!!!!!!!!!
>>
>> Can you please provide me sample code for it? I got the idea but i dont
>> know how to implement it.
>>
>> Thanks
>>
>>
>> On Tue, May 13, 2014 at 7:02 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>> You can tell the MergePolicy to limit the maximum size of segments it
>>> should merge.
>>>
>>> Also, you should try to upgrade: 3.0.1 is REALLY old.
>>>
>>> Mike McCandless
>>>
>>> http://blog.mikemccandless.com
>>>
>>>
>>> On Tue, May 13, 2014 at 1:58 AM, Yogesh patel
>>> <yo...@gmail.com> wrote:
>>> > HI
>>> >
>>> > I am using lucene 3.0.1. I am writing many documents with lucene
>>> > Indexwriter. But Indexwriter add all documents into file which becomes
>>> more
>>> > than 4GB in my case. so can i distribute files or partitioned ?
>>> >
>>> > --
>>> >
>>> >
>>> >
>>> >
>>> > *Regards,Yogesh Patel*
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>>
>> --
>>
>>
>>
>>
>> *Regards,Yogesh Patel*
>>
>>
>>
>
>
> --
>
>
>
>
> *Regards,Yogesh Patel*

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Lucene: Index Writer to write in multiple file instead make one heavy file

Posted by Yogesh patel <yo...@gmail.com>.
In my case, it creates CFS(Compound File) As 10 GB. Can we split that file
while optimize or writing index.

Thanks


On Wed, May 14, 2014 at 7:38 PM, Yogesh patel
<yo...@gmail.com>wrote:

> Thanks for reply!!!!!!!!!!!
>
> Can you please provide me sample code for it? I got the idea but i dont
> know how to implement it.
>
> Thanks
>
>
> On Tue, May 13, 2014 at 7:02 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> You can tell the MergePolicy to limit the maximum size of segments it
>> should merge.
>>
>> Also, you should try to upgrade: 3.0.1 is REALLY old.
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Tue, May 13, 2014 at 1:58 AM, Yogesh patel
>> <yo...@gmail.com> wrote:
>> > HI
>> >
>> > I am using lucene 3.0.1. I am writing many documents with lucene
>> > Indexwriter. But Indexwriter add all documents into file which becomes
>> more
>> > than 4GB in my case. so can i distribute files or partitioned ?
>> >
>> > --
>> >
>> >
>> >
>> >
>> > *Regards,Yogesh Patel*
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
>
>
>
>
> *Regards,Yogesh Patel*
>
>
>


-- 




*Regards,Yogesh Patel*

Re: Lucene: Index Writer to write in multiple file instead make one heavy file

Posted by Yogesh patel <yo...@gmail.com>.
Thanks for reply!!!!!!!!!!!

Can you please provide me sample code for it? I got the idea but i dont
know how to implement it.

Thanks


On Tue, May 13, 2014 at 7:02 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> You can tell the MergePolicy to limit the maximum size of segments it
> should merge.
>
> Also, you should try to upgrade: 3.0.1 is REALLY old.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, May 13, 2014 at 1:58 AM, Yogesh patel
> <yo...@gmail.com> wrote:
> > HI
> >
> > I am using lucene 3.0.1. I am writing many documents with lucene
> > Indexwriter. But Indexwriter add all documents into file which becomes
> more
> > than 4GB in my case. so can i distribute files or partitioned ?
> >
> > --
> >
> >
> >
> >
> > *Regards,Yogesh Patel*
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 




*Regards,Yogesh Patel*

Re: Lucene: Index Writer to write in multiple file instead make one heavy file

Posted by Michael McCandless <lu...@mikemccandless.com>.
You can tell the MergePolicy to limit the maximum size of segments it
should merge.

Also, you should try to upgrade: 3.0.1 is REALLY old.

Mike McCandless

http://blog.mikemccandless.com


On Tue, May 13, 2014 at 1:58 AM, Yogesh patel
<yo...@gmail.com> wrote:
> HI
>
> I am using lucene 3.0.1. I am writing many documents with lucene
> Indexwriter. But Indexwriter add all documents into file which becomes more
> than 4GB in my case. so can i distribute files or partitioned ?
>
> --
>
>
>
>
> *Regards,Yogesh Patel*

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: Lucene: Index Writer to write in multiple file instead make one heavy file

Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
Yogesh patel [yogeshpateldaiict@gmail.com] wrote:
> I am using lucene 3.0.1. I am writing many documents with lucene
> Indexwriter. But Indexwriter add all documents into file which becomes more
> than 4GB in my case. so can i distribute files or partitioned ?

Normally Lucene does not produce a single large file. I guess you are performing an optimize. Don't do that (it is not really recommended anyway) and you should have multiple smaller files.

If that was not clear, then please show us the part of your code that handles index updates.

- Toke Eskildsen
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org