You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by maureen tanuwidjaja <au...@yahoo.com> on 2007/02/01 15:22:13 UTC

Building lucene index using 100 Gb Mobile HardDisk

Dear All,
  
  I was indexing 660,000 XML documents.The unoptimized index file was  successfully built in about 17 hrs...This index file resides in my D  drive which has the free space 38 Gb.This space is insufficient for  optimizing the index file -->I read Lucene documentation said about  its requirement of substantial temporary free disk space
  
  Hence,I now put my index file in the Mobile HardDisk which has the capacity of 100 Gb,and restart indexing.
  
  However...there is an exception saying that :
  
  java.io.IOException: There is not enough space on the disk
  
  when I index the 120,000th documents
  
  
  It is a pretty weird thing!!...I keep thinking what is happening...Is  there anything to do with the file system used?my D drive use NTFS file  system whilst this Mobile HDD use FAT32...
  
  
  Any comments/suggestion pls?
  
  THanks and best Regards,
  Maureen

 
---------------------------------
Food fight? Enjoy some healthy debate
in the Yahoo! Answers Food & Drink Q&A.

Re: Building lucene index using 100 Gb Mobile HardDisk

Posted by John Haxby <jc...@scalix.com>.
maureen tanuwidjaja wrote:
> Oh is it?I didn't know about that...so Is it means I cant use this Mobile HDD..
>
> Damien McCarthy <da...@propylon.com> wrote:  FAT 32 imposes a lower file size limitation than NTF. Attempts to create
> files greater that 4Gig on FAT32 will throw error you are seeing.
>   
Not at all, you just need to re-format the disk using a sensible file 
system.   If you're using Linux, that's ext3, if you're using Windows, NTFS.

If you have data on there you really want to keep and you can't back it 
up elsewhere while you do the re-format, then hunt down "gparted" -- it 
has a live CD that you can use to grow, shrink and move partitions so 
you'll be able to move stuff around your mobile disk while you shrink 
the FAT32 partition and grow the NTFS partition.  The shrink/grow/move 
cycle can take several hours though, depending on how often you have to 
do it (which in turn depends on how full the disk is).

jch

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Building lucene index using 100 Gb Mobile HardDisk

Posted by Daniel Noll <da...@nuix.com>.
maureen tanuwidjaja wrote:
> Oh is it?I didn't know about that...so Is it means I cant use this Mobile HDD..

You can always reformat the hard drive as NTFS.

Daniel

-- 
Daniel Noll

Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia    Ph: +61 2 9280 0699
Web: http://nuix.com/                               Fax: +61 2 9212 6902

This message is intended only for the named recipient. If you are not
the intended recipient you are notified that disclosing, copying,
distributing or taking any action in reliance on the contents of this
message or attachment is strictly prohibited.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Locking in Lucene 2.0

Posted by Michael McCandless <lu...@mikemccandless.com>.
Kadlabalu, Hareesh wrote:
> Hi,
> I am starting to work with Lucene 2.0 and I noticed that we can no
> longer create an FSDirectory using a LockFactory. 
> 
> Could someone point me to some discussion or documentation related to
> locking and what has changed in terms of best practices? It appears that
> the only way to build custom locking is to write my own Directory
> implementation. 

I think you mean Lucene trunk (not 2.0)?  LockFactory hasn't been
released yet.

With the trunk, you can still instantiate an FSDirectory with your own
LockFactory using FSDirectory.getDirectory(*)?

Maybe you are referring to the boolean "create" argument?  That was
deprecated/removed from FSDirectory.getDirectory as part of:

     http://issues.apache.org/jira/browse/LUCENE-773

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Locking in Lucene 2.0

Posted by "Kadlabalu, Hareesh" <Ha...@fatwire.com>.
Hi,
I am starting to work with Lucene 2.0 and I noticed that we can no
longer create an FSDirectory using a LockFactory. 

Could someone point me to some discussion or documentation related to
locking and what has changed in terms of best practices? It appears that
the only way to build custom locking is to write my own Directory
implementation. 

Thanks 
-hareesh


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: Building lucene index using 100 Gb Mobile HardDisk

Posted by maureen tanuwidjaja <au...@yahoo.com>.
Oh is it?I didn't know about that...so Is it means I cant use this Mobile HDD..

Damien McCarthy <da...@propylon.com> wrote:  FAT 32 imposes a lower file size limitation than NTF. Attempts to create
files greater that 4Gig on FAT32 will throw error you are seeing.

-----Original Message-----
From: maureen tanuwidjaja [mailto:autumn_musique@yahoo.com] 
Sent: 01 February 2007 14:22
To: java-user@lucene.apache.org
Subject: Building lucene index using 100 Gb Mobile HardDisk

Dear All,
  
  I was indexing 660,000 XML documents.The unoptimized index file was
successfully built in about 17 hrs...This index file resides in my D  drive
which has the free space 38 Gb.This space is insufficient for  optimizing
the index file -->I read Lucene documentation said about  its requirement of
substantial temporary free disk space
  
  Hence,I now put my index file in the Mobile HardDisk which has the
capacity of 100 Gb,and restart indexing.
  
  However...there is an exception saying that :
  
  java.io.IOException: There is not enough space on the disk
  
  when I index the 120,000th documents
  
  
  It is a pretty weird thing!!...I keep thinking what is happening...Is
there anything to do with the file system used?my D drive use NTFS file
system whilst this Mobile HDD use FAT32...
  
  
  Any comments/suggestion pls?
  
  THanks and best Regards,
  Maureen

 
---------------------------------
Food fight? Enjoy some healthy debate
in the Yahoo! Answers Food & Drink Q&A.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org



 
---------------------------------
Never Miss an Email
Stay connected with Yahoo! Mail on your mobile. Get started!

RE: Building lucene index using 100 Gb Mobile HardDisk

Posted by Damien McCarthy <da...@propylon.com>.
FAT 32 imposes a lower file size limitation than NTF. Attempts to create
files greater that 4Gig on FAT32 will throw error you are seeing.

-----Original Message-----
From: maureen tanuwidjaja [mailto:autumn_musique@yahoo.com] 
Sent: 01 February 2007 14:22
To: java-user@lucene.apache.org
Subject: Building lucene index using 100 Gb Mobile HardDisk

Dear All,
  
  I was indexing 660,000 XML documents.The unoptimized index file was
successfully built in about 17 hrs...This index file resides in my D  drive
which has the free space 38 Gb.This space is insufficient for  optimizing
the index file -->I read Lucene documentation said about  its requirement of
substantial temporary free disk space
  
  Hence,I now put my index file in the Mobile HardDisk which has the
capacity of 100 Gb,and restart indexing.
  
  However...there is an exception saying that :
  
  java.io.IOException: There is not enough space on the disk
  
  when I index the 120,000th documents
  
  
  It is a pretty weird thing!!...I keep thinking what is happening...Is
there anything to do with the file system used?my D drive use NTFS file
system whilst this Mobile HDD use FAT32...
  
  
  Any comments/suggestion pls?
  
  THanks and best Regards,
  Maureen

 
---------------------------------
Food fight? Enjoy some healthy debate
in the Yahoo! Answers Food & Drink Q&A.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org