You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Doug Cutting <cu...@apache.org> on 2004/12/23 21:28:55 UTC

Re: CFS file and file formats

It would be useful to have a command-line utility (i.e., a static 
main(String[]) method somewhere) that lists the files and sizes 
contained inside a CFS file, and perhaps even an option to unpack it. 
Anyone care to contribute this method?

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: CFS file and file formats

Posted by Bernhard Messer <bm...@apache.org>.
Doug Cutting schrieb:

> Bernhard Messer wrote:
>
>> Why not implementing a small utility class, f.e CompoundFileUtil.java 
>> within the org.apache.lucene.index Package ? This class could be 
>> public and implement the necessary functionality. This is what i 
>> would prefer, because we don't have to change the visibility of 
>> CompoundFileReader or other parts of the API. The other option would 
>> be to add a public static method to IndexReader class. But i don't 
>> like to overwhelm IndexReader with a method, just a very small 
>> audience would use.
>
>
> Currently IndexWriter is the only public place in the API where the 
> compound format appears.  So, until we decide to expose index formats 
> more systematically, I think this should stay at the IndexReader level.
>
> Thus I would prefer a main() on IndexReader that had various commands, 
> perhaps something like:
>
>   java org.apache.lucene.index.IndexReader <dir> cfs list
>   java org.apache.lucene.index.IndexReader <dir> cfs extract
>   java org.apache.lucene.index.IndexReader <dir> unlock
>   java org.apache.lucene.index.IndexReader <dir> list-segments
>
this looks ok for me for the moment, but i still favorite an additional 
class within the same package. Something like "IndexUtil" which could 
hold all this kind of functionality. If there is some time over the 
weekend, i will have a look on it how much effort it will be to have 
those 4 methods in IndexReader.

Bernhard

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: CFS file and file formats

Posted by Doug Cutting <cu...@apache.org>.
Bernhard Messer wrote:
> Why not implementing a small utility class, f.e CompoundFileUtil.java 
> within the org.apache.lucene.index Package ? This class could be public 
> and implement the necessary functionality. This is what i would prefer, 
> because we don't have to change the visibility of CompoundFileReader or 
> other parts of the API. The other option would be to add a public static 
> method to IndexReader class. But i don't like to overwhelm IndexReader 
> with a method, just a very small audience would use.

Currently IndexWriter is the only public place in the API where the 
compound format appears.  So, until we decide to expose index formats 
more systematically, I think this should stay at the IndexReader level.

Thus I would prefer a main() on IndexReader that had various commands, 
perhaps something like:

   java org.apache.lucene.index.IndexReader <dir> cfs list
   java org.apache.lucene.index.IndexReader <dir> cfs extract
   java org.apache.lucene.index.IndexReader <dir> unlock
   java org.apache.lucene.index.IndexReader <dir> list-segments

etc.

Doug


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: CFS file and file formats

Posted by Bernhard Messer <bm...@apache.org>.
Doug Cutting schrieb:

> Bernhard Messer wrote:
>
>>> I understand the technical reason for main() there, but logically this
>>> belongs to an external utility class, I think.
>>>
>> Otis you are right, i already thought about it. It could be simply 
>> moved to a newly created class in org.apache.lucene.util package. But 
>> then we have to change the visibility of CompoundFileReader to 
>> public. I have no problems with a public CompoundFileReader class. 
>> Does anybody see a reason that the visibility of CompoundFileReader 
>> should not be changed to public ?
>
>
> Yes.  Historically we've tried hard in Lucene to not make things 
> public unless they're useful to users.  We try to keep the public API 
> as small as possible.  This makes the documentation easier to read and 
> also makes the system easier to maintain.  CompoundFileReader is an 
> implementation class, not a user class.
>
Why not implementing a small utility class, f.e CompoundFileUtil.java 
within the org.apache.lucene.index Package ? This class could be public 
and implement the necessary functionality. This is what i would prefer, 
because we don't have to change the visibility of CompoundFileReader or 
other parts of the API. The other option would be to add a public static 
method to IndexReader class. But i don't like to overwhelm IndexReader 
with a method, just a very small audience would use.

Bernhard

>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: CFS file and file formats

Posted by Doug Cutting <cu...@apache.org>.
Bernhard Messer wrote:
>> I understand the technical reason for main() there, but logically this
>> belongs to an external utility class, I think.
>>
> Otis you are right, i already thought about it. It could be simply moved 
> to a newly created class in org.apache.lucene.util package. But then we 
> have to change the visibility of CompoundFileReader to public. I have no 
> problems with a public CompoundFileReader class. Does anybody see a 
> reason that the visibility of CompoundFileReader should not be changed 
> to public ?

Yes.  Historically we've tried hard in Lucene to not make things public 
unless they're useful to users.  We try to keep the public API as small 
as possible.  This makes the documentation easier to read and also makes 
the system easier to maintain.  CompoundFileReader is an implementation 
class, not a user class.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: CFS file and file formats

Posted by Bernhard Messer <bm...@apache.org>.
>Hello,
>
>I understand the technical reason for main() there, but logically this
>belongs to an external utility class, I think.
>
>  
>
Otis you are right, i already thought about it. It could be simply moved 
to a newly created class in org.apache.lucene.util package. But then we 
have to change the visibility of CompoundFileReader to public. I have no 
problems with a public CompoundFileReader class. Does anybody see a 
reason that the visibility of CompoundFileReader should not be changed 
to public ?

Bernhard

>
>--- Bernhard Messer <bm...@apache.org> wrote:
>
>  
>
>>hi,
>>
>>i already had a look at Garrett's implementation. I made some smaller
>>
>>changes to improve the performance when extracting the files from the
>>
>>compound. All tests work fine and the index is usable after
>>extraction. 
>>The new functionality is added as a public static void main () to 
>>CompoundFileReader because of the reduced visibility (package) of 
>>CompoundFileReader itself. It will be committed this afternoon.
>>
>>Bernhard
>>
>>    
>>
>>>Doug Cutting wrote:
>>>
>>>      
>>>
>>>>It would be useful to have a command-line utility (i.e., a static 
>>>>main(String[]) method somewhere) that lists the files and sizes 
>>>>contained inside a CFS file, and perhaps even an option to unpack
>>>>        
>>>>
>>it. 
>>    
>>
>>>>Anyone care to contribute this method?
>>>>        
>>>>
>>>Here's a diff to add this functionality to CompoundFileReader.  
>>>Comments are of course welcome, as I'm not that fantastic a Java
>>>      
>>>
>>hacker.
>>    
>>
>>>-garrett
>>>
>>>      
>>>
>>------------------------------------------------------------------------
>>    
>>
>>>Index: src/java/org/apache/lucene/index/CompoundFileReader.java
>>>===================================================================
>>>RCS file:
>>>      
>>>
>/home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/index/CompoundFileReader.java,v
>  
>
>>>retrieving revision 1.14
>>>diff -u -r1.14 CompoundFileReader.java
>>>--- src/java/org/apache/lucene/index/CompoundFileReader.java	28 Sep
>>>      
>>>
>>2004 18:15:52 -0000	1.14
>>    
>>
>>>+++ src/java/org/apache/lucene/index/CompoundFileReader.java	25 Dec
>>>      
>>>
>>2004 04:25:11 -0000
>>    
>>
>>>@@ -17,12 +17,14 @@
>>> */
>>>
>>>import org.apache.lucene.store.Directory;
>>>+import org.apache.lucene.store.FSDirectory;
>>>import org.apache.lucene.store.IndexInput;
>>>import org.apache.lucene.store.BufferedIndexInput;
>>>import org.apache.lucene.store.IndexOutput;
>>>import org.apache.lucene.store.Lock;
>>>import java.util.HashMap;
>>>import java.io.IOException;
>>>+import java.io.FileOutputStream;
>>>
>>>
>>>/**
>>>@@ -233,5 +235,61 @@
>>>        }
>>>
>>>
>>>+    }
>>>+
>>>+    public static void main(String [] args) {
>>>+        String dirname = null, filename = null;
>>>+        boolean extract = false;
>>>+
>>>+        for (int i = 0; i < args.length; ++i) {
>>>+            if (args[i].equals("-extract")) {
>>>+                extract = true;
>>>+            } else if (dirname == null) {
>>>+                dirname = args[i];
>>>+            } else if (filename == null) {
>>>+                filename = args[i];
>>>+            }
>>>+        }
>>>+
>>>+        if (dirname == null || filename == null) {
>>>+            System.out.println("Usage: CompoundFileReader directory
>>>      
>>>
>>cfsfile");
>>    
>>
>>>+            System.out.println("");
>>>+            System.out.println("Prints the filename and size of
>>>      
>>>
>>each file "
>>    
>>
>>>+                               + "within cfsfile.");
>>>+            System.out.println("");
>>>+            System.out.println("Add the -extract flag to extract
>>>      
>>>
>>files to the "
>>    
>>
>>>+                               + "current working directory.");
>>>+
>>>+            return;
>>>+        }
>>>+
>>>+        try {
>>>+            Directory dir = FSDirectory.getDirectory(dirname,
>>>      
>>>
>>false);
>>    
>>
>>>+
>>>+            CompoundFileReader cfr = new CompoundFileReader(dir,
>>>      
>>>
>>filename);
>>    
>>
>>>+
>>>+            String [] files = cfr.list();
>>>+
>>>+            for (int i = 0; i < files.length; ++i) {
>>>+                long len = cfr.fileLength(files[i]);
>>>+
>>>+                System.out.println(files[i] + "\t: " + len + "
>>>      
>>>
>>bytes");
>>    
>>
>>>+
>>>+                if (extract) {
>>>+                    IndexInput ii = cfr.openInput(files[i]);
>>>+
>>>+                    FileOutputStream f = new
>>>      
>>>
>>FileOutputStream(files[i]);
>>    
>>
>>>+
>>>+                    while (len-- != 0) {
>>>+                        byte b = ii.readByte();
>>>+                        f.write(b);
>>>+                    }
>>>+
>>>+                    f.close();
>>>+                }
>>>+            }
>>>+        } catch (IOException ioe) {
>>>+            ioe.printStackTrace();
>>>+        }
>>>    }
>>>}
>>>
>>> 
>>>
>>>      
>>>
>>------------------------------------------------------------------------
>>    
>>
>>---------------------------------------------------------------------
>>    
>>
>>>To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>>For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>>
>>>      
>>>
>>    
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
>  
>


Re: CFS file and file formats

Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hello,

I understand the technical reason for main() there, but logically this
belongs to an external utility class, I think.

Otis


--- Bernhard Messer <bm...@apache.org> wrote:

> hi,
> 
> i already had a look at Garrett's implementation. I made some smaller
> 
> changes to improve the performance when extracting the files from the
> 
> compound. All tests work fine and the index is usable after
> extraction. 
> The new functionality is added as a public static void main () to 
> CompoundFileReader because of the reduced visibility (package) of 
> CompoundFileReader itself. It will be committed this afternoon.
> 
> Bernhard
> 
> > Doug Cutting wrote:
> >
> >> It would be useful to have a command-line utility (i.e., a static 
> >> main(String[]) method somewhere) that lists the files and sizes 
> >> contained inside a CFS file, and perhaps even an option to unpack
> it. 
> >> Anyone care to contribute this method?
> >
> >
> > Here's a diff to add this functionality to CompoundFileReader.  
> > Comments are of course welcome, as I'm not that fantastic a Java
> hacker.
> >
> > -garrett
> >
>
>------------------------------------------------------------------------
> >
> >Index: src/java/org/apache/lucene/index/CompoundFileReader.java
> >===================================================================
> >RCS file:
>
/home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/index/CompoundFileReader.java,v
> >retrieving revision 1.14
> >diff -u -r1.14 CompoundFileReader.java
> >--- src/java/org/apache/lucene/index/CompoundFileReader.java	28 Sep
> 2004 18:15:52 -0000	1.14
> >+++ src/java/org/apache/lucene/index/CompoundFileReader.java	25 Dec
> 2004 04:25:11 -0000
> >@@ -17,12 +17,14 @@
> >  */
> > 
> > import org.apache.lucene.store.Directory;
> >+import org.apache.lucene.store.FSDirectory;
> > import org.apache.lucene.store.IndexInput;
> > import org.apache.lucene.store.BufferedIndexInput;
> > import org.apache.lucene.store.IndexOutput;
> > import org.apache.lucene.store.Lock;
> > import java.util.HashMap;
> > import java.io.IOException;
> >+import java.io.FileOutputStream;
> > 
> > 
> > /**
> >@@ -233,5 +235,61 @@
> >         }
> > 
> > 
> >+    }
> >+
> >+    public static void main(String [] args) {
> >+        String dirname = null, filename = null;
> >+        boolean extract = false;
> >+
> >+        for (int i = 0; i < args.length; ++i) {
> >+            if (args[i].equals("-extract")) {
> >+                extract = true;
> >+            } else if (dirname == null) {
> >+                dirname = args[i];
> >+            } else if (filename == null) {
> >+                filename = args[i];
> >+            }
> >+        }
> >+
> >+        if (dirname == null || filename == null) {
> >+            System.out.println("Usage: CompoundFileReader directory
> cfsfile");
> >+            System.out.println("");
> >+            System.out.println("Prints the filename and size of
> each file "
> >+                               + "within cfsfile.");
> >+            System.out.println("");
> >+            System.out.println("Add the -extract flag to extract
> files to the "
> >+                               + "current working directory.");
> >+
> >+            return;
> >+        }
> >+
> >+        try {
> >+            Directory dir = FSDirectory.getDirectory(dirname,
> false);
> >+
> >+            CompoundFileReader cfr = new CompoundFileReader(dir,
> filename);
> >+
> >+            String [] files = cfr.list();
> >+
> >+            for (int i = 0; i < files.length; ++i) {
> >+                long len = cfr.fileLength(files[i]);
> >+
> >+                System.out.println(files[i] + "\t: " + len + "
> bytes");
> >+
> >+                if (extract) {
> >+                    IndexInput ii = cfr.openInput(files[i]);
> >+
> >+                    FileOutputStream f = new
> FileOutputStream(files[i]);
> >+
> >+                    while (len-- != 0) {
> >+                        byte b = ii.readByte();
> >+                        f.write(b);
> >+                    }
> >+
> >+                    f.close();
> >+                }
> >+            }
> >+        } catch (IOException ioe) {
> >+            ioe.printStackTrace();
> >+        }
> >     }
> > }
> >
> >  
> >
>
>------------------------------------------------------------------------
> >
>
>---------------------------------------------------------------------
> >To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> >For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
> >
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Re: CFS file and file formats

Posted by Bernhard Messer <bm...@apache.org>.
hi,

i already had a look at Garrett's implementation. I made some smaller 
changes to improve the performance when extracting the files from the 
compound. All tests work fine and the index is usable after extraction. 
The new functionality is added as a public static void main () to 
CompoundFileReader because of the reduced visibility (package) of 
CompoundFileReader itself. It will be committed this afternoon.

Bernhard

> Doug Cutting wrote:
>
>> It would be useful to have a command-line utility (i.e., a static 
>> main(String[]) method somewhere) that lists the files and sizes 
>> contained inside a CFS file, and perhaps even an option to unpack it. 
>> Anyone care to contribute this method?
>
>
> Here's a diff to add this functionality to CompoundFileReader.  
> Comments are of course welcome, as I'm not that fantastic a Java hacker.
>
> -garrett
>
>------------------------------------------------------------------------
>
>Index: src/java/org/apache/lucene/index/CompoundFileReader.java
>===================================================================
>RCS file: /home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/index/CompoundFileReader.java,v
>retrieving revision 1.14
>diff -u -r1.14 CompoundFileReader.java
>--- src/java/org/apache/lucene/index/CompoundFileReader.java	28 Sep 2004 18:15:52 -0000	1.14
>+++ src/java/org/apache/lucene/index/CompoundFileReader.java	25 Dec 2004 04:25:11 -0000
>@@ -17,12 +17,14 @@
>  */
> 
> import org.apache.lucene.store.Directory;
>+import org.apache.lucene.store.FSDirectory;
> import org.apache.lucene.store.IndexInput;
> import org.apache.lucene.store.BufferedIndexInput;
> import org.apache.lucene.store.IndexOutput;
> import org.apache.lucene.store.Lock;
> import java.util.HashMap;
> import java.io.IOException;
>+import java.io.FileOutputStream;
> 
> 
> /**
>@@ -233,5 +235,61 @@
>         }
> 
> 
>+    }
>+
>+    public static void main(String [] args) {
>+        String dirname = null, filename = null;
>+        boolean extract = false;
>+
>+        for (int i = 0; i < args.length; ++i) {
>+            if (args[i].equals("-extract")) {
>+                extract = true;
>+            } else if (dirname == null) {
>+                dirname = args[i];
>+            } else if (filename == null) {
>+                filename = args[i];
>+            }
>+        }
>+
>+        if (dirname == null || filename == null) {
>+            System.out.println("Usage: CompoundFileReader directory cfsfile");
>+            System.out.println("");
>+            System.out.println("Prints the filename and size of each file "
>+                               + "within cfsfile.");
>+            System.out.println("");
>+            System.out.println("Add the -extract flag to extract files to the "
>+                               + "current working directory.");
>+
>+            return;
>+        }
>+
>+        try {
>+            Directory dir = FSDirectory.getDirectory(dirname, false);
>+
>+            CompoundFileReader cfr = new CompoundFileReader(dir, filename);
>+
>+            String [] files = cfr.list();
>+
>+            for (int i = 0; i < files.length; ++i) {
>+                long len = cfr.fileLength(files[i]);
>+
>+                System.out.println(files[i] + "\t: " + len + " bytes");
>+
>+                if (extract) {
>+                    IndexInput ii = cfr.openInput(files[i]);
>+
>+                    FileOutputStream f = new FileOutputStream(files[i]);
>+
>+                    while (len-- != 0) {
>+                        byte b = ii.readByte();
>+                        f.write(b);
>+                    }
>+
>+                    f.close();
>+                }
>+            }
>+        } catch (IOException ioe) {
>+            ioe.printStackTrace();
>+        }
>     }
> }
>
>  
>
>------------------------------------------------------------------------
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>


Re: CFS file and file formats

Posted by Garrett Rooney <ro...@electricjellyfish.net>.
Doug Cutting wrote:
> It would be useful to have a command-line utility (i.e., a static 
> main(String[]) method somewhere) that lists the files and sizes 
> contained inside a CFS file, and perhaps even an option to unpack it. 
> Anyone care to contribute this method?

Here's a diff to add this functionality to CompoundFileReader.  Comments 
are of course welcome, as I'm not that fantastic a Java hacker.

-garrett

Re: CFS file and file formats

Posted by Garrett Rooney <ro...@electricjellyfish.net>.
Doug Cutting wrote:
> It would be useful to have a command-line utility (i.e., a static 
> main(String[]) method somewhere) that lists the files and sizes 
> contained inside a CFS file, and perhaps even an option to unpack it. 
> Anyone care to contribute this method?

I may have a chance to look at that over the weekend.  I'm going to need 
to figure out how the CFS files work anyway so I can implement them for 
lucene4c in the future, so this would be a good way to dive into the code.

-garrett

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org