You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Doug Cutting <cu...@apache.org> on 2004/12/23 21:28:55 UTC
Re: CFS file and file formats
It would be useful to have a command-line utility (i.e., a static
main(String[]) method somewhere) that lists the files and sizes
contained inside a CFS file, and perhaps even an option to unpack it.
Anyone care to contribute this method?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: CFS file and file formats
Posted by Bernhard Messer <bm...@apache.org>.
Doug Cutting schrieb:
> Bernhard Messer wrote:
>
>> Why not implementing a small utility class, f.e CompoundFileUtil.java
>> within the org.apache.lucene.index Package ? This class could be
>> public and implement the necessary functionality. This is what i
>> would prefer, because we don't have to change the visibility of
>> CompoundFileReader or other parts of the API. The other option would
>> be to add a public static method to IndexReader class. But i don't
>> like to overwhelm IndexReader with a method, just a very small
>> audience would use.
>
>
> Currently IndexWriter is the only public place in the API where the
> compound format appears. So, until we decide to expose index formats
> more systematically, I think this should stay at the IndexReader level.
>
> Thus I would prefer a main() on IndexReader that had various commands,
> perhaps something like:
>
> java org.apache.lucene.index.IndexReader <dir> cfs list
> java org.apache.lucene.index.IndexReader <dir> cfs extract
> java org.apache.lucene.index.IndexReader <dir> unlock
> java org.apache.lucene.index.IndexReader <dir> list-segments
>
this looks ok for me for the moment, but i still favorite an additional
class within the same package. Something like "IndexUtil" which could
hold all this kind of functionality. If there is some time over the
weekend, i will have a look on it how much effort it will be to have
those 4 methods in IndexReader.
Bernhard
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: CFS file and file formats
Posted by Doug Cutting <cu...@apache.org>.
Bernhard Messer wrote:
> Why not implementing a small utility class, f.e CompoundFileUtil.java
> within the org.apache.lucene.index Package ? This class could be public
> and implement the necessary functionality. This is what i would prefer,
> because we don't have to change the visibility of CompoundFileReader or
> other parts of the API. The other option would be to add a public static
> method to IndexReader class. But i don't like to overwhelm IndexReader
> with a method, just a very small audience would use.
Currently IndexWriter is the only public place in the API where the
compound format appears. So, until we decide to expose index formats
more systematically, I think this should stay at the IndexReader level.
Thus I would prefer a main() on IndexReader that had various commands,
perhaps something like:
java org.apache.lucene.index.IndexReader <dir> cfs list
java org.apache.lucene.index.IndexReader <dir> cfs extract
java org.apache.lucene.index.IndexReader <dir> unlock
java org.apache.lucene.index.IndexReader <dir> list-segments
etc.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: CFS file and file formats
Posted by Bernhard Messer <bm...@apache.org>.
Doug Cutting schrieb:
> Bernhard Messer wrote:
>
>>> I understand the technical reason for main() there, but logically this
>>> belongs to an external utility class, I think.
>>>
>> Otis you are right, i already thought about it. It could be simply
>> moved to a newly created class in org.apache.lucene.util package. But
>> then we have to change the visibility of CompoundFileReader to
>> public. I have no problems with a public CompoundFileReader class.
>> Does anybody see a reason that the visibility of CompoundFileReader
>> should not be changed to public ?
>
>
> Yes. Historically we've tried hard in Lucene to not make things
> public unless they're useful to users. We try to keep the public API
> as small as possible. This makes the documentation easier to read and
> also makes the system easier to maintain. CompoundFileReader is an
> implementation class, not a user class.
>
Why not implementing a small utility class, f.e CompoundFileUtil.java
within the org.apache.lucene.index Package ? This class could be public
and implement the necessary functionality. This is what i would prefer,
because we don't have to change the visibility of CompoundFileReader or
other parts of the API. The other option would be to add a public static
method to IndexReader class. But i don't like to overwhelm IndexReader
with a method, just a very small audience would use.
Bernhard
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: CFS file and file formats
Posted by Doug Cutting <cu...@apache.org>.
Bernhard Messer wrote:
>> I understand the technical reason for main() there, but logically this
>> belongs to an external utility class, I think.
>>
> Otis you are right, i already thought about it. It could be simply moved
> to a newly created class in org.apache.lucene.util package. But then we
> have to change the visibility of CompoundFileReader to public. I have no
> problems with a public CompoundFileReader class. Does anybody see a
> reason that the visibility of CompoundFileReader should not be changed
> to public ?
Yes. Historically we've tried hard in Lucene to not make things public
unless they're useful to users. We try to keep the public API as small
as possible. This makes the documentation easier to read and also makes
the system easier to maintain. CompoundFileReader is an implementation
class, not a user class.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: CFS file and file formats
Posted by Bernhard Messer <bm...@apache.org>.
>Hello,
>
>I understand the technical reason for main() there, but logically this
>belongs to an external utility class, I think.
>
>
>
Otis you are right, i already thought about it. It could be simply moved
to a newly created class in org.apache.lucene.util package. But then we
have to change the visibility of CompoundFileReader to public. I have no
problems with a public CompoundFileReader class. Does anybody see a
reason that the visibility of CompoundFileReader should not be changed
to public ?
Bernhard
>
>--- Bernhard Messer <bm...@apache.org> wrote:
>
>
>
>>hi,
>>
>>i already had a look at Garrett's implementation. I made some smaller
>>
>>changes to improve the performance when extracting the files from the
>>
>>compound. All tests work fine and the index is usable after
>>extraction.
>>The new functionality is added as a public static void main () to
>>CompoundFileReader because of the reduced visibility (package) of
>>CompoundFileReader itself. It will be committed this afternoon.
>>
>>Bernhard
>>
>>
>>
>>>Doug Cutting wrote:
>>>
>>>
>>>
>>>>It would be useful to have a command-line utility (i.e., a static
>>>>main(String[]) method somewhere) that lists the files and sizes
>>>>contained inside a CFS file, and perhaps even an option to unpack
>>>>
>>>>
>>it.
>>
>>
>>>>Anyone care to contribute this method?
>>>>
>>>>
>>>Here's a diff to add this functionality to CompoundFileReader.
>>>Comments are of course welcome, as I'm not that fantastic a Java
>>>
>>>
>>hacker.
>>
>>
>>>-garrett
>>>
>>>
>>>
>>------------------------------------------------------------------------
>>
>>
>>>Index: src/java/org/apache/lucene/index/CompoundFileReader.java
>>>===================================================================
>>>RCS file:
>>>
>>>
>/home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/index/CompoundFileReader.java,v
>
>
>>>retrieving revision 1.14
>>>diff -u -r1.14 CompoundFileReader.java
>>>--- src/java/org/apache/lucene/index/CompoundFileReader.java 28 Sep
>>>
>>>
>>2004 18:15:52 -0000 1.14
>>
>>
>>>+++ src/java/org/apache/lucene/index/CompoundFileReader.java 25 Dec
>>>
>>>
>>2004 04:25:11 -0000
>>
>>
>>>@@ -17,12 +17,14 @@
>>> */
>>>
>>>import org.apache.lucene.store.Directory;
>>>+import org.apache.lucene.store.FSDirectory;
>>>import org.apache.lucene.store.IndexInput;
>>>import org.apache.lucene.store.BufferedIndexInput;
>>>import org.apache.lucene.store.IndexOutput;
>>>import org.apache.lucene.store.Lock;
>>>import java.util.HashMap;
>>>import java.io.IOException;
>>>+import java.io.FileOutputStream;
>>>
>>>
>>>/**
>>>@@ -233,5 +235,61 @@
>>> }
>>>
>>>
>>>+ }
>>>+
>>>+ public static void main(String [] args) {
>>>+ String dirname = null, filename = null;
>>>+ boolean extract = false;
>>>+
>>>+ for (int i = 0; i < args.length; ++i) {
>>>+ if (args[i].equals("-extract")) {
>>>+ extract = true;
>>>+ } else if (dirname == null) {
>>>+ dirname = args[i];
>>>+ } else if (filename == null) {
>>>+ filename = args[i];
>>>+ }
>>>+ }
>>>+
>>>+ if (dirname == null || filename == null) {
>>>+ System.out.println("Usage: CompoundFileReader directory
>>>
>>>
>>cfsfile");
>>
>>
>>>+ System.out.println("");
>>>+ System.out.println("Prints the filename and size of
>>>
>>>
>>each file "
>>
>>
>>>+ + "within cfsfile.");
>>>+ System.out.println("");
>>>+ System.out.println("Add the -extract flag to extract
>>>
>>>
>>files to the "
>>
>>
>>>+ + "current working directory.");
>>>+
>>>+ return;
>>>+ }
>>>+
>>>+ try {
>>>+ Directory dir = FSDirectory.getDirectory(dirname,
>>>
>>>
>>false);
>>
>>
>>>+
>>>+ CompoundFileReader cfr = new CompoundFileReader(dir,
>>>
>>>
>>filename);
>>
>>
>>>+
>>>+ String [] files = cfr.list();
>>>+
>>>+ for (int i = 0; i < files.length; ++i) {
>>>+ long len = cfr.fileLength(files[i]);
>>>+
>>>+ System.out.println(files[i] + "\t: " + len + "
>>>
>>>
>>bytes");
>>
>>
>>>+
>>>+ if (extract) {
>>>+ IndexInput ii = cfr.openInput(files[i]);
>>>+
>>>+ FileOutputStream f = new
>>>
>>>
>>FileOutputStream(files[i]);
>>
>>
>>>+
>>>+ while (len-- != 0) {
>>>+ byte b = ii.readByte();
>>>+ f.write(b);
>>>+ }
>>>+
>>>+ f.close();
>>>+ }
>>>+ }
>>>+ } catch (IOException ioe) {
>>>+ ioe.printStackTrace();
>>>+ }
>>> }
>>>}
>>>
>>>
>>>
>>>
>>>
>>------------------------------------------------------------------------
>>
>>
>>---------------------------------------------------------------------
>>
>>
>>>To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>>For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>>
>>>
>>>
>>
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
>
>
Re: CFS file and file formats
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hello,
I understand the technical reason for main() there, but logically this
belongs to an external utility class, I think.
Otis
--- Bernhard Messer <bm...@apache.org> wrote:
> hi,
>
> i already had a look at Garrett's implementation. I made some smaller
>
> changes to improve the performance when extracting the files from the
>
> compound. All tests work fine and the index is usable after
> extraction.
> The new functionality is added as a public static void main () to
> CompoundFileReader because of the reduced visibility (package) of
> CompoundFileReader itself. It will be committed this afternoon.
>
> Bernhard
>
> > Doug Cutting wrote:
> >
> >> It would be useful to have a command-line utility (i.e., a static
> >> main(String[]) method somewhere) that lists the files and sizes
> >> contained inside a CFS file, and perhaps even an option to unpack
> it.
> >> Anyone care to contribute this method?
> >
> >
> > Here's a diff to add this functionality to CompoundFileReader.
> > Comments are of course welcome, as I'm not that fantastic a Java
> hacker.
> >
> > -garrett
> >
>
>------------------------------------------------------------------------
> >
> >Index: src/java/org/apache/lucene/index/CompoundFileReader.java
> >===================================================================
> >RCS file:
>
/home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/index/CompoundFileReader.java,v
> >retrieving revision 1.14
> >diff -u -r1.14 CompoundFileReader.java
> >--- src/java/org/apache/lucene/index/CompoundFileReader.java 28 Sep
> 2004 18:15:52 -0000 1.14
> >+++ src/java/org/apache/lucene/index/CompoundFileReader.java 25 Dec
> 2004 04:25:11 -0000
> >@@ -17,12 +17,14 @@
> > */
> >
> > import org.apache.lucene.store.Directory;
> >+import org.apache.lucene.store.FSDirectory;
> > import org.apache.lucene.store.IndexInput;
> > import org.apache.lucene.store.BufferedIndexInput;
> > import org.apache.lucene.store.IndexOutput;
> > import org.apache.lucene.store.Lock;
> > import java.util.HashMap;
> > import java.io.IOException;
> >+import java.io.FileOutputStream;
> >
> >
> > /**
> >@@ -233,5 +235,61 @@
> > }
> >
> >
> >+ }
> >+
> >+ public static void main(String [] args) {
> >+ String dirname = null, filename = null;
> >+ boolean extract = false;
> >+
> >+ for (int i = 0; i < args.length; ++i) {
> >+ if (args[i].equals("-extract")) {
> >+ extract = true;
> >+ } else if (dirname == null) {
> >+ dirname = args[i];
> >+ } else if (filename == null) {
> >+ filename = args[i];
> >+ }
> >+ }
> >+
> >+ if (dirname == null || filename == null) {
> >+ System.out.println("Usage: CompoundFileReader directory
> cfsfile");
> >+ System.out.println("");
> >+ System.out.println("Prints the filename and size of
> each file "
> >+ + "within cfsfile.");
> >+ System.out.println("");
> >+ System.out.println("Add the -extract flag to extract
> files to the "
> >+ + "current working directory.");
> >+
> >+ return;
> >+ }
> >+
> >+ try {
> >+ Directory dir = FSDirectory.getDirectory(dirname,
> false);
> >+
> >+ CompoundFileReader cfr = new CompoundFileReader(dir,
> filename);
> >+
> >+ String [] files = cfr.list();
> >+
> >+ for (int i = 0; i < files.length; ++i) {
> >+ long len = cfr.fileLength(files[i]);
> >+
> >+ System.out.println(files[i] + "\t: " + len + "
> bytes");
> >+
> >+ if (extract) {
> >+ IndexInput ii = cfr.openInput(files[i]);
> >+
> >+ FileOutputStream f = new
> FileOutputStream(files[i]);
> >+
> >+ while (len-- != 0) {
> >+ byte b = ii.readByte();
> >+ f.write(b);
> >+ }
> >+
> >+ f.close();
> >+ }
> >+ }
> >+ } catch (IOException ioe) {
> >+ ioe.printStackTrace();
> >+ }
> > }
> > }
> >
> >
> >
>
>------------------------------------------------------------------------
> >
>
>---------------------------------------------------------------------
> >To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> >For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: CFS file and file formats
Posted by Bernhard Messer <bm...@apache.org>.
hi,
i already had a look at Garrett's implementation. I made some smaller
changes to improve the performance when extracting the files from the
compound. All tests work fine and the index is usable after extraction.
The new functionality is added as a public static void main () to
CompoundFileReader because of the reduced visibility (package) of
CompoundFileReader itself. It will be committed this afternoon.
Bernhard
> Doug Cutting wrote:
>
>> It would be useful to have a command-line utility (i.e., a static
>> main(String[]) method somewhere) that lists the files and sizes
>> contained inside a CFS file, and perhaps even an option to unpack it.
>> Anyone care to contribute this method?
>
>
> Here's a diff to add this functionality to CompoundFileReader.
> Comments are of course welcome, as I'm not that fantastic a Java hacker.
>
> -garrett
>
>------------------------------------------------------------------------
>
>Index: src/java/org/apache/lucene/index/CompoundFileReader.java
>===================================================================
>RCS file: /home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/index/CompoundFileReader.java,v
>retrieving revision 1.14
>diff -u -r1.14 CompoundFileReader.java
>--- src/java/org/apache/lucene/index/CompoundFileReader.java 28 Sep 2004 18:15:52 -0000 1.14
>+++ src/java/org/apache/lucene/index/CompoundFileReader.java 25 Dec 2004 04:25:11 -0000
>@@ -17,12 +17,14 @@
> */
>
> import org.apache.lucene.store.Directory;
>+import org.apache.lucene.store.FSDirectory;
> import org.apache.lucene.store.IndexInput;
> import org.apache.lucene.store.BufferedIndexInput;
> import org.apache.lucene.store.IndexOutput;
> import org.apache.lucene.store.Lock;
> import java.util.HashMap;
> import java.io.IOException;
>+import java.io.FileOutputStream;
>
>
> /**
>@@ -233,5 +235,61 @@
> }
>
>
>+ }
>+
>+ public static void main(String [] args) {
>+ String dirname = null, filename = null;
>+ boolean extract = false;
>+
>+ for (int i = 0; i < args.length; ++i) {
>+ if (args[i].equals("-extract")) {
>+ extract = true;
>+ } else if (dirname == null) {
>+ dirname = args[i];
>+ } else if (filename == null) {
>+ filename = args[i];
>+ }
>+ }
>+
>+ if (dirname == null || filename == null) {
>+ System.out.println("Usage: CompoundFileReader directory cfsfile");
>+ System.out.println("");
>+ System.out.println("Prints the filename and size of each file "
>+ + "within cfsfile.");
>+ System.out.println("");
>+ System.out.println("Add the -extract flag to extract files to the "
>+ + "current working directory.");
>+
>+ return;
>+ }
>+
>+ try {
>+ Directory dir = FSDirectory.getDirectory(dirname, false);
>+
>+ CompoundFileReader cfr = new CompoundFileReader(dir, filename);
>+
>+ String [] files = cfr.list();
>+
>+ for (int i = 0; i < files.length; ++i) {
>+ long len = cfr.fileLength(files[i]);
>+
>+ System.out.println(files[i] + "\t: " + len + " bytes");
>+
>+ if (extract) {
>+ IndexInput ii = cfr.openInput(files[i]);
>+
>+ FileOutputStream f = new FileOutputStream(files[i]);
>+
>+ while (len-- != 0) {
>+ byte b = ii.readByte();
>+ f.write(b);
>+ }
>+
>+ f.close();
>+ }
>+ }
>+ } catch (IOException ioe) {
>+ ioe.printStackTrace();
>+ }
> }
> }
>
>
>
>------------------------------------------------------------------------
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
Re: CFS file and file formats
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
Doug Cutting wrote:
> It would be useful to have a command-line utility (i.e., a static
> main(String[]) method somewhere) that lists the files and sizes
> contained inside a CFS file, and perhaps even an option to unpack it.
> Anyone care to contribute this method?
Here's a diff to add this functionality to CompoundFileReader. Comments
are of course welcome, as I'm not that fantastic a Java hacker.
-garrett
Re: CFS file and file formats
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
Doug Cutting wrote:
> It would be useful to have a command-line utility (i.e., a static
> main(String[]) method somewhere) that lists the files and sizes
> contained inside a CFS file, and perhaps even an option to unpack it.
> Anyone care to contribute this method?
I may have a chance to look at that over the weekend. I'm going to need
to figure out how the CFS files work anyway so I can implement them for
lucene4c in the future, so this would be a good way to dive into the code.
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org