Is there a StrlenFilter yet?

["Spencer, Dave" <da...@lumos.com>]

Use case - you want to protect yourself against pathalogical docs
such as one with a string of a million consectutive characters - any
normal tokenizer will consider this one big token but there's probably
no point in indexing a string that is a million characters long.
One example is indexing a mailing list which could contain uuencoded
attachments - there
could be lots of lamo lines 72 or so chars long.

Anyway - I've attached a possible impl.

Discussion question is, let's say the filter is told to only return
tokens <= 5 chars long (note:
I think 16 or so would be more realistic for most docs -this is just for
sake of example).

What if there is one 6 chars long then i.e. longer than the limit - say
it 
is "abcdef".

Then either:

[a] we ignore "abcdef" and assume it is garbage
or
[b] we return "abcde" and "bcdef" i.e. all 5 char substrings
of it, so that if someone wants to search on the 6 char string they
sort of still can (at least w/ a carefully chosen query...hmmm..).

Anyway here's some code.
If popular it could be put into StandardAnalyzer.

--------
package com.tropo.lucene;

import java.io.IOException;
import org.apache.lucene.analysis.*;

/**
 * Removes words that are too long and too short from the stream
 */
public final class StrlenFilter
	extends TokenFilter
{
	/**
	 * Build a filter that removes words that are too long or too
short from the text.
	 */
	public StrlenFilter(TokenStream in, int min, int max)
	{
		input = in;
		this.min = min;
		this.max =max;
	}

	/** Returns the next input Token whose termText() is the right
len
	 */
	public final Token next() throws IOException
	{
		// return the first non-stop word found
		for (Token token = input.next(); token != null; token =
input.next())
		{
			final int len = token.termText().length();
			if ( len >= min && len <= max)
				return token;
			// note: else we ignore it but should we index
each part of it?
		}
		// reached EOS -- return null		
		return null;
	}
	final int min;
	final int max;
	}



--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>

Exception while merging indexes

[Rajive Dave <ha...@yahoo.com>]

Hi All,

I am trying to merge different indexes into one index
and I get following excpetion. The program I used to
merge the indexes is at the end of the email. 

When I list the segmentnames in the segments files in 
the corresponding index directory I see the particular
segment (_87aa) in the index but no files associated
with the index. I have no idea what happened. 

I did try to merge this index with others onece beofre
and it successfully completed. But its failing now.
More interestingly it fails while optimizing ( I had
print line is optimize call ). 

Looks like there is some problem with remerging the
index which has already been merged before.
Theoretically it shoudln't be the problem becuase
merge program is just reading the index and writing a
new merged one . Any help will be appreciated.

Thanks
Rajive 

java.io.FileNotFoundException:
/data0/indexRep/index/_87aa.fnm (No such file or
directory)
        at java.io.RandomAccessFile.open(Native
Method)
        at
java.io.RandomAccessFile.<init>(RandomAccessFile.java:93)
        at
java.io.RandomAccessFile.<init>(RandomAccessFile.java:138)
        at
org.apache.lucene.store.FSInputStream$Descriptor.<init>(FSDirectory.java:253)
        at
org.apache.lucene.store.FSInputStream.<init>(FSDirectory.java:261)
        at
org.apache.lucene.store.FSDirectory.openFile(FSDirectory.java:210)
        at
org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:78)
        at
org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:103)
        at
org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:381)
        at
org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:365)
        at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:261)
        at
org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:318)
        at MergeIndex.main(MergeIndex.java:26)


public class  MergeIndex
{
  public static void main(String[] args) {
    try {
      Date start = new Date();
      System.out.println(start.getTime());
      IndexWriter writer = new IndexWriter("index",
new StandardAnalyzer(), true);

      String [] dirs = args  ;
      FSDirectory [] luceneDirs = new FSDirectory
[dirs.length];
      for ( int i = 0 ; i < dirs.length ; i ++ )
      {
         luceneDirs[i] = 
FSDirectory.getDirectory(dirs[i],false);
      }

      writer.addIndexes(luceneDirs);

      writer.optimize();
      writer.close();

      Date end = new Date();

      System.out.print(end.getTime() -
start.getTime());
      System.out.println(" total milliseconds");

    } catch (Exception e) {
      e.printStackTrace();
      System.out.println(" caught a " + e.getClass() +
                         "\n with message: " +
e.getMessage());
    }
  }
}


__________________________________________________
Do you Yahoo!?
Y! Web Hosting - Let the expert host your web site
http://webhosting.yahoo.com/

--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>