Term position offset question

[Ben West <bw...@yahoo.com>]

Hey all,

I'm trying to implement the SynonymFilter from Lucene In Action (p. 133 in the 2nd edition). (The sample code can be downloaded from http://www.manning.com/hatcher3/ albeit in Java form).

Basically, I have a something like:

class SynonymFilter : TokenFilter {
public override bool IncrementToken() {
  termAttribute.SetTermBuffer("synonym");
  positionIncrementAttribute.SetPositionIncrement(0);
}
}

Despite me setting the position increment to 0, it still seems to store it with an increment of 1 when I look at it in Luke. The net result of which is that my phrase queries don't work.

(e.g. the synonyms "syn1 syn2 syn3" have positions 1, 2 and 3 instead of 1, 1, 1)

This is the last filter in my chain, so it's not being overridden somewhere else.

Is there some reason why the Java version of this would be different from .NET? Or am I doing something weird? I can provide a full code sample if that is helpful.

Thanks,
-Ben


      

RE: Term position offset question

["Granroth, Neal V." <ne...@thermofisher.com>]

As the Lucene.NET unit-tests pass; I imagine you're correct, that your code is "doing something weird". :)

In the Java example, the incrementToken() method appears to preserve and restore the "state" of the token stream.  Are you sure that your code is doing something similar?  The method also returns true, just after the call to SetPositionIncrement(0), you should check that your code also exits correctly.


- Neal


-----Original Message-----
From: Ben West [mailto:bwsithspawn00@yahoo.com] 
Sent: Monday, December 13, 2010 2:24 PM
To: lucene-net-user@lucene.apache.org
Subject: Term position offset question

Hey all,

I'm trying to implement the SynonymFilter from Lucene In Action (p. 133 in the 2nd edition). (The sample code can be downloaded from http://www.manning.com/hatcher3/ albeit in Java form).

Basically, I have a something like:

class SynonymFilter : TokenFilter {
public override bool IncrementToken() {
  termAttribute.SetTermBuffer("synonym");
  positionIncrementAttribute.SetPositionIncrement(0);
}
}

Despite me setting the position increment to 0, it still seems to store it with an increment of 1 when I look at it in Luke. The net result of which is that my phrase queries don't work.

(e.g. the synonyms "syn1 syn2 syn3" have positions 1, 2 and 3 instead of 1, 1, 1)

This is the last filter in my chain, so it's not being overridden somewhere else.

Is there some reason why the Java version of this would be different from .NET? Or am I doing something weird? I can provide a full code sample if that is helpful.

Thanks,
-Ben