synonym filter and offsets

[Joe Calderon <ca...@gmail.com>]

hello *, im having issues with the synonym filter altering token offsets,

my input text is
"saturday night live"
its is tokenized by the whitespace tokenizer yielding 3 tokens
[saturday, 0,8], [night, 9, 14], [live, 15,19]

on indexing these are passed through a synonym filter that has this line
saturday night live => snl, saturday night live


i now end up with four tokens
[saturday, 0, 19], [snl, 0, 19], [night, 0, 19], [live, 0,19]

what i want is
[saturday, 0,8], [snl, 0,19], [night, 9, 14], [live, 15,19]


when using the highlighter i want to make it so only the relevant part
of the text is highlighted, how can i fix my filter chain?


thx much
--joe

Re: synonym filter and offsets

[Chris Hostetter <ho...@fucit.org>]

: on indexing these are passed through a synonym filter that has this line
: saturday night live => snl, saturday night live

: i now end up with four tokens
: [saturday, 0, 19], [snl, 0, 19], [night, 0, 19], [live, 0,19]
: 
: what i want is
: [saturday, 0,8], [snl, 0,19], [night, 9, 14], [live, 15,19]


Hmmm ... I don't think there's any way to make SYnonymFilterFactory do 
that.

The problem is that while in your case it seems obvious that the 
final "saturday" token should get the same start/end offset info as the 
initial "saturday" token from the tokenizer, in the general case there 
isn't a one to one mapping.  the synonym file could just as easily look 
like...

saturday-night live => snl, saturday night live

...in which case it would have no idea what offsets to apply to "saturday" 

That said: i suppose it could make sense to put in explicit logic that if 
a multi-token input produces multi-token output, and some of hte output 
tokens have the same charbuffer as tkens from the input, then they should 
get identical offsets ... but i haven't thought it threw enough to be sure 
there wouldn't be any nasty gotchas.

wanna open a feature request in jira?




-Hoss