You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Liudvikas Bukys <bu...@cs.rochester.edu> on 2004/04/27 20:34:31 UTC
Re: gappy text
My gappy text problem went away when I modified my personal Bayes
tokenizer to recognize strings of gappy text and concatenate them
into a single token.
This tokenizer modification is always an improvement, as it takes
clues that would otherwise be completely discarded and makes some
use of them.
I was planning on recommending it to the SA Bayes implementors
after trying it for a while. So far I can report complete success.
Unfortunately, I have no code patch, because I tested it in my
non-SA Bayes hacking implementation. An UNTRIED code snippet
would be to add
my tmp;
s/(\s\S\s(\S\s)+)/(($tmp = $1), ($tmp =~ s!\s!!g), (" $tmp "))/ge;
somewhere in tokenize_line