You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2015/02/19 19:58:04 UTC
[Bug 7141] New: Bayes truncates ('skip') long tokens on bytes,
should it count characters instead?
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7141
Bug ID: 7141
Summary: Bayes truncates ('skip') long tokens on bytes, should
it count characters instead?
Product: Spamassassin
Version: 3.4.0
Hardware: All
OS: All
Status: NEW
Severity: enhancement
Priority: P5
Component: Plugins
Assignee: dev@spamassassin.apache.org
Reporter: Mark.Martinec@ijs.si
Observing reported bayes tokens, some of these truncated (skip) tokens
stand out, e.g.:
dbg: bayes: token 'sk:Раз�' => 0.986543689320388
dbg: bayes: token 'sk:Бес�' => 0.986543689320388
dbg: bayes: token 'sk:сбо�' => 0.986543689320388
dbg: bayes: token 'sk:уча�' => 0.993172413793104
dbg: bayes: token 'sk:Ува�' => 0.986543689320388
dbg: bayes: token 'sk:сог�' => 0.0156699029126214
Seems like truncation to 7 bytes is unnatural for text in such
UTF-8 encoding which use mostly 2-byte encoded pairs (which goes
for alphabets like Cyrillic, Greek, Latin diacritics - but not CJK
or US-ASCII). The last character is chopped in the middle of its
2-byte sequence.
Should we increase truncation by one - to 8 bytes, which would
at least preserve most of the 2-byte characters in such alphabets.
Or maybe the limit of 7 should be interpreted as 7 *characters*
(not bytes).
Here is the current relevant code section from Plugin/Bayes.pm:
# How long a token should we hold onto? (note: German speakers
# typically will require a longer token than English ones.)
use constant MAX_TOKEN_LENGTH => 15;
[...]
if (($region == 0 && HDRS_TOKENIZE_LONG_TOKENS_AS_SKIPS)
|| ($region == 1 && BODY_TOKENIZE_LONG_TOKENS_AS_SKIPS)
|| ($region == 2 && URIS_TOKENIZE_LONG_TOKENS_AS_SKIPS))
{
# if (TOKENIZE_LONG_TOKENS_AS_SKIPS)
# Spambayes trick via Matt: Just retain 7 chars. Do not retain the
# length, it does not help; see jm's mail to -devel on Nov 20 2002 at
# http://sourceforge.net/p/spamassassin/mailman/message/12977605/
# "sk:" stands for "skip".
$token = "sk:".substr($token, 0, 7);
}
}
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7141] Bayes truncates ('skip') long tokens on bytes, should it
count characters instead?
Posted by bu...@bugzilla.spamassassin.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7141
RW <rw...@googlemail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |rwmaillists@googlemail.com
--- Comment #1 from RW <rw...@googlemail.com> ---
Does this still have a point? I'm wondering if it was simply implemented at a
time when tokens weren't hashed to a constant size.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7141] Bayes truncates ('skip') long tokens on bytes, should it
count characters instead?
Posted by bu...@bugzilla.spamassassin.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7141
--- Comment #2 from Mark Martinec <Ma...@ijs.si> ---
(In reply to RW from comment #1)
> Does this still have a point? I'm wondering if it was simply implemented at
> a time when tokens weren't hashed to a constant size.
That was my thought too - I don't really know. Although looking
at what gets 'skipped' (like fragments of long e-mail addresses
or URLs, long numbers, some gibberish, some really long words),
perhaps this stemming approximation does have some merit. It may be
worth finding out why it was introduced back in 2002 or thereabout.
Regardless of whether we disable it eventually or not,
here is what could be used meanwhile:
--- lib/Mail/SpamAssassin/Plugin/Bayes.pm (revision 1661153)
+++ lib/Mail/SpamAssassin/Plugin/Bayes.pm (working copy)
@@ -1240,7 +1241,14 @@
# length, it does not help; see jm's mail to -devel on Nov 20 2002 at
# http://sourceforge.net/p/spamassassin/mailman/message/12977605/
# "sk:" stands for "skip".
- $token = "sk:".substr($token, 0, 7);
+ # Bug 7141: retain seven UTF-8 chars (or other bytes),
+ # if followed by at least two bytes
+ $token =~ s{ ^ ( (?> (?: [\x00-\x7F\xF5-\xFF] |
+ [\xC0-\xDF][\x80-\xBF] |
+ [\xE0-\xEF][\x80-\xBF]{2} |
+ [\xF0-\xF4][\x80-\xBF]{3} | . ){7} ))
+ .{2,} \z }{sk:$1}xs;
+ ## (was:) $token = "sk:".substr($token, 0, 7); # seven bytes
}
}
As this code section is not entered often, additional complexity
does not matter much here.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7141] Bayes truncates ('skip') long tokens on bytes, should it
count characters instead?
Posted by bu...@bugzilla.spamassassin.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7141
Mark Martinec <Ma...@ijs.si> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
Target Milestone|Undefined |3.4.1
--- Comment #4 from Mark Martinec <Ma...@ijs.si> ---
> I'm folding the above in. I think this follows the spirit of the original
> code (takes 7 characters)
Closing.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7141] Bayes truncates ('skip') long tokens on bytes, should it
count characters instead?
Posted by bu...@bugzilla.spamassassin.org.
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7141
--- Comment #3 from Mark Martinec <Ma...@ijs.si> ---
I'm folding the above in. I think this follows the spirit of the original
code (takes 7 characters), except that back in 2002 single-byte encodings
(extended ASCII like Latin etc.) were much more prevalent than today's UTF-8,
(even without normalize_charset turned on).
Bug 7141: Bayes truncates ('skip') long tokens on bytes,
should it count characters instead?
Sending lib/Mail/SpamAssassin/Plugin/Bayes.pm
Committed revision 1661633.
Re-assessing usefulness of skips (and/or their length) might still be
a good thing to do. If someone has a will to do so is most welcome.
--
You are receiving this mail because:
You are the assignee for the bug.