You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by yblin <yb...@ewave-technology.com> on 2016/04/29 11:30:14 UTC
normalize_charset
Hi! I've encountered a problem.I want to filter Chinese spams，and
through some documents I know that SpamAssassin can transform non-UTF-8
encoding emails into UTF-8 encoding ones.So I think that SpamAssassin
can filter not only UTF-8 encoding emails but also non-UTF-8 encoding
emails if I enable the option "normalize_charset".However,when I use
this option, a strange problem occures---I can not filter UTF-8 encoding
or non-UTF-8 encoding emails with this option enabled but can filter
UTF-8 encoding emails with this option disabled.
Could you tell me what the problem is and how to solve it?
Re: normalize_charset
Posted by RW <rw...@googlemail.com>.
On Sat, 30 Apr 2016 11:44:03 +0800
yblin wrote:
Please stop replying off-list.
>
> There is an example.I've used a UTF-8 encoding mail to debug
> SpamAssassin to test a rule called CN_BODY_1 in local.cf as the
> picture shows.
>
>
> The picture below shows the debuggin result when /normal_charset/
> disabled:
Images aren't very useful. We need to find a reproducible problem.
It's very easy to find example of CJK rules that do work with
"normalize_charset 1" on UTF-8 text, the following example came from
text from a random spam.
$ grep normalize .spamassassin/user_prefs
normalize_charset 1
$ grep T_CJK_TEST1 /usr/local/etc/mail/spamassassin/local-rules.cf
body T_CJK_TEST1 /如以上連接無法打開/
$ printf "\n如以上連接無法打開" | spamassassin | grep "T_CJK_TEST1"
T_CJK_TEST1 version=3.4.1
* 0.0 T_CJK_TEST1 BODY: No description available.
Re: normalize_charset
Posted by RW <rw...@googlemail.com>.
> Am 29.04.2016 um 11:54 schrieb yblin:
> > I can't filter the mail by body rules.However ,I can filter utf-8
> > encoding mails with /normalize_charset/ option disabled
Do you have an example of a rule that doesn't work? Preferably a
simple one.
Re: normalize_charset
Posted by Reindl Harald <h....@thelounge.net>.
Am 29.04.2016 um 11:54 schrieb yblin:
> I can't filter the mail by body rules.However ,I can filter utf-8
> encoding mails with /normalize_charset/ option disabled
stay on list!
as said you need to utf8-enocde your body rules
when it's done properly they work, proven by 1200 body-rules here
> 在 2016年04月29日 17:36, Reindl Harald 写道:
>>
>>
>> Am 29.04.2016 um 11:30 schrieb yblin:
>>> Hi! I've encountered a problem.I want to filter Chinese spams，and
>>> through some documents I know that SpamAssassin can transform non-UTF-8
>>> encoding emails into UTF-8 encoding ones.So I think that SpamAssassin
>>> can filter not only UTF-8 encoding emails but also non-UTF-8 encoding
>>> emails if I enable the option "normalize_charset".However,when I use
>>> this option, a strange problem occures---I can not filter UTF-8 encoding
>>> or non-UTF-8 encoding emails with this option enabled but can filter
>>> UTF-8 encoding emails with this option disabled.
>>> Could you tell me what the problem is and how to solve it?
>>
>> define "can not filter"
>>
>> bayes or rules?
>>
>> bayes:
>> you need to rebuild your bayes from scratch after change that option,
>> hence one should *always* keep the messages he used to train it for
>> such a rebuild and if it's only for the case that the tokenizer in a
>> later version get better capabilities (the same tokenizer is used for
>> split mails before train them as while test incoming mail against
>> trained tokens)
>>
>> body rules:
>> you need to utf8-encode the rules in your configuration
Re: normalize_charset
Posted by Reindl Harald <h....@thelounge.net>.
Am 29.04.2016 um 11:30 schrieb yblin:
> Hi! I've encountered a problem.I want to filter Chinese spams，and
> through some documents I know that SpamAssassin can transform non-UTF-8
> encoding emails into UTF-8 encoding ones.So I think that SpamAssassin
> can filter not only UTF-8 encoding emails but also non-UTF-8 encoding
> emails if I enable the option "normalize_charset".However,when I use
> this option, a strange problem occures---I can not filter UTF-8 encoding
> or non-UTF-8 encoding emails with this option enabled but can filter
> UTF-8 encoding emails with this option disabled.
> Could you tell me what the problem is and how to solve it?
define "can not filter"
bayes or rules?
bayes:
you need to rebuild your bayes from scratch after change that option,
hence one should *always* keep the messages he used to train it for such
a rebuild and if it's only for the case that the tokenizer in a later
version get better capabilities (the same tokenizer is used for split
mails before train them as while test incoming mail against trained tokens)
body rules:
you need to utf8-encode the rules in your configuration