You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2014/07/12 07:50:47 UTC
[Bug 7068] New: Plugin which counts unicode entities in text/plain
MIME parts
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7068
Bug ID: 7068
Summary: Plugin which counts unicode entities in text/plain
MIME parts
Product: Spamassassin
Version: SVN Trunk (Latest Devel Version)
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: Plugins
Assignee: dev@spamassassin.apache.org
Reporter: philipp@redfish-solutions.com
Created attachment 5221
--> https://issues.apache.org/SpamAssassin/attachment.cgi?id=5221&action=edit
First attempt, patch against svn trunk
A MIME part claiming to be text/plain and containing Unicode characters must be
encoded as quoted-printable or base64. Any message in 7bit or 8bit encoding
containing (HTML) Unicode entities will not render them as Unicode, but
literally.
Thus a few such sequences might occur on a mailing list of developers
discussing such characters, but a message with a high density of such
characters is likely spam.
Here we propose such a check (attached as diffs), and an example rule:
if can(Mail::SpamAssassin::Plugin::MIMEEval::has_check_abundant_unicode_ratio)
body T_PP_TOO_MUCH_UNICODE
eval:check_abundant_unicode_ratio('T_PP_TOO_MUCH_UNICODE', 0.05)
describe T_PP_TOO_MUCH_UNICODE Is text/plain but has many unicode escapes
score T_PP_TOO_MUCH_UNICODE 1.0
endif
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7068] Plugin which counts unicode entities in text/plain MIME
parts
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7068
--- Comment #6 from Philip Prindeville <ph...@redfish-solutions.com> ---
(In reply to Kevin A. McGrail from comment #5)
> (In reply to Philip Prindeville from comment #3)
> > I can attach a documentation patch if you want.
>
> PLease check over the docs I wrote. They were a quick pass.
Documentation looked fine.
> > requires 1 less 'undef' in the list?
>
> Yes, that's my belief (and what I implemented in the code).
Okay, in that case the rule name probably wasn't required in bug #7063 either.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7068] Plugin which counts unicode entities in text/plain MIME
parts
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7068
--- Comment #8 from Kevin A. McGrail <km...@pccc.com> ---
Philip, it appears I never committed the rules through a misspelling of the
word "add".
svn commit -m 'Adding rules for bug 7068'
Adding kmcgrail/20_bug_7068.cf
Transmitting file data .
Committed revision 1629622.
--
You are receiving this mail because:
You are the assignee for the bug.
Re: [Bug 7068] Plugin which counts unicode entities in text/plain
MIME parts
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 8/1/2014 7:06 PM, Quanah Gibson-Mount wrote:
> --On Friday, August 01, 2014 7:42 PM -0400 "Kevin A. McGrail"
> <KM...@PCCC.com> wrote:
>
>> PREFACE: You should discuss this type of stuff on the mailing list
>> not in
>> bugzilla. Bugzilla is not a support forum.
>
> Hm, my thought was if anyone else decides to do it, then it's tracked
> in bugzilla, so quick and easy to find.
Perhaps but you weren't reporting a bug, statistics on the
implementation, etc. You were just asking how to implement it.
Regards,
KAM
Re: [Bug 7068] Plugin which counts unicode entities in text/plain
MIME parts
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Friday, August 01, 2014 5:06 PM -0700 Quanah Gibson-Mount
<qu...@zimbra.com> wrote:
>> The PP_TOO_MUCH_UNICODE05 and 02 are being tested in RuleQA to see which
>> is most effective so you might use both, might use one, might pick a
>> different percentage etc. and please report your findings.
.02 is definitely hitting a lot. So far today, it has hit 10 times, all of
which were spam emails per RAZOR and DCC.
I haven't had a hit yet today on .05, but will keep checking.
--Quanah
--
Quanah Gibson-Mount
Server Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: [Bug 7068] Plugin which counts unicode entities in text/plain
MIME parts
Posted by Quanah Gibson-Mount <qu...@zimbra.com>.
--On Friday, August 01, 2014 7:42 PM -0400 "Kevin A. McGrail"
<KM...@PCCC.com> wrote:
> PREFACE: You should discuss this type of stuff on the mailing list not in
> bugzilla. Bugzilla is not a support forum.
Hm, my thought was if anyone else decides to do it, then it's tracked in
bugzilla, so quick and easy to find.
> Looks good.
>
> The PP_TOO_MUCH_UNICODE05 and 02 are being tested in RuleQA to see which
> is most effective so you might use both, might use one, might pick a
> different percentage etc. and please report your findings.
You bet!
--Quanah
--
Quanah Gibson-Mount
Server Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
Re: [Bug 7068] Plugin which counts unicode entities in text/plain
MIME parts
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 8/1/2014 6:38 PM, bugzilla-daemon@issues.apache.org wrote:
> https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7068
>
> Quanah Gibson-Mount <qu...@zimbra.com> changed:
>
> What |Removed |Added
> ----------------------------------------------------------------------------
> CC| |quanah@zimbra.com
>
> --- Comment #7 from Quanah Gibson-Mount <qu...@zimbra.com> ---
> I just grabbed the patch out of the subversion repo. I assume I want to do the
> following in my custom rules file:
>
> ifplugin Mail::SpamAssassin::Plugin::MIMEEval
> if
> can(Mail::SpamAssassin::Plugin::MIMEEval::has_check_abundant_unicode_ratio)
> body PP_TOO_MUCH_UNICODE05
> eval:check_abundant_unicode_ratio(0.05)
> describe PP_TOO_MUCH_UNICODE05 Is text/plain but has many unicode
> escapes
> score PP_TOO_MUCH_UNICODE05 1.0
>
> body PP_TOO_MUCH_UNICODE02 eval:check_abundant_unicode_ratio(0.02)
> describe PP_TOO_MUCH_UNICODE02 Is text/plain but has many unicode
> escapes
> score PP_TOO_MUCH_UNICODE02 0.5
> endif
> endif
>
> Thanks! :)
PREFACE: You should discuss this type of stuff on the mailing list not
in bugzilla. Bugzilla is not a support forum.
Looks good.
The PP_TOO_MUCH_UNICODE05 and 02 are being tested in RuleQA to see which
is most effective so you might use both, might use one, might pick a
different percentage etc. and please report your findings.
[Bug 7068] Plugin which counts unicode entities in text/plain MIME
parts
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7068
Quanah Gibson-Mount <qu...@zimbra.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |quanah@zimbra.com
--- Comment #7 from Quanah Gibson-Mount <qu...@zimbra.com> ---
I just grabbed the patch out of the subversion repo. I assume I want to do the
following in my custom rules file:
ifplugin Mail::SpamAssassin::Plugin::MIMEEval
if
can(Mail::SpamAssassin::Plugin::MIMEEval::has_check_abundant_unicode_ratio)
body PP_TOO_MUCH_UNICODE05
eval:check_abundant_unicode_ratio(0.05)
describe PP_TOO_MUCH_UNICODE05 Is text/plain but has many unicode
escapes
score PP_TOO_MUCH_UNICODE05 1.0
body PP_TOO_MUCH_UNICODE02 eval:check_abundant_unicode_ratio(0.02)
describe PP_TOO_MUCH_UNICODE02 Is text/plain but has many unicode
escapes
score PP_TOO_MUCH_UNICODE02 0.5
endif
endif
Thanks! :)
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7068] Plugin which counts unicode entities in text/plain MIME
parts
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7068
Kevin A. McGrail <km...@pccc.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
--- Comment #5 from Kevin A. McGrail <km...@pccc.com> ---
(In reply to Philip Prindeville from comment #3)
> (In reply to Kevin A. McGrail from comment #2)
>
> > - the biggest issue is that regardless of existing code, please take the
> > time to add POD documentation so that new features not only get added but
> > used in the future.
>
> I can attach a documentation patch if you want.
PLease check over the docs I wrote. They were a quick pass.
> > - I don't see why you are passing the rule name to the eval and I removed
> > that
>
> Sorry, passing it where? In the test rules? Okay, then does that mean that
> the 2nd line of:
>
> sub check_abundant_unicode_ratio {
> my ($self, $pms, undef, undef, $ratio) = @_;
>
> requires 1 less 'undef' in the list?
Yes, that's my belief (and what I implemented in the code).
> > - I added a rule with a 5% and a 2% threshold to see how it goes
> >
> > body PP_TOO_MUCH_UNICODE05 eval:check_abundant_unicode_ratio(0.05)
> > describe PP_TOO_MUCH_UNICODE05 Is text/plain but has many unicode
> > escapes
> > score PP_TOO_MUCH_UNICODE05 1.0
> > tflags PP_TOO_MUCH_UNICODE05 publish
> >
> > body PP_TOO_MUCH_UNICODE02
> > eval:check_abundant_unicode_ratio(0.02)
> > describe PP_TOO_MUCH_UNICODE02 Is text/plain but has many unicode
> > escapes
> > score PP_TOO_MUCH_UNICODE02 0.5
> > tflags PP_TOO_MUCH_UNICODE02 publish
> >
> > If it passes test, I'll commit and you can report more back.
>
> Awesome! Thanks.
Thank you for the patch. Next priority for me is analyzing why we are starved
on the ruleqa but that will be Monday night.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7068] Plugin which counts unicode entities in text/plain MIME
parts
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7068
Kevin A. McGrail <km...@pccc.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #5221|0 |1
is obsolete| |
--- Comment #4 from Kevin A. McGrail <km...@pccc.com> ---
Created attachment 5225
--> https://issues.apache.org/SpamAssassin/attachment.cgi?id=5225&action=edit
Slightly changed patch with docs and rules
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7068] Plugin which counts unicode entities in text/plain MIME
parts
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7068
Philip Prindeville <ph...@redfish-solutions.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |guenther@rudersport.de
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7068] Plugin which counts unicode entities in text/plain MIME
parts
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7068
--- Comment #1 from Philip Prindeville <ph...@redfish-solutions.com> ---
Created attachment 5222
--> https://issues.apache.org/SpamAssassin/attachment.cgi?id=5222&action=edit
Exemplary message which inspired writing the test
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7068] Plugin which counts unicode entities in text/plain MIME
parts
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7068
--- Comment #3 from Philip Prindeville <ph...@redfish-solutions.com> ---
(In reply to Kevin A. McGrail from comment #2)
> - the biggest issue is that regardless of existing code, please take the
> time to add POD documentation so that new features not only get added but
> used in the future.
I can attach a documentation patch if you want.
> - I don't see why you are passing the rule name to the eval and I removed
> that
Sorry, passing it where? In the test rules? Okay, then does that mean that
the 2nd line of:
sub check_abundant_unicode_ratio {
my ($self, $pms, undef, undef, $ratio) = @_;
requires 1 less 'undef' in the list?
> - I added a rule with a 5% and a 2% threshold to see how it goes
>
> body PP_TOO_MUCH_UNICODE05 eval:check_abundant_unicode_ratio(0.05)
> describe PP_TOO_MUCH_UNICODE05 Is text/plain but has many unicode
> escapes
> score PP_TOO_MUCH_UNICODE05 1.0
> tflags PP_TOO_MUCH_UNICODE05 publish
>
> body PP_TOO_MUCH_UNICODE02
> eval:check_abundant_unicode_ratio(0.02)
> describe PP_TOO_MUCH_UNICODE02 Is text/plain but has many unicode
> escapes
> score PP_TOO_MUCH_UNICODE02 0.5
> tflags PP_TOO_MUCH_UNICODE02 publish
>
> If it passes test, I'll commit and you can report more back.
Awesome! Thanks.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7068] Plugin which counts unicode entities in text/plain MIME
parts
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7068
Philip Prindeville <ph...@redfish-solutions.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #5221|0 |1
is patch| |
CC| |philipp@redfish-solutions.c
| |om
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7068] Plugin which counts unicode entities in text/plain MIME
parts
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7068
--- Comment #2 from Kevin A. McGrail <km...@pccc.com> ---
Some comments:
- the biggest issue is that regardless of existing code, please take the time
to add POD documentation so that new features not only get added but used in
the future.
- I don't see why you are passing the rule name to the eval and I removed that
- You don't need T_ for the rule. It's a test rule automatically until it's
promoted.
- I added a rule with a 5% and a 2% threshold to see how it goes
body PP_TOO_MUCH_UNICODE05 eval:check_abundant_unicode_ratio(0.05)
describe PP_TOO_MUCH_UNICODE05 Is text/plain but has many unicode
escapes
score PP_TOO_MUCH_UNICODE05 1.0
tflags PP_TOO_MUCH_UNICODE05 publish
body PP_TOO_MUCH_UNICODE02 eval:check_abundant_unicode_ratio(0.02)
describe PP_TOO_MUCH_UNICODE02 Is text/plain but has many unicode
escapes
score PP_TOO_MUCH_UNICODE02 0.5
tflags PP_TOO_MUCH_UNICODE02 publish
If it passes test, I'll commit and you can report more back.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 7068] Plugin which counts unicode entities in text/plain MIME
parts
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7068
Philip Prindeville <ph...@redfish-solutions.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |kmcgrail@pccc.com
--
You are receiving this mail because:
You are the assignee for the bug.