You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Loren Wilton <lw...@earthlink.net> on 2004/02/28 04:46:42 UTC
Trying to detect bogus end tags
I'm trying to come up with a way to detect bogus end tags, and so far I'm
not having much luck.
What I'm specifically trying to catch are things like
</table>
</belch></huntsville></delusion></wilma></boswell></attune>
</vasectomy></centum></surf></yeasty></molt></autocollimate>
</acrobat></harvest></gage></flagrant></fumble></nowadays>
</BODY>
</HTML>
Now, it looks like there is an html_tag_balance eval that would catch the
fact that there is no "<belch" to match the "</belch>" in the above hunk of
spam, if only there were some way that I could feed "belch" into the eval.
I can detect end tags eash enough with a regexp, but I can't find any way
that works to pull the found tag out and feed it to the eval routine within
an SA rule definition.
Alternately, is there a way to write a regexp that will let me look backward
for <belch once I have found </belch>? I can't seem to figure this one out
either.
Thanks,
Loren
Re: Trying to detect bogus end tags
Posted by Loren Wilton <lw...@earthlink.net>.
I've finally got a strange regexp that *almost* works. I can see it getting
the right results internally, but it never seems to return true or whatever
a regexp returns on success. At this point I have no idea if I'm dealing
with bugs, features, or a strong misunderstanding of the language.
If anyone wants to try to figure out what is wrong with it (or can see it at
a glance) here is the ugly little thing:
full BAD__HTML m|<BODY>(.*)</BODY>(??{ (scalar reverse $1) =~
/>([a-z]+)\/<(?!.*[\s>]\1<)/is; })|s
Loren
Re: Trying to detect bogus end tags
Posted by Berend De Schouwer <bd...@jhb.ucs.co.za>.
On Monday 01 March 2004 11:54, Berend De Schouwer wrote:
> On Saturday 28 February 2004 05:46, Loren Wilton wrote:
> > I'm trying to come up with a way to detect bogus end tags, and so
> > far I'm not having much luck.
> >
> > What I'm specifically trying to catch are things like
> >
> > </table>
> > </belch></huntsville></delusion></wilma></boswell></attune>
> > </vasectomy></centum></surf></yeasty></molt></autocollimate>
> > </acrobat></harvest></gage></flagrant></fumble></nowadays>
> > </BODY>
> > </HTML>
This does catch them:
full __HTML_BAYES_POISON /(\<\/[a-zA-Z]{3,}\>[\s]*){20,}/
meta BDS_HTML_BAYES_POISON (HTML_MESSAGE && __HTML_BAYES_POISON)
describe BDS_HTML_BAYES_POISON Multiple garbage HTML close tag
score BDS_HTML_BAYES_POISON 2.0
You might get false positives if you regularly get HTML e-mail with real
statements like (/b)(/font)(/tr)(/td)(/table), etc. but I think that
won't be common. At least not 20 tags in a row. To counter false
positives, I've made it count 3 letter words only, so (/b) and (/tr)
won't count, and I've used 'full' instead of 'rawbody' so I don't have
a problem with \r\n.
The ideal solution is for SpamAssassin, or a plugin, to search for
unmatched HTML tags, in HTML MIME parts, except for (p) and (br), and
score 0.1 for each unmatched tag. In the absense of perfection, try
this.
--
Berend De Schouwer
Re: Trying to detect bogus end tags
Posted by Berend De Schouwer <bd...@jhb.ucs.co.za>.
On Saturday 28 February 2004 05:46, Loren Wilton wrote:
> I'm trying to come up with a way to detect bogus end tags, and so far
> I'm not having much luck.
>
> What I'm specifically trying to catch are things like
>
> </table>
> </belch></huntsville></delusion></wilma></boswell></attune>
> </vasectomy></centum></surf></yeasty></molt></autocollimate>
> </acrobat></harvest></gage></flagrant></fumble></nowadays>
> </BODY>
> </HTML>
>
> Now, it looks like there is an html_tag_balance eval that would catch
> the fact that there is no "<belch" to match the "</belch>" in the
> above hunk of spam, if only there were some way that I could feed
> "belch" into the eval. I can detect end tags eash enough with a
> regexp, but I can't find any way that works to pull the found tag out
> and feed it to the eval routine within an SA rule definition.
I see this too. I don't know the proper answer, but you could just look
for 30 close tags in a row.
rawbody TOO_MANY_CLOSE_TAGS /(\<[\/]?[^\>]{1,}\>){20,}/i
I don't think this is too effective in the long term, but it'll cat some
spams.
> Alternately, is there a way to write a regexp that will let me look
> backward for <belch once I have found </belch>? I can't seem to
> figure this one out either.
>
> Thanks,
> Loren
--
Berend De Schouwer
Re: Trying to detect bogus end tags
Posted by Loren Wilton <lw...@earthlink.net>.
> The list of valid HTML tags is finite. You could try something like:
>
> rawbody BOGUS_HTML_TAG /<\/(?<!(list|of|valid|tags|...)>)[a-z]+>/i
>
> N.B.: I'm still having trouble wrapping my brain around zero-length
> assertions - does the above look right?
I think the above would do it for HTML tags. However, someone else pointed
out that there are a whole lot of other SGML-formatted things that can
appear in mail, and the list of tags for such things is essentially
infinite. Thus, the above test might not be manageable. Which is why I was
trying to match an end tag to a missing begin tag. In my rather limited
understanding of things SGML-like, I don't think it is valid to have an end
tag without a corresponding (proplerly nested) begin tag. Assuming that
basic assumption was correct, a check for any begin tag to match a given end
tag (without bothering with nesting and true balance checks) would be a big
step forward.
Loren
Re: Trying to detect bogus end tags
Posted by John Hardin <jo...@aproposretail.com>.
On Fri, 2004-02-27 at 19:46, Loren Wilton wrote:
> I'm trying to come up with a way to detect bogus end tags, and so far I'm
> not having much luck.
>
> What I'm specifically trying to catch are things like
>
> </table>
> </belch></huntsville></delusion></wilma></boswell></attune>
> </vasectomy></centum></surf></yeasty></molt></autocollimate>
> </acrobat></harvest></gage></flagrant></fumble></nowadays>
> </BODY>
> </HTML>
The list of valid HTML tags is finite. You could try something like:
rawbody BOGUS_HTML_TAG /<\/(?<!(list|of|valid|tags|...)>)[a-z]+>/i
Here is where additive scoring would be of benefit. I wouldn't want to
score high on a handful of bogus tags, but a dozen or more should score
fairly high.
N.B.: I'm still having trouble wrapping my brain around zero-length
assertions - does the above look right?
--
John Hardin KA7OHZ
Internal Systems Administrator/Guru voice: (425) 672-1304
Apropos Retail Management Systems, Inc. fax: (425) 672-0192
-----------------------------------------------------------------------
Failure to plan ahead on someone else's part does not constitute an
emergency on my part.
- David W. Barts in a.s.r
Re: Trying to detect bogus end tags
Posted by Daniel Quinlan <qu...@pathname.com>.
"Loren Wilton" <lw...@earthlink.net> writes:
> I'm trying to come up with a way to detect bogus end tags, and so far I'm
> not having much luck.
3.0 will have a test for this, although it just looks for tags in
general. Someone could try enhancing the test to also notice when an
end element is used for a start-only tag.
These two tests overlap quite a bit, I'm leaving it to the score
optimizer to figure out the final scores...
OVERALL% SPAM% HAM% S/O RANK SCORE NAME
387188 306777 80411 0.792 0.00 0.00 (all messages)
100.000 79.2321 20.7679 0.792 0.00 0.00 (all messages as %)
2.305 2.7818 0.4863 0.851 0.58 1.00 HTML_BADTAG_00_10
4.236 5.3309 0.0597 0.989 0.92 1.00 HTML_BADTAG_10_20
1.165 1.4678 0.0087 0.994 0.93 1.00 HTML_BADTAG_20_30
1.917 2.4171 0.0075 0.997 0.93 1.00 HTML_BADTAG_30_40
15.944 20.1234 0.0000 1.000 0.97 1.00 HTML_BADTAG_40_50
0.454 0.5731 0.0000 1.000 0.94 1.00 HTML_BADTAG_50_60
1.023 1.2915 0.0000 1.000 0.94 1.00 HTML_BADTAG_60_70
0.367 0.4635 0.0000 1.000 0.94 1.00 HTML_BADTAG_70_80
0.127 0.1604 0.0000 1.000 0.94 1.00 HTML_BADTAG_80_90
0.015 0.0186 0.0000 1.000 0.94 1.00 HTML_BADTAG_90_100
1.312 1.5777 0.2985 0.841 0.56 1.00 HTML_NONELEMENT_00_10
0.600 0.7168 0.1530 0.824 0.53 1.00 HTML_NONELEMENT_10_20
0.598 0.7409 0.0510 0.936 0.77 1.00 HTML_NONELEMENT_20_30
2.936 3.6994 0.0236 0.994 0.93 1.00 HTML_NONELEMENT_30_40
0.966 1.2159 0.0112 0.991 0.92 1.00 HTML_NONELEMENT_40_50
15.548 19.6214 0.0075 1.000 0.97 1.00 HTML_NONELEMENT_50_60
1.477 1.8626 0.0037 0.998 0.94 1.00 HTML_NONELEMENT_60_70
1.409 1.7749 0.0112 0.994 0.92 1.00 HTML_NONELEMENT_70_80
1.556 1.9627 0.0025 0.999 0.94 1.00 HTML_NONELEMENT_80_90
1.153 1.4558 0.0000 1.000 0.94 1.00 HTML_NONELEMENT_90_100
For HTML_MESSAGE messages only:
OVERALL% SPAM% HAM% S/O RANK SCORE NAME
265707 260392 5315 0.980 0.00 0.00 (all messages)
100.000 97.9997 2.0003 0.980 0.00 0.00 (all messages as %)
3.359 3.2774 7.3565 0.308 0.03 1.00 HTML_BADTAG_00_10
6.173 6.2805 0.9031 0.874 0.65 1.00 HTML_BADTAG_10_20
1.697 1.7293 0.1317 0.929 0.77 1.00 HTML_BADTAG_20_30
2.793 2.8476 0.1129 0.962 0.86 1.00 HTML_BADTAG_30_40
23.234 23.7081 0.0000 1.000 0.99 1.00 HTML_BADTAG_40_50
0.662 0.6751 0.0000 1.000 0.96 1.00 HTML_BADTAG_50_60
1.491 1.5216 0.0000 1.000 0.96 1.00 HTML_BADTAG_60_70
0.535 0.5461 0.0000 1.000 0.96 1.00 HTML_BADTAG_70_80
0.185 0.1889 0.0000 1.000 0.96 1.00 HTML_BADTAG_80_90
0.021 0.0219 0.0000 1.000 0.96 1.00 HTML_BADTAG_90_100
1.912 1.8587 4.5155 0.292 0.03 1.00 HTML_NONELEMENT_00_10
0.874 0.8445 2.3142 0.267 0.02 1.00 HTML_NONELEMENT_10_20
0.871 0.8729 0.7714 0.531 0.14 1.00 HTML_NONELEMENT_20_30
4.278 4.3584 0.3575 0.924 0.76 1.00 HTML_NONELEMENT_30_40
1.407 1.4325 0.1693 0.894 0.69 1.00 HTML_NONELEMENT_40_50
22.657 23.1167 0.1129 0.995 0.98 1.00 HTML_NONELEMENT_50_60
2.152 2.1944 0.0564 0.975 0.89 1.00 HTML_NONELEMENT_60_70
2.053 2.0911 0.1693 0.925 0.76 1.00 HTML_NONELEMENT_70_80
2.267 2.3123 0.0376 0.984 0.91 1.00 HTML_NONELEMENT_80_90
1.681 1.7151 0.0000 1.000 0.96 1.00 HTML_NONELEMENT_90_100
Daniel
--
Daniel Quinlan anti-spam (SpamAssassin), Linux,
http://www.pathname.com/~quinlan/ and open source consulting