Re: More text/plain questions

[ha...@t-online.de]

>> >> I got the following MIME body part below, and I�m wondering if it would make sense to filter on this as well.
>> >> Given that it�s text/plain with an implicit charset=�us-ascii� and an implicit content-transfer-encoding of 7bit, the sequence &#x[0-9A-F]{4} doesn�t really parse into a 16-bit character, would it? That would be a broken MUA that made such a leap...
>> >> Wouldn�t that normally render as the character �&�, �#�, �x�, etc. rather than the unicode16 or UTF-8 character with that hex value?
>> >> There might be times when someone has sent an attachment improperly encoded this way which might have embedded binary values in it, but that�s kind of buggy anyway� it should have been done as base64 and application/octet-stream in the worst of cases if it has arbitrary binary data.
>> >> I wouldn�t want a message where someone gives a couple of examples of encoding &#x0400 for instance being flagged as SPAM, but if the text is 20% or more of these sequences then I would say that�s SPAM-sign.
>> >> Anyway, here�s the body I saw:
>> >> --1388-8200-b67c-e579-9c27-df36-12fa-a2eb
>> Content-Type: text/plain;
>> >> Thе Rеаl RеаѕоnThе Ꮯоmіng Ꮯоllарѕе...Thе rеаl rеаѕоn ᎳHY HоmеlаndSеcurіtу rеcеntlу рurchаѕеd1.7 Bіllіоn Rоundѕ оf аmmunіtіоn...Ꮃhаt Yоu Muѕt Dо Tо Ꭼnѕurе YоurSаfеtуHоmеlаnd ѕеcurіtу іѕ thеrе tо ѕеcurеthе hоmеlаnd оnlу... Sо thеѕе Ьullеtѕаrе rеаlу mеаnt fоr thеThіѕ іѕ аn еmаіlаdvеrtіѕеmеnt thаt wаѕ ѕеnt tо уоu Ьу Ρаtrіоt Survіvаl Ρlаn. If уоuwіѕh tо nоlоngеr rеcеіvе mеѕѕаgеѕ thаt рrоmоtе ѕurvіvаl tірѕ, рlеаѕеclіck hеrе tо unѕuЬѕcrіЬе.4 Unstable as water, thou shalt not excel because thou wentest up to thy fathers bed then defiledst thou it he went up to my couch.34 And Pharaohnechoh made Eliakim the son of Josiah king in the room of Josiah his father, and turned his name to Jehoiakim, and took Jehoahaz away and he came to Egypt, and died there.37  And the thing was good in the eyes of Pharaoh, and in the eyes o!
>> f all his servants.
>> >> --1388-8200-b67c-e579-9c27-df36-12fa-a2eb

Hi,

while this is certainly not correct - and likely does not display in every mail client - it would
probably work in several webmailers. Perhaps this is the configuration the author of that
crap tested.
Now, I am somewhat reluctant to classify badly formatted mails as spam: there are many
systems around, even from major players, that send legitimate mails like order confirmation,
delivery notification, opted-in newsletters but do many of the formal things more right than wrong
On the other side, looking at the actual characters shows that the message is spam: these are
cyrillic letters that happen to look exactly like western ones (a, e, o or such) so the obvious intent
is to avoid detection of the strings. We have seen the same with IDN domain names that might
use a cyrillic a to register a domain that looks like, e.g. paypal.com
The list of characters is fairly short, so maybe checking for these characters in all commonly
used variants (html entities, utf8 encoded, +u0430, \u0430. IDN encoded) would be a good
spam indication

Regards
Wolfgang

Re: More text/plain questions

["David F. Skoll" <df...@roaringpenguin.com>]

On Mon, 07 Jul 2014 19:29:11 -0400
Daniel Staal <DS...@usa.net> wrote:

> Just to start the discussion: I'd say default to UTF-8 if not
> otherwise specified and can't be worked out.  (How hard to work on
> 'working it out' is a question, of course.)  It's the growing
> standard, as far as I can tell.

+1.  UTF-8 is the best choice.  (Modern) Perl handles it very nicely.
Even non-UTF-8 messages should be recoded into UTF-8 for body rules;
otherwise, making a rule that looks for things like "抵押" will be
well-nigh impossible.

Regards,

David.

Re: More text/plain questions

[Daniel Staal <DS...@usa.net>]

--As of July 7, 2014 5:20:01 PM -0400, Kevin A. McGrail is alleged to have 
said:

> On 7/7/2014 5:09 PM, Philip Prindeville wrote:
>> On Jul 7, 2014, at 7:15 AM, Kevin A. McGrail <KM...@PCCC.com> wrote:
>>
>>> On 7/7/2014 2:28 AM, John Wilcock wrote:
>>>> Le 05/07/2014 19:08, Philip Prindeville a écrit :
>>>>> As for encoding a cyrillic small a: there are many ways to do this.
>>>>> iso-8859-4, utf-8, jp2212, gb2312, win1252, etc. I don’t think this
>>>>> would be very efficient—there are just too many charsets possible.
>>>> Normalising the input message to UTF-8 before body checks would help
>>>> somewhat with that. I seem to remember there's been talk of doing this.
>>>>
>>> Yes, or utf-16...  I think that will be necessary to keep SA effective
>>> in the modern world sooner than later.
>>
>> Okay, but… if the message body is non-ASCII and the CTE is 8bit or
>> base64 and no explicit charset has been given, how do you know which
>> translation to perform?
>>
>> I get a lot of Han SPAM in GB2312 where the charset is never specified
>> (apparently it’s a national default in China, despite the requirements
>> stated in RFC-2045 and -2046).
> Sorry, I haven't even started delving into the devilish details but I
> know it's looming as a needed feature.

--As for the rest, it is mine.

Just to start the discussion: I'd say default to UTF-8 if not otherwise 
specified and can't be worked out.  (How hard to work on 'working it out' 
is a question, of course.)  It's the growing standard, as far as I can tell.

Even if it's wrong in a particular case, it would probably be useful: It 
would give rule writers something to work with.

Daniel T. Staal

---------------------------------------------------------------
This email copyright the author.  Unless otherwise noted, you
are expressly allowed to retransmit, quote, or otherwise use
the contents for non-commercial purposes.  This copyright will
expire 5 years after the author's death, or in 30 years,
whichever is longer, unless such a period is in excess of
local copyright law.
---------------------------------------------------------------

Re: More text/plain questions

["Kevin A. McGrail" <KM...@PCCC.com>]

On 7/7/2014 5:09 PM, Philip Prindeville wrote:
> On Jul 7, 2014, at 7:15 AM, Kevin A. McGrail <KM...@PCCC.com> wrote:
>
>> On 7/7/2014 2:28 AM, John Wilcock wrote:
>>> Le 05/07/2014 19:08, Philip Prindeville a écrit :
>>>> As for encoding a cyrillic small a: there are many ways to do this.
>>>> iso-8859-4, utf-8, jp2212, gb2312, win1252, etc. I don’t think this
>>>> would be very efficient—there are just too many charsets possible.
>>> Normalising the input message to UTF-8 before body checks would help somewhat with that. I seem to remember there's been talk of doing this.
>>>
>> Yes, or utf-16...  I think that will be necessary to keep SA effective in the modern world sooner than later.
>
> Okay, but… if the message body is non-ASCII and the CTE is 8bit or base64 and no explicit charset has been given, how do you know which translation to perform?
>
> I get a lot of Han SPAM in GB2312 where the charset is never specified (apparently it’s a national default in China, despite the requirements stated in RFC-2045 and -2046).
Sorry, I haven't even started delving into the devilish details but I 
know it's looming as a needed feature.

regards,
KAM

Re: More text/plain questions

[Philip Prindeville <ph...@redfish-solutions.com>]

On Jul 7, 2014, at 7:15 AM, Kevin A. McGrail <KM...@PCCC.com> wrote:

> On 7/7/2014 2:28 AM, John Wilcock wrote:
>> Le 05/07/2014 19:08, Philip Prindeville a écrit :
>>> As for encoding a cyrillic small a: there are many ways to do this.
>>> iso-8859-4, utf-8, jp2212, gb2312, win1252, etc. I don’t think this
>>> would be very efficient—there are just too many charsets possible.
>> 
>> Normalising the input message to UTF-8 before body checks would help somewhat with that. I seem to remember there's been talk of doing this.
>> 
> Yes, or utf-16...  I think that will be necessary to keep SA effective in the modern world sooner than later.


Okay, but… if the message body is non-ASCII and the CTE is 8bit or base64 and no explicit charset has been given, how do you know which translation to perform?

I get a lot of Han SPAM in GB2312 where the charset is never specified (apparently it’s a national default in China, despite the requirements stated in RFC-2045 and -2046).

-Philip

Re: More text/plain questions

["Kevin A. McGrail" <KM...@PCCC.com>]

On 7/7/2014 2:28 AM, John Wilcock wrote:
> Le 05/07/2014 19:08, Philip Prindeville a écrit :
>> As for encoding a cyrillic small a: there are many ways to do this.
>> iso-8859-4, utf-8, jp2212, gb2312, win1252, etc. I don’t think this
>> would be very efficient—there are just too many charsets possible.
>
> Normalising the input message to UTF-8 before body checks would help 
> somewhat with that. I seem to remember there's been talk of doing this.
>
Yes, or utf-16...  I think that will be necessary to keep SA effective 
in the modern world sooner than later.

Re: More text/plain questions

[John Wilcock <jo...@tradoc.fr>]

Le 05/07/2014 19:08, Philip Prindeville a écrit :
> As for encoding a cyrillic small a: there are many ways to do this.
> iso-8859-4, utf-8, jp2212, gb2312, win1252, etc. I don’t think this
> would be very efficient—there are just too many charsets possible.

Normalising the input message to UTF-8 before body checks would help 
somewhat with that. I seem to remember there's been talk of doing this.

-- 
John

Re: More text/plain questions

[Philip Prindeville <ph...@redfish-solutions.com>]

On Jul 4, 2014, at 12:08 AM, hamann.w@t-online.de wrote:

> 
> Hi,
> 
> while this is certainly not correct - and likely does not display in every mail client - it would
> probably work in several webmailers. Perhaps this is the configuration the author of that
> crap tested.
> Now, I am somewhat reluctant to classify badly formatted mails as spam: there are many
> systems around, even from major players, that send legitimate mails like order confirmation,
> delivery notification, opted-in newsletters but do many of the formal things more right than wrong
> On the other side, looking at the actual characters shows that the message is spam: these are
> cyrillic letters that happen to look exactly like western ones (a, e, o or such) so the obvious intent
> is to avoid detection of the strings. We have seen the same with IDN domain names that might
> use a cyrillic a to register a domain that looks like, e.g. paypal.com
> The list of characters is fairly short, so maybe checking for these characters in all commonly
> used variants (html entities, utf8 encoded, +u0430, \u0430. IDN encoded) would be a good
> spam indication
> 
> Regards
> Wolfgang
> 
> 

I think you’re overlooking what a lot of tests already do: test for poor formatting.

INVALID_DATE
UNPARSEABLE_RELAY
HTML_MISSING_CTYPE
MISSING_HEADERS
MISSING_DATE

As for encoding a cyrillic small a: there are many ways to do this. iso-8859-4, utf-8, jp2212, gb2312, win1252, etc. I don’t think this would be very efficient—there are just too many charsets possible.

-Philip