You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Eddy Beliveau <ed...@hec.ca> on 2009/01/08 22:05:42 UTC

Malformed UTF-8 character with SA 3.2.5

Hi!

I sent this email to the amavisd-new group but didn't received any replies

I give it a spin on this group

Maybe someone can help

Thanks in advance 
Eddy

----------------- original email

Hi!

We are using Postfix 2.5.5 on our RHEL AS release 4 (Nahant Update 6)
academic server.

amavisd-new-2.6.2 (20081215), Unicode aware, LANG="fr"
with spamassassin 3.2.5

Our log file contains many:
 amavis[19738]: (19738-05) _WARN: Malformed UTF-8 character (unexpected 
continuation byte 0x8e, with no preceding start byte) in pattern match 
(m//) at 
/var/lib/spamassassin/3.002005/70_sare_specific_cf_sare_sa-update_dostech_net/200605280300.cf, 
rule SARE_SPEC_REPL_OBFU2, line 1, <GEN16> line 3620.

That rule is version 01.03.13

Can someone help ?

Many thanks
Eddy
--

Eddy Beliveau
HEC Montreal
Montreal (Quebec)
Canada


Re: Malformed UTF-8 character with SA 3.2.5

Posted by Eddy Beliveau <ed...@hec.ca>.
Hi! Mark,

Many thanks for your reply.

I also have similar WARN error
Jan  9 09:20:02 smtpext2 amavis[20636]: (20636-16) _WARN: Malformed 
UTF-8 character (unexpected continuation byte 0xba, with no preceding 
start byte) in pattern match (m//) at 
/usr/lib/perl5/site_perl/5.8.5/Mail/SpamAssassin/Plugin/BodyEval.pm line 
243, <GEN16> line 10082.

As suggested, I created my first spamassassin bug report (6042) and 
include a sample

Thanks,
Eddy

-------- Message original --------
Sujet : Re: Malformed UTF-8 character with SA 3.2.5
De : Mark Martinec <Ma...@ijs.si>
Pour : users@spamassassin.apache.org
Date : 2009-01-08 18:55

> Eddy,
> 
>> I sent this email to the amavisd-new group but didn't received any replies
>> I give it a spin on this group
>> Maybe someone can help
> 
> Yes, this is probably a more suitable place for this question.
> 
>> We are using Postfix 2.5.5 on our RHEL AS release 4 (Nahant Update 6)
>> academic server.
>>
>> amavisd-new-2.6.2 (20081215), Unicode aware, LANG="fr"
>> with spamassassin 3.2.5
>>
>> Our log file contains many:
>>  amavis[19738]: (19738-05) _WARN: Malformed UTF-8 character (unexpected
>> continuation byte 0x8e, with no preceding start byte) in pattern match
>> (m//) at
>> /var/lib/spamassassin/3.002005/70_sare_specific_cf_sare_sa-update_dostech_n
>> et/200605280300.cf, rule SARE_SPEC_REPL_OBFU2, line 1, <GEN16> line 3620.
>>
>> That rule is version 01.03.13
>> Can someone help ?
> 
> I searched our logs for something similar and came up with a possibly related
> case, but in a different code section. Here is mine (using SA 3.3):
> 
> rules: failed to run TVD_STOCK1 test, skipping:                                                   
>         (Malformed UTF-8 character (fatal) 
> at /usr/local/lib/perl5/site_perl/5.10.0/Mail/SpamAssassin/Plugin/BodyEval.pm 
> line 250, <GEN11> line 499.
> 
> This one is  within sub _check_stock_info, evaluating the regexp:
>   $rnd_chunk =~ /^\s*([^:\s][^:\n]{2,29})\s*:\s*\S/mg
> on a perfectly valid UTF-8 string. It turns out it is a bug in
> perl5.8.8, 5.8.9 and in 5.10.0 - the bug goes away if the string
> is not tainted.
> 
> I filed a Perl bug report yesterday:
>   perlbug: [perl #62048]
>   Unwarranted "Malformed UTF-8 character" on tainted variable
> 
> I don't know if your case of "Malformed UTF-8 character" is due
> to the same bug, or is it a SpamAssassin problem.
> 
> Can you make one of your mail samples available for examination?
> With any luck the problem is reproducible, making it much easier
> to resolve or provide a workaround. It's probably best to open
> a SpamAssassin bug case and attach a sample.
> 
>   Mark
> 

-- 
Eddy Beliveau
HEC Montreal
Montreal (Quebec)
Canada

Re: Malformed UTF-8 character with SA 3.2.5

Posted by Sergey Kovalev <sp...@kovalev.com.ru>.
Mark Martinec wrote:
> 
>> Our log file contains many:
>>  amavis[19738]: (19738-05) _WARN: Malformed UTF-8 character (unexpected
>> continuation byte 0x8e, with no preceding start byte) in pattern match
>> (m//) at
>> /var/lib/spamassassin/3.002005/70_sare_specific_cf_sare_sa-update_dostech_n
>> et/200605280300.cf, rule SARE_SPEC_REPL_OBFU2, line 1, <GEN16> line 3620.
> 
> I searched our logs for something similar and came up with a possibly related
> case, but in a different code section. Here is mine (using SA 3.3):
> 
> rules: failed to run TVD_STOCK1 test, skipping:                                                   
>         (Malformed UTF-8 character (fatal) 
> at /usr/local/lib/perl5/site_perl/5.10.0/Mail/SpamAssassin/Plugin/BodyEval.pm 
> line 250, <GEN11> line 499.
> 
> This one is  within sub _check_stock_info, evaluating the regexp:
>   $rnd_chunk =~ /^\s*([^:\s][^:\n]{2,29})\s*:\s*\S/mg
> on a perfectly valid UTF-8 string. It turns out it is a bug in
> perl5.8.8, 5.8.9 and in 5.10.0 - the bug goes away if the string
> is not tainted.

I'm not quite familiar with utf-8 handling in perl, but it seems to me 
that there are some different flavours of UTF :)
I'm trying to score cyrillic pr0n messages. And it is not simple.
In order SA to start hitting my regexp rules I've modified Check.pm with 
`use utf8;' string and added normalize_charset 1 to local.cf.
Now it detects bad words but spamd generates warnings at startup/reload

Jan  9 01:54:40 mx spamd[19814]: Malformed UTF-8 character (unexpected 
non-continuation byte 0xe8, immediately after start byte 0xe9) in eval 
"string" at 
/var/db/spamassassin/3.002005/updates_spamassassin_org/20_advance_fee.cf, 
rule __FRAUD_GAN, line 1.
Jan  9 01:54:40 mx spamd[19814]: Malformed UTF-8 character (unexpected 
non-continuation byte 0x5d, immediately after start byte 0xe8) in eval 
"string" at 
/var/db/spamassassin/3.002005/updates_spamassassin_org/20_advance_fee.cf, 
rule __FRAUD_GAN, line 1.

BTW, I don't need 20_advance_fee.cf but want to use 
updates.spamassassin.org channel. Is it possible to somehow ignore some 
*.cf?

Re: Malformed UTF-8 character with SA 3.2.5

Posted by Mark Martinec <Ma...@ijs.si>.
Eddy,

> I sent this email to the amavisd-new group but didn't received any replies
> I give it a spin on this group
> Maybe someone can help

Yes, this is probably a more suitable place for this question.

> We are using Postfix 2.5.5 on our RHEL AS release 4 (Nahant Update 6)
> academic server.
>
> amavisd-new-2.6.2 (20081215), Unicode aware, LANG="fr"
> with spamassassin 3.2.5
>
> Our log file contains many:
>  amavis[19738]: (19738-05) _WARN: Malformed UTF-8 character (unexpected
> continuation byte 0x8e, with no preceding start byte) in pattern match
> (m//) at
> /var/lib/spamassassin/3.002005/70_sare_specific_cf_sare_sa-update_dostech_n
>et/200605280300.cf, rule SARE_SPEC_REPL_OBFU2, line 1, <GEN16> line 3620.
>
> That rule is version 01.03.13
> Can someone help ?

I searched our logs for something similar and came up with a possibly related
case, but in a different code section. Here is mine (using SA 3.3):

rules: failed to run TVD_STOCK1 test, skipping:                                                   
        (Malformed UTF-8 character (fatal) 
at /usr/local/lib/perl5/site_perl/5.10.0/Mail/SpamAssassin/Plugin/BodyEval.pm 
line 250, <GEN11> line 499.

This one is  within sub _check_stock_info, evaluating the regexp:
  $rnd_chunk =~ /^\s*([^:\s][^:\n]{2,29})\s*:\s*\S/mg
on a perfectly valid UTF-8 string. It turns out it is a bug in
perl5.8.8, 5.8.9 and in 5.10.0 - the bug goes away if the string
is not tainted.

I filed a Perl bug report yesterday:
  perlbug: [perl #62048]
  Unwarranted "Malformed UTF-8 character" on tainted variable

I don't know if your case of "Malformed UTF-8 character" is due
to the same bug, or is it a SpamAssassin problem.

Can you make one of your mail samples available for examination?
With any luck the problem is reproducible, making it much easier
to resolve or provide a workaround. It's probably best to open
a SpamAssassin bug case and attach a sample.

  Mark