You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2013/06/19 22:32:14 UTC
[Bug 6945] New: sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6945
Bug ID: 6945
Summary: sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
Product: Spamassassin
Version: 3.3.1
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: Learner
Assignee: dev@spamassassin.apache.org
Reporter: sa_bugzilla@m.nbs-eng.ru
sa-learn dies with the following error if a message contains non-ASCII
characters in host name portion of Message-ID:
plugin: eval failed: bayes: (in learn) Wide character in subroutine entry at
/usr/share/perl5/Mail/SpamAssassin/BayesStore/DBM.pm line 826.
ERROR: the Bayes learn function returned an error, please re-run with -D for
more information at /usr/bin/sa-learn line 493.
In this particular case the offending Message-ID line contains Cyrillic
characters encoded in CP-1251,
If decoded from CP-1251 to UTF-8 the line would read:
Message-ID: <8B8D9181744049F381C34F99B078DC9B@Костя-ПК>
After removing this line from the message sa-learn worked just fine.
A sample (heavily reducted and massaged) message in mbox format is attached.
Try the following command to reproduce the problem:
LANG=C sa-learn --spam --mbox sample.mbox
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6945] sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6945
--- Comment #2 from AXB <ax...@gmail.com> ---
(In reply to Alexander.A from comment #1)
> Created attachment 5154 [details]
> sample mbox which crashes sa-learn
Could you please post a "real world" sample?
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6945] sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6945
Alexander.A <sa...@m.nbs-eng.ru> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |sa_bugzilla@m.nbs-eng.ru
--- Comment #1 from Alexander.A <sa...@m.nbs-eng.ru> ---
Created attachment 5154
--> https://issues.apache.org/SpamAssassin/attachment.cgi?id=5154&action=edit
sample mbox which crashes sa-learn
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6945] sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6945
--- Comment #7 from Mark Martinec <Ma...@ijs.si> ---
Created attachment 5155
--> https://issues.apache.org/SpamAssassin/attachment.cgi?id=5155&action=edit
prevents Message-ID and similar header fields to be associated with a character
set
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6945] sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6945
--- Comment #5 from Alexander.A <sa...@m.nbs-eng.ru> ---
I think I found the culprit, but not the solution.
To reproduce the bug on clean Debian 6.0 installation, please, enable option
normalize_charset 1
in ~/.spamassassin/user_prefs of /etc/spamassassin/local.cf
Note, that you need the Encode::Detect module installed for this option to have
effect. I am using Encode::Detect installed from the APT repository:
$ perl -le 'use Encode::Detect; print Encode::Detect->VERSION'
1.01
Hope, now the bug is reproducible.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6945] sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6945
Kevin A. McGrail <km...@pccc.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |kmcgrail@pccc.com
Severity|normal |blocker
--- Comment #10 from Kevin A. McGrail <km...@pccc.com> ---
The patch is clean and works but I don't work with alternate character sets
enough to know if this works/doesn't work in the real-world.
Changing to a blocker for 3.4.0 based on your email. Please review.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6945] sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6945
--- Comment #11 from Mark Martinec <Ma...@ijs.si> ---
Bug 6945 - fix a taint bug in dbg() call
Sending lib/Mail/SpamAssassin/BayesStore.pm
Committed revision 1495637.
Bug 6945: sa-learn dies on non-ASCII characters in Message-ID
Sending lib/Mail/SpamAssassin/Message/Node.pm
Committed revision 1495638.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6945] sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6945
--- Comment #3 from Mark Martinec <Ma...@ijs.si> ---
Can't reproduce it even with SpamAssassin 3.3.1,
although I can imagine how this issue could arise.
Please show:
$ perl -le 'use DB_File; print DB_File->VERSION'
$ perl -le 'use SHA1; print SHA1->VERSION'
$ perl -le 'use SHA; print SHA->VERSION'
$ perl -v
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6945] sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6945
--- Comment #8 from Alexander.A <sa...@m.nbs-eng.ru> ---
(In reply to Mark Martinec from comment #7)
> Created attachment 5155 [details]
The patch worked for me.
(on Debian 6.0, stock SA 3.3.1)
Fantastic! Just two days and the problem that bugged me for years is gone!
Thanks a lot.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6945] sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6945
--- Comment #6 from Mark Martinec <Ma...@ijs.si> ---
> I think I found the culprit, but not the solution.
> To reproduce the bug on clean Debian 6.0 installation, please, enable option
> normalize_charset 1
Ah, yes, that explains. A bug indeed, also still in trunk.
Please try the attached patch. It is a diff against trunk, but should
also apply to 3.3.1 with some offset.
It does three things:
- makes sure some header fields (like Message-ID, Received, ...) are
never attempted to be MIME-decoded - these header fields are just
strings of bytes, no character set can be associated with them;
- of the remaining header fields only those with explicit encodings
are decoded;
- fix a taint bug in BayesStore.pm dbg() call
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6945] sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6945
--- Comment #9 from Mark Martinec <Ma...@ijs.si> ---
> The patch worked for me.
> (on Debian 6.0, stock SA 3.3.1)
> Fantastic! Just two days and the problem that bugged me for years is gone!
Greats, thanks for troubleshooting and for trying it out.
I don't think this is a showstopper for 3.4.0-rc2, but
the patch should be considered for the final (or a next rc if any).
Regarding the closeness to a release and a (small) change in
behaviour, I guess it would be appropriate to vote on this change.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6945] sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6945
--- Comment #4 from Alexander.A <sa...@m.nbs-eng.ru> ---
Here you are:
$ perl -le 'use DB_File; print DB_File->VERSION'
1.82
SHA1 and SHA are not found:
$ perl -le 'use SHA1; print SHA1->VERSION'
Can't locate SHA1.pm in @INC..... [rest is skipped]
However, Digest::SHA* are present:
$ perl -le 'use Digest::SHA1; print Digest::SHA1->VERSION'
2.13
$ perl -le 'use Digest::SHA; print Digest::SHA->VERSION'
5.47
$ perl -v
This is perl, v5.10.1 (*) built for i486-linux-gnu-thread-multi
(with 59 registered patches, see perl -V for more detail)
[...Copyright info removed...]
Please, feel free to ask me to run more tests...
FYI: I am running Debian 6.0 (Squeeze), however the installation is quite old.
I do not remember exactly, but I probably first installed SA on Debian 4.0 or
even 3.1. The installation was probably via CPAN at that time, but then, during
an upgrade of the OS, I converted SA to stock package from apt repository. Do
not remember when this happend, or how I "converted" the installation.
Regarding the sample message: it is a "real world" spam message, with
everything non-essential removed. My installation of sa-learn dies on this
reducted message, also the full original message with just one line
(Message-ID:) removed is processed correctly.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6945] sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6945
Mark Martinec <Ma...@ijs.si> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
--- Comment #12 from Mark Martinec <Ma...@ijs.si> ---
I've been running now with 'normalize_charset 1' for a week now,
so far so good. Closing.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 6945] sa-learn dies on non-ASCII characters in Message-ID:
(when LANG=C)
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6945
Mark Martinec <Ma...@ijs.si> changed:
What |Removed |Added
----------------------------------------------------------------------------
Target Milestone|Undefined |3.4.0
--
You are receiving this mail because:
You are the assignee for the bug.