You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2006/07/13 13:02:48 UTC
[Bug 4988] New: mass-check vs report_safe encapsulation: From lines cause empty messages
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
Summary: mass-check vs report_safe encapsulation: From lines
cause empty messages
Product: Spamassassin
Version: SVN Trunk (Latest Devel Version)
Platform: Other
OS/Version: other
Status: NEW
Severity: major
Priority: P5
Component: Masses
AssignedTo: dev@spamassassin.apache.org
ReportedBy: jm@jmason.org
I noticed in
http://ruleqa.spamassassin.org/20060713-r421549-n/NO_RECEIVED/detail
that there are a *lot* of hits on NO_RECEIVED, EMPTY_MESSAGE, MISSING_DATE
and a few other rules in zmi's mass-check mboxes. This typically
indicates problems with empty messages in the data.
I investigated -- here's some lines for example:
. 4 /home/bbmass/rawcor/zmi/spam/masslearn_spam.15992.XzD16459.mbox.68831423
EMPTY_MESSAGE,MISSING_DATE,MISSING_HEADERS,MISSING_SUBJECT,NO_RECEIVED,
NO_RELAYS,TO_CC_NONE,TVD_SPACE_RATIO,__ENV_AND_HDR_FROM_MATCH,__UNUSABLE_MSGID
time=1149528224,sc antime=0,format=m,reuse=no . 4
/home/bbmass/rawcor/zmi/spam/masslearn_spam.16304.Qay16749.mbox.68831423
EMPTY_MESSAGE,MISSING_DATE,MISSING_HEADERS,MISSING_SUBJECT,NO_RECEIVED,
NO_RELAYS,TO_CC_NONE,TVD_SPACE_RATIO,__ENV_AND_HDR_FROM_MATCH,__UNUSABLE_MSGID
time=1149528224,sc antime=0,format=m,reuse=no
I extracted a sample from
/home/bbmass/rawcor/zmi/spam/masslearn_spam.15992.XzD16459.mbox , removed
all the identifying info, and uploaded it to
http://taint.org/xfer/2006/report_safe_issue.txt .
The interesting thing about it is that it contains a "report_safe"
encapsulated message, encapsulated at zmi's site. So far so good. This
however contains a "From" line:
------------=_4484A1B3.C6CD6B4D
Content-Type: message/rfc822; x-spam-type=original
Content-Description: original message before SpamAssassin
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
From rqugps@donnlordphoto.com Tue Jul 11 00:38:16 2006
Return-Path: <rq...@donnlordphoto.com>
X-Original-To: REMOVED@REMOVED.com
(leading spaces do not appear in the original btw)
This appears to be the problem. The mbox parser in mass-check sees this
"From" line, and incorrectly concludes that it's the start of the next
message, ending the current one. It then de-encapsulates the report_safe
encapsulation, extracting the (empty) "message" and mass-checking that.
- Issue 1. It should not attempt to mass-check an empty message; if the
MIME end-boundary for the report_safe encapsulation is not found, it
should ignore the message as incomplete. (In fact, it should attempt
to keep reading until it does find the MIME end boundary. But that may
get complex, since breaking up mboxes into messages takes place in
ArchiveIterator, which doesn't know about report_safe.)
As a kludge, ignoring the empty messages should work OK for now, since
ArchiveIterator will have found 2 messages: 1. the empty encapsulation,
and 2. the encapsulated message, with an additional single line MIME
end-boundary (which is ignored).
- Issue 2. should report_safe include the "From" line? I think not, since
the message/rfc822 MIME format (which is what the report_safe
encapsulation uses) is not a UNIX mbox. It should either remove it
entirely in the encapsulated message, or escape it as ">From",
"X-Original-From-Line" or similar. (I think it should remove it,
personally.)
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
------- Additional Comments From jm@jmason.org 2006-07-15 20:33 -------
pasting some mails from Michael --
OK, I've put the original unmunged message here:
http://zmi.at/x/bug4988-message.txt
I have all my messages in an IMAP folder. Some time ago I changed the
markup of SPAM for some account, causing messages like this one to be
safe_report-ed (included as original message in an attachment).
Then I run my masscheck, collecting via fetchmail into an mbox:
fetchmail -k -a -s -n -p IMAP --folder 'masscheck_SPAM' --auth
'password' -m "formail -d -I \"From \" -a \"From \" -s >>$spamfile"
power2u.zmi.at || { echo "fetchmail SPAM error, exit"; exit; }
This command creates this file:
http://zmi.at/x/bug4988-message-mbox.txt
A quick diff:
# diff /www/zmi.at/x/bug4988-messa*
1,4d0
< From rqugps@donnlordphoto.com Sat Jul 15 00:39:27 2006
< Received: from power2u.zmi.at [195.202.170.130]
< by localhost with IMAP (fetchmail-6.2.5)
< for sa_train@localhost (single-drop); Sat, 15 Jul 2006 00:39:27
+0200 (CEST)
55a52
> X-UID: 14695
111d107
< From rqugps@donnlordphoto.com Sat Jul 15 00:39:27 2006
I can imagine that's a fetchmail bug, as it seems to only hit messages
that are encapsulated. Or are any non safe_report messages causing that
problem?
BTW, yes, I always upload my SPAM to
rsync.spamassassin.org/mailcorpus_zmi/spam/
so that you can look at it. I do that for the dev's, so in cases like
this one you can directly look at the problematic parts.
For any debugging you need, you can use/post my SPAM samples on the dev
list in unmunged format.
[jm: thanks!]
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
------- Additional Comments From jm@jmason.org 2006-07-13 21:00 -------
'My guess is the original message was passed in incorrectly such
that it had 2 mbox separators, and then when the message got encapsulated, we
preserved that information.'
hmm, that sounds quite unlikely! ;) Michael (Monnerie) -- are you reading
this? can you comment, since the message came from your collection? (thanks ;)
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
------- Additional Comments From felicity@apache.org 2006-07-13 21:08 -------
(In reply to comment #4)
> hmm, that sounds quite unlikely! ;) Michael (Monnerie) -- are you reading
> this? can you comment, since the message came from your collection? (thanks ;)
I'd also accept "show me a non-corrupt input message that generates a corrupted
encapsulated ouptut". But for now I'm sticking with my GIGO view of the situation.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
------- Additional Comments From jm@jmason.org 2006-07-15 19:34 -------
>> Actually Theo -- even if there are two "From" lines in the input, *none* should
>> be making it into the output message/rfc822 part. It may be GIGO, but if
>
>That's not true at all. There's nothing special about a mbox separator as far
>as SA is concerned, and having one in the encapsulated mime part is perfectly
>fine depending on what is processing the message outside of SA (in file format
>for instance, the mbox separator has no meaning).
what I mean is that RFC-822 does not permit a "From" line. that line is part of
the UNIX mbox format, not RFC-822.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
Re: [Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by Michael Monnerie <mi...@it-management.at>.
On Samstag, 15. Juli 2006 22:43
bugzilla-daemon@bugzilla.spamassassin.org wrote:
> Here is the culprit -- formail's "-d" switch:
> remove that switch, and it works fine. ;)
> Michael, would it be possible to alter the fetchmail command line
> accordingly?
I just did it. Sorry for the inconvenience. It's been there since I do
mass checks, and was just triggered because now I have some spam within
SA-markup ("safe_report") format. *pfff*
mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660/4156531 .network.your.ideas.
// PGP Key: "curl -s http://zmi.at/zmi3.asc | gpg --import"
// Fingerprint: 44A3 C1EC B71E C71A B4C2 9AA6 C818 847C 55CB A4EE
// Keyserver: www.keyserver.net Key-ID: 0x55CBA4EE
[Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
jm@jmason.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |INVALID
------- Additional Comments From jm@jmason.org 2006-07-15 20:43 -------
ok, good news -- it wasn't produced by SpamAssassin, it comes from
the formail command. See:
: jm 270...; formail -d -I "From " -a "From " -s < ~/DL/bug4988-message.txt > o
: jm 271...; diff ~/DL/bug4988-message-mbox.txt o
1,4c1
< From rqugps@donnlordphoto.com Sat Jul 15 00:39:27 2006
< Received: from power2u.zmi.at [195.202.170.130]
< by localhost with IMAP (fetchmail-6.2.5)
< for sa_train@localhost (single-drop); Sat, 15 Jul 2006 00:39:27 +0200 (CEST)
---
> From rqugps@donnlordphoto.com Sat Jul 15 21:39:05 2006
55a53
> X-UID: 14695
111c109
< From rqugps@donnlordphoto.com Sat Jul 15 00:39:27 2006
---
> From rqugps@donnlordphoto.com Sat Jul 15 21:39:05 2006
(in other words, it's missing the harmless fetchmail Received hdr,
and the datestamps in the From lines are different -- but the
second, problematic From line was added.)
Here is the culprit -- formail's "-d" switch:
-d Tell formail that the messages it is supposed to split need not be in
strict mailbox format (i.e., allows you to split digests/articles or
non-standard mailbox formats). This disables recognition of the Con-
tent-Length: field.
remove that switch, and it works fine. ;)
Michael, would it be possible to alter the fetchmail command line accordingly?
Regarding the other issue: what mass-check should do -- I think we can
simply close this bug, since this is an "artificial" issue created entirely
by that formail command; it wasn't caused by the "real" spam itself in any
way. Marking INVALID.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
------- Additional Comments From lwilton@earthlink.net 2006-07-13 17:28 -------
Hum. The line at the start of the encapsulation says
Content-Description: original message before SpamAssassin
If that is the original message, and you remove the From line, then it isn't
the original message any more!
I use this format for spam, and I can certainly say that I look at that From
line there, rather than scrolling back a page to the encapsulating header to
look for that one line. I'm not even positive that they may not be different
between the encapsulating header and the real header. For instance, you can
stick tags in the From header. Sometimes it is interesting to know what the
REAL header looked like before SA mucked with it.
Isn't the whole problem here that the mbox mail picker has a bug and doesn't
correctly recognize encapsulated mime messages? This can't be an inherent mbox
problem - mail clients can pick out mimed messages just fine.
I'd say the right solution here is to fix the mbox mail reader to correctly
recognize the end of a mimed message, even if it includes content that happens
to look like a mail message.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
Re: [Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by Michael Monnerie <mi...@it-management.at>.
On Freitag, 14. Juli 2006 13:47
bugzilla-daemon@bugzilla.spamassassin.org wrote:
> oh, if Michael gives me permission, btw, I'll post the raw unmunged
> sample, too.
BTW, yes, I always upload my SPAM to
rsync.spamassassin.org/mailcorpus_zmi/spam/
so that you can look at it. I do that for the dev's, so in cases like
this one you can directly look at the problematic parts.
For any debugging you need, you can use/post my SPAM samples on the dev
list in unmunged format.
mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660/4156531 .network.your.ideas.
// PGP Key: "curl -s http://zmi.at/zmi3.asc | gpg --import"
// Fingerprint: 44A3 C1EC B71E C71A B4C2 9AA6 C818 847C 55CB A4EE
// Keyserver: www.keyserver.net Key-ID: 0x55CBA4EE
[Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
------- Additional Comments From jm@jmason.org 2006-07-14 11:47 -------
oh, if Michael gives me permission, btw, I'll post the raw unmunged sample, too.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
------- Additional Comments From jm@jmason.org 2006-07-14 11:49 -------
> > - Issue 1. It should not attempt to mass-check an empty message; if the
> > MIME end-boundary for the report_safe encapsulation is not found, it should
> > ignore the message as incomplete. (In fact, it should attempt to keep
> > reading until it does find the MIME end boundary. But that may get
> > complex, since breaking up mboxes into messages takes place in
> > ArchiveIterator, which doesn't know about report_safe.)
>
> -1 -- this isn't a bug in mbox parsing, and we definitely don't want to try
> adding logic for it to figure out if something looks "valid" or not. it's
> job is to parse an mbox and pass the message on. if the input is corrupt,
> then the problem is the input.
Hold on; there are two parts involved. One is the mbox parsing in
ArchiveIterator, and I agree, changing that would be messy and inappropriate;
however, the other part is the calling code, in "mass-check" itself, and it'd
be fine to add a hack-fix for this there, in my opinion. That's what I'd
suggest doing.
Loren:
> If that is the original message, and you remove the From line, then it isn't
> the original message any more!
no, the From line should not be there. Having it there is breaking the MIME
message/rfc822 format rules. It has to go.
Actually Theo -- even if there are two "From" lines in the input, *none* should
be making it into the output message/rfc822 part. It may be GIGO, but if
SpamAssassin accepts it, parses it, and marks it as spam correctly, why not fix
it on the way out so that we are producing valid message/rfc822 output?
"Be liberal in what you accept, strict in what you produce" etc.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
Re: [Bug 4988] New: mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by Michael Monnerie <mi...@it-management.at>.
On Donnerstag, 13. Juli 2006 13:02
bugzilla-daemon@bugzilla.spamassassin.org wrote:
> http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
OK, I've put the original unmunged message here:
http://zmi.at/x/bug4988-message.txt
I have all my messages in an IMAP folder. Some time ago I changed the
markup of SPAM for some account, causing messages like this one to be
safe_report-ed (included as original message in an attachment).
Then I run my masscheck, collecting via fetchmail into an mbox:
fetchmail -k -a -s -n -p IMAP --folder 'masscheck_SPAM' --auth
'password' -m "formail -d -I \"From \" -a \"From \" -s >>$spamfile"
power2u.zmi.at || { echo "fetchmail SPAM error, exit"; exit; }
This command creates this file:
http://zmi.at/x/bug4988-message-mbox.txt
A quick diff:
# diff /www/zmi.at/x/bug4988-messa*
1,4d0
< From rqugps@donnlordphoto.com Sat Jul 15 00:39:27 2006
< Received: from power2u.zmi.at [195.202.170.130]
< by localhost with IMAP (fetchmail-6.2.5)
< for sa_train@localhost (single-drop); Sat, 15 Jul 2006 00:39:27
+0200 (CEST)
55a52
> X-UID: 14695
111d107
< From rqugps@donnlordphoto.com Sat Jul 15 00:39:27 2006
I can imagine that's a fetchmail bug, as it seems to only hit messages
that are encapsulated. Or are any non safe_report messages causing that
problem?
mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660/4156531 .network.your.ideas.
// PGP Key: "curl -s http://zmi.at/zmi3.asc | gpg --import"
// Fingerprint: 44A3 C1EC B71E C71A B4C2 9AA6 C818 847C 55CB A4EE
// Keyserver: www.keyserver.net Key-ID: 0x55CBA4EE
[Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
------- Additional Comments From felicity@apache.org 2006-07-13 18:43 -------
(In reply to comment #0)
> http://taint.org/xfer/2006/report_safe_issue.txt .
interesting.
> The interesting thing about it is that it contains a "report_safe"
> encapsulated message, encapsulated at zmi's site. So far so good. This
> however contains a "From" line:
That would be a bug, if it were reproducable... ;) 3.1.4 and 3.2.0 work fine
for me when I run that message through (or any of my test messages).
> ------------=_4484A1B3.C6CD6B4D
> Content-Type: message/rfc822; x-spam-type=original
> Content-Description: original message before SpamAssassin
> Content-Disposition: inline
> Content-Transfer-Encoding: 8bit
>
> From rqugps@donnlordphoto.com Tue Jul 11 00:38:16 2006
> Return-Path: <rq...@donnlordphoto.com>
> X-Original-To: REMOVED@REMOVED.com
as far as our code goes, the mbox separator is removed in the message parse
phase. Then when the generated messages happens, the mbox separator is used at
the top, and only the message contents (no separator) goes in the mime part.
> This appears to be the problem. The mbox parser in mass-check sees this
> "From" line, and incorrectly concludes that it's the start of the next
> message, ending the current one. It then de-encapsulates the report_safe
> encapsulation, extracting the (empty) "message" and mass-checking that.
Well, not exactly. The parser is doing exactly what it's supposed to be doing.
The problem is that there's an extra mbox separator in the message when there
shouldn't be which corrupts the mbox.
> - Issue 1. It should not attempt to mass-check an empty message; if the
> MIME end-boundary for the report_safe encapsulation is not found, it
> should ignore the message as incomplete. (In fact, it should attempt
> to keep reading until it does find the MIME end boundary. But that may
> get complex, since breaking up mboxes into messages takes place in
> ArchiveIterator, which doesn't know about report_safe.)
-1 -- this isn't a bug in mbox parsing, and we definitely don't want to try
adding logic for it to figure out if something looks "valid" or not. it's job
is to parse an mbox and pass the message on. if the input is corrupt, then the
problem is the input.
> - Issue 2. should report_safe include the "From" line? I think not, since
absolutely not, and it shouldn't based on our current code. It feels like the
issue here is that an improperly formatted message was passed into SA.
> the message/rfc822 MIME format (which is what the report_safe
> encapsulation uses) is not a UNIX mbox. It should either remove it
> entirely in the encapsulated message, or escape it as ">From",
which it does.
> "X-Original-From-Line" or similar. (I think it should remove it,
> personally.)
we don't want to get into that game -- the rewrite section doesn't know what
format the message came in as anyway since it's irrelevant.
So in short, there is no bug in the mbox handling -- it's working exactly as
it's supposed to. The problem is the mbox is corrupt. So that begs the
question: why? My guess is the original message was passed in incorrectly such
that it had 2 mbox separators, and then when the message got encapsulated, we
preserved that information.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
jm@jmason.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Target Milestone|Undefined |3.1.4
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
------- Additional Comments From jm@jmason.org 2006-07-20 18:14 -------
of course RFC-822 allows lines (in the body) to start with "From ". I'm talking
about the UNIX mbox-format "From " line, preceding the RFC-822 headers in an mbox.
to be more specific, RFC-822 does not permit the header block to start with a
UNIX-mbox-format-style "From " line.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
------- Additional Comments From felicity@apache.org 2006-07-14 15:46 -------
(In reply to comment #7)
> Actually Theo -- even if there are two "From" lines in the input, *none* should
> be making it into the output message/rfc822 part. It may be GIGO, but if
That's not true at all. There's nothing special about a mbox separator as far
as SA is concerned, and having one in the encapsulated mime part is perfectly
fine depending on what is processing the message outside of SA (in file format
for instance, the mbox separator has no meaning).
So far, the only way I've been able to replicate the behavior as seen in the
sample message is to pass a message w/ 2 mbox separators into SA via --file (the
default method). The output is exactly as expected in that case. If the same
input is passed in via --mbox, SA sees 2 messages, as expected. In this case,
if the thing calling SA is expecting mbox format but passes in file format,
that's a bug in the caller -- hence GIGO, we're doing exactly what we're
supposed to do.
> SpamAssassin accepts it, parses it, and marks it as spam correctly, why not fix
> it on the way out so that we are producing valid message/rfc822 output?
> "Be liberal in what you accept, strict in what you produce" etc.
You can't fix what isn't broken.
I'm sorry if I'm getting a bit defensive here. I just really don't see a
problem with the information provided.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
------- Additional Comments From jgmyers@proofpoint.com 2006-07-20 16:38 -------
(In reply to comment #9)
> what I mean is that RFC-822 does not permit a "From" line.
Nonsense. Nothing in either RFC 822, RFC 2822, or MIME prohibits lines starting
with "From ". Please provide a citation to relevant standards to back up your
claim.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4988] mass-check vs report_safe encapsulation: From lines cause empty messages
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4988
------- Additional Comments From jm@jmason.org 2006-07-13 11:04 -------
Created an attachment (id=3580)
--> (http://issues.apache.org/SpamAssassin/attachment.cgi?id=3580&action=view)
report_safe_issues.txt
here's that message sample, as an attachment
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.