You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Keith Whyte <ke...@media-solutions.ie> on 2005/01/01 23:17:58 UTC
feed MimeDefanged mail to sa-learn?
Maybe another, more simple way to ask the question I asked yesterday:
When my MimeDefang/Spamassassin system tags a message as spam, it adds
the Spamassassin report as an attachment to the original message.
Can somebody confirm for me that it would be a BAD idea to feed these MD
modified messages to sa-learn?
Thanks.
Keith.
RE: feed MimeDefanged mail to sa-learn?
Posted by Gary Funck <ga...@intrepid.com>.
> From: Keith Whyte
> Sent: Tuesday, January 11, 2005 11:15 PM
[...]
>
> anyway, on the messages where that regex does trigger, the resulting
> mail is unreadable in Pine, which complains about not being able to open
> text attachment, and thunderbird on windows via imap, which in some
> cases just showed me an empty mailbox.
>
> maybe i'll get a chance to look at it over the next few days.
Keith, if you can send me a copy of the mail messages (with full headers)
that should've been converted and weren't converted correctly, I'll take a
look.
thanks, - Gary
Re: feed MimeDefanged mail to sa-learn?
Posted by Keith Whyte <ke...@media-solutions.ie>.
Gary Funck wrote:
>Attached, is a Perl script, mdf2sa.pl, which converts spam messages
>that have been marked up by MIMEDefang, into a form that is similar
>to the message format used by SA. Also attached is a simple
>
>
>
Hi Gary, just tried out that script, and sorry to say it didn't work at all.
a lot of the time, it fails here:
goto copy_original if !m{$report_hdr_pat};
even though it's not obvious to me (not being a total perl regex guru)
why, as the messages all contain:
Content-Type: text/plain; name="SpamAssassinReport.txt"
Content-Disposition: inline; filename="SpamAssassinReport.txt"
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0
X-Mailer: MIME-tools 5.413 (Entity 5.413)
anyway, on the messages where that regex does trigger, the resulting
mail is unreadable in Pine, which complains about not being able to open
text attachment, and thunderbird on windows via imap, which in some
cases just showed me an empty mailbox.
maybe i'll get a chance to look at it over the next few days.
thanks,
Keith.
RE: feed MimeDefanged mail to sa-learn?
Posted by Gary Funck <ga...@intrepid.com>.
Attached, is a Perl script, mdf2sa.pl, which converts spam messages
that have been marked up by MIMEDefang, into a form that is similar
to the message format used by SA. Also attached is a simple
procmail script, mdf2sa.rc, which drives the perl script, and
can be invoked via formail as follows:
formail -s procmail -m mdf2sa.rc < spam.mbox > new_spam.mbox
(the procmail script writes to a logfile named x.log).
The conversion could be made upon delivery to a spam mailbox
as follows:
:0 fbhw
* ^X-Spam-Score:
* ^X-Scanned-By: MIMEDefang
* B ?? ^Content-Type: text/plain; name=\"SpamAssassinReport\.txt\"
| perl $HOME/mdf2sa.pl
:0:
* ^X-Spam-Score:.*\(\*\*\*\*\*
spam-mail
disclaimer: the Perl script is only lightly tested. I'd appreciate hearing
any suggestions to improve the Perl coding style, or bug fixes/improvements.
PS: the code that removes a couple of Received headers when reconstructing
the message header is implmentation and needs to be tuned for your
installation.
Re: feed MimeDefanged mail to sa-learn?
Posted by Alex S Moore <as...@edge.net>.
On Sun, 02 Jan 2005 12:27:52 -0600
Keith Whyte <ke...@media-solutions.ie> wrote:
> what occured to me later is that when MD sends a message to
> spamassassin to analyse, it hasn't modified it. If that message scores
> more that the auto_learn_threshold, then sa will auto learn the
> original, right? it's only after SA returns the spam score to MD that
> md goes about it's insertion of header, attaching the report etc.
> maybe that's the best way, just let it autolearn. I get quite a lot of
> spam scoring 20+
> then i just need to feed it ham.
> there's another question - if i feed a system-wide bayes ham from only
> one user, will that cause wierdness?
>
Good luck on auto-learn. In the past, I had that working in MD. But a
few months ago, it stopped working and I have not determined how to fix
it.
Regarding the report, we may be speaking of two different things, so be
sure to test. Yes, MD adds headers and can attach a report. Now that I
remember the details, I do this very differently from the stock MD
filter.
You should train and not rely on autolearn to learn everything that
bayes needs to learn. If the source for ham is basically the same for
all users, then one user may be enough. You will know when someone
complains. If they complain, tell them to copy ham into a folder for
you to use for training bayes.
Alex
Re: feed MimeDefanged mail to sa-learn?
Posted by Loren Wilton <lw...@earthlink.net>.
> what occured to me later is that when MD sends a message to spamassassin
> to analyse, it hasn't modified it. If that message scores more that the
> auto_learn_threshold, then sa will auto learn the original, right?
This might cause problems, depending on how "original" things are. I think
MD needs to put a valid Received header in there, or you may have probems
with the trust zone and end up triggering ALL_TRUSTED for a lot of things
that shouldn't.
> there's another question - if i feed a system-wide bayes ham from only
> one user, will that cause wierdness?
Depends. If the one user has ham that is pretty much like everyone else's
then you will probably be ok. But if other people get other kinds of things
then you could end up getting FPs and possibly FNs for other people.
Bayes is only as good as the data it is fed. Ideally it should have a
fairly representative sample of ham for all of the users to do a good job,
when using site-wide bayes.
If you have users using Outlook or OE you coudl set up some shared folders
that were IMAP folders, and let users drop spam and ham into them, then scan
it yourself (for people dropping the wrong stuff in the wrong folders) and
feed it to sa-learn.
Loren
Re: feed MimeDefanged mail to sa-learn?
Posted by Keith Whyte <ke...@media-solutions.ie>.
Alex S Moore wrote:
>On Sat, 01 Jan 2005 16:17:58 -0600
>Keith Whyte <ke...@media-solutions.ie> wrote:
>
>
>
>>Maybe another, more simple way to ask the question I asked yesterday:
>>
>>When my MimeDefang/Spamassassin system tags a message as spam, it adds
>>
>>the Spamassassin report as an attachment to the original message.
>>
>>Can somebody confirm for me that it would be a BAD idea to feed these
>>MD modified messages to sa-learn?
>>
>>
>
>That is the only way that bayes is trained at my location for probably
>close to two years. My MIMEDefang filter adds the SA report to the
>e-mail. So spam mail has an additional part called something like
>SpamAssassinReport.txt. The bayes database contains the past two to
>three months of tokens. It works very, very well.
>
>The list for MD could be more help if you are having problems with this.
>
>Alex
>
>
what occured to me later is that when MD sends a message to spamassassin
to analyse, it hasn't modified it. If that message scores more that the
auto_learn_threshold, then sa will auto learn the original, right?
it's only after SA returns the spam score to MD that md goes about it's
insertion of header, attaching the report etc.
maybe that's the best way, just let it autolearn. I get quite a lot of
spam scoring 20+
then i just need to feed it ham.
there's another question - if i feed a system-wide bayes ham from only
one user, will that cause wierdness?
thanks all!
Keith.
Re: feed MimeDefanged mail to sa-learn?
Posted by Alex S Moore <as...@edge.net>.
On Sat, 01 Jan 2005 16:17:58 -0600
Keith Whyte <ke...@media-solutions.ie> wrote:
> Maybe another, more simple way to ask the question I asked yesterday:
>
> When my MimeDefang/Spamassassin system tags a message as spam, it adds
>
> the Spamassassin report as an attachment to the original message.
>
> Can somebody confirm for me that it would be a BAD idea to feed these
> MD modified messages to sa-learn?
That is the only way that bayes is trained at my location for probably
close to two years. My MIMEDefang filter adds the SA report to the
e-mail. So spam mail has an additional part called something like
SpamAssassinReport.txt. The bayes database contains the past two to
three months of tokens. It works very, very well.
The list for MD could be more help if you are having problems with this.
Alex
Re: feed MimeDefanged mail to sa-learn?
Posted by Keith Whyte <ke...@media-solutions.ie>.
Loren Wilton wrote:
>>When my MimeDefang/Spamassassin system tags a message as spam, it adds
>>the Spamassassin report as an attachment to the original message.
>>
>>Can somebody confirm for me that it would be a BAD idea to feed these MD
>>modified messages to sa-learn?
>>
>>
>
>No, it wouldn't be a bad idea. SA knows how to remove its own markup.
>
>Where you get into trouble is when you forward a spam to a mailbox to feed
>to SA. SA won't know to remove the last forwarding step in the spam.
>
>
Ok I understand that SA can remove it's own markup and i also note the
configuration option bayes_header_ignore,
but the markup from Mimedefang is not SA's own markup.
maybe some experimentation is in order.
thanks.
Re: feed MimeDefanged mail to sa-learn?
Posted by Jeff Grossman <je...@stikman.com>.
Loren Wilton <lw...@earthlink.net> wrote:
>> When my MimeDefang/Spamassassin system tags a message as spam, it adds
>> the Spamassassin report as an attachment to the original message.
>>
>> Can somebody confirm for me that it would be a BAD idea to feed these MD
>> modified messages to sa-learn?
>
> No, it wouldn't be a bad idea. SA knows how to remove its own markup.
>
> Where you get into trouble is when you forward a spam to a mailbox to feed
> to SA. SA won't know to remove the last forwarding step in the spam.
How about redirecting an e-mail to a mailbox to feed into SA. Would
that work, or is it the same as being forwarded?
Jeff
Re: feed MimeDefanged mail to sa-learn?
Posted by Loren Wilton <lw...@earthlink.net>.
> When my MimeDefang/Spamassassin system tags a message as spam, it adds
> the Spamassassin report as an attachment to the original message.
>
> Can somebody confirm for me that it would be a BAD idea to feed these MD
> modified messages to sa-learn?
No, it wouldn't be a bad idea. SA knows how to remove its own markup.
Where you get into trouble is when you forward a spam to a mailbox to feed
to SA. SA won't know to remove the last forwarding step in the spam.
RE: feed MimeDefanged mail to sa-learn?
Posted by Gary Funck <ga...@intrepid.com>.
> -----Original Message-----
> From: Keith Whyte [mailto:keith@media-solutions.ie]
> Sent: Saturday, January 01, 2005 2:18 PM
> To: users@spamassassin.apache.org
> Subject: feed MimeDefanged mail to sa-learn?
>
>
> Maybe another, more simple way to ask the question I asked yesterday:
>
> When my MimeDefang/Spamassassin system tags a message as spam, it adds
> the Spamassassin report as an attachment to the original message.
>
> Can somebody confirm for me that it would be a BAD idea to feed these MD
> modified messages to sa-learn?
Since the reports will mainly only appear in spam, then the tokens unique to
the reports will appear to be spammy. Since typically those spammy tokens
won't appear in regualar e-mails (except for spam-related lists), then
probably all that will happen is that the spammy tokens will tend to push
out other tokens that appeared in the base-level messages.
I asked a related question on the Mimedefang list: How to change the
spam attachment reporting format so that it agrees with Spamassassin's
format? No reply to date.
http://lists.roaringpenguin.com/pipermail/mimedefang/2004-December/025691.ht
ml