You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Joe Acquisto-j4 <jo...@j4computers.com> on 2019/11/28 16:56:19 UTC

Yet another simple question - how to reprocess an email

Well, here goes, asbestos pants on.  I did, honest, do some searching before asking this.

I want to be able to reprocess a particular email, marked as SPAM, after making some SA tweaks.   

Basically I have saved the email, which was received as an attachment, as a text file.   Thinking to simply drop this file into a queue somewhere in the receive process and just let her rip.  However I was given pause by the message numbering that sendmail seems to use. 

The process is mail is delivered by a "fetching" server, to a sendmail box, which also hosts SA, after processing is delivered to another on prem box.   

Thanks for any assistance.

-- 
+++++++++++++++++++++++
         joea@@j4computers.com
      https://www.j4computers.com
               845-687-3734
+++++++++++++++++++++++

Re: Yet another simple question - how to reprocess an email

Posted by Martin Gregorie <ma...@gregorie.org>.
On Thu, 2019-11-28 at 22:34 +0000, RW wrote:
> A lot of us rely on SA stipping X-Spam-* headers, so header-based
> filtering into a spam folder works correctly. This includes numerous
> mail hosting and freemail providers. 
> 
Interesting: I've just rechecked this using your last message in this
conversation and it has worked as you expected it to: 

- saved as an mbox format file by Evolution
- fed to spamd using spamc and grabbed on its return by less for
  inspection
- the original three X-Spam headers were replaced by three new ones
  recognised by a different hostname (expected) and slightly different 
  SA score.

I don't know when this started to work: I've been using SA since around
2004 and back then I was certainly seeing both the old and new sets of
X-Spam headers if I ran a message through my SA test systrem, otherwise
I'd never have written the 'cleaner' script.

As it happens, 'cleaner' does nothing to a message except copy it if
there are no X-Spam headers in the file, so no harm done apart from a
few wasted cycles.

Thanks for prodding me into checking it just now.

Martin




Re: Yet another simple question - how to reprocess an email

Posted by RW <rw...@googlemail.com>.
On Thu, 28 Nov 2019 21:32:29 +0000
Martin Gregorie wrote:

> On Thu, 2019-11-28 at 21:00 +0000, RW wrote:
> 
> > There shouldn't be any need for this as SA strips such headers
> > itself. 
> Yes, I've seem that said several times, BUT every time I capture some
> spam from Evolution by using "File:Save as mbox" to capture it as a
> .txt file and then feed it into SA as I showed:
> 
> $ spamc --max-size=2000000 <spam.txt | less
> 
> what is displayed always contains both the X-Spam headers added as the
> message hit my system and a second set of X-Spam headers added by that
> test shot. 


That's strange:

$ printf 'X-Spam-Status: 12345\n' | spamc -u test | grep X-Spam-Status 
X-Spam-Status: Yes, score=11.4 required=5.0 autolearn=disabled

$ printf 'X-Spam-Foo: 12345\n' | spamc -u test | grep X-Spam-Foo
$

A lot of us rely on SA stipping X-Spam-* headers, so header-based
filtering into a spam folder works correctly. This includes numerous
mail hosting and freemail providers. 

My guess is that when you saw this you had corrupted the headers in some
way.

 


Re: Yet another simple question - how to reprocess an email

Posted by Martin Gregorie <ma...@gregorie.org>.
On Thu, 2019-11-28 at 21:00 +0000, RW wrote:

> There shouldn't be any need for this as SA strips such headers itself.
>
Yes, I've seem that said several times, BUT every time I capture some
spam from Evolution by using "File:Save as mbox" to capture it as a .txt
file and then feed it into SA as I showed:

$ spamc --max-size=2000000 <spam.txt | less

what is displayed always contains both the X-Spam headers added as the
message hit my system and a second set of X-Spam headers added by that
test shot. 

This is why I went to the trouble of writing the 'cleaner' script -
believe it or not, this isn't another case of NIH.

The inbound mail flow is:

incoming mail -> domainhost -> ISP --getmail--> SA --sendmail--> Postfix

because this allows me to send and receive mail without having any ports
open in my firewall.
 
Martin



Re: Yet another simple question - how to reprocess an email

Posted by RW <rw...@googlemail.com>.
On Thu, 28 Nov 2019 17:42:34 +0000
Martin Gregorie wrote:


> I do something similar with with collection of test messages, mostly
> received spam, that I use to test my local SA rule set.
> 
> Essentially, all I do is:
> 
> 1) remove all headers starting with 'X-Spam', otherwise the X-Spam
>    headers injected when the piece if spam was received will still be
>    there after I've run the test. This is confusing rather than
> harmful, but the cleaner is just a script using awk. See below.

There shouldn't be any need for this as SA strips such headers itself.

Re: Yet another simple question - how to reprocess an email

Posted by Martin Gregorie <ma...@gregorie.org>.
On Thu, 2019-11-28 at 18:38 -0500, Joe Acquisto-j4 wrote:

> > > Is there any tangent down this path were I can get the dropped
> > > "test" message to actually flow through, in "normal" fashion?    
> 
> > From logs I can see that spamd does seem to give the message a
> > taste,  as I can follow /
> var/log/mail and see activity at the precise time I feed it, but the
> message  does not seem to be delivered.
> 
> I've tried some other off box methods as well, including using CURL
> which is purported to have smtp ability, yet I get syntax
> errors or invalid option as the touted features do not exist in
> versions in use here.
> 
What are you doing with the message after it comes back from spamc? 

spamc should be in some sort of pipeline that grabs the message after it
has had X-Spam headers inserted and pass it to whatever will queue it
for the intended recipient's MUA.

My set up is a little odd in that my pipeline used getmail to retrieve
mail from my ISP's smarthost and precedes my MTA with a pipeline like
this, where 'spamkiller' is a simple C program that looks at the spam
headers to see whether its spam or ham. Spam is sent to a holding area
and ham is passed to Postfix for delivery. Here's a diagram:

getmail --> spamc --> spamkiller --ham--> sendmail --> postfix
                               |
                               +--spam--> spam quarantine store

Most people simply splice spamc into Postfix's internal pipeline,
defined in master.cf, which connects its mail reception process to its
delivery process. 

Martin



Re: Yet another simple question - how to reprocess an email

Posted by Joe Acquisto-j4 <jo...@j4computers.com>.
>>>
>>>>
>> On Thu, 2019-11-28 at 11:56 -0500, Joe Acquisto-j4 wrote:
I want to be able to reprocess a particular email, marked as SPAM,
>>> after making some SA tweaks.   
>>> 
>> I do something similar with with collection of test messages, mostly
>> received spam, that I use to test my local SA rule set.
>> 
>> Essentially, all I do is:
>> 
>> 1) remove all headers starting with 'X-Spam', otherwise the X-Spam
>>    headers injected when the piece if spam was received will still be
>>    there after I've run the test. This is confusing rather than harmful,
>>    but the cleaner is just a script using awk. See below.
>> 
>> 2) pass the message through spamd by running:
>> 
>> 	spamc --max-size=2000000 <testmag.txt | less
>> 
>>    and examine the result
>> 
> 
> Thanks. Helpful.   I did not test the cleaner as I had already, in effect, 
> "sanitized" the message.
> 
> What I had hoped to do was allow this to flow though and be delivered to the 
> end point, as if under normal circumstances.
> 
> joe a. 
> 

Gentlefolks . . .

Is there any tangent down this path were I can get the dropped "test" message to actually flow through, in "normal" fashion?    

From logs I can see that spamd does seem to give the message a taste,  as I can follow /
var/log/mail and see activity at the precise time I feed it, but the message  does not seem to be delivered.

I've tried some other off box methods as well, including using CURL which is purported to have smtp ability, yet I get syntax
errors or invalid option as the touted features do not exist in versions in use here.





-- 
+++++++++++++++++++++++
         joea@@j4computers.com
      https://www.j4computers.com
               845-687-3734
+++++++++++++++++++++++

Re: Yet another simple question - how to reprocess an email

Posted by Joe Acquisto-j4 <jo...@j4computers.com>.
>>>
> On Thu, 2019-11-28 at 11:56 -0500, Joe Acquisto-j4 wrote:
>> I want to be able to reprocess a particular email, marked as SPAM,
>> after making some SA tweaks.   
>> 
> I do something similar with with collection of test messages, mostly
> received spam, that I use to test my local SA rule set.
> 
> Essentially, all I do is:
> 
> 1) remove all headers starting with 'X-Spam', otherwise the X-Spam
>    headers injected when the piece if spam was received will still be
>    there after I've run the test. This is confusing rather than harmful,
>    but the cleaner is just a script using awk. See below.
> 
> 2) pass the message through spamd by running:
> 
> 	spamc --max-size=2000000 <testmag.txt | less
> 
>    and examine the result
> 

Thanks. Helpful.   I did not test the cleaner as I had already, in effect, "sanitized" the message.

What I had hoped to do was allow this to flow though and be delivered to the end point, as if under normal circumstances.

joe a. 

-- 
+++++++++++++++++++++++
         joea@@j4computers.com
      https://www.j4computers.com
               845-687-3734
+++++++++++++++++++++++

Re: Yet another simple question - how to reprocess an email

Posted by Martin Gregorie <ma...@gregorie.org>.
On Thu, 2019-11-28 at 11:56 -0500, Joe Acquisto-j4 wrote:
> I want to be able to reprocess a particular email, marked as SPAM,
> after making some SA tweaks.   
> 
I do something similar with with collection of test messages, mostly
received spam, that I use to test my local SA rule set.

Essentially, all I do is:

1) remove all headers starting with 'X-Spam', otherwise the X-Spam
   headers injected when the piece if spam was received will still be
   there after I've run the test. This is confusing rather than harmful,
   but the cleaner is just a script using awk. See below.

2) pass the message through spamd by running:

	spamc --max-size=2000000 <testmag.txt | less

   and examine the result

Here's the cleaner script. 
My spam corpus lives in the 'data' directory in my spam test user:
============================== cleaner ==============================
#!/bin/bash
#
#help
#Syntax:   cleaner [file...]
#Function: Remove SA headers from example message(s)
#          If a list of files is present they are processed.
#          If no list is supplied, all files in data/* are processed.
#Options:  none
#end
#
if [ "$1" == '-?' ]
then
        script_help cleaner
        exit
fi

function clean()
{
        echo "Cleaning $1" 
        gawk '
                BEGIN           { act = "copy";
                                  body = "no";
                                }
                /^[A-Za-z]/     { act = "copy"    }
                /^X-Spam/       { act = "skip"    }
                /^$/            { body = "yes"; }
                                {  
                                  if (act == "copy" || body == "yes")
                                        { print }
                                }
        ' <$1 >temp.txt
        mv temp.txt $1
}

if [ $# -gt 0 ]
then
        for f in $*
        do
                clean $f
        done
else
        for f in data/*.txt
        do
                clean $f
        done
fi
========================== end of cleaner ===========================

Martin