You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Jari Fredriksson <ja...@iki.fi> on 2009/10/02 19:45:58 UTC
Daily statistics into email
Some just mentioned sa-stats.pl statistics, and I then wrote a script for me to post daily stats for me into email.
This is not nuclear science, but I still share it.
It is HTML formatted because I use Outlook Express to read mail, but it is easy to fix
The file is named so that it runs just before 00-logrotate in Debian Linux.
Non HTML version:
--------------- /etc/cron.daily/00a-sa-stats -----------------------
#!/bin/sh
/usr/local/bin/sa-stats.pl
--------------------------------------------------------------------
HTML version for GUI users
--------------- /etc/cron.daily/00a-sa-stats -----------------------
#!/bin/sh
FILE=/tmp/stats.mail
echo To: jarif@wellington>$FILE
echo Subject: SpamAssassin statistics>>$FILE
echo "Content-Type: text/html; charset=\"us-ascii\"">>$FILE
echo>>$FILE
/usr/local/bin/sa-stats.pl | /usr/bin/txt2html >>$FILE
/usr/sbin/sendmail jarif@wellington < $FILE
rm $FILE
--------------------------------------------------------------------
The HTML version requires txt2html program which is installable on Debian apt, and propably others too. If not, maybe cpan..
Sendmail command is available with sendmail and postfix emailers, dunno about others.
Re: Daily statistics into email
Posted by Martin Gregorie <ma...@gregorie.org>.
On Sat, 2009-10-03 at 03:57 +0300, Jari Fredriksson wrote:
> But let us keep in mind that it is the client that renders the mail
> for us to see.
>
> it must be some format the the client must understand.
> postfix.sendmail is not a client, and whatever it does must be
> understandable by the client.
>
Of course. I use Evolution as my MUA, which did the job just as you'd
expect.
Martin
Re: Daily statistics into email
Posted by Jari Fredriksson <ja...@iki.fi>.
> On Sat, 2009-10-03 at 00:03 +0300, Jari Fredriksson wrote:
>> This is something that I have no knowledge.
>>
> It was a surprise to me too!
>
>> Could you see the "source" format of the mail? I can't
>> think anything except it being in HTML format, as there
>> is no AFAIK no other formats for "rich text" in email.
>>
> I didn't keep any examples, but from memory the headers
> set the content to base64 and the body was a single
> base64 block. Normally the body is not MIME - its simply
> a plain ASCII body. During the backup
> I accumulate the report in a temp file and right at the
> end I cat it to stdout.
>
>> Maybe sendmail (was it really sendmail?) can convert
>> when it sees ANSI or something codes in the data..
>>
> AFAIK its just postfix.sendmail. I suppose its possible
> that crond does the encoding internally, but that's not
> mentioned in its manpage.
>
But let us keep in mind that it is the client that renders the mail for us to see.
it must be some format the the client must understand. postfix.sendmail is not a client, and whatever it does must be understandable by the client.
Re: Daily statistics into email
Posted by Martin Gregorie <ma...@gregorie.org>.
On Sat, 2009-10-03 at 00:03 +0300, Jari Fredriksson wrote:
> This is something that I have no knowledge.
>
It was a surprise to me too!
> Could you see the "source" format of the mail? I can't think anything
> except it being in HTML format, as there is no AFAIK no other formats
> for "rich text" in email.
>
I didn't keep any examples, but from memory the headers set the content
to base64 and the body was a single base64 block. Normally the body is
not MIME - its simply a plain ASCII body. During the backup
I accumulate the report in a temp file and right at the end I cat it to
stdout.
> Maybe sendmail (was it really sendmail?) can convert when it sees ANSI
> or something codes in the data..
>
AFAIK its just postfix.sendmail. I suppose its possible that crond does
the encoding internally, but that's not mentioned in its manpage.
Martin
Re: Daily statistics into email
Posted by Jari Fredriksson <ja...@iki.fi>.
> On Fri, 2009-10-02 at 23:28 +0300, Jari Fredriksson wrote:
>> There is a blank line between Content-Type and Hello,
>> but the Content-Type line WILL get to the body, and the
>> html gets injected after it as raw html code, not as
>> html (because the actual content type will be text not
>> html).
>>
>> Cron does NOT allow manipulating the headers. ALL the
>> output is assumed to be payload.
>>
> Fair comment: I asked the question and got your answer.
> Thanks.
>
> However, did you know that sendmail isn't completely dumb?
>
> I have a nightly backup job that stops Postgres, does the
> backup and restarts it. The Postgres stop and start are
> done using 'service', which returns a "Stopping Postgres
> [FAILED]" message if Postgres fails to stop. FAILED is
> red, so the line contains non-ASCII characters. Until I
> fixed this failure sendmail was converting the entire
> message body into base64 encoding due to the X-term
> sequences being used to mark the start and stop of the
> red foreground.
>
> I can't see any reference to this behavior in the
> sendmail manpage.
This is something that I have no knowledge.
Could you see the "source" format of the mail? I can't think anything except it being in HTML format, as there is no AFAIK no other formats for "rich text" in email.
Maybe sendmail (was it really sendmail?) can convert when it sees ANSI or something codes in the data..
Interesting.
Re: Daily statistics into email
Posted by Martin Gregorie <ma...@gregorie.org>.
On Fri, 2009-10-02 at 23:28 +0300, Jari Fredriksson wrote:
> There is a blank line between Content-Type and Hello, but the
> Content-Type line WILL get to the body, and the html gets injected
> after it as raw html code, not as html (because the actual content
> type will be text not html).
>
> Cron does NOT allow manipulating the headers. ALL the output is
> assumed to be payload.
>
Fair comment: I asked the question and got your answer. Thanks.
However, did you know that sendmail isn't completely dumb?
I have a nightly backup job that stops Postgres, does the backup and
restarts it. The Postgres stop and start are done using 'service', which
returns a "Stopping Postgres [FAILED]" message if Postgres fails to
stop. FAILED is red, so the line contains non-ASCII characters. Until I
fixed this failure sendmail was converting the entire message body into
base64 encoding due to the X-term sequences being used to mark the start
and stop of the red foreground.
I can't see any reference to this behavior in the sendmail manpage.
Martin
Re: Daily statistics into email
Posted by Jari Fredriksson <ja...@iki.fi>.
> On Fri, 2009-10-02 at 21:33 +0300, Jari Fredriksson wrote:
>>> On Fri, 2009-10-02 at 20:45 +0300, Jari Fredriksson
>>> wrote:
>>>
>>>> Sendmail command is available with sendmail and postfix
>>>> emailers, dunno about others.
>>>>
>>> You don't need to use sendmail: if the cron job writes
>>> anything to stdout (or stderr) this is automatically
>>> mailed to root.
>>>
>>> If you'd rather that mail sent to root comes to you
>>> instead, just add a redirection line to /etc/aliases.
>>> Don't forget to regenerate the aliases database by
>>> running 'aliases' or your redirection won't take effect.
>>>
>>>
>>
>> That HTML version needs to add a header for
>> Content-Type. That is not possible by just echoing
>> somehing, as those go automatically to the body.
>>
>> The non-html version uses cron's default behaviour, but
>> the html version must use sendmail.
>>
> As crond must also use sendmail to ship any text that is
> left for it to deal with and you've already inserted the
> MIME header, it seems to me that the HTML processing
> would happen anyway, regardless of whether the call to
> sendmail was implicit or explicit. Hence I assumed that
> you'd just sendmail to avoid using the aliases system.
> What did I miss?
>
As I have understood it, crond grabs the stdout of the process it runs, and puts that as the "payload" of the email it generates. It writes the mail headers for that mail itself.
An email contains two parts, the headers and the body. They are separated by a blank line.
First blank line in the stream is the separator. It is not possible to inject headers in the stdout of the cron job, as cron injects the blank line between the body and the headers it generates.
If you do not belive, just make a file like this:
------------------- text.sh ----------------------
echo "Content-Type: text/html; charset=\"us-ascii\""
echo ""
echo Hello | txt2html
---------------------------------------------------
And then command
batch text.sh
There is a blank line between Content-Type and Hello, but the Content-Type line WILL get to the body, and the html gets injected after it as raw html code, not as html (because the actual content type will be text not html).
Cron does NOT allow manipulating the headers. ALL the output is assumed to be payload.
If you know better, please post a sample.
Re: Daily statistics into email
Posted by Martin Gregorie <ma...@gregorie.org>.
On Fri, 2009-10-02 at 21:33 +0300, Jari Fredriksson wrote:
> > On Fri, 2009-10-02 at 20:45 +0300, Jari Fredriksson wrote:
> >
> >> Sendmail command is available with sendmail and postfix
> >> emailers, dunno about others.
> >>
> > You don't need to use sendmail: if the cron job writes
> > anything to stdout (or stderr) this is automatically
> > mailed to root.
> >
> > If you'd rather that mail sent to root comes to you
> > instead, just add a redirection line to /etc/aliases.
> > Don't forget to regenerate the aliases database by
> > running 'aliases' or your redirection won't take effect.
> >
> >
>
> That HTML version needs to add a header for Content-Type. That is not possible by just echoing somehing, as those go automatically to the body.
>
> The non-html version uses cron's default behaviour, but the html version must use sendmail.
>
As crond must also use sendmail to ship any text that is left for it to
deal with and you've already inserted the MIME header, it seems to me
that the HTML processing would happen anyway, regardless of whether the
call to sendmail was implicit or explicit. Hence I assumed that you'd
just sendmail to avoid using the aliases system. What did I miss?
Martin
Re: Daily statistics into email
Posted by Jari Fredriksson <ja...@iki.fi>.
> On Fri, 2009-10-02 at 20:45 +0300, Jari Fredriksson wrote:
>
>> Sendmail command is available with sendmail and postfix
>> emailers, dunno about others.
>>
> You don't need to use sendmail: if the cron job writes
> anything to stdout (or stderr) this is automatically
> mailed to root.
>
> If you'd rather that mail sent to root comes to you
> instead, just add a redirection line to /etc/aliases.
> Don't forget to regenerate the aliases database by
> running 'aliases' or your redirection won't take effect.
>
>
That HTML version needs to add a header for Content-Type. That is not possible by just echoing somehing, as those go automatically to the body.
The non-html version uses cron's default behaviour, but the html version must use sendmail.
Re: Daily statistics into email
Posted by Martin Gregorie <ma...@gregorie.org>.
On Fri, 2009-10-02 at 20:45 +0300, Jari Fredriksson wrote:
> Sendmail command is available with sendmail and postfix emailers,
> dunno about others.
>
You don't need to use sendmail: if the cron job writes anything to
stdout (or stderr) this is automatically mailed to root.
If you'd rather that mail sent to root comes to you instead, just add a
redirection line to /etc/aliases. Don't forget to regenerate the aliases
database by running 'aliases' or your redirection won't take effect.
Martin
Re: Daily statistics into email
Posted by Jari Fredriksson <ja...@iki.fi>.
>> http://ruleqa.spamassassin.org/
>> If you are capable of processing your mail nightly in
>> cron, why don't you join the nightly mass check? You can
>> help to test the rules and make the sa-update channel
>> better. We especially need non-English ham in the
>> nightly masscheck.
>>
>> http://wiki.apache.org/spamassassin/NightlyMassCheck
>> Here's HOWTO. The documentation is a bit confusing. I'm
>> working on a much simpler version of this.
>>
>> What distro do you use?
>>
>> Warren Togami
>> wtogami@redhat.com
>
> I'm using Debian Lenny. This really interests me as a
> SpamFighter! Going into the links now.
A second thought.. My hardware does not allow extra work. I currently have two hosts with SpamAssassin in my network, both of them use only one child. There is a 3rd host for Amavisd-new, and it also has only one thread for virusscanners.
I get about 10000-15000 spam per month, and this setup barely manages it without great delays. I don't think I can run those tests with these machines.
I dream about a multicore server, but currently I'm laid off from my day work and wondering where to get bread and beer from.
I will join later when that is possible, but not now.
Re: Daily statistics into email
Posted by Jari Fredriksson <ja...@iki.fi>.
> http://ruleqa.spamassassin.org/
> If you are capable of processing your mail nightly in
> cron, why don't you join the nightly mass check? You can
> help to test the rules and make the sa-update channel
> better. We especially need non-English ham in the
> nightly masscheck.
>
> http://wiki.apache.org/spamassassin/NightlyMassCheck
> Here's HOWTO. The documentation is a bit confusing. I'm
> working on a much simpler version of this.
>
> What distro do you use?
>
> Warren Togami
> wtogami@redhat.com
I'm using Debian Lenny. This really interests me as a SpamFighter! Going into the links now.
Re: Daily statistics into email
Posted by Warren Togami <wt...@redhat.com>.
http://ruleqa.spamassassin.org/
If you are capable of processing your mail nightly in cron, why don't
you join the nightly mass check? You can help to test the rules and
make the sa-update channel better. We especially need non-English ham
in the nightly masscheck.
http://wiki.apache.org/spamassassin/NightlyMassCheck
Here's HOWTO. The documentation is a bit confusing. I'm working on a
much simpler version of this.
What distro do you use?
Warren Togami
wtogami@redhat.com