You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Jim McLachlan <ji...@oss-ltd.com> on 2017/04/04 21:09:05 UTC
Problem with massive log files
Hi,
I have a problem with the huge amount of messages being logged by spamassassin.
I have around 10 active e-mail users on the system, none of whom have any
unusual e-mail usage. This is what I've seen in the last 2 hours:
$ date
Mon 3 Apr 08:00:50 UTC 2017
$ ls -l /var/log/mail.log
-rw-r----- 1 syslog adm 86370829860 Apr 3 08:00 /var/log/mail.log
-rw-r----- 1 syslog adm 331608479025 Apr 2 09:20 /var/log/mail.log.1
$ spamassassin --version
SpamAssassin version 3.4.1
running on Perl version 5.22.1
My set up consists of Postfix, Postgrey, Spamassassin, Clam-AV, Amavis-new and
Dovecot.
When I send an e-mail through the system, it immediately starts churning out a
long list of log messages that implies it's checking messages from last October
(when I set up the server). It goes through thousands of messages like this and
then settles down again until another e-mail is processed.
My initial e-mail with attachment didn't work, so an excerpt from the log file
can be found here:
http://pasted.co/5e546e7a
Can someone please explain to me why it's repeating all this work and all these
messages for every e-mail that gets processed and what I can do to fix this. I
reduced the problem slightly yesterday by preventing all these messages getting
logged to syslog at the same time.
Kind regards.
Jim.
Re: Problem with massive log files
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 4/4/2017 9:34 PM, John Hardin wrote:
> "grep -v" of what? The logged info: lines (assuming they aren't being
> discarded at the moment)?
>
> That does work for identifying hosts, but it won't tell you what's on
> the other end of the connection.
I was just looking for other hosts. I didn't realize your tcpdump would
help identify what was on the other end.
Re: Problem with massive log files
Posted by John Hardin <jh...@impsec.org>.
On Tue, 4 Apr 2017, Kevin A. McGrail wrote:
> On 4/4/2017 9:14 PM, John Hardin wrote:
>>
>> At the most basic you'd filter for the port spamd is listening on:
>
> Hmm, thinking about my firewall question in context with this issue of how to
> use tcpdump. I think we already have this info with this line:
>
> Apr 2 10:31:26 oss2 spamfilter: Sat Oct 15 16:24:54 2016 [2758] info: spamd:
> connection from ip6-localhost [::1]:56238 to port 783, fd 5
Yeah, I did see that. The netstat would probably catch something running
on the SA host itself. It actually would catch stuff from other hosts, but
it takes a snapshot where tcpdump captures and reports traffic as long as
it's running.
> So a grep -v should give the same info which from a spotcheck of the log
> snippet isn't going to identify another host.
"grep -v" of what? The logged info: lines (assuming they aren't being
discarded at the moment)?
That does work for identifying hosts, but it won't tell you what's on the
other end of the connection.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The tree of freedom must be freshened from time to time
with the blood of tyrants and tyrannosaurs.
-- DW, commenting on the GM6 Lynx .50BMG bullpup
-----------------------------------------------------------------------
9 days until Thomas Jefferson's 274th Birthday
Re: Problem with massive log files
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 4/4/2017 9:14 PM, John Hardin wrote:
>
> At the most basic you'd filter for the port spamd is listening on:
Hmm, thinking about my firewall question in context with this issue of
how to use tcpdump. I think we already have this info with this line:
Apr 2 10:31:26 oss2 spamfilter: Sat Oct 15 16:24:54 2016 [2758] info: spamd: connection from ip6-localhost [::1]:56238 to port 783, fd 5
So a grep -v should give the same info which from a spotcheck of the log
snippet isn't going to identify another host.
regards,
KAM
Re: Problem with massive log files
Posted by John Hardin <jh...@impsec.org>.
On Wed, 5 Apr 2017, Jim McLachlan wrote:
> Hi John,
>
> That sounds like a good move. I don't have a lot of experience using
> tcpdump. Could you help prevent me from fumbling around like a ****wit with
> it and let me know what I need to do with it to identify the source of the
> spamd traffic?
At the most basic you'd filter for the port spamd is listening on:
tcpdump -n tcp dst port 783
You'd probably also want to filter for SYN packets to reduce the clutter:
tcpdump -n tcp dst port 783 and 'tcp[tcpflags] & tcp-syn != 0'
Note the source IP address. If it's a different host you can look there
for what's sending the traffic.
You can also run this to get an idea of what programs are calling spamd:
netstat -natp | grep ":783 "
You could run the netstat first, under the assumption that the source is
most likely local rather than remote, then run tcpdump if you can't find a
plausible local source.
All of this needs to be done as root, of course.
> On 05/04/17 01:21, John Hardin wrote:
>>
>> Perhaps run tcpdump and see where the spamd traffic is coming from, to
>> rule out something other than the local MTAs?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The more you believe you can create heaven on earth
the more likely you are to set up guillotines in the public square
to hasten the process. -- James Lileks
-----------------------------------------------------------------------
9 days until Thomas Jefferson's 274th Birthday
Re: Problem with massive log files
Posted by Jim McLachlan <ji...@oss-ltd.com>.
Hi John,
That sounds like a good move. I don't have a lot of experience using tcpdump.
Could you help prevent me from fumbling around like a ****wit with it and let me
know what I need to do with it to identify the source of the spamd traffic?
Thanks.
Kind regards.
Jim.
On 05/04/17 01:21, John Hardin wrote:
> On Tue, 4 Apr 2017, Kevin A. McGrail wrote:
>
>> On 4/4/2017 8:04 PM, John Hardin wrote:
>>> If all else fails, you may want to visit syslog.conf and tell it to ignore
>>> mail.info level messages.
>>
>> Hmm, normally I agree with you, John but I'd strongly recommend against that.
>> He's got something hitting spamd approximately 500x more than is needed and
>> that would just hide the issue.
>>
>> Plus, I'd likely say the machine is CPU and IO bound from that much processing
>> anyway... The syslog is just a tip of the iceberg.
>
> Oh, agreed.
>
> Perhaps run tcpdump and see where the spamd traffic is coming from, to rule out
> something other than the local MTAs?
>
--
James R. McLachlan PGDCCI(Open)
Managing Director
Objective Software Services Ltd.
Web : http://www.oss-ltd.com
Tel : +44 (0)1397 708550
Mob : +44 (0)7971 232717
Fax : +44 (0)7970 117580
e-mail: jim@oss-ltd.com
Objective Software Services Ltd. is a company registered in England and Wales
with company number 2892148.
Registered office: 11 Percy Terrace, Tunbridge Wells, Kent, TN4 9RH
Re: Problem with massive log files
Posted by John Hardin <jh...@impsec.org>.
On Tue, 4 Apr 2017, Kevin A. McGrail wrote:
> On 4/4/2017 8:04 PM, John Hardin wrote:
>> If all else fails, you may want to visit syslog.conf and tell it to ignore
>> mail.info level messages.
>
> Hmm, normally I agree with you, John but I'd strongly recommend against that.
> He's got something hitting spamd approximately 500x more than is needed and
> that would just hide the issue.
>
> Plus, I'd likely say the machine is CPU and IO bound from that much
> processing anyway... The syslog is just a tip of the iceberg.
Oh, agreed.
Perhaps run tcpdump and see where the spamd traffic is coming from, to
rule out something other than the local MTAs?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The more you believe you can create heaven on earth
the more likely you are to set up guillotines in the public square
to hasten the process. -- James Lileks
-----------------------------------------------------------------------
9 days until Thomas Jefferson's 274th Birthday
Re: Problem with massive log files
Posted by Martin Gregorie <ma...@gregorie.org>.
It occurs to me that anything grinding through enough mail to generate
that much logging should also be eating a lot of CPU - so much so that
it might even be identified by seeing what is using unexpectedly large
amounts of CPU time.�
Running 'top' and watching it for a while to see what patterns develop
is one possibility.�
Using a pipeline like this is another:
ps -ef | sort -k=4 -r | head
It lists the 10 heaviest CPU users in descending order each time its
run.
Martin
Re: Problem with massive log files
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 4/4/2017 8:04 PM, John Hardin wrote:
> If all else fails, you may want to visit syslog.conf and tell it to
> ignore mail.info level messages.
Hmm, normally I agree with you, John but I'd strongly recommend against
that. He's got something hitting spamd approximately 500x more than is
needed and that would just hide the issue.
Plus, I'd likely say the machine is CPU and IO bound from that much
processing anyway... The syslog is just a tip of the iceberg.
Re: Problem with massive log files
Posted by Jim McLachlan <ji...@oss-ltd.com>.
Hi John,
I did that a couple of days ago after I ran out of disk space. It's helped
quite a lot, but only in that it's removed a symptom.
-rw-r----- 1 syslog adm 457498 Apr 5 00:09 /var/log/syslog
-rw-r----- 1 syslog adm 652564 Apr 4 06:33 /var/log/syslog.1
-rw-r----- 1 syslog adm 2766939968 Apr 3 06:45 /var/log/syslog.2.gz
-rw-r----- 1 syslog adm 7074131258 Apr 2 09:20 /var/log/syslog.3.gz
Kind regards.
Jim.
On 05/04/17 01:04, John Hardin wrote:
> On Wed, 5 Apr 2017, Jim McLachlan wrote:
>
>> The text "info: spamd: processing message" appears in that 162,761 times.
>
> If all else fails, you may want to visit syslog.conf and tell it to ignore
> mail.info level messages.
>
>
--
James R. McLachlan PGDCCI(Open)
Managing Director
Objective Software Services Ltd.
Web : http://www.oss-ltd.com
Tel : +44 (0)1397 708550
Mob : +44 (0)7971 232717
Fax : +44 (0)7970 117580
e-mail: jim@oss-ltd.com
Objective Software Services Ltd. is a company registered in England and Wales
with company number 2892148.
Registered office: 11 Percy Terrace, Tunbridge Wells, Kent, TN4 9RH
Re: Problem with massive log files
Posted by John Hardin <jh...@impsec.org>.
On Wed, 5 Apr 2017, Jim McLachlan wrote:
> The text "info: spamd: processing message" appears in that 162,761
> times.
If all else fails, you may want to visit syslog.conf and tell it to ignore
mail.info level messages.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The more you believe you can create heaven on earth
the more likely you are to set up guillotines in the public square
to hasten the process. -- James Lileks
-----------------------------------------------------------------------
9 days until Thomas Jefferson's 274th Birthday
Re: Problem with massive log files
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 4/4/2017 7:58 PM, Jim McLachlan wrote:
>
> I'm not sure which message I'm looking for, but for that same file
> of 1,000,000 lines, I used this line to cut out all occurrences of
> "postfix" and count them:
So ~300 vs 160K or something bizarre...
Is there anything using that spamfilter.sh that might be integrated with
existing email folders like something in dovecot? Perhaps it's grinding
through retesting everything?
I'm stretching here but you are looking for something processing mail
that isn't postfix it appears...
Re: Problem with massive log files
Posted by Jim McLachlan <ji...@oss-ltd.com>.
Hi KAM,
I'm not sure which message I'm looking for, but for that same file of 1,000,000
lines, I used this line to cut out all occurrences of "postfix" and count them:
$ grep postfix /tmp/mail_sample.log | cut -d " " -f 6- | cut -d "[" -f 1 | sort
| uniq -c
7 postfix/cleanup
3 postfix/pickup
6 postfix/pipe
16 postfix/qmgr
3 postfix/smtp
293 postfix/smtpd
Port 783 isn't open to the outside world.
Kind regards.
Jim.
On 05/04/17 00:48, Kevin A. McGrail wrote:
> On 4/4/2017 7:35 PM, Jim McLachlan wrote:
>> The text "info: spamd: processing message" appears in that 162,761 times.
> Neat... And how many times do you have a line indicating a new message from
> postfix in the same period?
>
> Firewall off port 783 on the box. It's a longshot but perhaps something is
> connecting to your spamd?
>
> Regards,
> KAM
--
James R. McLachlan PGDCCI(Open)
Managing Director
Objective Software Services Ltd.
Web : http://www.oss-ltd.com
Tel : +44 (0)1397 708550
Mob : +44 (0)7971 232717
Fax : +44 (0)7970 117580
e-mail: jim@oss-ltd.com
Objective Software Services Ltd. is a company registered in England and Wales
with company number 2892148.
Registered office: 11 Percy Terrace, Tunbridge Wells, Kent, TN4 9RH
Re: Problem with massive log files
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 4/4/2017 7:35 PM, Jim McLachlan wrote:
> The text "info: spamd: processing message" appears in that 162,761
> times.
Neat... And how many times do you have a line indicating a new message
from postfix in the same period?
Firewall off port 783 on the box. It's a longshot but perhaps something
is connecting to your spamd?
Regards,
KAM
Re: Problem with massive log files
Posted by Jim McLachlan <ji...@oss-ltd.com>.
Hi KAM,
Well, I have an application that sends out around 400 to 500 e-mails per day,
but other than that, there are about a dozen or so personal e-mail accounts.
That should be < 20,000 per month.
I just hived off the last 1,000,000 lines of the current mail.log file for a
quick check. The messages cover the 6 minute period:
Apr 4 23:24:26 to 23:30:40
The text "info: spamd: processing message" appears in that 162,761 times.
Kind regards.
Jim.
On 05/04/17 00:24, Kevin A. McGrail wrote:
> On 4/4/2017 7:06 PM, Jim McLachlan wrote:
>> It looks like my mail.* logs are rotated weekly. I'll change that so
>> they're rotated daily. That will certainly help, but I'm sure it would be
>> good for the disk and CPU if I can reduce the amount of data being logged.
>
> Well, how many emails are you processing because I process a million or so a
> month and my logs are much MUCH smaller with MORE logging enabled?
>
> I'm just wondering if you process a stuff load of email and your logs represent
> that mail volume. I didn't see anything under the "too much logging" category
> in the log sample.
>
> Regards,
> KAM
--
James R. McLachlan PGDCCI(Open)
Managing Director
Objective Software Services Ltd.
Web : http://www.oss-ltd.com
Tel : +44 (0)1397 708550
Mob : +44 (0)7971 232717
Fax : +44 (0)7970 117580
e-mail: jim@oss-ltd.com
Objective Software Services Ltd. is a company registered in England and Wales
with company number 2892148.
Registered office: 11 Percy Terrace, Tunbridge Wells, Kent, TN4 9RH
Re: Problem with massive log files
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 4/4/2017 7:06 PM, Jim McLachlan wrote:
> It looks like my mail.* logs are rotated weekly. I'll change that
> so they're rotated daily. That will certainly help, but I'm sure it
> would be good for the disk and CPU if I can reduce the amount of data
> being logged.
Well, how many emails are you processing because I process a million or
so a month and my logs are much MUCH smaller with MORE logging enabled?
I'm just wondering if you process a stuff load of email and your logs
represent that mail volume. I didn't see anything under the "too much
logging" category in the log sample.
Regards,
KAM
Re: Problem with massive log files
Posted by Jim McLachlan <ji...@oss-ltd.com>.
Sorry, I did a direct reply instead of a reply to the list. I hope this
corrects that.
Hi KAM,
You're confused Not as much as me. I'm completely baffled
I've posted my master.cf to http://pasted.co/ba783cac just in case that
might be useful.
It looks like my mail.* logs are rotated weekly. I'll change that so
they're rotated daily. That will certainly help, but I'm sure it would be good
for the disk and CPU if I can reduce the amount of data being logged.
Kind regards.
Jim.
On 04/04/17 23:55, Kevin A. McGrail wrote:
> On 4/4/2017 6:42 PM, Jim McLachlan wrote:
>> amavis 1680 1 0 2016 ? 00:01:40 /usr/sbin/amavisd-new (master)
>> amavis 10898 1680 0 17:29 ? 00:00:01 /usr/sbin/amavisd-new (ch7-avail)
>> amavis 15292 1680 0 22:16 ? 00:00:00 /usr/sbin/amavisd-new (ch1-avail)
>> postfix 15337 7599 0 22:19 ? 00:00:00 smtp -n amavis -t unix -u -o
>> smtp_data_done_timeout=1200 -o smtp_send_xforward_command=yes -o
>> disable_dns_lookups=yes -o max_use=20
>> amavis 15344 1680 0 22:19 ? 00:00:00 /usr/sbin/amavisd-new (ch1-avail)
>>
>>
>> I've posted the spamfilter.sh file to http://pasted.co/7b794ccd
>>
>> I don't see anything in there about verbose logging
>
> Quick points:
>
> 1 - The verbose logging (which I don't think is the issue) would be in your
> postfix master.cf indicated by -v on smtpd. Reviewing the log snippet, I saw
> nothing that looked like too much logging, anything looping, etc.
>
> 2 - I am confused if you have amavis why you would also have spamfilter.sh. I
> don't use amavisd-new but I'm sure some here do and can comment.
>
> Regards,
> KAM
--
James R. McLachlan PGDCCI(Open)
Managing Director
Objective Software Services Ltd.
Web : http://www.oss-ltd.com
Tel : +44 (0)1397 708550
Mob : +44 (0)7971 232717
Fax : +44 (0)7970 117580
e-mail: jim@oss-ltd.com
Objective Software Services Ltd. is a company registered in England and Wales
with company number 2892148.
Registered office: 11 Percy Terrace, Tunbridge Wells, Kent, TN4 9RH
Re: Problem with massive log files
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 4/4/2017 6:42 PM, Jim McLachlan wrote:
> amavis 1680 1 0 2016 ? 00:01:40 /usr/sbin/amavisd-new
> (master)
> amavis 10898 1680 0 17:29 ? 00:00:01 /usr/sbin/amavisd-new
> (ch7-avail)
> amavis 15292 1680 0 22:16 ? 00:00:00 /usr/sbin/amavisd-new
> (ch1-avail)
> postfix 15337 7599 0 22:19 ? 00:00:00 smtp -n amavis -t unix
> -u -o smtp_data_done_timeout=1200 -o smtp_send_xforward_command=yes -o
> disable_dns_lookups=yes -o max_use=20
> amavis 15344 1680 0 22:19 ? 00:00:00 /usr/sbin/amavisd-new
> (ch1-avail)
>
>
> I've posted the spamfilter.sh file to http://pasted.co/7b794ccd
>
> I don't see anything in there about verbose logging
Quick points:
1 - The verbose logging (which I don't think is the issue) would be in
your postfix master.cf indicated by -v on smtpd. Reviewing the log
snippet, I saw nothing that looked like too much logging, anything
looping, etc.
2 - I am confused if you have amavis why you would also have
spamfilter.sh. I don't use amavisd-new but I'm sure some here do and
can comment.
Regards,
KAM
Re: Problem with massive log files
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 4/4/2017 8:45 PM, Jim McLachlan wrote:
> I noticed a lot of dovecot processes, mainly dovecot/imap and
> dovecot/imap-login. I restarted that, but it only cleared them
> temporarily and they're back now.
You should have those if you are running an IMAP server. If you
shutdown dovecot, does the logging to maillog return to "normal" levels?
> I made a copy of spamfilter.sh to my_spamfilter.sh, then did the
> chmod -x on the original. I updated master.cf to refer to
> my_spamfilter.sh and restarted postfix and spamassassin.
>
> Sadly, no luck.
>
> If I tail -f /var/log/mail.log and then send an e-mail, it
> instantly starts spitting out those log lines.
To be clear, I'm expecting log lines. You should have SOME logging.
And it should be like 10 to 1 ration of spamfilter logs to postfix as a
random off-the-cuff number.
> It's also 1:44am here at the moment, so I'm going to have to go to
> bed now. Please accept my apologies for breaking off this conversation
> at this stage.
Sleep is for the weak... Or sleep for a week...
Re: Problem with massive log files
Posted by Jim McLachlan <ji...@oss-ltd.com>.
Hi KAM,
I noticed a lot of dovecot processes, mainly dovecot/imap and
dovecot/imap-login. I restarted that, but it only cleared them temporarily and
they're back now.
I made a copy of spamfilter.sh to my_spamfilter.sh, then did the chmod -x on
the original. I updated master.cf to refer to my_spamfilter.sh and restarted
postfix and spamassassin.
Sadly, no luck.
If I tail -f /var/log/mail.log and then send an e-mail, it instantly starts
spitting out those log lines.
It's also 1:44am here at the moment, so I'm going to have to go to bed now.
Please accept my apologies for breaking off this conversation at this stage.
Kind regards.
Jim.
On 05/04/17 01:29, Kevin A. McGrail wrote:
> On 4/4/2017 6:42 PM, Jim McLachlan wrote:
>>
>> https://www.digitalocean.com/community/tutorials/how-to-configure-a-mail-server-using-postfix-dovecot-mysql-and-spamassassin
>>
>>
>> More recently, I found this one:
>>
>> https://www.exratione.com/2016/05/a-mailserver-on-ubuntu-16-04-postfix-dovecot-mysql/
>>
>>
>> Which I've used to check what I've got. It /seems/ to be set up OK.
> Neither reference spamfilter.sh that I can see.
>
> What I would recommend is chmod -x spamfilter.sh and rename it to something else
> and change your postfix to only reference the new name.
>
> See if that doesn't slow things down...
>
--
James R. McLachlan PGDCCI(Open)
Managing Director
Objective Software Services Ltd.
Web : http://www.oss-ltd.com
Tel : +44 (0)1397 708550
Mob : +44 (0)7971 232717
Fax : +44 (0)7970 117580
e-mail: jim@oss-ltd.com
Objective Software Services Ltd. is a company registered in England and Wales
with company number 2892148.
Registered office: 11 Percy Terrace, Tunbridge Wells, Kent, TN4 9RH
Re: Problem with massive log files
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 4/4/2017 6:42 PM, Jim McLachlan wrote:
>
> https://www.digitalocean.com/community/tutorials/how-to-configure-a-mail-server-using-postfix-dovecot-mysql-and-spamassassin
>
>
> More recently, I found this one:
>
> https://www.exratione.com/2016/05/a-mailserver-on-ubuntu-16-04-postfix-dovecot-mysql/
>
>
> Which I've used to check what I've got. It /seems/ to be set up OK.
Neither reference spamfilter.sh that I can see.
What I would recommend is chmod -x spamfilter.sh and rename it to
something else and change your postfix to only reference the new name.
See if that doesn't slow things down...
Re: Problem with massive log files
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 4/4/2017 8:51 PM, Jim McLachlan wrote:
> Thanks. I tried them both with the same results, several e-mail
> details, then the summary:
>
> 61 Kbytes in 8 Requests.
>
> They all look like valid e-mails.
They are alternatives for the same command.
I would expect some entries. 8 sounds about right.
Not have a billion queued there is all we were checking.
Re: Problem with massive log files
Posted by Jim McLachlan <ji...@oss-ltd.com>.
Hi KAM,
Thanks. I tried them both with the same results, several e-mail details, then
the summary:
61 Kbytes in 8 Requests.
They all look like valid e-mails.
Kind regards.
Jim.
On 05/04/17 01:46, Kevin A. McGrail wrote:
> On 4/4/2017 8:39 PM, Jim McLachlan wrote:
>> Could you let me know where I should look for the temporary files you mentioned?
>
> One thing might be postfix queues but I'd expect postfix lines in the maillogs...
>
> mailq or postqueue -p
>
> Regards,
> KAM
--
James R. McLachlan PGDCCI(Open)
Managing Director
Objective Software Services Ltd.
Web : http://www.oss-ltd.com
Tel : +44 (0)1397 708550
Mob : +44 (0)7971 232717
Fax : +44 (0)7970 117580
e-mail: jim@oss-ltd.com
Objective Software Services Ltd. is a company registered in England and Wales
with company number 2892148.
Registered office: 11 Percy Terrace, Tunbridge Wells, Kent, TN4 9RH
Re: Problem with massive log files
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 4/4/2017 8:39 PM, Jim McLachlan wrote:
> Could you let me know where I should look for the temporary files you
> mentioned?
One thing might be postfix queues but I'd expect postfix lines in the
maillogs...
mailq or postqueue -p
Regards,
KAM
Re: Problem with massive log files
Posted by ap-ml <ap...@soadns.co.uk>.
Usually the directories will exist somewhere in /var or /usr, my linux
is rusty, but try this command line in a new terminal window
inotifywait -rme modify,attrib,move,close_write,create,delete,delete_self /dname
change dname to appropriate directory. inotify is part of iotify-tools
on Centos, im sure you can find/install it easily enough. try different
directory roots perhaps , start mail server in another window, see what
is being accessed.
its early here too ;-)
On 05/04/2017 01:39, Jim McLachlan wrote:
> Hi ap-ml,
>
> This sounds interesting. Could you let me know where I should
> look for the temporary files you mentioned?
>
> I'm on the edges of my knowledge of e-mail and networking here :-)
>
> Kind regards.
>
> Jim.
>
> On 05/04/17 01:11, ap-ml wrote:
>> Its almost as though there is a build-up of messages that are being
>> continually
>> scanned through, I had a similar issue once where due to incorrect
>> permissions,
>> temp files were not being deleted. Perhaps check temp & working
>> directories for
>> such a logjam of emails. Have you also checked for the message IDs
>> in the
>> logfiles in email headers to see if the files are in user emaildirs
>> or some
>> other folder?
>>
>>
>> On 05/04/2017 01:06, Dave Wreski wrote:
>>> Hi,
>>>
>>>>
>>>> I've posted the spamfilter.sh file to http://pasted.co/7b794ccd
>>>>
>>>> I don't see anything in there about verbose logging, but there are
>>>> two lines in there with a resemblance to your suggestion:
>>>>
>>>> logger -f $SALOG -p mail.notice -t spamfilter <<<"Spam filter
>>>> piping to
>>>> SpamAssassin: $SPAMASSASSIN -x -E -s $MAX_MESSAGE_SIZE"
>>>>
>>>> and
>>>>
>>>> logger -s -p mail.notice -t spamfilter <<<"OK. Piping to sendmail:
>>>> $SENDMAIL $@"
>>>>
>>>> The second one seems to be after-the-fact, so I think I could
>>>> modify
>>>> the first one. Should this be cut down to:
>>>>
>>>> $SPAMASSASSIN -x -E -s $MAX_MESSAGE_SIZE
>>>>
>>>> to avoid the logging process?
>>>
>>> Yes, you can comment out this line:
>>>
>>> logger -f $SALOG -p mail.notice -t spamfilter <<<"Spam filter piping
>>> ...
>>>
>>> and this one:
>>>
>>> logger -s -p mail.notice -t spamfilter <<<"OK. Piping to sendmail: ...
>>>
>>> Basically, the first and last "logger" lines. That's a good start.
>>>
>>> Regards,
>>> Dave
>
Re: Problem with massive log files
Posted by Jim McLachlan <ji...@oss-ltd.com>.
Hi ap-ml,
This sounds interesting. Could you let me know where I should look for the
temporary files you mentioned?
I'm on the edges of my knowledge of e-mail and networking here :-)
Kind regards.
Jim.
On 05/04/17 01:11, ap-ml wrote:
> Its almost as though there is a build-up of messages that are being continually
> scanned through, I had a similar issue once where due to incorrect permissions,
> temp files were not being deleted. Perhaps check temp & working directories for
> such a logjam of emails. Have you also checked for the message IDs in the
> logfiles in email headers to see if the files are in user emaildirs or some
> other folder?
>
>
> On 05/04/2017 01:06, Dave Wreski wrote:
>> Hi,
>>
>>>
>>> I've posted the spamfilter.sh file to http://pasted.co/7b794ccd
>>>
>>> I don't see anything in there about verbose logging, but there are
>>> two lines in there with a resemblance to your suggestion:
>>>
>>> logger -f $SALOG -p mail.notice -t spamfilter <<<"Spam filter piping to
>>> SpamAssassin: $SPAMASSASSIN -x -E -s $MAX_MESSAGE_SIZE"
>>>
>>> and
>>>
>>> logger -s -p mail.notice -t spamfilter <<<"OK. Piping to sendmail:
>>> $SENDMAIL $@"
>>>
>>> The second one seems to be after-the-fact, so I think I could modify
>>> the first one. Should this be cut down to:
>>>
>>> $SPAMASSASSIN -x -E -s $MAX_MESSAGE_SIZE
>>>
>>> to avoid the logging process?
>>
>> Yes, you can comment out this line:
>>
>> logger -f $SALOG -p mail.notice -t spamfilter <<<"Spam filter piping ...
>>
>> and this one:
>>
>> logger -s -p mail.notice -t spamfilter <<<"OK. Piping to sendmail: ...
>>
>> Basically, the first and last "logger" lines. That's a good start.
>>
>> Regards,
>> Dave
--
James R. McLachlan PGDCCI(Open)
Managing Director
Objective Software Services Ltd.
Web : http://www.oss-ltd.com
Tel : +44 (0)1397 708550
Mob : +44 (0)7971 232717
Fax : +44 (0)7970 117580
e-mail: jim@oss-ltd.com
Objective Software Services Ltd. is a company registered in England and Wales
with company number 2892148.
Registered office: 11 Percy Terrace, Tunbridge Wells, Kent, TN4 9RH
Re: Problem with massive log files
Posted by ap-ml <ap...@soadns.co.uk>.
Its almost as though there is a build-up of messages that are being
continually scanned through, I had a similar issue once where due to
incorrect permissions, temp files were not being deleted. Perhaps check
temp & working directories for such a logjam of emails. Have you also
checked for the message IDs in the logfiles in email headers to see if
the files are in user emaildirs or some other folder?
On 05/04/2017 01:06, Dave Wreski wrote:
> Hi,
>
>>
>> I've posted the spamfilter.sh file to http://pasted.co/7b794ccd
>>
>> I don't see anything in there about verbose logging, but there are
>> two lines in there with a resemblance to your suggestion:
>>
>> logger -f $SALOG -p mail.notice -t spamfilter <<<"Spam filter piping to
>> SpamAssassin: $SPAMASSASSIN -x -E -s $MAX_MESSAGE_SIZE"
>>
>> and
>>
>> logger -s -p mail.notice -t spamfilter <<<"OK. Piping to sendmail:
>> $SENDMAIL $@"
>>
>> The second one seems to be after-the-fact, so I think I could modify
>> the first one. Should this be cut down to:
>>
>> $SPAMASSASSIN -x -E -s $MAX_MESSAGE_SIZE
>>
>> to avoid the logging process?
>
> Yes, you can comment out this line:
>
> logger -f $SALOG -p mail.notice -t spamfilter <<<"Spam filter piping ...
>
> and this one:
>
> logger -s -p mail.notice -t spamfilter <<<"OK. Piping to sendmail: ...
>
> Basically, the first and last "logger" lines. That's a good start.
>
> Regards,
> Dave
Re: Problem with massive log files
Posted by Dave Wreski <dw...@guardiandigital.com>.
Hi,
>
> I've posted the spamfilter.sh file to http://pasted.co/7b794ccd
>
> I don't see anything in there about verbose logging, but there are
> two lines in there with a resemblance to your suggestion:
>
> logger -f $SALOG -p mail.notice -t spamfilter <<<"Spam filter piping to
> SpamAssassin: $SPAMASSASSIN -x -E -s $MAX_MESSAGE_SIZE"
>
> and
>
> logger -s -p mail.notice -t spamfilter <<<"OK. Piping to sendmail:
> $SENDMAIL $@"
>
> The second one seems to be after-the-fact, so I think I could modify
> the first one. Should this be cut down to:
>
> $SPAMASSASSIN -x -E -s $MAX_MESSAGE_SIZE
>
> to avoid the logging process?
Yes, you can comment out this line:
logger -f $SALOG -p mail.notice -t spamfilter <<<"Spam filter piping ...
and this one:
logger -s -p mail.notice -t spamfilter <<<"OK. Piping to sendmail: ...
Basically, the first and last "logger" lines. That's a good start.
Regards,
Dave
Re: Problem with massive log files
Posted by Jim McLachlan <ji...@oss-ltd.com>.
Hi Dave,
I used the following instructions to set up the system, but they weren't
followed verbatim because I already had some things in place.
https://www.digitalocean.com/community/tutorials/how-to-configure-a-mail-server-using-postfix-dovecot-mysql-and-spamassassin
More recently, I found this one:
https://www.exratione.com/2016/05/a-mailserver-on-ubuntu-16-04-postfix-dovecot-mysql/
Which I've used to check what I've got. It /seems/ to be set up OK.
I don't know how much of this is useful or helpful, but you mentioned
amavis-new. My process list currently contains:
clamav 1001 1 0 2016 ? 00:38:54 /usr/bin/freshclam -d
--foreground=true
clamav 1018 1 0 2016 ? 01:46:43 /usr/sbin/clamd --foreground=true
amavis 1680 1 0 2016 ? 00:01:40 /usr/sbin/amavisd-new (master)
amavis 10898 1680 0 17:29 ? 00:00:01 /usr/sbin/amavisd-new (ch7-avail)
amavis 15292 1680 0 22:16 ? 00:00:00 /usr/sbin/amavisd-new (ch1-avail)
postfix 15337 7599 0 22:19 ? 00:00:00 smtp -n amavis -t unix -u -o
smtp_data_done_timeout=1200 -o smtp_send_xforward_command=yes -o
disable_dns_lookups=yes -o max_use=20
amavis 15344 1680 0 22:19 ? 00:00:00 /usr/sbin/amavisd-new (ch1-avail)
I've posted the spamfilter.sh file to http://pasted.co/7b794ccd
I don't see anything in there about verbose logging, but there are two lines in
there with a resemblance to your suggestion:
logger -f $SALOG -p mail.notice -t spamfilter <<<"Spam filter piping to
SpamAssassin: $SPAMASSASSIN -x -E -s $MAX_MESSAGE_SIZE"
and
logger -s -p mail.notice -t spamfilter <<<"OK. Piping to sendmail: $SENDMAIL $@"
The second one seems to be after-the-fact, so I think I could modify the first
one. Should this be cut down to:
$SPAMASSASSIN -x -E -s $MAX_MESSAGE_SIZE
to avoid the logging process?
Kind regards.
Jim.
On 04/04/17 23:19, Dave Wreski wrote:
> Hi,
>
>> I thought spamfilter was spamassassin.
>>
>> Looking through my config files, the postfix master.cf file contains
>> the line:
>>
>> flags=Rq user=spamd argv=/usr/bin/spamfilter.sh -oi -f ${sender}
>> ${recipient}
>>
>> /usr/bin/spamfilter.sh is described in the comments as:
>
> Where did you get the instructions to put this whole system together? You
> mentioned amavisd-new, but this doesn't look to be using amavisd-new at all.
>
> Look through the script for something relating to "debug" or "verbose" or
> "info:" or something pertaining to logging and disable it.
>
> In fact, a little googling shows this:
>
> logger <<<"Spam filter piping to SpamAssassin, then to: $SENDMAIL $@"
> ${SPAMASSASSIN} | ${SENDMAIL} "$@"
>
> You should be able to comment that out and instead use the following in place:
>
> ${SENDMAIL} "$@"
>
>> Current log file is up to 165 Gb.
>
> You should look at your logging and/or log rotating system to get this under
> control. I believe that's going to be /etc/logrotate.d/
>
> Regards,
> Dave
>
>
>>
>> Kind regards.
>>
>> Jim.
>>
>>
>>
>> On 04/04/17 22:41, Dave Wreski wrote:
>>> Hi,
>>>
>>>> My set up consists of Postfix, Postgrey, Spamassassin, Clam-AV,
>>>> Amavis-new and Dovecot.
>>>
>>> What is "spamfilter"?
>>>
>>> Apr 2 10:31:26 oss2 spamfilter: Sun Oct 16 07:24:13 2016 [16208]
>>> info: spamd:
>>> connection from ip6-localhost [::1]:53930 to port 783, fd 5
>>>
>>> What operating system?
>>>
>>> Regards,
>>> Dave
>>
--
James R. McLachlan PGDCCI(Open)
Managing Director
Objective Software Services Ltd.
Web : http://www.oss-ltd.com
Tel : +44 (0)1397 708550
Mob : +44 (0)7971 232717
Fax : +44 (0)7970 117580
e-mail: jim@oss-ltd.com
Objective Software Services Ltd. is a company registered in England and Wales
with company number 2892148.
Registered office: 11 Percy Terrace, Tunbridge Wells, Kent, TN4 9RH
Re: Problem with massive log files
Posted by Dave Wreski <dw...@guardiandigital.com>.
Hi,
> I thought spamfilter was spamassassin.
>
> Looking through my config files, the postfix master.cf file contains
> the line:
>
> flags=Rq user=spamd argv=/usr/bin/spamfilter.sh -oi -f ${sender}
> ${recipient}
>
> /usr/bin/spamfilter.sh is described in the comments as:
Where did you get the instructions to put this whole system together?
You mentioned amavisd-new, but this doesn't look to be using amavisd-new
at all.
Look through the script for something relating to "debug" or "verbose"
or "info:" or something pertaining to logging and disable it.
In fact, a little googling shows this:
logger <<<"Spam filter piping to SpamAssassin, then to: $SENDMAIL $@"
${SPAMASSASSIN} | ${SENDMAIL} "$@"
You should be able to comment that out and instead use the following in
place:
${SENDMAIL} "$@"
> Current log file is up to 165 Gb.
You should look at your logging and/or log rotating system to get this
under control. I believe that's going to be /etc/logrotate.d/
Regards,
Dave
>
> Kind regards.
>
> Jim.
>
>
>
> On 04/04/17 22:41, Dave Wreski wrote:
>> Hi,
>>
>>> My set up consists of Postfix, Postgrey, Spamassassin, Clam-AV,
>>> Amavis-new and Dovecot.
>>
>> What is "spamfilter"?
>>
>> Apr 2 10:31:26 oss2 spamfilter: Sun Oct 16 07:24:13 2016 [16208]
>> info: spamd:
>> connection from ip6-localhost [::1]:53930 to port 783, fd 5
>>
>> What operating system?
>>
>> Regards,
>> Dave
>
Re: Problem with massive log files
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 4/4/2017 6:53 PM, Jim McLachlan wrote:
> Do you know why the spamfilter entries in the log file have dates
> going back to October? Is the normal spamassassin behaviour that
> isn't usually logged, or is it doing something unusual?
>
> It seems to check all of them and log each check every time a new
> e-mail gets processed.
Are you rotating your maillog's nightly?
Regards,
KAM
Re: Problem with massive log files
Posted by Jim McLachlan <ji...@oss-ltd.com>.
Hi KAM,
No, there's nothing in the master.cf them indicates anything to do with logging
verbosely. No occurrences of "-v" and no mention of "log" or logging, etc.
Do you know why the spamfilter entries in the log file have dates going back to
October? Is the normal spamassassin behaviour that isn't usually logged, or is
it doing something unusual?
It seems to check all of them and log each check every time a new e-mail gets
processed.
Something must be set to verbose somewhere, but I haven't found it yet.
Kind regards.
Jim.
On 04/04/17 23:19, Kevin A. McGrail wrote:
> On 4/4/2017 6:08 PM, Jim McLachlan wrote:
>> I thought spamfilter was spamassassin.
> No, it's not. It's what we would call the glue. It's a content filter script
> that is reaching out to a spamassassin daemon called spamd using a lightweight c
> program called spamc.
>
> SpamD allows for spamassassin to run and daemonize so you don't have to wait for
> perl compilation on each and every call.
>
> So it's:
>
> SMTP -> Postfix -> Spamfilter.sh->spamc->spamd and back again
>
>
>
> In your master.cf for postfix, do you have anything like smtpd -v to turn on
> verbose logging?
>
>
> Looking at your logs, I see tons of spamfilter calls but not very many smtp
> calls. The logging doesn't have a queueid but checking the size= and the
> message id's they appear to be different messages. So how are emails getting to
> spamfilter.sh? Perhaps that is just because you took a small snapshot of the logs.
>
> Overall I don't see repetitive logging but the GB size of the logs is impressive
> for a server running only 2 SPAMD children...
>
> Regards,
> KAM
--
James R. McLachlan PGDCCI(Open)
Managing Director
Objective Software Services Ltd.
Web : http://www.oss-ltd.com
Tel : +44 (0)1397 708550
Mob : +44 (0)7971 232717
Fax : +44 (0)7970 117580
e-mail: jim@oss-ltd.com
Objective Software Services Ltd. is a company registered in England and Wales
with company number 2892148.
Registered office: 11 Percy Terrace, Tunbridge Wells, Kent, TN4 9RH
Re: Problem with massive log files
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 4/4/2017 6:08 PM, Jim McLachlan wrote:
> I thought spamfilter was spamassassin.
No, it's not. It's what we would call the glue. It's a content filter
script that is reaching out to a spamassassin daemon called spamd using
a lightweight c program called spamc.
SpamD allows for spamassassin to run and daemonize so you don't have to
wait for perl compilation on each and every call.
So it's:
SMTP -> Postfix -> Spamfilter.sh->spamc->spamd and back again
In your master.cf for postfix, do you have anything like smtpd -v to
turn on verbose logging?
Looking at your logs, I see tons of spamfilter calls but not very many
smtp calls. The logging doesn't have a queueid but checking the size=
and the message id's they appear to be different messages. So how are
emails getting to spamfilter.sh? Perhaps that is just because you took
a small snapshot of the logs.
Overall I don't see repetitive logging but the GB size of the logs is
impressive for a server running only 2 SPAMD children...
Regards,
KAM
Re: Problem with massive log files
Posted by Jim McLachlan <ji...@oss-ltd.com>.
Hi Dave,
Thanks for the quick response.
The OS is Ubuntu 16.04.
I thought spamfilter was spamassassin.
Looking through my config files, the postfix master.cf file contains the line:
flags=Rq user=spamd argv=/usr/bin/spamfilter.sh -oi -f ${sender} ${recipient}
/usr/bin/spamfilter.sh is described in the comments as:
# spamfilter.sh
#
# Advanced filter to plug SpamAssassin into the Postfix MTA
# Only discards messages that are above the threshold that is
# passed in to spamc; other spam messages that are below this
# threshold will be rewritten, but not discarded.
#
# Modified by Jeremy Morton
I can post the whole file if it would help. It's only 54 lines.
Hope that helps.
Current log file is up to 165 Gb.
Kind regards.
Jim.
On 04/04/17 22:41, Dave Wreski wrote:
> Hi,
>
>> My set up consists of Postfix, Postgrey, Spamassassin, Clam-AV,
>> Amavis-new and Dovecot.
>
> What is "spamfilter"?
>
> Apr 2 10:31:26 oss2 spamfilter: Sun Oct 16 07:24:13 2016 [16208] info: spamd:
> connection from ip6-localhost [::1]:53930 to port 783, fd 5
>
> What operating system?
>
> Regards,
> Dave
--
James R. McLachlan PGDCCI(Open)
Managing Director
Objective Software Services Ltd.
Web : http://www.oss-ltd.com
Tel : +44 (0)1397 708550
Mob : +44 (0)7971 232717
Fax : +44 (0)7970 117580
e-mail: jim@oss-ltd.com
Objective Software Services Ltd. is a company registered in England and Wales
with company number 2892148.
Registered office: 11 Percy Terrace, Tunbridge Wells, Kent, TN4 9RH
Re: Problem with massive log files
Posted by Dave Wreski <dw...@guardiandigital.com>.
Hi,
> My set up consists of Postfix, Postgrey, Spamassassin, Clam-AV,
> Amavis-new and Dovecot.
What is "spamfilter"?
Apr 2 10:31:26 oss2 spamfilter: Sun Oct 16 07:24:13 2016 [16208] info:
spamd: connection from ip6-localhost [::1]:53930 to port 783, fd 5
What operating system?
Regards,
Dave
Re: Problem with massive log files
Posted by Jim McLachlan <ji...@oss-ltd.com>.
Hi Tom,
Oh wow. I think you might have nailed it!
I posted spamfilter.sh last night to http://pasted.co/7b794ccd
The file in there (SALOG) looks like the culprit. It's 86 Mb and starts with:
Sat Oct 15 13:01:30 2016 [2753] info: logger: removing stderr method
Sat Oct 15 13:01:31 2016 [2756] info: zoom: able to use 346/346 'body_0'
compiled rules (100%)
Sat Oct 15 13:01:32 2016 [2756] info: spamd: server started on IO::Socket::IP
[::1]:783, IO::Socket::IP [127.0.0.1]:783 (running version 3.4.1)
Sat Oct 15 13:01:32 2016 [2756] info: spamd: server pid: 2756
Sat Oct 15 13:01:32 2016 [2756] info: spamd: server successfully spawned child
process, pid 2758
Which matches the lines I caught in the original log snippet.
So it is re-dumping a load of old messages back into the mail.log file, but now
I know where all that data is coming from :-) What's the best course of action?
If I delete/truncate that file, it will just grow back like a weed.
Dave Wreski suggested that I comment out the first and last logging from the
spamfilter.sh script.
I think I need to do that and then also add that file to the daily logrotate
list to prevent it growing continuously.
Does this make sense to everyone?
Thank you all! I'm so grateful for your time, expertise and effort on this.
Kind regards.
Jim.
On 05/04/17 08:32, Tom Hendrikx wrote:
> Hi,
>
> The thing that immediately caught my eye was the fact that in a line
> such as:
>
> Apr 2 10:31:26 oss2 spamfilter: Sat Oct 15 15:20:22 2016 [2758] info:
> spamd: connection from ip6-localhost [::1]:55708 to port 783, fd 5
>
> There are 2 timestamps, far away from each other. After some pondering,
> my guess is that spamfilter.sh is writing away log lines to a temporary
> file for each delivery, and them spewing them again when handling is
> complete. But there is a bug where spamfilter.sh does not cleanup after
> itself, and new lines are appended to the existing temporary file, and
> then the complete contents of the file are sent to syslog. Thus for each
> single delivery the logging for all messages in the past half year (Oct
> 15 -> Apr x) is sent to syslog.
>
> Please post the full contents of the spamfilter.sh, and examine the
> contents of any temporary files that is it using.
>
> Kind regards,
>
> Tom
>
>
> On 04-04-17 23:09, Jim McLachlan wrote:
>> Hi,
>>
>> I have a problem with the huge amount of messages being logged by
>> spamassassin. I have around 10 active e-mail users on the system, none
>> of whom have any unusual e-mail usage. This is what I've seen in the
>> last 2 hours:
>>
>> $ date
>> Mon 3 Apr 08:00:50 UTC 2017
>>
>> $ ls -l /var/log/mail.log
>> -rw-r----- 1 syslog adm 86370829860 Apr 3 08:00 /var/log/mail.log
>> -rw-r----- 1 syslog adm 331608479025 Apr 2 09:20 /var/log/mail.log.1
>>
>>
>> $ spamassassin --version
>> SpamAssassin version 3.4.1
>> running on Perl version 5.22.1
>>
>> My set up consists of Postfix, Postgrey, Spamassassin, Clam-AV,
>> Amavis-new and Dovecot.
>>
>> When I send an e-mail through the system, it immediately starts
>> churning out a long list of log messages that implies it's checking
>> messages from last October (when I set up the server). It goes through
>> thousands of messages like this and then settles down again until
>> another e-mail is processed.
>>
>> My initial e-mail with attachment didn't work, so an excerpt from
>> the log file can be found here:
>>
>> http://pasted.co/5e546e7a
>>
>> Can someone please explain to me why it's repeating all this work
>> and all these messages for every e-mail that gets processed and what I
>> can do to fix this. I reduced the problem slightly yesterday by
>> preventing all these messages getting logged to syslog at the same time.
>>
>> Kind regards.
>>
>> Jim.
--
James R. McLachlan PGDCCI(Open)
Managing Director
Objective Software Services Ltd.
Web : http://www.oss-ltd.com
Tel : +44 (0)1397 708550
Mob : +44 (0)7971 232717
Fax : +44 (0)7970 117580
e-mail: jim@oss-ltd.com
Objective Software Services Ltd. is a company registered in England and Wales
with company number 2892148.
Registered office: 11 Percy Terrace, Tunbridge Wells, Kent, TN4 9RH
Re: Problem with massive log files
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
On 05.04.17 11:13, Jim McLachlan wrote:
>Hi everyone,
>
> Thank you all :-) That has sorted out.
>
> The spamfilter.sh file was referring to a log file:
>
> /var/log/spamassassin/spamd.log
>
> The logrotate system was trying to rotate:
>
> /var/log/spamd.log
>
> I've updated spamfilter.sh to point to the right file and mail.log
>has finally settled down to a sensible behaviour.
you should drop the home-brew spamfilter.sh and use glue like
amavisd-milter or similar.
Why at all does spamfilter.sh dump output contents of log file back to syslog?
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
"Where do you want to go to die?" [Microsoft]
Re: Problem with massive log files
Posted by Jim McLachlan <ji...@oss-ltd.com>.
Hi everyone,
Thank you all :-) That has sorted out.
The spamfilter.sh file was referring to a log file:
/var/log/spamassassin/spamd.log
The logrotate system was trying to rotate:
/var/log/spamd.log
I've updated spamfilter.sh to point to the right file and mail.log has finally
settled down to a sensible behaviour.
Thanks again. I'm not sure where this would have ended up without your help.
It was becoming unrealistic to keep the server going as it was.
http://serverfault.com/questions/842082/spamassassin-logging/842710#842710
Kind regards.
Jim.
On 05/04/17 08:32, Tom Hendrikx wrote:
> Hi,
>
> The thing that immediately caught my eye was the fact that in a line
> such as:
>
> Apr 2 10:31:26 oss2 spamfilter: Sat Oct 15 15:20:22 2016 [2758] info:
> spamd: connection from ip6-localhost [::1]:55708 to port 783, fd 5
>
> There are 2 timestamps, far away from each other. After some pondering,
> my guess is that spamfilter.sh is writing away log lines to a temporary
> file for each delivery, and them spewing them again when handling is
> complete. But there is a bug where spamfilter.sh does not cleanup after
> itself, and new lines are appended to the existing temporary file, and
> then the complete contents of the file are sent to syslog. Thus for each
> single delivery the logging for all messages in the past half year (Oct
> 15 -> Apr x) is sent to syslog.
>
> Please post the full contents of the spamfilter.sh, and examine the
> contents of any temporary files that is it using.
>
> Kind regards,
>
> Tom
>
>
> On 04-04-17 23:09, Jim McLachlan wrote:
>> Hi,
>>
>> I have a problem with the huge amount of messages being logged by
>> spamassassin. I have around 10 active e-mail users on the system, none
>> of whom have any unusual e-mail usage. This is what I've seen in the
>> last 2 hours:
>>
>> $ date
>> Mon 3 Apr 08:00:50 UTC 2017
>>
>> $ ls -l /var/log/mail.log
>> -rw-r----- 1 syslog adm 86370829860 Apr 3 08:00 /var/log/mail.log
>> -rw-r----- 1 syslog adm 331608479025 Apr 2 09:20 /var/log/mail.log.1
>>
>>
>> $ spamassassin --version
>> SpamAssassin version 3.4.1
>> running on Perl version 5.22.1
>>
>> My set up consists of Postfix, Postgrey, Spamassassin, Clam-AV,
>> Amavis-new and Dovecot.
>>
>> When I send an e-mail through the system, it immediately starts
>> churning out a long list of log messages that implies it's checking
>> messages from last October (when I set up the server). It goes through
>> thousands of messages like this and then settles down again until
>> another e-mail is processed.
>>
>> My initial e-mail with attachment didn't work, so an excerpt from
>> the log file can be found here:
>>
>> http://pasted.co/5e546e7a
>>
>> Can someone please explain to me why it's repeating all this work
>> and all these messages for every e-mail that gets processed and what I
>> can do to fix this. I reduced the problem slightly yesterday by
>> preventing all these messages getting logged to syslog at the same time.
>>
>> Kind regards.
>>
>> Jim.
--
James R. McLachlan PGDCCI(Open)
Managing Director
Objective Software Services Ltd.
Web : http://www.oss-ltd.com
Tel : +44 (0)1397 708550
Mob : +44 (0)7971 232717
Fax : +44 (0)7970 117580
e-mail: jim@oss-ltd.com
Objective Software Services Ltd. is a company registered in England and Wales
with company number 2892148.
Registered office: 11 Percy Terrace, Tunbridge Wells, Kent, TN4 9RH
Re: Problem with massive log files
Posted by Tom Hendrikx <to...@whyscream.net>.
Hi,
The thing that immediately caught my eye was the fact that in a line
such as:
Apr 2 10:31:26 oss2 spamfilter: Sat Oct 15 15:20:22 2016 [2758] info:
spamd: connection from ip6-localhost [::1]:55708 to port 783, fd 5
There are 2 timestamps, far away from each other. After some pondering,
my guess is that spamfilter.sh is writing away log lines to a temporary
file for each delivery, and them spewing them again when handling is
complete. But there is a bug where spamfilter.sh does not cleanup after
itself, and new lines are appended to the existing temporary file, and
then the complete contents of the file are sent to syslog. Thus for each
single delivery the logging for all messages in the past half year (Oct
15 -> Apr x) is sent to syslog.
Please post the full contents of the spamfilter.sh, and examine the
contents of any temporary files that is it using.
Kind regards,
Tom
On 04-04-17 23:09, Jim McLachlan wrote:
> Hi,
>
> I have a problem with the huge amount of messages being logged by
> spamassassin. I have around 10 active e-mail users on the system, none
> of whom have any unusual e-mail usage. This is what I've seen in the
> last 2 hours:
>
> $ date
> Mon 3 Apr 08:00:50 UTC 2017
>
> $ ls -l /var/log/mail.log
> -rw-r----- 1 syslog adm 86370829860 Apr 3 08:00 /var/log/mail.log
> -rw-r----- 1 syslog adm 331608479025 Apr 2 09:20 /var/log/mail.log.1
>
>
> $ spamassassin --version
> SpamAssassin version 3.4.1
> running on Perl version 5.22.1
>
> My set up consists of Postfix, Postgrey, Spamassassin, Clam-AV,
> Amavis-new and Dovecot.
>
> When I send an e-mail through the system, it immediately starts
> churning out a long list of log messages that implies it's checking
> messages from last October (when I set up the server). It goes through
> thousands of messages like this and then settles down again until
> another e-mail is processed.
>
> My initial e-mail with attachment didn't work, so an excerpt from
> the log file can be found here:
>
> http://pasted.co/5e546e7a
>
> Can someone please explain to me why it's repeating all this work
> and all these messages for every e-mail that gets processed and what I
> can do to fix this. I reduced the problem slightly yesterday by
> preventing all these messages getting logged to syslog at the same time.
>
> Kind regards.
>
> Jim.