You are viewing a plain text version of this content. The canonical link for it is here.
Posted to sysadmins@spamassassin.apache.org by Bill Cole <sa...@billmail.scconsult.com> on 2024/07/07 16:55:59 UTC
Re: Rule update broken
On 2024-07-07 at 10:49:41 UTC-0400 (Sun, 07 Jul 2024 10:49:41 -0400)
Bill Cole <sa...@billmail.scconsult.com>
is rumored to have said:
> Does anyone know a proximal cause for all the error messages coming
> out of the mirror check and updates jobs? I don't want to interfere
> with anything anyone else was trying to do, but it needs fixing.
I've looked, and all I can see that seems odd is multiple "OOM killer"
events apparently associated with heavy volume on the RuleQA site. First
cut log analysis says these are coming out of 2 ranges of the Huawei
Singapore Cloud. But it is hard to say what's "normal" volume, given the
short log life.
I continue to investigate.
--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo@toad.social and many *@billmail.scconsult.com
addresses)
Not Currently Available For Hire
Re: Rule update broken
Posted by John Hardin <jh...@impsec.org>.
On Sun, 7 Jul 2024, Bill Cole wrote:
> On 2024-07-07 at 13:56:54 UTC-0400 (Sun, 07 Jul 2024 13:56:54 -0400)
> Bill Cole <sy...@spamassassin.apache.org>
> is rumored to have said:
>
>> On 2024-07-07 at 12:55:59 UTC-0400 (Sun, 07 Jul 2024 12:55:59 -0400)
>> Bill Cole <sy...@spamassassin.apache.org>
>> is rumored to have said:
>>
>>> On 2024-07-07 at 10:49:41 UTC-0400 (Sun, 07 Jul 2024 10:49:41 -0400)
>>> Bill Cole <sa...@billmail.scconsult.com>
>>> is rumored to have said:
>>>
>> [...]
>>> I continue to investigate.
>>
>> Interim update:
>
> Looks like a Polish machine at 93.105.184.179 which is also a node in the
> "nymaim" botnet (or a proxy serving multiple miscreants) is to blame for the
> largest part of the excess traffic on RuleQA, apparently crawling the whole
> history of rule test results down to the rule detail level.
>
> Blocking that IP for now.
I was doing some rule dev and testing yesterday and watched the
ruleQA website collapse in real time. Didn't have a chance to
troubleshoot it, sorry.
I can confirm that the ruleQA website is again responding properly.
Thanks, Bill!
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Joan Peterson is like that: you expect at least a pseudological
argument, but instead you get the weird ramblings of a woman with
the critical thinking abilities of an 18th century peasant. -- Ken
-----------------------------------------------------------------------
Today: Robert Heinlein's 117th birthday
Re: Rule update broken
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 2024-07-07 at 14:23:34 UTC-0400 (Sun, 07 Jul 2024 14:23:34 -0400)
Bill Cole <sy...@spamassassin.apache.org>
is rumored to have said:
> On 2024-07-07 at 13:56:54 UTC-0400 (Sun, 07 Jul 2024 13:56:54 -0400)
> Bill Cole <sy...@spamassassin.apache.org>
> is rumored to have said:
>
>> On 2024-07-07 at 12:55:59 UTC-0400 (Sun, 07 Jul 2024 12:55:59 -0400)
>> Bill Cole <sy...@spamassassin.apache.org>
>> is rumored to have said:
>>
>>> On 2024-07-07 at 10:49:41 UTC-0400 (Sun, 07 Jul 2024 10:49:41 -0400)
>>> Bill Cole <sa...@billmail.scconsult.com>
>>> is rumored to have said:
>>>
>> [...]
>>> I continue to investigate.
>>
>> Interim update:
>
> Looks like a Polish machine at 93.105.184.179 which is also a node in
> the "nymaim" botnet (or a proxy serving multiple miscreants) is to
> blame for the largest part of the excess traffic on RuleQA, apparently
> crawling the whole history of rule test results down to the rule
> detail level.
>
> Blocking that IP for now.
I did not do so, as a closer look determined that dozens of others
contributed to the burden, and whack-a-mole is not sustainable.
The total hit rate jumped about 4X around 2024-07-05 16:00 UTC. Excess
hits came from a broad collection of IPs and were predominantly on
single rule details but also various other deep links. The all run the
ruleqa.cgi Perl script and that may decompress old archives using pigz.
This combined process burden used enough memory to trigger multiple
OOM-killer events under the added demands of rule updates. The hit rate
dropped again today around 16:00 UTC.
I wasn't able to get into the VM (connect, key exchange, then nothing)
until after the stress started to resolve and load average was still
over 255 when I logged in.
All I've done is to add entries for rule-following but useless bots to
robots.txt for the main site, and I'm not sure what more would be
useful. This looks like a carefully structured DoS attack, distributed
at the network layer but clearly coordinated at the ultimate source by
someone who had looked at the ruleqa.cgi code. There are not any usable
invariants in the requests that could be used to block them, unless we
want to start blocking cheap VPS provider blocks in bulk.
Anyway: looks OK now.
--
Bill Cole
Re: Rule update broken
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 2024-07-07 at 13:56:54 UTC-0400 (Sun, 07 Jul 2024 13:56:54 -0400)
Bill Cole <sy...@spamassassin.apache.org>
is rumored to have said:
> On 2024-07-07 at 12:55:59 UTC-0400 (Sun, 07 Jul 2024 12:55:59 -0400)
> Bill Cole <sy...@spamassassin.apache.org>
> is rumored to have said:
>
>> On 2024-07-07 at 10:49:41 UTC-0400 (Sun, 07 Jul 2024 10:49:41 -0400)
>> Bill Cole <sa...@billmail.scconsult.com>
>> is rumored to have said:
>>
> [...]
>> I continue to investigate.
>
> Interim update:
Looks like a Polish machine at 93.105.184.179 which is also a node in
the "nymaim" botnet (or a proxy serving multiple miscreants) is to blame
for the largest part of the excess traffic on RuleQA, apparently
crawling the whole history of rule test results down to the rule detail
level.
Blocking that IP for now.
--
Bill Cole
Re: Rule update broken
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 2024-07-07 at 12:55:59 UTC-0400 (Sun, 07 Jul 2024 12:55:59 -0400)
Bill Cole <sy...@spamassassin.apache.org>
is rumored to have said:
> On 2024-07-07 at 10:49:41 UTC-0400 (Sun, 07 Jul 2024 10:49:41 -0400)
> Bill Cole <sa...@billmail.scconsult.com>
> is rumored to have said:
>
[...]
> I continue to investigate.
Interim update:
Some of the excess traffic on the main site was from various sham
"search engine" bots, some of which seem to be looking for every file in
the Sun Java api reference docs. At least 2 (ahrefs and mj12) I know
from direct experience honor exclusion via robots.txt, so I've added
them and others with similar behaviors to robots.txt on the main site.
As far as I have been able to find so far, the ruleqa site already has a
robots.txt disallowing all bots.
--
Bill Cole