You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by "Daryl C. W. O'Shea" <sp...@dostech.ca> on 2011/03/22 02:57:02 UTC
nightly sa-update updates are being published
We're back above the corpus thresholds for nightly sa-update updates. Yay!
Daryl
-------- Original Message --------
Subject: Output from "cron" command
Date: Sun, 20 Mar 2011 02:55:12 GMT
From: Rule Updates Daemon <up...@spamassassin.zones.apache.org>
To: updatesd@spamassassin.zones.apache.org
Your "cron" job on spamassassin.zones.apache.org
bash
/export/home/updatesd/svn/mkupdates-with-scores/do-stable-update-with-scores
produced the following output:
(set-0 nightly)
HAM: 181190 (150000 required)
SPAM: 161161 (150000 required)
(set-1 weekly)
HAM: 166559 (150000 required)
SPAM: 165409 (150000 required)
Re: nightly sa-update updates are being published
Posted by John Hardin <jh...@impsec.org>.
On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
> On 21/03/2011 10:53 PM, John Hardin wrote:
>> On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
>>
>> > On 21/03/2011 10:34 PM, John Hardin wrote:
>> > > On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
>> > >
>> > > > We're back above the corpus thresholds for nightly sa-update
>> > > > updates.
>> > > > Yay!
>> > >
>> > > And the update that just went out still contains the PILL_PRICE
>> > > rules...
>> > >
>> > > Maybe tomorrow. Dang.
>> >
>> > No, it'll take a week, I think. A side-effect of only doing
>> > net-enabled checks once a week... I think it uses the rules from the
>> > oldest version, but I can't remember without looking.
>>
>> I don't think so. The one that just went out was for commit 1083377 and
>> that was a regular nightly masscheck. If the updates were only for
>
> Ooops... I was looking at 1083705 which is for trunk.
>
>> weekly -net masschecks then how did we get two on consecutive days?
>> (Unless somebody pushed the second...)
>
> There's an update every day, but the set-1 results only change once a week,
> so I think the rules that are published are tied to set-1.... or something
> like that. Basically you're screwed for a week between net-checks if a bad
> rule gets out.
Well, we'll know for sure tomorrow when the update based on a commit after
I removed the PILL_PRICE rules goes out.
Dang.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The third basic rule of firearms safety:
Keep your booger hook off the bang switch!
-----------------------------------------------------------------------
8 days until the M1911 is 100 years old - and still going strong!
Re: nightly sa-update updates are being published
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 21/03/2011 10:53 PM, John Hardin wrote:
> On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
>
>> On 21/03/2011 10:34 PM, John Hardin wrote:
>>> On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
>>>
>>> > We're back above the corpus thresholds for nightly sa-update updates.
>>> > Yay!
>>>
>>> And the update that just went out still contains the PILL_PRICE rules...
>>>
>>> Maybe tomorrow. Dang.
>>
>> No, it'll take a week, I think. A side-effect of only doing
>> net-enabled checks once a week... I think it uses the rules from the
>> oldest version, but I can't remember without looking.
>
> I don't think so. The one that just went out was for commit 1083377 and
> that was a regular nightly masscheck. If the updates were only for
Ooops... I was looking at 1083705 which is for trunk.
> weekly -net masschecks then how did we get two on consecutive days?
> (Unless somebody pushed the second...)
There's an update every day, but the set-1 results only change once a
week, so I think the rules that are published are tied to set-1.... or
something like that. Basically you're screwed for a week between
net-checks if a bad rule gets out.
Daryl
Re: nightly sa-update updates are being published
Posted by John Hardin <jh...@impsec.org>.
On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
> On 21/03/2011 10:34 PM, John Hardin wrote:
>> On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
>>
>> > We're back above the corpus thresholds for nightly sa-update updates.
>> > Yay!
>>
>> And the update that just went out still contains the PILL_PRICE rules...
>>
>> Maybe tomorrow. Dang.
>
> No, it'll take a week, I think. A side-effect of only doing net-enabled
> checks once a week... I think it uses the rules from the oldest version, but
> I can't remember without looking.
I don't think so. The one that just went out was for commit 1083377 and
that was a regular nightly masscheck. If the updates were only for weekly
-net masschecks then how did we get two on consecutive days? (Unless
somebody pushed the second...)
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
If someone has a gun and is trying to kill you, it would be
reasonable to shoot back with your own gun.
-- the Dalai Lama, May 15, 2001
-----------------------------------------------------------------------
8 days until the M1911 is 100 years old - and still going strong!
Re: nightly sa-update updates are being published
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 22/03/2011 9:51 PM, Mark Martinec wrote:
> On Tuesday March 22 2011 04:29:13 Daryl C. W. O'Shea wrote:
>> OK, I've added the three (0) metas to the bottom of 72_active.cf
>> and rolled and pushed an update.
>
> Thanks!
>
> There is also an urgent need for pushing the fixed RCVD_ILLEGAL_IP
> rule in 3.2 branch to its updates - Bug 6552. Its scores are quite
> high (3.199 3.196 2.902 1.908) and cause false positives for more
> recently assigned IPv4 networks - as we read recurring complaints.
>
> The updated rule is in the 3.2 branch, backported unchanged from 3.3
> and trunk.
Hrm. Let me see what I can do. I'll have to refresh my memory on what
state 3.2 is at. There were some big changes to how everything worked,
but I can't remember when.
At worst, I'll build rules from 3.2 branch and publish them. I'd like
to test it against each version automatically... I guess I could do it
manually if I have to.
It'll probably be at least the weekend before I could look at it. I
want to look at re-enabling 3.3 updates first (which probably won't be
until the weekend either).
Daryl
>
> Any chance we could do that? I don't know how.
>
> Mark
Re: nightly sa-update updates are being published
Posted by Mark Martinec <Ma...@ijs.si>.
On Tuesday March 22 2011 04:29:13 Daryl C. W. O'Shea wrote:
> OK, I've added the three (0) metas to the bottom of 72_active.cf
> and rolled and pushed an update.
Thanks!
There is also an urgent need for pushing the fixed RCVD_ILLEGAL_IP
rule in 3.2 branch to its updates - Bug 6552. Its scores are quite
high (3.199 3.196 2.902 1.908) and cause false positives for more
recently assigned IPv4 networks - as we read recurring complaints.
The updated rule is in the 3.2 branch, backported unchanged from 3.3
and trunk.
Any chance we could do that? I don't know how.
Mark
Re: nightly sa-update updates are being published
Posted by "Warren Togami Jr." <wt...@gmail.com>.
On 3/23/2011 6:51 PM, Daryl C. W. O'Shea wrote:
> On 23/03/2011 7:21 AM, Warren Togami Jr. wrote:
>> I am afraid we already need another emergency rule update for 3.3.
>>
>> Please take a look at Bug #6220 and #6560. In the short-term we need to
>> push another emergency rule update to disable all six of those network
>> rules. Then we need to figure out how it managed to ignore the
>> "#testrules" mark within the sandbox file.
>
> We've rolled back to the less problematic update that was live until
> last week. Update 1052462 has been released as update 1083704. It will
> be live in DNS in 7 minutes.
>
> Daryl
This means FS_RU_URL is back, but this is prior to the introduction of
Bug 6558 right?
Warren
Re: nightly sa-update updates are being published
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 23/03/2011 7:21 AM, Warren Togami Jr. wrote:
> I am afraid we already need another emergency rule update for 3.3.
>
> Please take a look at Bug #6220 and #6560. In the short-term we need to
> push another emergency rule update to disable all six of those network
> rules. Then we need to figure out how it managed to ignore the
> "#testrules" mark within the sandbox file.
We've rolled back to the less problematic update that was live until
last week. Update 1052462 has been released as update 1083704. It will
be live in DNS in 7 minutes.
Daryl
Re: nightly sa-update updates are being published
Posted by "Warren Togami Jr." <wt...@gmail.com>.
On 3/21/2011 5:29 PM, Daryl C. W. O'Shea wrote:
> On 21/03/2011 11:08 PM, Karsten Bräckelmann wrote:
>> On Mon, 2011-03-21 at 20:01 -0700, John Hardin wrote:
>>> On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
>>
>>>> Actually... I can't find any sign of PILL_PRICE in the latest
>>>> update. Are you
>>>> sure there's still an issue with it?
>>>
>>> Here's what I did:
>> [...]
>>> If I did that wrong I'll be happy to say "d'oh!".
>>
>> Nope, already confirmed myself.
>>
>> The __PILL_PRICE_x rules are still there, with the tflags multiple.
>
> OK, I've added the three (0) metas to the bottom of 72_active.cf and
> rolled and pushed an update. I didn't test it, because, well, I'm going
> to bed. Hopefully I didn't screw it up.
>
> I've disabled stable branch updates for now. If they are re-enabled
> there's a chance that the fix will be clobbered and the bad version will
> be published again. I'll confirm things are OK later in the week before
> re-enabling.
>
> I think my first step to resolving this "fix a broken update" issue is
> to write a script to just roll-back to a given (manually chosen) update
> revision number, by having it copied and published as a new (higher)
> revision number. It doesn't fix things like the Y2K10 issue we had, but
> it would resolve issues like we just had now.
>
> Daryl
Hi Daryl,
I am afraid we already need another emergency rule update for 3.3.
Please take a look at Bug #6220 and #6560. In the short-term we need to
push another emergency rule update to disable all six of those network
rules. Then we need to figure out how it managed to ignore the
"#testrules" mark within the sandbox file.
Warren Togami
warren@togami.com
Re: nightly sa-update updates are being published
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 21/03/2011 11:08 PM, Karsten Bräckelmann wrote:
> On Mon, 2011-03-21 at 20:01 -0700, John Hardin wrote:
>> On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
>
>>> Actually... I can't find any sign of PILL_PRICE in the latest update. Are you
>>> sure there's still an issue with it?
>>
>> Here's what I did:
> [...]
>> If I did that wrong I'll be happy to say "d'oh!".
>
> Nope, already confirmed myself.
>
> The __PILL_PRICE_x rules are still there, with the tflags multiple.
OK, I've added the three (0) metas to the bottom of 72_active.cf and
rolled and pushed an update. I didn't test it, because, well, I'm going
to bed. Hopefully I didn't screw it up.
I've disabled stable branch updates for now. If they are re-enabled
there's a chance that the fix will be clobbered and the bad version will
be published again. I'll confirm things are OK later in the week before
re-enabling.
I think my first step to resolving this "fix a broken update" issue is
to write a script to just roll-back to a given (manually chosen) update
revision number, by having it copied and published as a new (higher)
revision number. It doesn't fix things like the Y2K10 issue we had, but
it would resolve issues like we just had now.
Daryl
Re: nightly sa-update updates are being published
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Mon, 2011-03-21 at 20:01 -0700, John Hardin wrote:
> On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
> > Actually... I can't find any sign of PILL_PRICE in the latest update. Are you
> > sure there's still an issue with it?
>
> Here's what I did:
[...]
> If I did that wrong I'll be happy to say "d'oh!".
Nope, already confirmed myself.
The __PILL_PRICE_x rules are still there, with the tflags multiple.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: nightly sa-update updates are being published
Posted by John Hardin <jh...@impsec.org>.
On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
> On 21/03/2011 10:39 PM, Daryl C. W. O'Shea wrote:
>> On 21/03/2011 10:34 PM, John Hardin wrote:
>> > On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
>> >
>> > > We're back above the corpus thresholds for nightly sa-update updates.
>> > > Yay!
>> >
>> > And the update that just went out still contains the PILL_PRICE rules...
>> >
>> > Maybe tomorrow. Dang.
>>
>> No, it'll take a week, I think. A side-effect of only doing net-enabled
>> checks once a week... I think it uses the rules from the oldest version,
>> but I can't remember without looking.
>
> Actually... I can't find any sign of PILL_PRICE in the latest update. Are you
> sure there's still an issue with it?
Here's what I did:
jhardin@dendarii ~/develop/spamassassin/sa-updates $ dig +short TXT 2.3.3.updates.spamassassin.org
"1083377"
jhardin@dendarii ~/develop/spamassassin/sa-updates $ wget http://buildbot.spamassassin.org.nyud.net:8090/updatestage/1083377.tar.gz
jhardin@dendarii ~/develop/spamassassin/sa-updates $ tar zxvf 1083377.tar.gz 72_active.cf
72_active.cf
jhardin@dendarii ~/develop/spamassassin/sa-updates $ grep PILL 72_active.cf
##{ MANY_PILL_PRICE
meta MANY_PILL_PRICE (__PILL_PRICE_1 + __PILL_PRICE_2 +
__PILL_PRICE_3) > 2
describe MANY_PILL_PRICE Prices for pills
##} MANY_PILL_PRICE
body __PILL_PRICE_1
m;\$?[\d\s.]{3,8}(?:/|per|each)\s?(?:pill|tablet|cap(?:sule|let));i
tflags __PILL_PRICE_1 multiple
body __PILL_PRICE_2
/(?:pill|tablet|cap(?:sule|let))s\s\$?[\d\s.]{3,8}/i
tflags __PILL_PRICE_2 multiple
body __PILL_PRICE_3
/free\s(?:pill|tablet|cap(?:sule|let))s/i
tflags __PILL_PRICE_3 multiple
If I did that wrong I'll be happy to say "d'oh!".
> I think the actual logic is that it uses the newest set of rules and that
> scores generated for the "older set" (that is, the net set after a non-net
> mass-check night, and vice-versa) are pruned of any rules that are no longer
> in the current set.
>
> So... I'm not going to worry about pushing an update now. Doesn't look like
> there's a need for one.
>
> Daryl
>
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
If someone has a gun and is trying to kill you, it would be
reasonable to shoot back with your own gun.
-- the Dalai Lama, May 15, 2001
-----------------------------------------------------------------------
8 days until the M1911 is 100 years old - and still going strong!
Re: nightly sa-update updates are being published
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 21/03/2011 10:39 PM, Daryl C. W. O'Shea wrote:
> On 21/03/2011 10:34 PM, John Hardin wrote:
>> On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
>>
>>> We're back above the corpus thresholds for nightly sa-update updates.
>>> Yay!
>>
>> And the update that just went out still contains the PILL_PRICE rules...
>>
>> Maybe tomorrow. Dang.
>
> No, it'll take a week, I think. A side-effect of only doing net-enabled
> checks once a week... I think it uses the rules from the oldest version,
> but I can't remember without looking.
Actually... I can't find any sign of PILL_PRICE in the latest update.
Are you sure there's still an issue with it?
I think the actual logic is that it uses the newest set of rules and
that scores generated for the "older set" (that is, the net set after a
non-net mass-check night, and vice-versa) are pruned of any rules that
are no longer in the current set.
So... I'm not going to worry about pushing an update now. Doesn't look
like there's a need for one.
Daryl
Re: nightly sa-update updates are being published
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 21/03/2011 10:34 PM, John Hardin wrote:
> On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
>
>> We're back above the corpus thresholds for nightly sa-update updates.
>> Yay!
>
> And the update that just went out still contains the PILL_PRICE rules...
>
> Maybe tomorrow. Dang.
No, it'll take a week, I think. A side-effect of only doing net-enabled
checks once a week... I think it uses the rules from the oldest version,
but I can't remember without looking.
Daryl
Re: nightly sa-update updates are being published
Posted by John Hardin <jh...@impsec.org>.
On Mon, 21 Mar 2011, Daryl C. W. O'Shea wrote:
> We're back above the corpus thresholds for nightly sa-update updates. Yay!
And the update that just went out still contains the PILL_PRICE rules...
Maybe tomorrow. Dang.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
If someone has a gun and is trying to kill you, it would be
reasonable to shoot back with your own gun.
-- the Dalai Lama, May 15, 2001
-----------------------------------------------------------------------
8 days until the M1911 is 100 years old - and still going strong!