You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Michael Scheidell <sc...@secnap.net> on 2009/05/14 18:45:02 UTC
ever resolve sa-compile taking 45mins issue?
Last time in frostbite falls, our intrepid heroes were faced with a big
question:
why does sa-compile take 5 mins on some boxes and 45mins on other.? same
cpu, same os, same ram, same load.
I think we all came to the conclustion that
sa rules (700 of them)? by themselves might take 5/10 mins to compile
Adding in sars (3000 rules), might add in 2 mins.
adding in sought rules (2500 lines) takes that 5 to 10 mins compile and
make it take 45 mins.
(so, if you are updating sought rules every 4 hours, 25% of that time is
taken up downloading and compiling rules)
anyone else see the same thing?
anyone else see something totally different?
anyone know why? and how to mitigate it?
one more thing.. I think it got worse when we went from rec2 0.12.* to
0.13.x
--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
> *| *SECNAP Network Security Corporation
* Certified SNORT Integrator
* 2008-9 Hot Company Award Winner, World Executive Alliance
* Five-Star Partner Program 2009, VARBusiness
* Best Anti-Spam Product 2008, Network Products Guide
* King of Spam Filters, SC Magazine 2008
_________________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
_________________________________________________________________________
Re: ever resolve sa-compile taking 45mins issue?
Posted by John Hardin <jh...@impsec.org>.
On Fri, 15 May 2009, Michael Scheidell wrote:
> any idea why the sought rules affect it so much?
Caching doesn't help if the ruleset changes _every time_ you try to
recompile it...
Justin, did you see my suggestion that the base string caching should also
include checking the sa-update directories, and that the sought generator
(et. al.) should also do a base-string extract and include the results in
the update? If base string extraction is a major factor in sa-compile
performance, and the results as based solely on the rule REs, then it
makes a _lot_ of sense to ship pre-extracted base strings out with the
maintained rules...
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
It is not the place of government to make right every tragedy and
woe that befalls every resident of the nation.
-----------------------------------------------------------------------
6 days until the 5th anniversary of SpaceshipOne winning the X-prize
Re: ever resolve sa-compile taking 45mins issue?
Posted by Michael Scheidell <sc...@secnap.net>.
Justin Mason wrote:
> On Fri, May 15, 2009 at 11:47, Michael Scheidell <sc...@secnap.net> wrote:
>
>> Justin Mason wrote:
>>
>> no -- I mean it was released in 3.2.4 ;
>>
>> Im running 3.2.5 and it still takes 45 mins to compile the sought rules.
>>
>
> in that case, the patch isn't helping you, and attempting to reapply
> it won't, either.
>
>
any idea why the sought rules affect it so much?
looks like if you do lots of
(this | that | the other | more | of | these| than | can | fit | on |
one | line)
seems to affect sacompile time (sought rules, and what was other KAM.cf?
--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
> *| *SECNAP Network Security Corporation
* Certified SNORT Integrator
* 2008-9 Hot Company Award Winner, World Executive Alliance
* Five-Star Partner Program 2009, VARBusiness
* Best Anti-Spam Product 2008, Network Products Guide
* King of Spam Filters, SC Magazine 2008
_________________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
_________________________________________________________________________
Re: ever resolve sa-compile taking 45mins issue?
Posted by Justin Mason <jm...@jmason.org>.
On Fri, May 15, 2009 at 11:47, Michael Scheidell <sc...@secnap.net> wrote:
>
>
> Justin Mason wrote:
>
> no -- I mean it was released in 3.2.4 ;
>
> Im running 3.2.5 and it still takes 45 mins to compile the sought rules.
in that case, the patch isn't helping you, and attempting to reapply
it won't, either.
> (so, why does the bug say 'resolved' and not closed?
that's how we do it in SA-land.
> and why does SOME of
> the patch apply?
> chunks 1,2, 5,6 and 7 seemed to apply.
sometimes "patch" will attempt to apply a patch, producing duplicated
or broken code. it's not
going to work in this case since the code is already in there.
--j.
Re: ever resolve sa-compile taking 45mins issue?
Posted by Michael Scheidell <sc...@secnap.net>.
Justin Mason wrote:
> no -- I mean it was released in 3.2.4 ;)
>
>
Im running 3.2.5 and it still takes 45 mins to compile the sought rules.
(so, why does the bug say 'resolved' and not closed? and why does SOME
of the patch apply?
chunks 1,2, 5,6 and 7 seemed to apply.
--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
> *| *SECNAP Network Security Corporation
* Certified SNORT Integrator
* 2008-9 Hot Company Award Winner, World Executive Alliance
* Five-Star Partner Program 2009, VARBusiness
* Best Anti-Spam Product 2008, Network Products Guide
* King of Spam Filters, SC Magazine 2008
_________________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
_________________________________________________________________________
Re: ever resolve sa-compile taking 45mins issue?
Posted by Justin Mason <jm...@jmason.org>.
no -- I mean it was released in 3.2.4 ;)
On Fri, May 15, 2009 at 10:40, Michael Scheidell <sc...@secnap.net> wrote:
>
>
> Justin Mason wrote:
>
> this has been done ages ago.
> https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5594
>
>
>
> maybe we can get it committed to 3.2.6?
>
> patch has been out there 16 months and 4 weeks.
>
> best I can tell, from all the notes, we need to apply this patch to 3.2.x
> branch, right?
> (only this one, right?)
>
> <https://issues.apache.org/SpamAssassin/attachment.cgi?id=4132>
>
> might need updating for 3.2.5: (applied it to perl/lib, not source?)
>
> Hunk #3 failed at 115.
> Hunk #4 failed at 163.
> Hunk #5 succeeded at 225 (offset 1 line).
> Hunk #6 succeeded at 271 (offset 1 line).
> Hunk #7 succeeded at 885 (offset 5 lines).
>
> --
> Michael Scheidell, CTO
> Phone: 561-999-5000, x 1259
>> | SECNAP Network Security Corporation
>
> Certified SNORT Integrator
> 2008-9 Hot Company Award Winner, World Executive Alliance
> Five-Star Partner Program 2009, VARBusiness
> Best Anti-Spam Product 2008, Network Products Guide
> King of Spam Filters, SC Magazine 2008
>
> ________________________________
>
> This email has been scanned and certified safe by SpammerTrap®.
> For Information please see www.secnap.com/products/spammertrap/
>
> ________________________________
>
Re: ever resolve sa-compile taking 45mins issue?
Posted by Michael Scheidell <sc...@secnap.net>.
Justin Mason wrote:
>
>
> this has been done ages ago.
> https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5594
>
>
maybe we can get it committed to 3.2.6?
patch has been out there 16 months and 4 weeks.
best I can tell, from all the notes, we need to apply this patch to
3.2.x branch, right?
(only this one, right?)
<https://issues.apache.org/SpamAssassin/attachment.cgi?id=4132>
might need updating for 3.2.5: (applied it to perl/lib, not source?)
Hunk #3 failed at 115.
Hunk #4 failed at 163.
Hunk #5 succeeded at 225 (offset 1 line).
Hunk #6 succeeded at 271 (offset 1 line).
Hunk #7 succeeded at 885 (offset 5 lines).
--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
> *| *SECNAP Network Security Corporation
* Certified SNORT Integrator
* 2008-9 Hot Company Award Winner, World Executive Alliance
* Five-Star Partner Program 2009, VARBusiness
* Best Anti-Spam Product 2008, Network Products Guide
* King of Spam Filters, SC Magazine 2008
_________________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
_________________________________________________________________________
Re: ever resolve sa-compile taking 45mins issue?
Posted by Justin Mason <jm...@jmason.org>.
On Thu, May 14, 2009 at 22:44, John Hardin <jh...@impsec.org> wrote:
> On Thu, 14 May 2009, John Hardin wrote:
>
>> On Thu, 14 May 2009, Michael Scheidell wrote:
>>
>>> Id LOVE to get back to 15 min compiles.
>>
>> I'm not familiar with the details of RE compilation - would it be possible
>> for Justing to have his bot compile the sought rulesets after generating
>> them and include that in the update? Or do all of the rules have to be
>> compiled together?
>
> Ah. This has come up (sort of) before...
>
> http://www.mailinglistarchive.com/users@spamassassin.apache.org/msg11754.html
>
> Perhaps now's the time to crack sa-compile's chest and add base string
> caching - ideally in a way that allows the base strings to be part of the
> update with the .cf files?
this has been done ages ago.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5594
--j.
Re: ever resolve sa-compile taking 45mins issue?
Posted by John Hardin <jh...@impsec.org>.
On Thu, 14 May 2009, John Hardin wrote:
> On Thu, 14 May 2009, Michael Scheidell wrote:
>
>> Id LOVE to get back to 15 min compiles.
>
> I'm not familiar with the details of RE compilation - would it be possible
> for Justing to have his bot compile the sought rulesets after generating them
> and include that in the update? Or do all of the rules have to be compiled
> together?
Ah. This has come up (sort of) before...
http://www.mailinglistarchive.com/users@spamassassin.apache.org/msg11754.html
Perhaps now's the time to crack sa-compile's chest and add base string
caching - ideally in a way that allows the base strings to be part of
the update with the .cf files?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Usually Microsoft doesn't develop products, we buy products.
-- Arno Edelmann, Microsoft product manager
-----------------------------------------------------------------------
7 days until the 5th anniversary of SpaceshipOne winning the X-prize
Re: ever resolve sa-compile taking 45mins issue?
Posted by Michael Scheidell <sc...@secnap.net>.
and on different processors and on different platforms, and with
different optimizations.
rec2 creates c object libraries.
needs to be platform, os, os version, processor dependent.
so, no, he can't.
John Hardin wrote:
> On Thu, 14 May 2009, Michael Scheidell wrote:
>
>> Id LOVE to get back to 15 min compiles.
>
> I'm not familiar with the details of RE compilation - would it be
> possible for Justing to have his bot compile the sought rulesets after
> generating them and include that in the update? Or do all of the rules
> have to be compiled together?
>
--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
> *| *SECNAP Network Security Corporation
* Certified SNORT Integrator
* 2008-9 Hot Company Award Winner, World Executive Alliance
* Five-Star Partner Program 2009, VARBusiness
* Best Anti-Spam Product 2008, Network Products Guide
* King of Spam Filters, SC Magazine 2008
_________________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
_________________________________________________________________________
Re: ever resolve sa-compile taking 45mins issue?
Posted by John Hardin <jh...@impsec.org>.
On Thu, 14 May 2009, Michael Scheidell wrote:
> Id LOVE to get back to 15 min compiles.
I'm not familiar with the details of RE compilation - would it be possible
for Justing to have his bot compile the sought rulesets after generating
them and include that in the update? Or do all of the rules have to be
compiled together?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Usually Microsoft doesn't develop products, we buy products.
-- Arno Edelmann, Microsoft product manager
-----------------------------------------------------------------------
7 days until the 5th anniversary of SpaceshipOne winning the X-prize
Re: ever resolve sa-compile taking 45mins issue?
Posted by Michael Scheidell <sc...@secnap.net>.
Andrews Carl 448 wrote:
> I am having the same problem. I have not figured it out yet but I have a
> question for you on this subject, are you using KAM.cf? If so does your
> sa-compile run any faster if you remove KAM.cf? Mine goes from 45
> minutes with KAM.cf down to 15 minutes which is still a long time
>
>
no, I looked it it, has lots of the same things as sought rules.
Id LOVE to get back to 15 min compiles.
--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
> *| *SECNAP Network Security Corporation
* Certified SNORT Integrator
* 2008-9 Hot Company Award Winner, World Executive Alliance
* Five-Star Partner Program 2009, VARBusiness
* Best Anti-Spam Product 2008, Network Products Guide
* King of Spam Filters, SC Magazine 2008
_________________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
_________________________________________________________________________
RE: ever resolve sa-compile taking 45mins issue?
Posted by Andrews Carl 448 <Ca...@crackerbarrel.com>.
I am having the same problem. I have not figured it out yet but I have a
question for you on this subject, are you using KAM.cf? If so does your
sa-compile run any faster if you remove KAM.cf? Mine goes from 45
minutes with KAM.cf down to 15 minutes which is still a long time
-----Original Message-----
From: Michael Scheidell [mailto:scheidell@secnap.net]
Sent: Thursday, May 14, 2009 11:45 AM
To: SpamAssassin Users List
Subject: ever resolve sa-compile taking 45mins issue?
Last time in frostbite falls, our intrepid heroes were faced with a big
question:
why does sa-compile take 5 mins on some boxes and 45mins on other.? same
cpu, same os, same ram, same load.
I think we all came to the conclustion that sa rules (700 of them)? by
themselves might take 5/10 mins to compile
Adding in sars (3000 rules), might add in 2 mins.
adding in sought rules (2500 lines) takes that 5 to 10 mins compile and
make it take 45 mins.
(so, if you are updating sought rules every 4 hours, 25% of that time is
taken up downloading and compiling rules)
anyone else see the same thing?
anyone else see something totally different?
anyone know why? and how to mitigate it?
one more thing.. I think it got worse when we went from rec2 0.12.* to
0.13.x
--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
> *| *SECNAP Network Security Corporation
* Certified SNORT Integrator
* 2008-9 Hot Company Award Winner, World Executive Alliance
* Five-Star Partner Program 2009, VARBusiness
* Best Anti-Spam Product 2008, Network Products Guide
* King of Spam Filters, SC Magazine 2008
________________________________________________________________________
_
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
________________________________________________________________________
_