You are viewing a plain text version of this content. The canonical link for it is here.
Posted to sysadmins@spamassassin.apache.org by "Kevin A. McGrail" <ke...@mcgrail.com> on 2017/06/02 12:40:45 UTC
Re: Potential problem with do-stable-update-with-scores cron job
I think you are right. And that the system needs a run to create the
files for the scores on the new server.
On 5/30/2017 8:25 PM, Kevin A. McGrail wrote:
> It's quite possible bb does update rule qa. Don't take what I say as absolute... Will look at this more tomorrow.
> Regards,
> KAM
>
> On May 30, 2017 8:15:16 PM EDT, Dave Jones <da...@apache.org> wrote:
>> If you look on the sa-vm1 box in /etc/cron.d/automc you will see all of
>>
>> the cron jobs that I found from the /home/kmcgrail/SAcron mbox. I
>> updated the bottom of the InfraNote2017 wiki page with what these cron
>> jobs do. I think this is the core of the ruleqa process that allows us
>>
>> to validate rules from the submitted corpus and start sa-update working
>>
>> again but the http://ruleqa.spamassassin.org site is not getting
>> updated. This is my next thing to tackle. Now that I know buildbot
>> wasn't involved that will same me a lot of time. I am concerned that
>> where ever this part was running wasn't backed up. I may have to take
>> your advice and post on the dev mailing list to see if anyone knows
>> anything or has some backups somewhere.
>>
>> Dave
>>
>> On 05/30/2017 07:00 PM, Kevin A. McGrail wrote:
>>> Ahh. This was for the quick masschecks for the preflight rules. I
>>> don't think it is used nor is the corpora it is run against
>> maintained.
>>> Is there parts of rule qa that aren't working other than these quick
>>> checks?
>>> Regards,
>>> KAM
>>>
>>> On May 30, 2017 7:50:40 PM EDT, Dave Jones <da...@apache.org> wrote:
>>>
>>> In looking into what updates the RuleQAApp web interface for
>>> http://ruleqa.spamassassin.org, I found this:
>>>
>>> https://wiki.apache.org/spamassassin/PreflightBuildBot
>>>
>>> Something has to do some processing to update the data folder
>> that the
>>> ruleqa.cgi uses for the http://ruleqa.spamassassin site. If we
>> can
>>> simply run the commands/steps from a cron'd script, that would be
>> better
>>> but I am not finding any such script so far. May have to create
>> one.
>>> Buildbot configs are in SVN:
>>>
>>>
>> https://svn.apache.org/repos/asf/spamassassin/trunk/build/buildbot/
>>> Dave
>>>
>>> On 05/30/2017 05:54 PM, Kevin A. McGrail wrote:
>>>
>>> Well, what are we using buildbot for? I'm trying to remember
>>> and it might not be needed. On 5/30/2017 5:08 PM, Dave Jones
>>> wrote:
>>>
>>> I think I have figured out the primary hurdle I was
>>> hitting when I wrote that last email on 5/27. I found
>> some
>>> stuff in the backups the very long and hard way to get a
>>> little closer. Now I am at the buildbot setup and
>>> discovery stage. I installed buildbot on sa-vm1 but there
>>> is a huge change in versions so the old master config
>> file
>>> in SVN has to be converted into the new version which
>> will
>>> take me some time as I have to learn buildbot from
>>> scratch. Dave On 05/30/2017 12:17 PM, Kevin A. McGrail
>> wrote:
>>> On 5/27/2017 3:12 PM, Dave Jones wrote:
>>>
>>> This script:
>>>
>> https://svn.apache.org/repos/asf/spamassassin/trunk/masses/rule-update-score-gen/do-nightly-rescore-example
>>> refers to
>>> '/home/dos/sa-score-gen/nightly-rescore-via-cron'
>>> which is not in any backups under
>>> /usr/local/spamassassin/backups. A few lines down
>>> in the script it refers to 'generate-new-scores'
>>> which exists in:
>>>
>> https://svn.apache.org/repos/asf/spamassassin/trunk/masses/rule-update-score-gen/
>>> It appears that his is a different
>>> 'generate-new-scores' than what might have been
>> in
>>> the missing /home/dos/sa-score-gen since it's
>>> doing some rsync'ing to pull in corpus which
>>> should have been on the same server? Where the
>>> servers separated before? Was
>>> rsync.spamassassin.org
>>> <http://rsync.spamassassin.org> on a different
>> box
>>> than where these cron jobs ran?
>>>
>>> Dave, What machines ran what, splitting and
>>> recombining servers, etc. is too much for me to
>>> remember, sorry. I can try and dig too but it's
>>> confusing to me as well. However, an important thing
>>> is are we running the correct scripts. Since I don't
>>> see /home/dos on spamassassin-vm1 in my backups, I'm
>>> guessing it was on a different server. Regards, KAM
>>>
Re: Potential problem with do-stable-update-with-scores cron job
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
On 6/2/2017 9:10 AM, Dave Jones wrote:
> On the ruleqa list yesterday, John Hardin confirmed that buildbot was
> running before to centrally process the full uploaded corpora. Sigh.
> This is going to be difficult to figure out and get going again so we
> need to focus on the distributed masscheck contributions first which
> is close to being good.
>
> The http://ruleqa.spamassassin.org site is not going to be updating
> until we figure out what was running that.
Understood. Yes, we had system that ran on uploaded corpora. I think
that is significantly less priority than our other rules process.
In effect, it was an rsync upload and masscheck system.
I personally think that sa-vm1 will not be anywhere near fast enough to
handle the load.
Regards,
KAM
Re: Potential problem with do-stable-update-with-scores cron job
Posted by Dave Jones <da...@apache.org>.
On 06/02/2017 07:40 AM, Kevin A. McGrail wrote:
> I think you are right. And that the system needs a run to create the
> files for the scores on the new server.
I ran a manual "force" with 9 masscheck contributors last night so we
have a test file out there that I have asked the ruleqa list for help
with QA'ing the update file before I enable DNS updates for sa-update:
wget http://bbmass.spamassassin.org/updates/1797164.tar.gz
wget http://bbmass.spamassassin.org/updates/1797164.tar.gz.sha1
wget http://bbmass.spamassassin.org/updates/1797164.tar.gz.asc
sa-update -D -v --install=1797164.tar.gz
>
> On 5/30/2017 8:25 PM, Kevin A. McGrail wrote:
>> It's quite possible bb does update rule qa. Don't take what I say as
>> absolute... Will look at this more tomorrow.
>> Regards,
>> KAM
>>
On the ruleqa list yesterday, John Hardin confirmed that buildbot was
running before to centrally process the full uploaded corpora. Sigh.
This is going to be difficult to figure out and get going again so we
need to focus on the distributed masscheck contributions first which is
close to being good.
The http://ruleqa.spamassassin.org site is not going to be updating
until we figure out what was running that.
Dave
>> On May 30, 2017 8:15:16 PM EDT, Dave Jones <da...@apache.org> wrote:
>>> If you look on the sa-vm1 box in /etc/cron.d/automc you will see all of
>>>
>>> the cron jobs that I found from the /home/kmcgrail/SAcron mbox. I
>>> updated the bottom of the InfraNote2017 wiki page with what these cron
>>> jobs do. I think this is the core of the ruleqa process that allows us
>>>
>>> to validate rules from the submitted corpus and start sa-update working
>>>
>>> again but the http://ruleqa.spamassassin.org site is not getting
>>> updated. This is my next thing to tackle. Now that I know buildbot
>>> wasn't involved that will same me a lot of time. I am concerned that
>>> where ever this part was running wasn't backed up. I may have to take
>>> your advice and post on the dev mailing list to see if anyone knows
>>> anything or has some backups somewhere.
>>>
>>> Dave
>>>
>>> On 05/30/2017 07:00 PM, Kevin A. McGrail wrote:
>>>> Ahh. This was for the quick masschecks for the preflight rules. I
>>>> don't think it is used nor is the corpora it is run against
>>> maintained.
>>>> Is there parts of rule qa that aren't working other than these quick
>>>> checks?
>>>> Regards,
>>>> KAM
>>>>
>>>> On May 30, 2017 7:50:40 PM EDT, Dave Jones <da...@apache.org> wrote:
>>>>
>>>> In looking into what updates the RuleQAApp web interface for
>>>> http://ruleqa.spamassassin.org, I found this:
>>>>
>>>> https://wiki.apache.org/spamassassin/PreflightBuildBot
>>>>
>>>> Something has to do some processing to update the data folder
>>> that the
>>>> ruleqa.cgi uses for the http://ruleqa.spamassassin site. If we
>>> can
>>>> simply run the commands/steps from a cron'd script, that would be
>>> better
>>>> but I am not finding any such script so far. May have to create
>>> one.
>>>> Buildbot configs are in SVN:
>>>>
>>> https://svn.apache.org/repos/asf/spamassassin/trunk/build/buildbot/
>>>> Dave
>>>>
>>>> On 05/30/2017 05:54 PM, Kevin A. McGrail wrote:
>>>>
>>>> Well, what are we using buildbot for? I'm trying to remember
>>>> and it might not be needed. On 5/30/2017 5:08 PM, Dave Jones
>>>> wrote:
>>>>
>>>> I think I have figured out the primary hurdle I was
>>>> hitting when I wrote that last email on 5/27. I found
>>> some
>>>> stuff in the backups the very long and hard way to get a
>>>> little closer. Now I am at the buildbot setup and
>>>> discovery stage. I installed buildbot on sa-vm1 but there
>>>> is a huge change in versions so the old master config
>>> file
>>>> in SVN has to be converted into the new version which
>>> will
>>>> take me some time as I have to learn buildbot from
>>>> scratch. Dave On 05/30/2017 12:17 PM, Kevin A. McGrail
>>> wrote:
>>>> On 5/27/2017 3:12 PM, Dave Jones wrote:
>>>>
>>>> This script:
>>> https://svn.apache.org/repos/asf/spamassassin/trunk/masses/rule-update-score-gen/do-nightly-rescore-example
>>>
>>>> refers to
>>>> '/home/dos/sa-score-gen/nightly-rescore-via-cron'
>>>> which is not in any backups under
>>>> /usr/local/spamassassin/backups. A few lines down
>>>> in the script it refers to 'generate-new-scores'
>>>> which exists in:
>>> https://svn.apache.org/repos/asf/spamassassin/trunk/masses/rule-update-score-gen/
>>>
>>>> It appears that his is a different
>>>> 'generate-new-scores' than what might have been
>>> in
>>>> the missing /home/dos/sa-score-gen since it's
>>>> doing some rsync'ing to pull in corpus which
>>>> should have been on the same server? Where the
>>>> servers separated before? Was
>>>> rsync.spamassassin.org
>>>> <http://rsync.spamassassin.org> on a different
>>> box
>>>> than where these cron jobs ran?
>>>>
>>>> Dave, What machines ran what, splitting and
>>>> recombining servers, etc. is too much for me to
>>>> remember, sorry. I can try and dig too but it's
>>>> confusing to me as well. However, an important thing
>>>> is are we running the correct scripts. Since I don't
>>>> see /home/dos on spamassassin-vm1 in my backups, I'm
>>>> guessing it was on a different server. Regards, KAM
>>>>
>