You are viewing a plain text version of this content. The canonical link for it is here.
Posted to sysadmins@spamassassin.apache.org by Dave Jones <da...@apache.org> on 2017/11/12 14:21:05 UTC
Re: ruleqa.cgi issue
On 11/11/2017 11:09 AM, Merijn van den Kroonenberg wrote:
>> On 11/11/2017 07:45 AM, Merijn van den Kroonenberg wrote:
>>> I saw you applied my patch for ruleqa.cgi and if I read the apache conf
>>> correctly, then the webserver serves pages directly from the checkout
>>> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>>>
> Damn stupid me.
> As I explained above the webserver serves directly from
> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>
> Thats the *svn/masses* !
>
I am so glad you are helping and able to keep this stuff straight. I
worked on this stuff a lot back in the summer and have forgotten a lot
of what I did. :) I meant to have a cron job setup for the automc user
to do an "svn up ~/svn/masses" but it wasn't there until a few minutes
ago. I had patched the ruleqa.cgi and committed it but the "svn up
~/svn/masses" was needed. I just did this part so it's now active.
Sorry about that.
>>> So the change should have gone live immediately if you applied the patch
>>> directly in svn/trunk/masses (and not in svn/masses).
> Thats the wrong way around! Actually for the change to go live it should
> be patched in
> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>
> But thats in the readonly working copy. I just verified in the resync
> account and indeed the ruleqa.cgi has not been patched so my change did
> NOT go live.
It should be live now. I manually verified the file at that path.
http://ruleqa.spamassassin.org/?longdatelist=1
So far good news on my latest patch to the SVN REVISION detection to
find the majority REVISION instead of the latetest REVISION.
llanga does stand out as being off a bit. It seems to be a day behind
at least by the date stamp. Do I need to put in a temporary cleanup for
this masschecker to toss it out?
>>> But it seems the patch had not the hoped for effect so I guess I have to
>>> keep looking then ;)
>>> Please keep in this change for now.
>>>
>> Ok
>>
>>>> On 11/10/2017 05:00 AM, Merijn van den Kroonenberg wrote:
>>>>>> I need some help understanding this problem now. The "promotions
>>>>>> validated" have failed for 3 weeks now so the active.list is not
>>>>>> updating. There seems to be some relationship to the masscheck
>>>>>> ruleqa
>>>>>> web output that is stuck on "day 2" failing. See the output at the
>>>>>> very
>>>>>> bottom.
>>>>> I am trying to see whats going on. I would really like some insight in
>>>>> the
>>>>> file:
>>>>> /usr/local/spamassassin/automc/html/ruleqa.cache
>>>>>
>>>>> And i would like to have some insight into which masscheck results are
>>>>> present on disk. So the output of something like this:
>>>>>
>>>>> ls -la /usr/local/spamassassin/automc/html/2017*
>>>>>
>>>>> ls -la /usr/local/spamassassin/automc/html/20171*/*
>>>>>
>>>>> or something which gives me a tree view.
>>>>>
>>>>>> Can someone look at the perl script "listpromotable" and figure out
>>>>>> what
>>>>>> is going on? I can tell it's looking at the output by day to find
>>>>>> "mcviewing" and "mcsubmitters".
>>>>> Would you say:
>>>>> http://ruleqa.spamassassin.org/?longdatelist=1
>>>>> those are really all masschecks? This seems unlikely to me, I would
>>>>> expect
>>>>> to see masschecks with many contributors much more often (every day?)
>>>>>
>>>>>
>>>>>> This is day 1 match which is green when you view it in a browser:
>>>>>>
>>>>>> <td class="daterevtd mcviewing" width='20%'>
>>>>>> <span class="daterev_masscheck_description mcviewing">
>>>>>> <p>
>>>>>> <a name="r20171106_r1814390_n"
>>>>>> href="/20171106-r1814390-n?xml=1"><strong>
>>>>>> <span class="dr">20171106-r1814390-n</span>
>>>>>> </strong></a> <b>(Viewing)</b>
>>>>>> </p><p>
>>>>>> <em><span class="mcsubmitters">axb-coi-bulk axb-generic
>>>>>> axb-ham-misc darxus ena-week0 ena-week1 ena-week2 ena-week3 giovanni
>>>>>> jarif jbrooks llanga mmiroslaw-mails-ham mmiroslaw-mails-spam
>>>>>> thendrikx</span></em>
>>>>>> </x>
>>>>>> </p>
>>>>>>
>>>>>> Day 2 doesn't have that table with "mcviewing". The next question is
>>>>>> what is causing this problem. Is it related to new commits that
>>>>>> throw
>>>>>> off the masscheck processing?
>>>>> The 2 days ago doesn't highlight a current masscheck....but still it
>>>>> shows
>>>>> a result at the bottom...so its showing *something*. I think its
>>>>> likely
>>>>> it
>>>>> is the masxcheck as present in the datrev input field:
>>>>> 20171108-r1814560-n
>>>>> But that one isn't in any daterev liting, not even in the full
>>>>> listing.
>>>>>
>>>>> So i think something in the ruleqa.cgi which builds the daterev list
>>>>> is
>>>>> broken and leaves out some masschecks.
>>>>> If I get the cachefile and the ddirectory listings I can go debug
>>>>> where
>>>>> things go pear-shaped.
>>>>>
>>>>
Re: ruleqa.cgi issue
Posted by Dave Jones <da...@apache.org>.
On 11/12/2017 09:22 AM, Merijn van den Kroonenberg wrote:
>> On 11/11/2017 11:09 AM, Merijn van den Kroonenberg wrote:
>>>> On 11/11/2017 07:45 AM, Merijn van den Kroonenberg wrote:
>>>>> I saw you applied my patch for ruleqa.cgi and if I read the apache
>>>>> conf
>>>>> correctly, then the webserver serves pages directly from the checkout
>>>>> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>>>>>
>>> Damn stupid me.
>>> As I explained above the webserver serves directly from
>>> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>>>
>>> Thats the *svn/masses* !
>>>
>> I am so glad you are helping and able to keep this stuff straight. I
>> worked on this stuff a lot back in the summer and have forgotten a lot
>> of what I did. :) I meant to have a cron job setup for the automc user
>> to do an "svn up ~/svn/masses" but it wasn't there until a few minutes
>> ago. I had patched the ruleqa.cgi and committed it but the "svn up
>> ~/svn/masses" was needed. I just did this part so it's now active.Â
>> Sorry about that.
> No problem, it hard, like you saw above, even tho I analyzed the
> situation, I still managed to mess up my own conclusion the first time ;)
>
>>>>> So the change should have gone live immediately if you applied the
>>>>> patch
>>>>> directly in svn/trunk/masses (and not in svn/masses).
>>> Thats the wrong way around! Actually for the change to go live it should
>>> be patched in
>>> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>>>
>>> But thats in the readonly working copy. I just verified in the resync
>>> account and indeed the ruleqa.cgi has not been patched so my change did
>>> NOT go live.
>> It should be live now. I manually verified the file at that path.
>>
>> http://ruleqa.spamassassin.org/?longdatelist=1
> Yes I see, and the patch indeed fixes the problem with ruleqa. I think the
> 2-days-ago pages will now work as expected.
>
>> So far good news on my latest patch to the SVN REVISION detection to
>> find the majority REVISION instead of the latetest REVISION.
> But why did last night (net masscheck) fail, it says not enough
> contributors, but when you look at the corpus there seem to be enough
> contributors? Or were some late?
The logic that was in place in March required a minimum of 10
contributors for ham and spam. I added a "force" arg to work with 8
each to help move things along. I thought we were easily getting 15
good contributors so a few days ago I removed the "force" option from
the cron entry. I noticed this about 5 hours ago and put the "force"
option back. I know, I know. I shouldn't be changing things like this
right now. :)
>
>> llanga does stand out as being off a bit. It seems to be a day behind
>> at least by the date stamp. Do I need to put in a temporary cleanup for
>> this masschecker to toss it out?
> I don't think the problem is llanga itself, I suspect its some of our
> scripting messing up.
>
> See:
> http://ruleqa.spamassassin.org/20171106-r1814390-n
>
> It has:
>
> 20171106-r1814390-n (Viewing)
> axb-coi-bulk axb-generic axb-ham-misc darxus ena-week0 ena-week1 ena-week2
> ena-week3 giovanni jarif jbrooks llanga mmiroslaw-mails-ham
> mmiroslaw-mails-spam thendrikx
>
> and
>
> 20171107-r1814390-n
> axb-coi-bulk axb-generic axb-ham-misc jarif llanga
>
> So It has the same revision (1814390) masschecked to two consecutive days,
> BUT all contributors in 20171107 are also present in 20171106 and in this
> case its not only llanga.
I would think the tagged version in SVN that gets put in the staged
rsync directory would be exactly what is reported in the ruleqa.cgi but
the date must be tied to the local run timestamp. This might be
something else we need to fix after we get this going "good enough." I
would like to open bug issues to track this kind of problem once we get
things back to working how they did in March.
>
> So what triggers them to masscheck the same revision again, the next day?
> But they also masscheck the newer revision later that same day.
> What triggers the masscheck? (I don't know anything about that part yet)
>
>
1. I think your ruleqa.cgi patch is helping so the rule promotion script
should work tomorrow.
2. The masscheck processing is supposed to be cron'd to run just after 9
AM UTC as closely as possible by each masschecker but some choose to run
it much later for different reasons like the cost of electricity or
spare computing capacity.
Re: ruleqa.cgi issue
Posted by Merijn van den Kroonenberg <me...@web2all.nl>.
> On 11/11/2017 11:09 AM, Merijn van den Kroonenberg wrote:
>>> On 11/11/2017 07:45 AM, Merijn van den Kroonenberg wrote:
>>>> I saw you applied my patch for ruleqa.cgi and if I read the apache
>>>> conf
>>>> correctly, then the webserver serves pages directly from the checkout
>>>> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>>>>
>> Damn stupid me.
>> As I explained above the webserver serves directly from
>> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>>
>> Thats the *svn/masses* !
>>
>
> I am so glad you are helping and able to keep this stuff straight. I
> worked on this stuff a lot back in the summer and have forgotten a lot
> of what I did. :) I meant to have a cron job setup for the automc user
> to do an "svn up ~/svn/masses" but it wasn't there until a few minutes
> ago. I had patched the ruleqa.cgi and committed it but the "svn up
> ~/svn/masses" was needed. I just did this part so it's now active.Â
> Sorry about that.
No problem, it hard, like you saw above, even tho I analyzed the
situation, I still managed to mess up my own conclusion the first time ;)
>
>>>> So the change should have gone live immediately if you applied the
>>>> patch
>>>> directly in svn/trunk/masses (and not in svn/masses).
>> Thats the wrong way around! Actually for the change to go live it should
>> be patched in
>> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>>
>> But thats in the readonly working copy. I just verified in the resync
>> account and indeed the ruleqa.cgi has not been patched so my change did
>> NOT go live.
>
> It should be live now. I manually verified the file at that path.
>
> http://ruleqa.spamassassin.org/?longdatelist=1
Yes I see, and the patch indeed fixes the problem with ruleqa. I think the
2-days-ago pages will now work as expected.
>
> So far good news on my latest patch to the SVN REVISION detection to
> find the majority REVISION instead of the latetest REVISION.
But why did last night (net masscheck) fail, it says not enough
contributors, but when you look at the corpus there seem to be enough
contributors? Or were some late?
>
> llanga does stand out as being off a bit. It seems to be a day behind
> at least by the date stamp. Do I need to put in a temporary cleanup for
> this masschecker to toss it out?
I don't think the problem is llanga itself, I suspect its some of our
scripting messing up.
See:
http://ruleqa.spamassassin.org/20171106-r1814390-n
It has:
20171106-r1814390-n (Viewing)
axb-coi-bulk axb-generic axb-ham-misc darxus ena-week0 ena-week1 ena-week2
ena-week3 giovanni jarif jbrooks llanga mmiroslaw-mails-ham
mmiroslaw-mails-spam thendrikx
and
20171107-r1814390-n
axb-coi-bulk axb-generic axb-ham-misc jarif llanga
So It has the same revision (1814390) masschecked to two consecutive days,
BUT all contributors in 20171107 are also present in 20171106 and in this
case its not only llanga.
So what triggers them to masscheck the same revision again, the next day?
But they also masscheck the newer revision later that same day.
What triggers the masscheck? (I don't know anything about that part yet)