You are viewing a plain text version of this content. The canonical link for it is here.
Posted to sysadmins@spamassassin.apache.org by Dave Jones <da...@apache.org> on 2017/11/07 12:37:07 UTC
Re: Cron ~/svn/trunk/build/mkupdates/run_nightly |
/usr/bin/tee /var/www/automc.spamassassin.org/mkupdates/mkupdates.txt
I need some help understanding this problem now. The "promotions
validated" have failed for 3 weeks now so the active.list is not
updating. There seems to be some relationship to the masscheck ruleqa
web output that is stuck on "day 2" failing. See the output at the very
bottom.
Can someone look at the perl script "listpromotable" and figure out what
is going on? I can tell it's looking at the output by day to find
"mcviewing" and "mcsubmitters".
This is day 1 match which is green when you view it in a browser:
<td class="daterevtd mcviewing" width='20%'>
<span class="daterev_masscheck_description mcviewing">
<p>
<a name="r20171106_r1814390_n"
href="/20171106-r1814390-n?xml=1"><strong>
<span class="dr">20171106-r1814390-n</span>
</strong></a> <b>(Viewing)</b>
</p><p>
<em><span class="mcsubmitters">axb-coi-bulk axb-generic
axb-ham-misc darxus ena-week0 ena-week1 ena-week2 ena-week3 giovanni
jarif jbrooks llanga mmiroslaw-mails-ham mmiroslaw-mails-spam
thendrikx</span></em>
</x>
</p>
Day 2 doesn't have that table with "mcviewing". The next question is
what is causing this problem. Is it related to new commits that throw
off the masscheck processing?
P.S. This cron'd script used to redirect all output to /dev/null but I
changed it to a tee a few days ago so we could easily keep an eye on
this output via emails to this list.
Dave
On 11/07/2017 02:30 AM, Cron Daemon wrote:
> + promote_active_rules
> + pwd
> + /usr/bin/perl build/mkupdates/listpromotable
> /usr/local/spamassassin/automc/svn/trunk
> HTTP get: http://ruleqa.spamassassin.org/1-days-ago?xml=1
> HTTP get: http://ruleqa.spamassassin.org/2-days-ago?xml=1
> no 'mcviewing', 'mcsubmitters' microformats on day 2
> URL: http://ruleqa.spamassassin.org/2-days-ago?xml=1
> + exit 25
>
Re: Cron ~/svn/trunk/build/mkupdates/run_nightly |
/usr/bin/tee /var/www/automc.spamassassin.org/mkupdates/mkupdates.txt
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
Looking for some help for Dave. Anyone?
On 11/7/2017 7:37 AM, Dave Jones wrote:
> I need some help understanding this problem now. The "promotions
> validated" have failed for 3 weeks now so the active.list is not
> updating. There seems to be some relationship to the masscheck ruleqa
> web output that is stuck on "day 2" failing. See the output at the
> very bottom.
>
> Can someone look at the perl script "listpromotable" and figure out
> what is going on? I can tell it's looking at the output by day to
> find "mcviewing" and "mcsubmitters".
>
> This is day 1 match which is green when you view it in a browser:
>
> <td class="daterevtd mcviewing" width='20%'>
> <span class="daterev_masscheck_description mcviewing">
> <p>
> <a name="r20171106_r1814390_n"
> href="/20171106-r1814390-n?xml=1"><strong>
> <span class="dr">20171106-r1814390-n</span>
> </strong></a> <b>(Viewing)</b>
> </p><p>
> <em><span class="mcsubmitters">axb-coi-bulk axb-generic
> axb-ham-misc darxus ena-week0 ena-week1 ena-week2 ena-week3 giovanni
> jarif jbrooks llanga mmiroslaw-mails-ham mmiroslaw-mails-spam
> thendrikx</span></em>
> </x>
> </p>
>
> Day 2 doesn't have that table with "mcviewing". The next question is
> what is causing this problem. Is it related to new commits that throw
> off the masscheck processing?
>
> P.S. This cron'd script used to redirect all output to /dev/null but I
> changed it to a tee a few days ago so we could easily keep an eye on
> this output via emails to this list.
>
> Dave
>
> On 11/07/2017 02:30 AM, Cron Daemon wrote:
>> + promote_active_rules
>> + pwd
>> + /usr/bin/perl build/mkupdates/listpromotable
>> /usr/local/spamassassin/automc/svn/trunk
>> HTTP get: http://ruleqa.spamassassin.org/1-days-ago?xml=1
>> HTTP get: http://ruleqa.spamassassin.org/2-days-ago?xml=1
>> no 'mcviewing', 'mcsubmitters' microformats on day 2
>> URL: http://ruleqa.spamassassin.org/2-days-ago?xml=1
>> + exit 25
>>
Re: ruleqa user llanga
Posted by Dave Jones <da...@apache.org>.
On 11/10/2017 09:01 AM, Merijn van den Kroonenberg wrote:
>>
>>>> Day 2 doesn't have that table with "mcviewing". The next question is
>>>> what is causing this problem. Is it related to new commits that throw
>>>> off the masscheck processing?
>>>
>>> The 2 days ago doesn't highlight a current masscheck....but still it
>>> shows
>>> a result at the bottom...so its showing *something*. I think its likely
>>> it
>>> is the masxcheck as present in the datrev input field:
>>> 20171108-r1814560-n
>>> But that one isn't in any daterev liting, not even in the full listing.
>>>
>>> So i think something in the ruleqa.cgi which builds the daterev list is
>>> broken and leaves out some masschecks.
>>> If I get the cachefile and the ddirectory listings I can go debug where
>>> things go pear-shaped.
>>>
>>
>> I have found one dubious piece of code where the masschecks are indexed
>> based on their svn rev number. But that is not an unique value has the
>> same revision can be masschecked multiple times (by different
>> submitter/date).
>
> I think this is in fact the case.
> There is something weird with masscheck user llanga.
> Either something is off with the timing of masscheck result submission or
> that user submits the masscheck result twice (once more the next day for
> the same revision).
> I think thats what triggers the bug in the ruleqa page.
>
> ls -l html/20171108/r1814560-n/LOGS.all-*-llanga*
> 5356811 Nov 10 01:05 html/20171108/r1814560-n/LOGS.all-ham-llanga.log.gz
> 521798 Nov 10 01:06 html/20171108/r1814560-n/LOGS.all-spam-llanga.log.gz
>
> ls -l html/20171109/r1814560-n/LOGS.all-*-llanga*
> 5356811 Nov 10 08:12 html/20171109/r1814560-n/LOGS.all-ham-llanga.log.gz
> 521798 Nov 10 08:12 html/20171109/r1814560-n/LOGS.all-spam-llanga.log.gz
>
> b14039f7b3ef3329d6bbd80e8a2eb5e04eb62129
> html/20171108/r1814560-n/LOGS.all-ham-llanga.log.gz
> b14039f7b3ef3329d6bbd80e8a2eb5e04eb62129
> html/20171109/r1814560-n/LOGS.all-ham-llanga.log.gz
>
> same checksum so same files.
> The question is, does the user do something wrong or is some scripting
> messed up (maybe related to bad timing or timezone issues).
>
>>
>> Please see attached patch for masses/rulequa/ruleqa.cgi
>
> I think i failed to attach patch correctly but send it directly to dave.
>
>>
>> If this is not it then I suspect code around line 453 which trims some
>> revisions away. But its very hard to read code.
>
>
I think I figured out what was causing problems with the masscheck SVN
revision getting thrown off by commits and llanga. I was determining
the $REVISION in masses/rule-update-score-gen/generate-new-scores.sh
around line 123 by finding the newest SVN revision. I thought the
staging of the rsync dir and the SVN tagged versions would keep that SVN
revision locked in for a 24 hour period. Now I have updated the logic
to find the SVN revision with the most occurrences in all of the corpus
for that particular scoreset type.
It might work best if a tag file was dropped with the SVN revision by
the run_nightly scrip that stages the masscheck area so the
generate-new-scores.sh could be better matched to that SVN revision. If
an SVN command could be used to find the latest sa-update tagged
version, then that could be used instead of a tag file.
--
Dave
Re: ruleqa user llanga
Posted by Dave Jones <da...@apache.org>.
On 11/10/2017 09:01 AM, Merijn van den Kroonenberg wrote:
>>
>>>> Day 2 doesn't have that table with "mcviewing". The next question is
>>>> what is causing this problem. Is it related to new commits that throw
>>>> off the masscheck processing?
>>>
>>> The 2 days ago doesn't highlight a current masscheck....but still it
>>> shows
>>> a result at the bottom...so its showing *something*. I think its likely
>>> it
>>> is the masxcheck as present in the datrev input field:
>>> 20171108-r1814560-n
>>> But that one isn't in any daterev liting, not even in the full listing.
>>>
>>> So i think something in the ruleqa.cgi which builds the daterev list is
>>> broken and leaves out some masschecks.
>>> If I get the cachefile and the ddirectory listings I can go debug where
>>> things go pear-shaped.
>>>
>>
>> I have found one dubious piece of code where the masschecks are indexed
>> based on their svn rev number. But that is not an unique value has the
>> same revision can be masschecked multiple times (by different
>> submitter/date).
>
> I think this is in fact the case.
> There is something weird with masscheck user llanga.
> Either something is off with the timing of masscheck result submission or
> that user submits the masscheck result twice (once more the next day for
> the same revision).
> I think thats what triggers the bug in the ruleqa page.
>
> ls -l html/20171108/r1814560-n/LOGS.all-*-llanga*
> 5356811 Nov 10 01:05 html/20171108/r1814560-n/LOGS.all-ham-llanga.log.gz
> 521798 Nov 10 01:06 html/20171108/r1814560-n/LOGS.all-spam-llanga.log.gz
>
> ls -l html/20171109/r1814560-n/LOGS.all-*-llanga*
> 5356811 Nov 10 08:12 html/20171109/r1814560-n/LOGS.all-ham-llanga.log.gz
> 521798 Nov 10 08:12 html/20171109/r1814560-n/LOGS.all-spam-llanga.log.gz
>
> b14039f7b3ef3329d6bbd80e8a2eb5e04eb62129
> html/20171108/r1814560-n/LOGS.all-ham-llanga.log.gz
> b14039f7b3ef3329d6bbd80e8a2eb5e04eb62129
> html/20171109/r1814560-n/LOGS.all-ham-llanga.log.gz
>
> same checksum so same files.
> The question is, does the user do something wrong or is some scripting
> messed up (maybe related to bad timing or timezone issues).
>
I will look at this closer this evening in 4 or 5 hours. I do see that
this masschecker llanga is standing out on the
http://ruleqa.spamassassin.org pages:
http://ruleqa.spamassassin.org/1-days-ago?xml=1
http://ruleqa.spamassassin.org/2-days-ago?xml=1
http://ruleqa.spamassassin.org/3-days-ago?xml=1
The masscheck processing is supposed to filter out masscheck submissions
that don't match the SVN tagged revision plust some other minimum
requirements but it may not be handling this situation properly.
>>
>> Please see attached patch for masses/rulequa/ruleqa.cgi
>
> I think i failed to attach patch correctly but send it directly to dave.
I have committed and applied your patch to the working area.
>
>>
>> If this is not it then I suspect code around line 453 which trims some
>> revisions away. But its very hard to read code.
>
>
--
Dave
ruleqa user llanga
Posted by Merijn van den Kroonenberg <me...@web2all.nl>.
>
>>> Day 2 doesn't have that table with "mcviewing". The next question is
>>> what is causing this problem. Is it related to new commits that throw
>>> off the masscheck processing?
>>
>> The 2 days ago doesn't highlight a current masscheck....but still it
>> shows
>> a result at the bottom...so its showing *something*. I think its likely
>> it
>> is the masxcheck as present in the datrev input field:
>> 20171108-r1814560-n
>> But that one isn't in any daterev liting, not even in the full listing.
>>
>> So i think something in the ruleqa.cgi which builds the daterev list is
>> broken and leaves out some masschecks.
>> If I get the cachefile and the ddirectory listings I can go debug where
>> things go pear-shaped.
>>
>
> I have found one dubious piece of code where the masschecks are indexed
> based on their svn rev number. But that is not an unique value has the
> same revision can be masschecked multiple times (by different
> submitter/date).
I think this is in fact the case.
There is something weird with masscheck user llanga.
Either something is off with the timing of masscheck result submission or
that user submits the masscheck result twice (once more the next day for
the same revision).
I think thats what triggers the bug in the ruleqa page.
ls -l html/20171108/r1814560-n/LOGS.all-*-llanga*
5356811 Nov 10 01:05 html/20171108/r1814560-n/LOGS.all-ham-llanga.log.gz
521798 Nov 10 01:06 html/20171108/r1814560-n/LOGS.all-spam-llanga.log.gz
ls -l html/20171109/r1814560-n/LOGS.all-*-llanga*
5356811 Nov 10 08:12 html/20171109/r1814560-n/LOGS.all-ham-llanga.log.gz
521798 Nov 10 08:12 html/20171109/r1814560-n/LOGS.all-spam-llanga.log.gz
b14039f7b3ef3329d6bbd80e8a2eb5e04eb62129
html/20171108/r1814560-n/LOGS.all-ham-llanga.log.gz
b14039f7b3ef3329d6bbd80e8a2eb5e04eb62129
html/20171109/r1814560-n/LOGS.all-ham-llanga.log.gz
same checksum so same files.
The question is, does the user do something wrong or is some scripting
messed up (maybe related to bad timing or timezone issues).
>
> Please see attached patch for masses/rulequa/ruleqa.cgi
I think i failed to attach patch correctly but send it directly to dave.
>
> If this is not it then I suspect code around line 453 which trims some
> revisions away. But its very hard to read code.
Re: Cron ~/svn/trunk/build/mkupdates/run_nightly |
/usr/bin/tee /var/www/automc.spamassassin.org/mkupdates/mkupdates.txt
Posted by Merijn van den Kroonenberg <me...@web2all.nl>.
>> Day 2 doesn't have that table with "mcviewing". The next question is
>> what is causing this problem. Is it related to new commits that throw
>> off the masscheck processing?
>
> The 2 days ago doesn't highlight a current masscheck....but still it shows
> a result at the bottom...so its showing *something*. I think its likely it
> is the masxcheck as present in the datrev input field: 20171108-r1814560-n
> But that one isn't in any daterev liting, not even in the full listing.
>
> So i think something in the ruleqa.cgi which builds the daterev list is
> broken and leaves out some masschecks.
> If I get the cachefile and the ddirectory listings I can go debug where
> things go pear-shaped.
>
I have found one dubious piece of code where the masschecks are indexed
based on their svn rev number. But that is not an unique value has the
same revision can be masschecked multiple times (by different
submitter/date).
Please see attached patch for masses/rulequa/ruleqa.cgi
If this is not it then I suspect code around line 453 which trims some
revisions away. But its very hard to read code.
Re: ruleqa.cgi issue
Posted by Dave Jones <da...@apache.org>.
On 11/12/2017 09:22 AM, Merijn van den Kroonenberg wrote:
>> On 11/11/2017 11:09 AM, Merijn van den Kroonenberg wrote:
>>>> On 11/11/2017 07:45 AM, Merijn van den Kroonenberg wrote:
>>>>> I saw you applied my patch for ruleqa.cgi and if I read the apache
>>>>> conf
>>>>> correctly, then the webserver serves pages directly from the checkout
>>>>> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>>>>>
>>> Damn stupid me.
>>> As I explained above the webserver serves directly from
>>> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>>>
>>> Thats the *svn/masses* !
>>>
>> I am so glad you are helping and able to keep this stuff straight. I
>> worked on this stuff a lot back in the summer and have forgotten a lot
>> of what I did. :) I meant to have a cron job setup for the automc user
>> to do an "svn up ~/svn/masses" but it wasn't there until a few minutes
>> ago. I had patched the ruleqa.cgi and committed it but the "svn up
>> ~/svn/masses" was needed. I just did this part so it's now active.Â
>> Sorry about that.
> No problem, it hard, like you saw above, even tho I analyzed the
> situation, I still managed to mess up my own conclusion the first time ;)
>
>>>>> So the change should have gone live immediately if you applied the
>>>>> patch
>>>>> directly in svn/trunk/masses (and not in svn/masses).
>>> Thats the wrong way around! Actually for the change to go live it should
>>> be patched in
>>> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>>>
>>> But thats in the readonly working copy. I just verified in the resync
>>> account and indeed the ruleqa.cgi has not been patched so my change did
>>> NOT go live.
>> It should be live now. I manually verified the file at that path.
>>
>> http://ruleqa.spamassassin.org/?longdatelist=1
> Yes I see, and the patch indeed fixes the problem with ruleqa. I think the
> 2-days-ago pages will now work as expected.
>
>> So far good news on my latest patch to the SVN REVISION detection to
>> find the majority REVISION instead of the latetest REVISION.
> But why did last night (net masscheck) fail, it says not enough
> contributors, but when you look at the corpus there seem to be enough
> contributors? Or were some late?
The logic that was in place in March required a minimum of 10
contributors for ham and spam. I added a "force" arg to work with 8
each to help move things along. I thought we were easily getting 15
good contributors so a few days ago I removed the "force" option from
the cron entry. I noticed this about 5 hours ago and put the "force"
option back. I know, I know. I shouldn't be changing things like this
right now. :)
>
>> llanga does stand out as being off a bit. It seems to be a day behind
>> at least by the date stamp. Do I need to put in a temporary cleanup for
>> this masschecker to toss it out?
> I don't think the problem is llanga itself, I suspect its some of our
> scripting messing up.
>
> See:
> http://ruleqa.spamassassin.org/20171106-r1814390-n
>
> It has:
>
> 20171106-r1814390-n (Viewing)
> axb-coi-bulk axb-generic axb-ham-misc darxus ena-week0 ena-week1 ena-week2
> ena-week3 giovanni jarif jbrooks llanga mmiroslaw-mails-ham
> mmiroslaw-mails-spam thendrikx
>
> and
>
> 20171107-r1814390-n
> axb-coi-bulk axb-generic axb-ham-misc jarif llanga
>
> So It has the same revision (1814390) masschecked to two consecutive days,
> BUT all contributors in 20171107 are also present in 20171106 and in this
> case its not only llanga.
I would think the tagged version in SVN that gets put in the staged
rsync directory would be exactly what is reported in the ruleqa.cgi but
the date must be tied to the local run timestamp. This might be
something else we need to fix after we get this going "good enough." I
would like to open bug issues to track this kind of problem once we get
things back to working how they did in March.
>
> So what triggers them to masscheck the same revision again, the next day?
> But they also masscheck the newer revision later that same day.
> What triggers the masscheck? (I don't know anything about that part yet)
>
>
1. I think your ruleqa.cgi patch is helping so the rule promotion script
should work tomorrow.
2. The masscheck processing is supposed to be cron'd to run just after 9
AM UTC as closely as possible by each masschecker but some choose to run
it much later for different reasons like the cost of electricity or
spare computing capacity.
Re: ruleqa.cgi issue
Posted by Merijn van den Kroonenberg <me...@web2all.nl>.
> On 11/11/2017 11:09 AM, Merijn van den Kroonenberg wrote:
>>> On 11/11/2017 07:45 AM, Merijn van den Kroonenberg wrote:
>>>> I saw you applied my patch for ruleqa.cgi and if I read the apache
>>>> conf
>>>> correctly, then the webserver serves pages directly from the checkout
>>>> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>>>>
>> Damn stupid me.
>> As I explained above the webserver serves directly from
>> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>>
>> Thats the *svn/masses* !
>>
>
> I am so glad you are helping and able to keep this stuff straight. I
> worked on this stuff a lot back in the summer and have forgotten a lot
> of what I did. :) I meant to have a cron job setup for the automc user
> to do an "svn up ~/svn/masses" but it wasn't there until a few minutes
> ago. I had patched the ruleqa.cgi and committed it but the "svn up
> ~/svn/masses" was needed. I just did this part so it's now active.Â
> Sorry about that.
No problem, it hard, like you saw above, even tho I analyzed the
situation, I still managed to mess up my own conclusion the first time ;)
>
>>>> So the change should have gone live immediately if you applied the
>>>> patch
>>>> directly in svn/trunk/masses (and not in svn/masses).
>> Thats the wrong way around! Actually for the change to go live it should
>> be patched in
>> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>>
>> But thats in the readonly working copy. I just verified in the resync
>> account and indeed the ruleqa.cgi has not been patched so my change did
>> NOT go live.
>
> It should be live now. I manually verified the file at that path.
>
> http://ruleqa.spamassassin.org/?longdatelist=1
Yes I see, and the patch indeed fixes the problem with ruleqa. I think the
2-days-ago pages will now work as expected.
>
> So far good news on my latest patch to the SVN REVISION detection to
> find the majority REVISION instead of the latetest REVISION.
But why did last night (net masscheck) fail, it says not enough
contributors, but when you look at the corpus there seem to be enough
contributors? Or were some late?
>
> llanga does stand out as being off a bit. It seems to be a day behind
> at least by the date stamp. Do I need to put in a temporary cleanup for
> this masschecker to toss it out?
I don't think the problem is llanga itself, I suspect its some of our
scripting messing up.
See:
http://ruleqa.spamassassin.org/20171106-r1814390-n
It has:
20171106-r1814390-n (Viewing)
axb-coi-bulk axb-generic axb-ham-misc darxus ena-week0 ena-week1 ena-week2
ena-week3 giovanni jarif jbrooks llanga mmiroslaw-mails-ham
mmiroslaw-mails-spam thendrikx
and
20171107-r1814390-n
axb-coi-bulk axb-generic axb-ham-misc jarif llanga
So It has the same revision (1814390) masschecked to two consecutive days,
BUT all contributors in 20171107 are also present in 20171106 and in this
case its not only llanga.
So what triggers them to masscheck the same revision again, the next day?
But they also masscheck the newer revision later that same day.
What triggers the masscheck? (I don't know anything about that part yet)
Re: ruleqa.cgi issue
Posted by Dave Jones <da...@apache.org>.
On 11/11/2017 11:09 AM, Merijn van den Kroonenberg wrote:
>> On 11/11/2017 07:45 AM, Merijn van den Kroonenberg wrote:
>>> I saw you applied my patch for ruleqa.cgi and if I read the apache conf
>>> correctly, then the webserver serves pages directly from the checkout
>>> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>>>
> Damn stupid me.
> As I explained above the webserver serves directly from
> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>
> Thats the *svn/masses* !
>
I am so glad you are helping and able to keep this stuff straight. I
worked on this stuff a lot back in the summer and have forgotten a lot
of what I did. :) I meant to have a cron job setup for the automc user
to do an "svn up ~/svn/masses" but it wasn't there until a few minutes
ago. I had patched the ruleqa.cgi and committed it but the "svn up
~/svn/masses" was needed. I just did this part so it's now active.
Sorry about that.
>>> So the change should have gone live immediately if you applied the patch
>>> directly in svn/trunk/masses (and not in svn/masses).
> Thats the wrong way around! Actually for the change to go live it should
> be patched in
> /usr/local/spamassassin/automc/svn/masses/rule-qa/automc/ruleqa.cgi
>
> But thats in the readonly working copy. I just verified in the resync
> account and indeed the ruleqa.cgi has not been patched so my change did
> NOT go live.
It should be live now. I manually verified the file at that path.
http://ruleqa.spamassassin.org/?longdatelist=1
So far good news on my latest patch to the SVN REVISION detection to
find the majority REVISION instead of the latetest REVISION.
llanga does stand out as being off a bit. It seems to be a day behind
at least by the date stamp. Do I need to put in a temporary cleanup for
this masschecker to toss it out?
>>> But it seems the patch had not the hoped for effect so I guess I have to
>>> keep looking then ;)
>>> Please keep in this change for now.
>>>
>> Ok
>>
>>>> On 11/10/2017 05:00 AM, Merijn van den Kroonenberg wrote:
>>>>>> I need some help understanding this problem now. The "promotions
>>>>>> validated" have failed for 3 weeks now so the active.list is not
>>>>>> updating. There seems to be some relationship to the masscheck
>>>>>> ruleqa
>>>>>> web output that is stuck on "day 2" failing. See the output at the
>>>>>> very
>>>>>> bottom.
>>>>> I am trying to see whats going on. I would really like some insight in
>>>>> the
>>>>> file:
>>>>> /usr/local/spamassassin/automc/html/ruleqa.cache
>>>>>
>>>>> And i would like to have some insight into which masscheck results are
>>>>> present on disk. So the output of something like this:
>>>>>
>>>>> ls -la /usr/local/spamassassin/automc/html/2017*
>>>>>
>>>>> ls -la /usr/local/spamassassin/automc/html/20171*/*
>>>>>
>>>>> or something which gives me a tree view.
>>>>>
>>>>>> Can someone look at the perl script "listpromotable" and figure out
>>>>>> what
>>>>>> is going on? I can tell it's looking at the output by day to find
>>>>>> "mcviewing" and "mcsubmitters".
>>>>> Would you say:
>>>>> http://ruleqa.spamassassin.org/?longdatelist=1
>>>>> those are really all masschecks? This seems unlikely to me, I would
>>>>> expect
>>>>> to see masschecks with many contributors much more often (every day?)
>>>>>
>>>>>
>>>>>> This is day 1 match which is green when you view it in a browser:
>>>>>>
>>>>>> <td class="daterevtd mcviewing" width='20%'>
>>>>>> <span class="daterev_masscheck_description mcviewing">
>>>>>> <p>
>>>>>> <a name="r20171106_r1814390_n"
>>>>>> href="/20171106-r1814390-n?xml=1"><strong>
>>>>>> <span class="dr">20171106-r1814390-n</span>
>>>>>> </strong></a> <b>(Viewing)</b>
>>>>>> </p><p>
>>>>>> <em><span class="mcsubmitters">axb-coi-bulk axb-generic
>>>>>> axb-ham-misc darxus ena-week0 ena-week1 ena-week2 ena-week3 giovanni
>>>>>> jarif jbrooks llanga mmiroslaw-mails-ham mmiroslaw-mails-spam
>>>>>> thendrikx</span></em>
>>>>>> </x>
>>>>>> </p>
>>>>>>
>>>>>> Day 2 doesn't have that table with "mcviewing". The next question is
>>>>>> what is causing this problem. Is it related to new commits that
>>>>>> throw
>>>>>> off the masscheck processing?
>>>>> The 2 days ago doesn't highlight a current masscheck....but still it
>>>>> shows
>>>>> a result at the bottom...so its showing *something*. I think its
>>>>> likely
>>>>> it
>>>>> is the masxcheck as present in the datrev input field:
>>>>> 20171108-r1814560-n
>>>>> But that one isn't in any daterev liting, not even in the full
>>>>> listing.
>>>>>
>>>>> So i think something in the ruleqa.cgi which builds the daterev list
>>>>> is
>>>>> broken and leaves out some masschecks.
>>>>> If I get the cachefile and the ddirectory listings I can go debug
>>>>> where
>>>>> things go pear-shaped.
>>>>>
>>>>
Re: Cron ~/svn/trunk/build/mkupdates/run_nightly |
/usr/bin/tee /var/www/automc.spamassassin.org/mkupdates/mkupdates.txt
Posted by Merijn van den Kroonenberg <me...@web2all.nl>.
> I need some help understanding this problem now. The "promotions
> validated" have failed for 3 weeks now so the active.list is not
> updating. There seems to be some relationship to the masscheck ruleqa
> web output that is stuck on "day 2" failing. See the output at the very
> bottom.
I am trying to see whats going on. I would really like some insight in the
file:
/usr/local/spamassassin/automc/html/ruleqa.cache
And i would like to have some insight into which masscheck results are
present on disk. So the output of something like this:
ls -la /usr/local/spamassassin/automc/html/2017*
ls -la /usr/local/spamassassin/automc/html/20171*/*
or something which gives me a tree view.
>
> Can someone look at the perl script "listpromotable" and figure out what
> is going on? I can tell it's looking at the output by day to find
> "mcviewing" and "mcsubmitters".
Would you say:
http://ruleqa.spamassassin.org/?longdatelist=1
those are really all masschecks? This seems unlikely to me, I would expect
to see masschecks with many contributors much more often (every day?)
>
> This is day 1 match which is green when you view it in a browser:
>
> <td class="daterevtd mcviewing" width='20%'>
> <span class="daterev_masscheck_description mcviewing">
> <p>
> <a name="r20171106_r1814390_n"
> href="/20171106-r1814390-n?xml=1"><strong>
> <span class="dr">20171106-r1814390-n</span>
> </strong></a> <b>(Viewing)</b>
> </p><p>
> <em><span class="mcsubmitters">axb-coi-bulk axb-generic
> axb-ham-misc darxus ena-week0 ena-week1 ena-week2 ena-week3 giovanni
> jarif jbrooks llanga mmiroslaw-mails-ham mmiroslaw-mails-spam
> thendrikx</span></em>
> </x>
> </p>
>
> Day 2 doesn't have that table with "mcviewing". The next question is
> what is causing this problem. Is it related to new commits that throw
> off the masscheck processing?
The 2 days ago doesn't highlight a current masscheck....but still it shows
a result at the bottom...so its showing *something*. I think its likely it
is the masxcheck as present in the datrev input field: 20171108-r1814560-n
But that one isn't in any daterev liting, not even in the full listing.
So i think something in the ruleqa.cgi which builds the daterev list is
broken and leaves out some masschecks.
If I get the cachefile and the ddirectory listings I can go debug where
things go pear-shaped.
>
> P.S. This cron'd script used to redirect all output to /dev/null but I
> changed it to a tee a few days ago so we could easily keep an eye on
> this output via emails to this list.
>
> Dave
>
> On 11/07/2017 02:30 AM, Cron Daemon wrote:
>> + promote_active_rules
>> + pwd
>> + /usr/bin/perl build/mkupdates/listpromotable
>> /usr/local/spamassassin/automc/svn/trunk
>> HTTP get: http://ruleqa.spamassassin.org/1-days-ago?xml=1
>> HTTP get: http://ruleqa.spamassassin.org/2-days-ago?xml=1
>> no 'mcviewing', 'mcsubmitters' microformats on day 2
>> URL: http://ruleqa.spamassassin.org/2-days-ago?xml=1
>> + exit 25
>>
>
>