You are viewing a plain text version of this content. The canonical link for it is here.
Posted to sysadmins@spamassassin.apache.org by "Kevin A. McGrail" <km...@apache.org> on 2017/05/14 01:00:16 UTC
SysAdmin Tasklist was Re: Next priority to get running on
sa-vm1.apache.org
On 5/13/2017 4:56 PM, Dave Jones wrote:
> What's the next priority now that the rsync and httpd configs are active?
>
> I will work on the build next using this:
>
> https://svn.apache.org/repos/asf/spamassassin/trunk/build/README
PREFACE: I'm working on the build. If you would like to help, we need
to coordinate first.
Here is the promised list of items I've identified.
To me, the priority would be getting ruleqa/masscheck better documented
and back up and running would be ideal.
If we can get that system running smoother with a shorter lag to
publishing rules, I'd like to help more with it.
DONE - Touch a file called MIRROR.CHECK in
/var/www/bbmass.spamassassin.org/updates on SA-VM1 and test if it is
synced to the Mirrors. NOTE: I sync every 10 mins
- Document on the wiki that MIRRORED.BY contains the sa update mirror
contact names.
- Get the various files for running the sa-update aka bbmass website
into SVN. This would NOT be the update files but likely everything else
including the httpd.conf, MIRRORED.BY, etc.
- Get the email to root from sa-vm1 to go to sysadmins@ without
moderation so we have cron logs, etc. archived.
- KAM to Get the passwords for crashplan for SA into sysadmins repo
encrypted so we have multiple people who have access.
- Get the sa-update-mirror-check script (attached) running on SA-VM1 and
emailing sysadmins@ without moderation
- Get Darxus' rule update check script (attached) running on SA-VM1 and
emailing sysadmins@ without moderation. See SA Dev list example: Rule
updates are too old - 2017-05-08
- Get Darxus' check script updated for 3.4.2 and 3.3.2.
- Perhaps update the sa-update-mirror-check to use the MIRROR.CHECK with
a timestamp to confirm it's within a reasonable period of time.
- Find out who wrote the sa-update-mirror-check (likely on the list
archives), check the licensing on the post and hopefully ask who wrote
it to public domain or Apache license. Then add
attribution/license/copyright and add it to the sysadmins repo.
- Ask Darxus' if we can repo his script as well with
attribution/license/copyright as above
- Ask Darxus' to turn off his script that runs on his infrastructure
- Identify what we used to provide on the old servers. Some things KAM
believes we had that need to be verified and likely expanded on:
o Masscheck RSYNC for people to send us their Masscheck Logs
o An email system for people to email and it would send the results
of checking that email
o Masscheck Corpora RSYNC or perhaps SSH for people to send us their
corpora for us to run our own Masscheck server. NOTE: This is the most
sensitive data we would have I believe since it is other people's real
mail.
o For the above, I think I myself have this setup. I'd like to
identify where and extend it / improve it / make sure it's working, etc.
o Look at the rsync MOTD[1]
o Masscheck stuff:
https://wiki.apache.org/spamassassin/NightlyMassCheck - KAM sent notes a
few days ago about how he got this running on spamassassin-vm. if that
doesn't suffice, please let me know.
- Identify what jm was using talon1.pccc.com to provide so I can mimic
it. His cron jobs were disabled last January but I think they were
running items related to masscheck.
- Get the RuleQA Website running again.
- Identify what the incoming.spamassassin.org server did/does/can do for
us. NOTE: It might be the the same as below.
- Talk to Grant Kellar with Sonic about the traps they have in place and
where they are sent to make sure we are utilizing them.
- Clean up and remove unnecessary backup data on sa-vm1 - NO NEED TO BE
HASTY ON THIS, I'M JUST WRITING A COMPLETE LIST.
- Identify how much data we need if Infra can shrink the data storage
allocated for sa-vm1
- Talk to AXB about SOUGHT and SOUGHT2
- Update the documentation for InfraNotes2017 with another pass of
updates about machines, etc.
[1]
corpus
nightly mass-check result upload area. It is password protected.
If you would like a password, please send a request to
pmc@spamassassin.apache.org and request a "nightly" username and password.
submit
Score generation mass-check result upload area. It is password
protected. If you would like a password, please send a request to
pmc@spamassassin.apache.org and request a "score generation" username
and password. Generally these are only granted after a mass-check
announcement has been made on the spamassassin developer mailing list.
anoncorpus
mass-check result download area, available via anonymous access.
--
Kevin A. McGrail
Asst. Treasurer, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
Re: SysAdmin Tasklist was Re: Next priority to get running on sa-vm1.apache.org
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
Makes sense and thanks.
Regards,
KAM
On May 15, 2017 9:09:05 AM EDT, Dave Jones <da...@apache.org> wrote:
>
>
>On 05/14/2017 09:25 AM, Kevin A. McGrail wrote:
>> On 5/14/2017 10:11 AM, Dave Jones wrote:
>>>
>>> Do we want to subscribe root like this? It doesn't need to receive
>>> any of these emails that will just fill up the root mailbox or
>>> possibly create a mail loop. I was thinking about allowing it as a
>>> non-member poster. I am more familiar with Mailman that allows a
>list
>>> of addresses that can post without being a member. Do you know if
>>> there is something like this available on the ASF lists? If so, I
>>> assumed this would be a Jira task for someone with admin rights to
>the
>>> listserv.
>>>
>>
>> Good point. Can you open an infra ticket and ask? In the meantime
>just
>> subscribe root.
>
>I am not able to subscribe root@sa-vm1.apache.org since inbound mail to
>
>sa-vm1.apache.org is not setup currently since Postfix is only
>listening
>on localhost. It don't want to change anything with Postfix just for
>this.
>
>I opened a Jira task to add root@sa-vm1.apache.org to be allowed to
>post
>as a non-subscriber. It seems that ezmlm supports this feature but it
>has to be added manually to a file by the admins.
>
>>
>>> I tweaked the sa-update-mirror-check.sh script and it's now
>>> /usr/local/bin/checkSAupdateMirrors.sh and symlink'd to
>>> /etc/cron.daily to send this list an email with the status of all
>the
>>> mirrors.
>>
>> Good! Can you change it to hourly when you get it tweaked.
>>
>
>Done. It is now symlinked to /etc/cron.hourly.
Re: SysAdmin Tasklist was Re: Next priority to get running on
sa-vm1.apache.org
Posted by Dave Jones <da...@apache.org>.
On 05/14/2017 09:25 AM, Kevin A. McGrail wrote:
> On 5/14/2017 10:11 AM, Dave Jones wrote:
>>
>> Do we want to subscribe root like this? It doesn't need to receive
>> any of these emails that will just fill up the root mailbox or
>> possibly create a mail loop. I was thinking about allowing it as a
>> non-member poster. I am more familiar with Mailman that allows a list
>> of addresses that can post without being a member. Do you know if
>> there is something like this available on the ASF lists? If so, I
>> assumed this would be a Jira task for someone with admin rights to the
>> listserv.
>>
>
> Good point. Can you open an infra ticket and ask? In the meantime just
> subscribe root.
I am not able to subscribe root@sa-vm1.apache.org since inbound mail to
sa-vm1.apache.org is not setup currently since Postfix is only listening
on localhost. It don't want to change anything with Postfix just for this.
I opened a Jira task to add root@sa-vm1.apache.org to be allowed to post
as a non-subscriber. It seems that ezmlm supports this feature but it
has to be added manually to a file by the admins.
>
>> I tweaked the sa-update-mirror-check.sh script and it's now
>> /usr/local/bin/checkSAupdateMirrors.sh and symlink'd to
>> /etc/cron.daily to send this list an email with the status of all the
>> mirrors.
>
> Good! Can you change it to hourly when you get it tweaked.
>
Done. It is now symlinked to /etc/cron.hourly.
Re: SysAdmin Tasklist was Re: Next priority to get running on
sa-vm1.apache.org
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
On 5/14/2017 10:11 AM, Dave Jones wrote:
>
> Do we want to subscribe root like this? It doesn't need to receive
> any of these emails that will just fill up the root mailbox or
> possibly create a mail loop. I was thinking about allowing it as a
> non-member poster. I am more familiar with Mailman that allows a list
> of addresses that can post without being a member. Do you know if
> there is something like this available on the ASF lists? If so, I
> assumed this would be a Jira task for someone with admin rights to the
> listserv.
>
Good point. Can you open an infra ticket and ask? In the meantime just
subscribe root.
> I tweaked the sa-update-mirror-check.sh script and it's now
> /usr/local/bin/checkSAupdateMirrors.sh and symlink'd to
> /etc/cron.daily to send this list an email with the status of all the
> mirrors.
Good! Can you change it to hourly when you get it tweaked.
Re: SysAdmin Tasklist was Re: Next priority to get running on
sa-vm1.apache.org
Posted by Dave Jones <da...@apache.org>.
On 05/14/2017 08:31 AM, Kevin A. McGrail wrote:
> On 5/14/2017 7:38 AM, Dave Jones wrote:
>> Thank goodness! I took a look at that README and my head started to
>> explode. I am not a perl developer so that was going to be very time
>> consuming to get up to speed on that. I will let you have it... :)
> Well, if you want to learn, I'd love to mentor others. I've tried to
> make it more straightforward each time I edit that document. And the
> interesting things is that it's very multi-disciplinary and more
> sysadminny than perly.
>
I will definitely take you up on this later when we have all of these
other things fixed and working smoothly.
>>> To me, the priority would be getting ruleqa/masscheck better
>>> documented and back up and running would be ideal.
>> I will change to working on this as my next priority.
>
> Cool, there is at least one CGI so getting that website working along
> with rsync for submissions would be my first attack. I believe you are
> a masscheck submitter so you'll know when the rsync password file, etc.
> is working.
>
I am so I will work on this next.
> Once the website works again and the rsync is working, work on what's
> needed to get the scripts firing again.
>
> *IMPORTANT* I've captured a week + a few days of the cron output before
> the server was turned off. This is in /home/kmcgrail/SACron.gz.
>
> It's an mbox format to let us sleuth into things like cron jobs for
> masscheck and rules update that run differently once per week. It also
> will let us look at any errors the system might have been throwing
> before the server switch over so we don't chase down strawmen.
>
> This should be one of the last sources of information that I have that
> isn't shared with the rest of the group well. Going forward, the
> centralized cron logging to the sysadmin mailing list will give us some
> archives of this data. Plus I figure any sysadmin can filter out the
> noise.
>
>> This is referenced in DNS by mirrors.updates.spamassasin.org TXT
>> record pointing to
>> http://spamassassin.apache.org/updates/MIRRORED.BY. Do you have the
>> details on how this file gets updated on that apache.org server so we
>> can add this to the wiki?
> I'd document it in the wiki at least referring to the file as I'm trying
> to make the wiki a single source of truth. The fact that it doesn't
> show up at all is worrisome.
>
> To my knowledge, MIRRORED.BY is hand edited on the server doing the
> hosting itself.
>
> I'm working on instructions to setup your own channel which will nicely
> intersect with this work.
>
>>
>> root email will deliver to this list now. Need someone to setup the
>> root@sa-vm1.apache.org to be allowed without moderation. See test
>> email I recently sent from root to the list.
> OK, see https://www.apache.org/foundation/mailinglists.html where you
> should be a moderator of this list and subscribe away. Assuming
> incoming mail works to root, you can then confirm the subscription. Ask
> if you need help.
Do we want to subscribe root like this? It doesn't need to receive any
of these emails that will just fill up the root mailbox or possibly
create a mail loop. I was thinking about allowing it as a non-member
poster. I am more familiar with Mailman that allows a list of addresses
that can post without being a member. Do you know if there is something
like this available on the ASF lists? If so, I assumed this would be a
Jira task for someone with admin rights to the listserv.
>
> Regards,
> KAM
>
I tweaked the sa-update-mirror-check.sh script and it's now
/usr/local/bin/checkSAupdateMirrors.sh and symlink'd to /etc/cron.daily
to send this list an email with the status of all the mirrors.
Dave
Re: SysAdmin Tasklist was Re: Next priority to get running on
sa-vm1.apache.org
Posted by "Kevin A. McGrail" <km...@apache.org>.
On 5/14/2017 7:38 AM, Dave Jones wrote:
> Thank goodness! I took a look at that README and my head started to
> explode. I am not a perl developer so that was going to be very time
> consuming to get up to speed on that. I will let you have it... :)
Well, if you want to learn, I'd love to mentor others. I've tried to
make it more straightforward each time I edit that document. And the
interesting things is that it's very multi-disciplinary and more
sysadminny than perly.
>> To me, the priority would be getting ruleqa/masscheck better
>> documented and back up and running would be ideal.
> I will change to working on this as my next priority.
Cool, there is at least one CGI so getting that website working along
with rsync for submissions would be my first attack. I believe you are
a masscheck submitter so you'll know when the rsync password file, etc.
is working.
Once the website works again and the rsync is working, work on what's
needed to get the scripts firing again.
*IMPORTANT* I've captured a week + a few days of the cron output before
the server was turned off. This is in /home/kmcgrail/SACron.gz.
It's an mbox format to let us sleuth into things like cron jobs for
masscheck and rules update that run differently once per week. It also
will let us look at any errors the system might have been throwing
before the server switch over so we don't chase down strawmen.
This should be one of the last sources of information that I have that
isn't shared with the rest of the group well. Going forward, the
centralized cron logging to the sysadmin mailing list will give us some
archives of this data. Plus I figure any sysadmin can filter out the noise.
> This is referenced in DNS by mirrors.updates.spamassasin.org TXT
> record pointing to
> http://spamassassin.apache.org/updates/MIRRORED.BY. Do you have the
> details on how this file gets updated on that apache.org server so we
> can add this to the wiki?
I'd document it in the wiki at least referring to the file as I'm trying
to make the wiki a single source of truth. The fact that it doesn't
show up at all is worrisome.
To my knowledge, MIRRORED.BY is hand edited on the server doing the
hosting itself.
I'm working on instructions to setup your own channel which will nicely
intersect with this work.
>
> root email will deliver to this list now. Need someone to setup the
> root@sa-vm1.apache.org to be allowed without moderation. See test
> email I recently sent from root to the list.
OK, see https://www.apache.org/foundation/mailinglists.html where you
should be a moderator of this list and subscribe away. Assuming
incoming mail works to root, you can then confirm the subscription. Ask
if you need help.
Regards,
KAM
--
Kevin A. McGrail
Asst. Treasurer, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
Re: SysAdmin Tasklist was Re: Next priority to get running on
sa-vm1.apache.org
Posted by Dave Jones <da...@apache.org>.
On 05/13/2017 08:02 PM, Kevin A. McGrail wrote:
> RESENDING: Scripts were blocked for security reasons
>
> On 5/13/2017 4:56 PM, Dave Jones wrote:
>> What's the next priority now that the rsync and httpd configs are active?
>>
>> I will work on the build next using this:
>>
>> https://svn.apache.org/repos/asf/spamassassin/trunk/build/README
>
> PREFACE: I'm working on the build. If you would like to help, we need
> to coordinate first.
>
Thank goodness! I took a look at that README and my head started to
explode. I am not a perl developer so that was going to be very time
consuming to get up to speed on that. I will let you have it... :)
> Here is the promised list of items I've identified.
>
> To me, the priority would be getting ruleqa/masscheck better documented
> and back up and running would be ideal.
>
I will change to working on this as my next priority.
> If we can get that system running smoother with a shorter lag to
> publishing rules, I'd like to help more with it.
>
>
> DONE - Touch a file called MIRROR.CHECK in
> /var/www/bbmass.spamassassin.org/updates on SA-VM1 and test if it is
> synced to the Mirrors. NOTE: I sync every 10 mins
>
> - Document on the wiki that MIRRORED.BY contains the sa update mirror
> contact names.
>
This is referenced in DNS by mirrors.updates.spamassasin.org TXT record
pointing to http://spamassassin.apache.org/updates/MIRRORED.BY. Do you
have the details on how this file gets updated on that apache.org server
so we can add this to the wiki?
> - Get the various files for running the sa-update aka bbmass website
> into SVN. This would NOT be the update files but likely everything else
> including the httpd.conf, MIRRORED.BY, etc.
>
> - Get the email to root from sa-vm1 to go to sysadmins@ without
> moderation so we have cron logs, etc. archived.
>
root email will deliver to this list now. Need someone to setup the
root@sa-vm1.apache.org to be allowed without moderation. See test email
I recently sent from root to the list.
> - KAM to Get the passwords for crashplan for SA into sysadmins repo
> encrypted so we have multiple people who have access.
>
> - Get the sa-update-mirror-check script (attached) running on SA-VM1 and
> emailing sysadmins@ without moderation
>
> - Get Darxus' rule update check script (attached) running on SA-VM1 and
> emailing sysadmins@ without moderation. See SA Dev list example: Rule
> updates are too old - 2017-05-08
>
> - Get Darxus' check script updated for 3.4.2 and 3.3.2.
>
> - Perhaps update the sa-update-mirror-check to use the MIRROR.CHECK with
> a timestamp to confirm it's within a reasonable period of time.
>
> - Find out who wrote the sa-update-mirror-check (likely on the list
> archives), check the licensing on the post and hopefully ask who wrote
> it to public domain or Apache license. Then add
> attribution/license/copyright and add it to the sysadmins repo.
>
> - Ask Darxus' if we can repo his script as well with
> attribution/license/copyright as above
>
> - Ask Darxus' to turn off his script that runs on his infrastructure
>
> - Identify what we used to provide on the old servers. Some things KAM
> believes we had that need to be verified and likely expanded on:
>
> o Masscheck RSYNC for people to send us their Masscheck Logs
> o An email system for people to email and it would send the results
> of checking that email
> o Masscheck Corpora RSYNC or perhaps SSH for people to send us their
> corpora for us to run our own Masscheck server. NOTE: This is the most
> sensitive data we would have I believe since it is other people's real
> mail.
> o For the above, I think I myself have this setup. I'd like to
> identify where and extend it / improve it / make sure it's working, etc.
> o Look at the rsync MOTD[1]
> o Masscheck stuff:
> https://wiki.apache.org/spamassassin/NightlyMassCheck - KAM sent notes a
> few days ago about how he got this running on spamassassin-vm. if that
> doesn't suffice, please let me know.
>
Will check this out today and try to get this working on sa-vm1.
> - Identify what jm was using talon1.pccc.com to provide so I can mimic
> it. His cron jobs were disabled last January but I think they were
> running items related to masscheck.
>
> - Get the RuleQA Website running again.
>
Will check this out today.
> - Identify what the incoming.spamassassin.org server did/does/can do for
> us. NOTE: It might be the the same as below.
>
> - Talk to Grant Kellar with Sonic about the traps they have in place and
> where they are sent to make sure we are utilizing them.
>
> - Clean up and remove unnecessary backup data on sa-vm1 - NO NEED TO BE
> HASTY ON THIS, I'M JUST WRITING A COMPLETE LIST.
>
> - Identify how much data we need if Infra can shrink the data storage
> allocated for sa-vm1
>
> - Talk to AXB about SOUGHT and SOUGHT2
>
> - Update the documentation for InfraNotes2017 with another pass of
> updates about machines, etc.
>
>
> [1]
> corpus
> nightly mass-check result upload area. It is password protected.
> If you would like a password, please send a request to
> pmc@spamassassin.apache.org and request a "nightly" username and password.
>
> submit
> Score generation mass-check result upload area. It is password
> protected. If you would like a password, please send a request to
> pmc@spamassassin.apache.org and request a "score generation" username
> and password. Generally these are only granted after a mass-check
> announcement has been made on the spamassassin developer mailing list.
>
> anoncorpus
> mass-check result download area, available via anonymous access.
>
>
> --
> Kevin A. McGrail
> Asst. Treasurer, Apache Software Foundation
> Chair Emeritus Apache SpamAssassin Project
>
Re: SysAdmin Tasklist was Re: Next priority to get running on
sa-vm1.apache.org
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
RESENDING: Scripts were blocked for security reasons
On 5/13/2017 4:56 PM, Dave Jones wrote:
> What's the next priority now that the rsync and httpd configs are active?
>
> I will work on the build next using this:
>
> https://svn.apache.org/repos/asf/spamassassin/trunk/build/README
PREFACE: I'm working on the build. If you would like to help, we need
to coordinate first.
Here is the promised list of items I've identified.
To me, the priority would be getting ruleqa/masscheck better documented
and back up and running would be ideal.
If we can get that system running smoother with a shorter lag to
publishing rules, I'd like to help more with it.
DONE - Touch a file called MIRROR.CHECK in
/var/www/bbmass.spamassassin.org/updates on SA-VM1 and test if it is
synced to the Mirrors. NOTE: I sync every 10 mins
- Document on the wiki that MIRRORED.BY contains the sa update mirror
contact names.
- Get the various files for running the sa-update aka bbmass website
into SVN. This would NOT be the update files but likely everything else
including the httpd.conf, MIRRORED.BY, etc.
- Get the email to root from sa-vm1 to go to sysadmins@ without
moderation so we have cron logs, etc. archived.
- KAM to Get the passwords for crashplan for SA into sysadmins repo
encrypted so we have multiple people who have access.
- Get the sa-update-mirror-check script (attached) running on SA-VM1 and
emailing sysadmins@ without moderation
- Get Darxus' rule update check script (attached) running on SA-VM1 and
emailing sysadmins@ without moderation. See SA Dev list example: Rule
updates are too old - 2017-05-08
- Get Darxus' check script updated for 3.4.2 and 3.3.2.
- Perhaps update the sa-update-mirror-check to use the MIRROR.CHECK with
a timestamp to confirm it's within a reasonable period of time.
- Find out who wrote the sa-update-mirror-check (likely on the list
archives), check the licensing on the post and hopefully ask who wrote
it to public domain or Apache license. Then add
attribution/license/copyright and add it to the sysadmins repo.
- Ask Darxus' if we can repo his script as well with
attribution/license/copyright as above
- Ask Darxus' to turn off his script that runs on his infrastructure
- Identify what we used to provide on the old servers. Some things KAM
believes we had that need to be verified and likely expanded on:
o Masscheck RSYNC for people to send us their Masscheck Logs
o An email system for people to email and it would send the results
of checking that email
o Masscheck Corpora RSYNC or perhaps SSH for people to send us their
corpora for us to run our own Masscheck server. NOTE: This is the most
sensitive data we would have I believe since it is other people's real
mail.
o For the above, I think I myself have this setup. I'd like to
identify where and extend it / improve it / make sure it's working, etc.
o Look at the rsync MOTD[1]
o Masscheck stuff:
https://wiki.apache.org/spamassassin/NightlyMassCheck - KAM sent notes a
few days ago about how he got this running on spamassassin-vm. if that
doesn't suffice, please let me know.
- Identify what jm was using talon1.pccc.com to provide so I can mimic
it. His cron jobs were disabled last January but I think they were
running items related to masscheck.
- Get the RuleQA Website running again.
- Identify what the incoming.spamassassin.org server did/does/can do for
us. NOTE: It might be the the same as below.
- Talk to Grant Kellar with Sonic about the traps they have in place and
where they are sent to make sure we are utilizing them.
- Clean up and remove unnecessary backup data on sa-vm1 - NO NEED TO BE
HASTY ON THIS, I'M JUST WRITING A COMPLETE LIST.
- Identify how much data we need if Infra can shrink the data storage
allocated for sa-vm1
- Talk to AXB about SOUGHT and SOUGHT2
- Update the documentation for InfraNotes2017 with another pass of
updates about machines, etc.
[1]
corpus
nightly mass-check result upload area. It is password protected.
If you would like a password, please send a request to
pmc@spamassassin.apache.org and request a "nightly" username and password.
submit
Score generation mass-check result upload area. It is password
protected. If you would like a password, please send a request to
pmc@spamassassin.apache.org and request a "score generation" username
and password. Generally these are only granted after a mass-check
announcement has been made on the spamassassin developer mailing list.
anoncorpus
mass-check result download area, available via anonymous access.
--
Kevin A. McGrail
Asst. Treasurer, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project