svn commit: r1708714 - in /comdev/projects.apache.org: STRUCTURE.txt scripts/README.txt scripts/cronjobs/parsecommitteeinfo.py scripts/cronjobs/parsecommitteeinfoBETA.py

[se...@apache.org]

Author: sebb
Date: Wed Oct 14 23:28:15 2015
New Revision: 1708714

URL: http://svn.apache.org/viewvc?rev=1708714&view=rev
Log:
Let's give parsecommitteeinfo a go

Added:
    comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfo.py
      - copied, changed from r1708707, comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfoBETA.py
Removed:
    comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfoBETA.py
Modified:
    comdev/projects.apache.org/STRUCTURE.txt
    comdev/projects.apache.org/scripts/README.txt

Modified: comdev/projects.apache.org/STRUCTURE.txt
URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/STRUCTURE.txt?rev=1708714&r1=1708713&r2=1708714&view=diff
==============================================================================
--- comdev/projects.apache.org/STRUCTURE.txt (original)
+++ comdev/projects.apache.org/STRUCTURE.txt Wed Oct 14 23:28:15 2015
@@ -33,9 +33,10 @@ Suggested cron setup:
     scripts/cronjobs/podlings.py - daily
     scripts/cronjobs/countaccounts.py - weekly
     scripts/cronjobs/parsereleases.py - daily
+    scripts/cronjob/parsecommitteeinfo.py - daily
 
 Stuff to run manually when needed:
-    scripts/import/parsecommittees.py
+    scripts/import/parsecommittees.py - obsolete
     scripts/import/parseprojects.py
 
 Webserver required:
@@ -57,6 +58,7 @@ crontab -l -u www-data:
 00 00 * * * cd /var/www/projects.apache.org/scripts/cronjobs && python3 parsechairs.py
 00 00 * * * cd /var/www/projects.apache.org/scripts/cronjobs && python3 countaccounts.py
 00 00 * * * cd /var/www/projects.apache.org/scripts/cronjobs && python3 parsereleases.py
+00 01 * * * cd /var/www/projects.apache.org/scripts/cronjobs && python3 parsecommitteeinfo.py
 
 10 4 * * *  cd /var/www/projects.apache.org/site/json && ( svn status | awk '/^\? / {print $2}' | xargs -r svn add )
 

Modified: comdev/projects.apache.org/scripts/README.txt
URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/scripts/README.txt?rev=1708714&r1=1708713&r2=1708714&view=diff
==============================================================================
--- comdev/projects.apache.org/scripts/README.txt (original)
+++ comdev/projects.apache.org/scripts/README.txt Wed Oct 14 23:28:15 2015
@@ -11,6 +11,19 @@ various sources:
   in:  data/cache/committee-info.json (from whimsy, via committee_info module)
   out: site/json/foundation/chairs.json - (used by reporter.a.o only)
 
+- parsecommitteeinfo.py: Parses committee-info.json to detect new and retired committees
+  and imports PMC data (RDF) from PMC data files
+  in: site/json/foundation/committees.json
+    + site/json/foundation/committees-retired.json
+    + https://whimsy.apache.org/public/committee-info.json (via committee_info.py)
+    + data/committees.xml - list of where to find PMC description RDF files
+    + data/committees/*.rdf - local PMC description RDF files
+  out: site/json/foundation/committees.json (updated)
+     + site/json/foundation/committees-retired.json (updated)
+     + site/json/foundation/pmcs.json - (used by reporter.a.o only)
+     + site/doap/{committeeId}/pmc-doap.rdf - this is an exact copy of the file listed in data/committees.xml
+     + site/doap/{committeeId}/pmc.rdf - TODO: what purpose does this serve?
+
 - parsecommitters.py: Fetches and parses the committer (LDAP) list via
   people.apache.org.
   in: http://people.apache.org/committer-index.html
@@ -30,20 +43,12 @@ various sources:
   out: json/foundation/releases.json
      + json/foundation/releases-files.json
 
+
 2. importing data (import)
 
 - parsecommittees.py: Parses committee-info.txt to detect new and retired committees and imports PMC data (RDF) from
   PMC data files
-  in: site/json/foundation/committees.json
-    + site/json/foundation/committees-retired.json
-    + data/board/committee-info.txt (https://svn.apache.org/repos/private/committers/board/committee-info.txt)
-    + data/committees.xml - list of where to find PMC description RDF files
-    + data/committees/*.rdf - local PMC description RDF files
-  out: site/json/foundation/committees.json (updated)
-     + site/json/foundation/committees-retired.json (updated)
-     + site/json/foundation/pmcs.json - (used by reporter.a.o only)
-     + site/doap/{committeeId}/pmc-doap.rdf - this is an exact copy of the file listed in data/committees.xml
-     + site/doap/{committeeId}/pmc.rdf - TODO: what purpose does this serve?
+  No longer needed, use parsecommitteeinfo instead
 
 - parseprojects.py: Parses existing projects RDF(DOAP) files and turns them into JSON objects.
   in: data/projects.xml + projects' DOAP files
@@ -53,7 +58,7 @@ various sources:
 
 NOTICE: what prevents import scripts to be added to cron?
 1. parse committees.py requires committee-info.txt, which is not available on project-vm (require authentication)
-   TODO: Whimsy now supplies a JSON version of CI
+   Whimsy now supplies a JSON version of CI
 2. both scripts not only update files but sometimes need to add new files (new committees or new projects) or move
    (projects going to Attic or retired committees)
    TODO: any reason why scripts should not do this automatically?

Copied: comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfo.py (from r1708707, comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfoBETA.py)
URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfo.py?p2=comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfo.py&p1=comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfoBETA.py&r1=1708707&r2=1708714&rev=1708714&view=diff
==============================================================================
--- comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfoBETA.py (original)
+++ comdev/projects.apache.org/scripts/cronjobs/parsecommitteeinfo.py Wed Oct 14 23:28:15 2015
@@ -17,9 +17,6 @@ import committee_info
 
 """
 
-               THIS IS A PRELIMINARY VERSION
-                     DO NOT USE YET
-               =============================
 Reads:
 ../../site/json/foundation/people.json
 ../../data/committees.xml