svn commit: r1686701 - in /comdev/projects.apache.org: STRUCTURE.txt scripts/README.txt scripts/import/parsepmcs.py

[hb...@apache.org]

Author: hboutemy
Date: Sun Jun 21 10:48:01 2015
New Revision: 1686701

URL: http://svn.apache.org/r1686701
Log:
updated documentation

Modified:
    comdev/projects.apache.org/STRUCTURE.txt
    comdev/projects.apache.org/scripts/README.txt
    comdev/projects.apache.org/scripts/import/parsepmcs.py

Modified: comdev/projects.apache.org/STRUCTURE.txt
URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/STRUCTURE.txt?rev=1686701&r1=1686700&r2=1686701&view=diff
==============================================================================
--- comdev/projects.apache.org/STRUCTURE.txt (original)
+++ comdev/projects.apache.org/STRUCTURE.txt Sun Jun 21 10:48:01 2015
@@ -18,15 +18,22 @@ Layout of comdev/projects.apache.org:
 /site/json/projects:
         - Contains project-specific data extracted from projects' DOAP files.
 
+/site/doap:
+        - Contains the equivalent DOAP data storage: 1 directory per committee,
+          containing pmc.rdf generated rdf (by parsecommittees.py) + pmc-doap.rdf copied (by parsepmcs.py)
+          + copied DOAP files (by rdfparse.py) from projects managed by the PMC
+
 Suggested cron setup:
     scripts/cronjobs/parsechairs.py - daily
     scripts/cronjobs/parsecomitters.py - daily/hourly (whatever we need/want)
     scripts/cronjobs/podlings.py - daily
     scripts/cronjobs/countaccounts.py - weekly
+    scripts/cronjobs/parsereleases.py - daily
 
 Stuff to run manually when needed:
     scripts/import/parsecommittees.py - requires committee-info.txt to be present
-    scripts/import/addpmc.py - manual run whenever a new PMC is founded
+    scripts/import/parsepmcs.py
+    scripts/import/rdfparse.py
 
 Webserver required:
 To test the site locally, a webserver is required or you'll get

Modified: comdev/projects.apache.org/scripts/README.txt
URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/scripts/README.txt?rev=1686701&r1=1686700&r2=1686701&view=diff
==============================================================================
--- comdev/projects.apache.org/scripts/README.txt (original)
+++ comdev/projects.apache.org/scripts/README.txt Sun Jun 21 10:48:01 2015
@@ -4,41 +4,43 @@ various sources:
 1. updating data (cronjobs)
 
 - countaccounts.py: Extract from LDAP monthly statistics on Unix accounts created
-  in: foundation/accounts-evolution.json + ldapsearch
-  out: foundation/accounts-evolution.json
+  in: json/foundation/accounts-evolution.json + ldapsearch
+  out: json/foundation/accounts-evolution.json
 
 - parsechairs.py: Fetches current VPs from the foundation website.
   in: http://www.apache.org/foundation/
-  out: foundation/chairs.json
+  out: json/foundation/chairs.json
 
 - parsecommitters.py: Fetches and parses the committer (LDAP) list via
   people.apache.org.
   in: http://people.apache.org/committer-index.html
-  out: foundation/people.json + foundation/groups.json
+  out: json/foundation/people.json + json/foundation/groups.json
   List of committers with reference to groups (people.json) and groups with corresponding committers (groups.json)
 
 - podlings.py: Reads podlings.xml from the incubator site and creates a JSON
   with timeline data, as well as current podling projects information.
   in: http://incubator.apache.org/podlings.xml
-  out: foundation/podlings.json + foundation/podlings-evolution.json
+  out: json/foundation/podlings.json + json/foundation/podlings-evolution.json
   Current list of podlings (podlings.json) and monthly statisctics on podlings (podlings-evolution.json)
 
 - parsereleases.py
   in: http://www.apache.org/dist/
-  out: foundation/releases.json
+  out: json/foundation/releases.json
 
 
 2. importing data (import)
 
 - parsecommittees.py: Parses committee-info.txt to detect new and retired committees
-  in: foundation/committees.json + foundation/committees-retired.json + committee-info.txt (https://svn.apache.org/repos/private/committers/board/committee-info.txt)
-  out: foundation/committees.json + foundation/committees-retired.json
+  in: json/foundation/committees.json + json/foundation/committees-retired.json
+      + committee-info.txt (https://svn.apache.org/repos/private/committers/board/committee-info.txt)
+      + https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/projects/pmc_list.xml + PMC data .rdf files
+  out: json/foundation/committees.json + json/foundation/committees-retired.json + doap/{committeeId}/pmc.rdf
 
 - parsepmcs.py: imports PMC data (RDF) from the old project.apache.org site.
   in: https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/projects/pmc_list.xml + PMC data .rdf files
-  out: foundation/pmcs.json
+  out: json/foundation/pmcs.json + doap/{committeeId}/pmc-doap.rdf
 
 - rdfparse.py: Parses existing RDF(DOAP) files from the old projects.a.o and
   turns them into JSON objects.
   in: https://svn.apache.org/repos/asf/infrastructure/site-tools/trunk/projects/files.xml + projects' DOAP files
-  out: projects/*.json + foundation/projects.json
+  out: json/projects/*.json + json/foundation/projects.json

Modified: comdev/projects.apache.org/scripts/import/parsepmcs.py
URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/scripts/import/parsepmcs.py?rev=1686701&r1=1686700&r2=1686701&view=diff
==============================================================================
--- comdev/projects.apache.org/scripts/import/parsepmcs.py (original)
+++ comdev/projects.apache.org/scripts/import/parsepmcs.py Sun Jun 21 10:48:01 2015
@@ -1,3 +1,5 @@
+# TODO: merge into parsecommittees.py
+
 from xml.dom import minidom
 import xml.etree.ElementTree as ET
 import re, urllib.request