svn commit: r1708717 - in /comdev/projects.apache.org: STRUCTURE.txt scripts/README.txt scripts/cronjobs/parseprojects.py scripts/import/parseprojects.py

[se...@apache.org]

Author: sebb
Date: Wed Oct 14 23:44:40 2015
New Revision: 1708717

URL: http://svn.apache.org/viewvc?rev=1708717&view=rev
Log:
Run parseprojects.py as a cron job

Added:
    comdev/projects.apache.org/scripts/cronjobs/parseprojects.py
      - copied unchanged from r1704813, comdev/projects.apache.org/scripts/import/parseprojects.py
Removed:
    comdev/projects.apache.org/scripts/import/parseprojects.py
Modified:
    comdev/projects.apache.org/STRUCTURE.txt
    comdev/projects.apache.org/scripts/README.txt

Modified: comdev/projects.apache.org/STRUCTURE.txt
URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/STRUCTURE.txt?rev=1708717&r1=1708716&r2=1708717&view=diff
==============================================================================
--- comdev/projects.apache.org/STRUCTURE.txt (original)
+++ comdev/projects.apache.org/STRUCTURE.txt Wed Oct 14 23:44:40 2015
@@ -59,6 +59,7 @@ crontab -l -u www-data:
 00 00 * * * cd /var/www/projects.apache.org/scripts/cronjobs && ./python3logger.sh countaccounts.py
 00 00 * * * cd /var/www/projects.apache.org/scripts/cronjobs && ./python3logger.sh parsereleases.py
 00 01 * * * cd /var/www/projects.apache.org/scripts/cronjobs && ./python3logger.sh parsecommitteeinfo.py
+00 01 * * * cd /var/www/projects.apache.org/scripts/cronjobs && ./python3logger.sh parseprojects.py
 
 10 4 * * *  cd /var/www/projects.apache.org/site/json && ( svn status | awk '/^\? / {print $2}' | xargs -r svn add )
 

Modified: comdev/projects.apache.org/scripts/README.txt
URL: http://svn.apache.org/viewvc/comdev/projects.apache.org/scripts/README.txt?rev=1708717&r1=1708716&r2=1708717&view=diff
==============================================================================
--- comdev/projects.apache.org/scripts/README.txt (original)
+++ comdev/projects.apache.org/scripts/README.txt Wed Oct 14 23:44:40 2015
@@ -43,22 +43,8 @@ various sources:
   out: json/foundation/releases.json
      + json/foundation/releases-files.json
 
-
-2. importing data (import)
-
-- parsecommittees.py: Parses committee-info.txt to detect new and retired committees and imports PMC data (RDF) from
-  PMC data files
-  No longer needed, use parsecommitteeinfo instead
-
 - parseprojects.py: Parses existing projects RDF(DOAP) files and turns them into JSON objects.
   in: data/projects.xml + projects' DOAP files
   out: site/json/projects/*.json - JSON versions of DOAP files
      + site/json/foundation/projects.json - combined listing of all projects
      + site/doap/{committeeId}/{project}.rdf - these are exact copies of the DOAPs listed in data/projects.xml
-
-NOTICE: what prevents import scripts to be added to cron?
-1. parse committees.py requires committee-info.txt, which is not available on project-vm (require authentication)
-   Whimsy now supplies a JSON version of CI
-2. both scripts not only update files but sometimes need to add new files (new committees or new projects) or move
-   (projects going to Attic or retired committees)
-   TODO: any reason why scripts should not do this automatically?