You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@community.apache.org by rb...@apache.org on 2026/09/17 18:02:51 UTC
svn commit: r1938322 - comdev/projects.apache.org/trunk
Author: rbowen
Date: Thu Sep 17 18:02:51 2026
New Revision: 1938322
Log:
Updates STRUCTURE.txt to reflect the current reality.
Modified:
comdev/projects.apache.org/trunk/STRUCTURE.txt
Modified: comdev/projects.apache.org/trunk/STRUCTURE.txt
==============================================================================
--- comdev/projects.apache.org/trunk/STRUCTURE.txt Thu Sep 17 18:00:04 2026 (r1938321)
+++ comdev/projects.apache.org/trunk/STRUCTURE.txt Thu Sep 17 18:02:51 2026 (r1938322)
@@ -3,43 +3,123 @@ Layout of comdev/projects.apache.org:
/scripts:
- Contains scripts used for import and maintenance of foundation-wide
data, such as committer IDs/names, project VPs, founding dates,
- reporting cycles etc. See README.txt for more info.
+ reporting cycles etc. See /scripts/README.txt for details.
+
+/scripts/cronjobs:
+ - Cronjob scripts that run nightly on projects-vm3 to generate JSON
+ from authoritative sources (Whimsy, LDAP, Incubator, dist/).
+ Scripts:
+ parsecommitters.py - committer/group data from Whimsy LDAP
+ parsecommitteeinfo.py - PMC data from Whimsy committee-info.json + local RDF
+ podlings.py - podling data from Incubator podlings.xml
+ parsereleases.py - release data from dist.apache.org
+ parseprojects.py - project metadata from DOAP files
+ generaterepos.py - repository listing
+ pubsubber.py - listens for SVN commits (runs on reboot)
+ Utilities:
+ python3logger.sh - wrapper that logs script output
+ svnadd.sh - SVN-adds new JSON files before nightly commit
+ jsonutils.py - shared JSON helpers
+ urlutils.py - shared URL fetch helpers
+ errtee.py - error logging helper
+ sendmail.py - email notification helper
+ testlogging.py - logging test
/data:
- Contains data maintained by committees
+/data/committees.xml:
+ - List of where to find PMC description RDF files
+
+/data/committees/:
+ - Per-committee RDF description files (*.rdf)
+
+/data/committees-retired/:
+ - RDF files for retired committees
+
+/data/projects.xml:
+ - Master list of DOAP file locations for all projects
+
+/data/projects-override/:
+ - Local copies of DOAP files for retired projects (moved to Attic).
+ Used when the original project committers are unavailable to update
+ the DOAP themselves. See AAREADME.txt in that directory for the
+ update procedure.
+
+/data/cache/:
+ - Cached data from external sources
+
+/failures/:
+ - (currently empty) Placeholder for failed cronjob output
+
/site:
- Contains the HTML, images and javascript needed to run the site
+/site/index.html, committees.html, project.html, projects.html, etc.:
+ - Static HTML pages that load JSON data client-side via JavaScript
+
+/site/js/:
+ - JavaScript: projects.js (main logic), script.js, project_editor.js,
+ plus libraries (jQuery, DataTables, ECharts, Underscore)
+
+/site/css/:
+ - Stylesheets
+
+/site/images/:
+ - Images
+
+/site/indexes/:
+ - Alternate project listing views: alphabetical, by category, by
+ language, by PMC, releases, standards, quick search
+
+/site/edit/:
+ - DISABLED. Was an LDAP-authenticated DOAP editor (CGI-based).
+ See README.txt in that directory.
+
+/site/ns/:
+ - ASF extension namespace definition (asfext)
+
/site/json:
- - Contains the JSON data storage calculated from /data and external sources
- (notice: used by reporter.apache.org too, see getjson.py)
+ - Contains the JSON data storage calculated from /data and external
+ sources. Also used by reporter.apache.org (see getjson.py).
/site/json/foundation:
- - Contains foundation-wide JSON data (committers, chairs, podling
- evolution etc)
+ - Foundation-wide JSON data generated by cronjobs:
+ committees.json - PMC roster, chairs, charters
+ committees-retired.json - retired committees
+ people.json - committers with group membership
+ people_name.json - availid → public name mapping
+ groups.json - groups with their committers
+ podlings.json - current podlings
+ podlings-history.json - ended/graduated podlings
+ releases.json - release data per project
+ releases-files.json - individual release files
+ projects.json - combined listing from DOAP files
+ repositories.json - Git/SVN repository listing
+ accounts-evolution2.json - account creation timeline
/site/json/projects:
- - Contains project-specific data extracted from projects' DOAP files.
+ - Per-project JSON files parsed from DOAP files by parseprojects.py.
+ One file per project (e.g. kafka.json, httpd.json).
-N.B. The directory structure should be owned by the www-data login (or whatever is used for the webserver)
-This is because at least one of the scripts (parsecommitteeinfo.py) may invoke SVN commands
-
-Suggested cron setup:
- scripts/cronjobs/parsecomitters.py - daily/hourly (whatever we need/want)
- scripts/cronjobs/podlings.py - daily
- scripts/cronjobs/countaccounts.py - weekly
- scripts/cronjobs/parsereleases.py - daily
- scripts/cronjob/parsecommitteeinfo.py - daily
+/Docker support:
+ Dockerfile, compose.yml, DOCKER.md - local development setup.
+ docker-config/ - Apache httpd config (25-projects.conf) + setup script
+ docker-data/ - runtime data (logs/) for Docker containers
+
+Note: the directory structure on the VM should be owned by the www-data user
+(or whatever is used for the webserver), because at least one script
+(parsecommitteeinfo.py) may invoke SVN commands.
Webserver required:
-To test the site locally, a webserver is required or you'll get
-"Cross origin requests are only supported for HTTP" errors.
-An easy setup for development is: run "python3 -m http.server 8888" from
-site directory to have site available at http://localhost:8888/
+To test the site locally without Docker, a webserver is required or you'll
+get "Cross origin requests are only supported for HTTP" errors. An easy setup
+for development is: run "python3 -m http.server 8888" from the site directory
+to have the site available at http://localhost:8888/
+
-Current crontab settings:
+Current crontab settings (projects-vm3):
crontab root:
# m h dom mon dow command