You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@community.apache.org by rb...@apache.org on 2026/09/17 18:02:51 UTC

svn commit: r1938322 - comdev/projects.apache.org/trunk

Author: rbowen
Date: Thu Sep 17 18:02:51 2026
New Revision: 1938322

Log:
Updates STRUCTURE.txt to reflect the current reality.

Modified:
   comdev/projects.apache.org/trunk/STRUCTURE.txt

Modified: comdev/projects.apache.org/trunk/STRUCTURE.txt
==============================================================================
--- comdev/projects.apache.org/trunk/STRUCTURE.txt	Thu Sep 17 18:00:04 2026	(r1938321)
+++ comdev/projects.apache.org/trunk/STRUCTURE.txt	Thu Sep 17 18:02:51 2026	(r1938322)
@@ -3,43 +3,123 @@ Layout of comdev/projects.apache.org:
 /scripts:
         - Contains scripts used for import and maintenance of foundation-wide
           data, such as committer IDs/names, project VPs, founding dates,
-          reporting cycles etc. See README.txt for more info.
+          reporting cycles etc. See /scripts/README.txt for details.
+
+/scripts/cronjobs:
+        - Cronjob scripts that run nightly on projects-vm3 to generate JSON
+          from authoritative sources (Whimsy, LDAP, Incubator, dist/).
+          Scripts:
+            parsecommitters.py   - committer/group data from Whimsy LDAP
+            parsecommitteeinfo.py - PMC data from Whimsy committee-info.json + local RDF
+            podlings.py          - podling data from Incubator podlings.xml
+            parsereleases.py     - release data from dist.apache.org
+            parseprojects.py     - project metadata from DOAP files
+            generaterepos.py     - repository listing
+            pubsubber.py         - listens for SVN commits (runs on reboot)
+          Utilities:
+            python3logger.sh     - wrapper that logs script output
+            svnadd.sh            - SVN-adds new JSON files before nightly commit
+            jsonutils.py         - shared JSON helpers
+            urlutils.py          - shared URL fetch helpers
+            errtee.py            - error logging helper
+            sendmail.py          - email notification helper
+            testlogging.py       - logging test
 
 /data:
         - Contains data maintained by committees
 
+/data/committees.xml:
+        - List of where to find PMC description RDF files
+
+/data/committees/:
+        - Per-committee RDF description files (*.rdf)
+
+/data/committees-retired/:
+        - RDF files for retired committees
+
+/data/projects.xml:
+        - Master list of DOAP file locations for all projects
+
+/data/projects-override/:
+        - Local copies of DOAP files for retired projects (moved to Attic).
+          Used when the original project committers are unavailable to update
+          the DOAP themselves. See AAREADME.txt in that directory for the
+          update procedure.
+
+/data/cache/:
+        - Cached data from external sources
+
+/failures/:
+        - (currently empty) Placeholder for failed cronjob output
+
 /site:
         - Contains the HTML, images and javascript needed to run the site
 
+/site/index.html, committees.html, project.html, projects.html, etc.:
+        - Static HTML pages that load JSON data client-side via JavaScript
+
+/site/js/:
+        - JavaScript: projects.js (main logic), script.js, project_editor.js,
+          plus libraries (jQuery, DataTables, ECharts, Underscore)
+
+/site/css/:
+        - Stylesheets
+
+/site/images/:
+        - Images
+
+/site/indexes/:
+        - Alternate project listing views: alphabetical, by category, by
+          language, by PMC, releases, standards, quick search
+
+/site/edit/:
+        - DISABLED. Was an LDAP-authenticated DOAP editor (CGI-based).
+          See README.txt in that directory.
+
+/site/ns/:
+        - ASF extension namespace definition (asfext)
+
 /site/json:
-        - Contains the JSON data storage calculated from /data and external sources
-          (notice: used by reporter.apache.org too, see getjson.py)
+        - Contains the JSON data storage calculated from /data and external
+          sources. Also used by reporter.apache.org (see getjson.py).
 
 /site/json/foundation:
-        - Contains foundation-wide JSON data (committers, chairs, podling
-          evolution etc)
+        - Foundation-wide JSON data generated by cronjobs:
+            committees.json          - PMC roster, chairs, charters
+            committees-retired.json  - retired committees
+            people.json              - committers with group membership
+            people_name.json         - availid → public name mapping
+            groups.json              - groups with their committers
+            podlings.json            - current podlings
+            podlings-history.json    - ended/graduated podlings
+            releases.json            - release data per project
+            releases-files.json      - individual release files
+            projects.json            - combined listing from DOAP files
+            repositories.json        - Git/SVN repository listing
+            accounts-evolution2.json - account creation timeline
 
 /site/json/projects:
-        - Contains project-specific data extracted from projects' DOAP files.
+        - Per-project JSON files parsed from DOAP files by parseprojects.py.
+          One file per project (e.g. kafka.json, httpd.json).
 
-N.B. The directory structure should be owned by the www-data login (or whatever is used for the webserver)
-This is because at least one of the scripts (parsecommitteeinfo.py) may invoke SVN commands
-
-Suggested cron setup:
-    scripts/cronjobs/parsecomitters.py - daily/hourly (whatever we need/want)
-    scripts/cronjobs/podlings.py - daily
-    scripts/cronjobs/countaccounts.py - weekly
-    scripts/cronjobs/parsereleases.py - daily
-    scripts/cronjob/parsecommitteeinfo.py - daily
+/Docker support:
+        Dockerfile, compose.yml, DOCKER.md - local development setup.
+        docker-config/ - Apache httpd config (25-projects.conf) + setup script
+        docker-data/   - runtime data (logs/) for Docker containers
+
+Note: the directory structure on the VM should be owned by the www-data user
+(or whatever is used for the webserver), because at least one script
+(parsecommitteeinfo.py) may invoke SVN commands.
 
 
 Webserver required:
-To test the site locally, a webserver is required or you'll get
-"Cross origin requests are only supported for HTTP" errors.
-An easy setup for development is: run "python3 -m http.server 8888" from
-site directory to have site available at http://localhost:8888/
+To test the site locally without Docker, a webserver is required or you'll
+get "Cross origin requests are only supported for HTTP" errors. An easy setup
+for development is: run "python3 -m http.server 8888" from the site directory
+to have the site available at http://localhost:8888/
+
 
-Current crontab settings:
+Current crontab settings (projects-vm3):
 
 crontab root:
 # m h  dom mon dow   command