You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by sb...@apache.org on 2025/10/10 14:37:06 UTC
(tooling-trusted-releases) branch main updated: Restructure some documentation, and add a navigation build script
This is an automated email from the ASF dual-hosted git repository.
sbp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-trusted-releases.git
The following commit(s) were added to refs/heads/main by this push:
new d8f0289 Restructure some documentation, and add a navigation build script
d8f0289 is described below
commit d8f0289c9b8dc6394368b6ca1cf9a97d2744f437
Author: Sean B. Palmer <se...@miscoranda.com>
AuthorDate: Fri Oct 10 15:36:40 2025 +0100
Restructure some documentation, and add a navigation build script
---
Makefile | 1 +
atr/docs/build-processes.html | 14 ++
atr/docs/build-processes.md | 23 ++
atr/docs/contribution-guide.html | 2 -
atr/docs/contribution-guide.md | 3 -
atr/docs/database.html | 10 +
atr/docs/database.md | 15 ++
atr/docs/developer-guide.html | 84 ++-----
atr/docs/developer-guide.md | 132 ++---------
atr/docs/how-to-contribute.html | 10 +
atr/docs/how-to-contribute.md | 15 ++
atr/docs/index.html | 16 +-
atr/docs/index.md | 17 +-
atr/docs/introduction-to-atr.html | 6 +-
atr/docs/introduction-to-atr.md | 8 +-
...eloper-guide.html => overview-of-the-code.html} | 67 ++----
...{developer-guide.md => overview-of-the-code.md} | 92 ++------
atr/docs/running-the-server.html | 46 ++++
atr/docs/running-the-server.md | 71 ++++++
atr/docs/user-guide.html | 10 +
atr/docs/user-guide.md | 15 ++
atr/static/css/atr.css | 2 +-
scripts/build_docs.py | 255 +++++++++++++++++++++
23 files changed, 592 insertions(+), 322 deletions(-)
diff --git a/Makefile b/Makefile
index 38de84b..69df453 100644
--- a/Makefile
+++ b/Makefile
@@ -49,6 +49,7 @@ commit:
docs:
rm -f atr/docs/*.html docs/*.html
+ uv run python3 scripts/build_docs.py
for fn in atr/docs/*.md docs/*.md; \
do \
cmark "$$fn" > "$${fn%.md}.html"; \
diff --git a/atr/docs/build-processes.html b/atr/docs/build-processes.html
new file mode 100644
index 0000000..700b3f1
--- /dev/null
+++ b/atr/docs/build-processes.html
@@ -0,0 +1,14 @@
+<h1 id="4-build-processes">3.4. Build processes</h1>
+<p><strong>Up</strong>: <code>3.</code> <a href="developer-guide">Developer guide</a></p>
+<p><strong>Prev</strong>: <code>3.3.</code> <a href="database">Database</a></p>
+<p><strong>Next</strong>: <code>3.5.</code> <a href="how-to-contribute">How to contribute</a></p>
+<p><strong>Sections</strong>:</p>
+<ul>
+<li><a href="#documentation-build-script">Documentation build script</a></li>
+</ul>
+<h2 id="documentation-build-script">Documentation build script</h2>
+<p>The ATR documentation that you're reading right now is structured like a book, with numbered chapters, sections, and navigation links between pages. We could maintain all of this by hand, but that would be tedious and error-prone. Instead, we use <a href="/ref/scripts/build_docs.py"><code>scripts/build_docs.py</code></a> to generate the navigation automatically from a single table of contents.</p>
+<p>The script reads the table of contents in <a href="/ref/atr/docs/index.md"><code>atr/docs/index.md</code></a>, extracts the hierarchy of pages, and then updates every referenced page to include the correct navigation links, page numbers, and section listings. This means that when we want to reorganize the documentation (say, inserting a new chapter or moving sections around) we only need to edit the table of contents, run the script, and all the navigation is updated automatically.</p>
+<p>The implementation is straightforward. The <a href="/ref/scripts/build_docs.py:parse_toc"><code>parse_toc</code></a> function extracts entries from the table of contents section in the index, and <a href="/ref/scripts/build_docs.py:build_navigation"><code>build_navigation</code></a> computes the up, previous, and next relationships for each page. The <a href="/ref/scripts/build_docs.py:update_document"><code>update_document</code></a> function is then called for each page, which rewri [...]
+<p>The navigation block itself is generated by <a href="/ref/scripts/build_docs.py:generate_navigation_block"><code>generate_navigation_block</code></a>, which formats the up, previous, and next links, adds a list of subpages if any exist, and includes a table of contents for the page's sections as extracted by <a href="/ref/scripts/build_docs.py:extract_h2_headings"><code>extract_h2_headings</code></a>. This keeps all of the navigational machinery separate from the actual content, which [...]
+<p>We also validate that every page in the table of contents exists, and that there are no unlinked Markdown files in the documentation directory. The <a href="/ref/scripts/build_docs.py:validate_files"><code>validate_files</code></a> function performs these checks and fails with a descriptive error if anything is wrong. This prevents us from accidentally forgetting to add a page to the table of contents, or from leaving old pages lying around that we meant to delete.</p>
diff --git a/atr/docs/build-processes.md b/atr/docs/build-processes.md
new file mode 100644
index 0000000..fe7ca71
--- /dev/null
+++ b/atr/docs/build-processes.md
@@ -0,0 +1,23 @@
+# 3.4. Build processes
+
+**Up**: `3.` [Developer guide](developer-guide)
+
+**Prev**: `3.3.` [Database](database)
+
+**Next**: `3.5.` [How to contribute](how-to-contribute)
+
+**Sections**:
+
+* [Documentation build script](#documentation-build-script)
+
+## Documentation build script
+
+The ATR documentation that you're reading right now is structured like a book, with numbered chapters, sections, and navigation links between pages. We could maintain all of this by hand, but that would be tedious and error-prone. Instead, we use [`scripts/build_docs.py`](/ref/scripts/build_docs.py) to generate the navigation automatically from a single table of contents.
+
+The script reads the table of contents in [`atr/docs/index.md`](/ref/atr/docs/index.md), extracts the hierarchy of pages, and then updates every referenced page to include the correct navigation links, page numbers, and section listings. This means that when we want to reorganize the documentation (say, inserting a new chapter or moving sections around) we only need to edit the table of contents, run the script, and all the navigation is updated automatically.
+
+The implementation is straightforward. The [`parse_toc`](/ref/scripts/build_docs.py:parse_toc) function extracts entries from the table of contents section in the index, and [`build_navigation`](/ref/scripts/build_docs.py:build_navigation) computes the up, previous, and next relationships for each page. The [`update_document`](/ref/scripts/build_docs.py:update_document) function is then called for each page, which rewrites the page heading to match the canonical numbering from the `index [...]
+
+The navigation block itself is generated by [`generate_navigation_block`](/ref/scripts/build_docs.py:generate_navigation_block), which formats the up, previous, and next links, adds a list of subpages if any exist, and includes a table of contents for the page's sections as extracted by [`extract_h2_headings`](/ref/scripts/build_docs.py:extract_h2_headings). This keeps all of the navigational machinery separate from the actual content, which starts at the first `##` heading.
+
+We also validate that every page in the table of contents exists, and that there are no unlinked Markdown files in the documentation directory. The [`validate_files`](/ref/scripts/build_docs.py:validate_files) function performs these checks and fails with a descriptive error if anything is wrong. This prevents us from accidentally forgetting to add a page to the table of contents, or from leaving old pages lying around that we meant to delete.
diff --git a/atr/docs/contribution-guide.html b/atr/docs/contribution-guide.html
deleted file mode 100644
index e4e8c35..0000000
--- a/atr/docs/contribution-guide.html
+++ /dev/null
@@ -1,2 +0,0 @@
-<h1 id="1-contribution-guide">2.1 Contribution guide</h1>
-<p>TODO</p>
diff --git a/atr/docs/contribution-guide.md b/atr/docs/contribution-guide.md
deleted file mode 100644
index 1ceabcd..0000000
--- a/atr/docs/contribution-guide.md
+++ /dev/null
@@ -1,3 +0,0 @@
-# 2.1 Contribution guide
-
-TODO
diff --git a/atr/docs/database.html b/atr/docs/database.html
new file mode 100644
index 0000000..e274d6b
--- /dev/null
+++ b/atr/docs/database.html
@@ -0,0 +1,10 @@
+<h1 id="3-database">3.3. Database</h1>
+<p><strong>Up</strong>: <code>3.</code> <a href="developer-guide">Developer guide</a></p>
+<p><strong>Prev</strong>: <code>3.2.</code> <a href="overview-of-the-code">Overview of the code</a></p>
+<p><strong>Next</strong>: <code>3.4.</code> <a href="build-processes">Build processes</a></p>
+<p><strong>Sections</strong>:</p>
+<ul>
+<li><a href="#introduction">Introduction</a></li>
+</ul>
+<h2 id="introduction">Introduction</h2>
+<p>ATR stores all of its data in a SQLite database. The database schema is defined in <a href="/ref/atr/models/sql.py"><code>models.sql</code></a> using <a href="https://sqlmodel.tiangolo.com/">SQLModel</a>, which uses <a href="https://docs.pydantic.dev/latest/">Pydantic</a> for data validation and <a href="https://www.sqlalchemy.org/">SQLAlchemy</a> for database operations. This page explains the main features of the database schema to help you understand how data is structured in ATR.</p>
diff --git a/atr/docs/database.md b/atr/docs/database.md
new file mode 100644
index 0000000..4e49028
--- /dev/null
+++ b/atr/docs/database.md
@@ -0,0 +1,15 @@
+# 3.3. Database
+
+**Up**: `3.` [Developer guide](developer-guide)
+
+**Prev**: `3.2.` [Overview of the code](overview-of-the-code)
+
+**Next**: `3.4.` [Build processes](build-processes)
+
+**Sections**:
+
+* [Introduction](#introduction)
+
+## Introduction
+
+ATR stores all of its data in a SQLite database. The database schema is defined in [`models.sql`](/ref/atr/models/sql.py) using [SQLModel](https://sqlmodel.tiangolo.com/), which uses [Pydantic](https://docs.pydantic.dev/latest/) for data validation and [SQLAlchemy](https://www.sqlalchemy.org/) for database operations. This page explains the main features of the database schema to help you understand how data is structured in ATR.
diff --git a/atr/docs/developer-guide.html b/atr/docs/developer-guide.html
index dc1c045..a561f88 100644
--- a/atr/docs/developer-guide.html
+++ b/atr/docs/developer-guide.html
@@ -1,70 +1,18 @@
-<h1 id="developer-guide">2. Developer guide</h1>
-<p><strong>Up</strong>: <a href=".">Apache Trusted Releases documentation</a></p>
-<p><strong>Prev</strong>: 1. <a href="introduction-to-atr">Introduction to ATR</a></p>
-<p>This is a guide for developers of ATR, explaining how to make changes to the ATR source code. For more information about how to contribute those changes back to us, please read the <a href="contribution-guide">contribution guide</a> instead.</p>
-<h2 id="running-the-server">Running the server</h2>
-<p>To develop ATR locally, we manage dependencies using <a href="https://docs.astral.sh/uv/">uv</a>. To run ATR on ASF hardware, we run it in containers managed by Puppet, but since this guide is about development, we focus on using uv.</p>
-<h3 id="get-the-source">Get the source</h3>
-<p><a href="https://github.com/apache/tooling-trusted-releases/fork">Fork the source code</a> of <a href="https://github.com/apache/tooling-trusted-releases">ATR on GitHub</a>, and then <a href="https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository">clone your fork locally</a>.</p>
-<p>There are lots of files and directories in the root of the ATR Git repository. The most important thing to know is that <code>atr/</code> contains the source code. ATR is a Python application based on <a href="https://github.com/apache/infrastructure-asfquart">ASFQuart</a>, which is based on <a href="https://github.com/pallets/quart">Quart</a>. The Quart web framework is an asynchronous version of <a href="https://github.com/pallets/flask">Flask</a>, a very widely used synchronous web [...]
-<h3 id="install-dependencies">Install dependencies</h3>
-<p>To run ATR locally after cloning the source, you will need to install the following dependencies:</p>
+<h1 id="developer-guide">3. Developer guide</h1>
+<p><strong>Up</strong>: <a href=".">Documentation</a></p>
+<p><strong>Prev</strong>: <code>2.</code> <a href="user-guide">User guide</a></p>
+<p><strong>Next</strong>: (none)</p>
+<p><strong>Pages</strong>:</p>
+<ul>
+<li><code>3.1.</code> <a href="running-the-server">Running the server</a></li>
+<li><code>3.2.</code> <a href="overview-of-the-code">Overview of the code</a></li>
+<li><code>3.3.</code> <a href="database">Database</a></li>
+<li><code>3.4.</code> <a href="build-processes">Build processes</a></li>
+<li><code>3.5.</code> <a href="how-to-contribute">How to contribute</a></li>
+</ul>
+<p><strong>Sections</strong>:</p>
<ul>
-<li>Any <a href="https://en.wikipedia.org/wiki/POSIX">POSIX</a> compliant <a href="https://frippery.org/make/">make</a></li>
-<li><a href="https://github.com/FiloSottile/mkcert">mkcert</a></li>
-<li><a href="https://www.python.org/downloads/release/python-3138/">Python 3.13</a></li>
-<li><a href="https://docs.astral.sh/uv/#installation">uv</a></li>
+<li><a href="#introduction">Introduction</a></li>
</ul>
-<p>You can install Python 3.13 through your package manager or through uv. Here is how to install these dependencies on <a href="https://en.wikipedia.org/wiki/Alpine_Linux">Alpine Linux</a>:</p>
-<pre><code class="language-shell">apk add curl git make mkcert@testing
-curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/local/bin" sh
-uv python install 3.13
-</code></pre>
-<p>You do not need to have Alpine Linux to develop ATR. It should work in any POSIX style environment.</p>
-<h3 id="run-the-server">Run the server</h3>
-<p>Then, to run the server:</p>
-<pre><code class="language-shell">cd tooling-trusted-releases/
-mkdir state
-make certs-local
-make serve-local
-</code></pre>
-<p>The <code>certs-local</code> step runs <code>mkcert localhost.apache.org localhost 127.0.0.1 ::1</code> to generate a locally trusted TLS certificate. If the certificate is not trusted, you may have to follow the <a href="https://github.com/FiloSottile/mkcert/blob/master/README.md">mkcert guide</a> to resolve the issue.</p>
-<p>ATR requires TLS even for development because login is performed through the actual ASF OAuth server. This way, the development behavior aligns closely with the production behavior. We try to minimize differences between development and production environments.</p>
-<h3 id="load-the-site">Load the site</h3>
-<p>ATR will then be served on various hosts, but we recommend using only <code>localhost.apache.org</code>. This requires adding an entry to your <code>/etc/hosts</code> and potentially restarting your DNS server. If you do this, the following link should work:</p>
-<p><a href="https://localhost.apache.org:8080/"><code>https://localhost.apache.org:8080/</code></a></p>
-<p>If you do not want to change your <code>/etc/hosts</code>, you can use <code>127.0.0.1</code>. You should not use <code>localhost</code>. The following link should work:</p>
-<p><a href="https://127.0.0.1:8080/"><code>https://127.0.0.1:8080/</code></a></p>
-<p>Pick one or the other, because logging into the site on one host does not log you in to the site on any other host.</p>
-<h2 id="understanding-the-code">Understanding the code</h2>
-<p>Once you have the server running, you can test it. At this point, it is useful to understand how the ATR works in general. References to symbols in this section are given without their <code>atr.</code> prefix, for brevity, and are linked to the respective source code. You should understand e.g. <a href="/ref/atr/server.py:app"><code>server.app</code></a> to mean <code>atr.server.app</code>.</p>
-<h3 id="hypercorn-and-asgi">Hypercorn and ASGI</h3>
-<p>ATR is an <a href="https://github.com/apache/infrastructure-asfquart">ASFQuart</a> application, running in <a href="https://hypercorn.readthedocs.io/en/latest/index.html">Hypercorn</a>. The entry point for Hypercorn is <a href="/ref/atr/server.py:app"><code>server.app</code></a>, which is then called with a few standard arguments <a href="https://asgi.readthedocs.io/en/latest/specs/main.html#overview">as per the ASGI specification</a>. We create the <code>app</code> object using the f [...]
-<pre><code class="language-python">app = create_app(config.get())
-</code></pre>
-<p>The <a href="/ref/atr/server.py:create_app"><code>server.create_app</code></a> function performs a lot of setup, and if you're interested in how the server works then you can read it and the functions it calls to understand the process further. In general, however, when developing ATR we do not make modifications at the ASFQuart, Quart, and Hypercorn levels very often.</p>
-<h3 id="routes-and-database">Routes and database</h3>
-<p>Users request ATR pages over HTTPS, and the ATR server processes those requests in route handlers. Most of those handlers are in <a href="/ref/atr/routes/"><code>routes</code></a>, but not all. What each handler does varies, of course, from handler to handler, but most perform at least one access to the ATR SQLite database.</p>
-<p>The path of the SQLite database is configured in <a href="/ref/atr/config.py:SQLITE_DB_PATH"><code>config.AppConfig.SQLITE_DB_PATH</code></a> by default, and will usually appear as <code>state/atr.db</code> with related <code>shm</code> and <code>wal</code> files. We do not expect ATR to have so many users that we need to scale beyond SQLite.</p>
-<p>We use <a href="https://sqlmodel.tiangolo.com/">SQLModel</a>, an ORM utilising <a href="https://docs.pydantic.dev/latest/">Pydantic</a> and <a href="https://www.sqlalchemy.org/">SQLAlchemy</a>, to create Python models for the ATR database. The core models file is <a href="/ref/atr/models/sql.py"><code>models.sql</code></a>. The most important individual SQLite models in this module are <a href="/ref/atr/models/sql.py:Committee"><code>Committee</code></a>, <a href="/ref/atr/models/sql. [...]
-<p>It is technically possible to interact with SQLite directly, but we do not do that in the ATR source. We use various interfaces in <a href="/ref/atr/db/__init__.py"><code>db</code></a> for reads, and interfaces in <a href="/ref/atr/storage/"><code>storage</code></a> for writes. We plan to move the <code>db</code> code into <code>storage</code> too eventually, because <code>storage</code> is designed to have read components and write components. There is also a legacy <a href="/ref/atr [...]
-<p>These three interfaces, <a href="/ref/atr/routes/"><code>routes</code></a>, <a href="/ref/atr/models/sql.py"><code>models.sql</code></a>, and <a href="/ref/atr/storage/"><code>storage</code></a>, are where the majority of activity happens when developing ATR.</p>
-<h3 id="user-interface">User interface</h3>
-<p>ATR provides a web interface for users to interact with the platform, and the implementation of that interface is split across several modules. The web interface uses server-side rendering almost entirely, where HTML is generated on the server and sent to the browser.</p>
-<p>The template system in ATR is <a href="https://jinja.palletsprojects.com/">Jinja2</a>, always accessed through the ATR <a href="/ref/atr/template.py"><code>template</code></a> module. Template files in Jinja2 syntax are stored in <a href="/ref/atr/templates/"><code>templates/</code></a>, and route handlers render them using the asynchronous <a href="/ref/atr/template.py:render"><code>template.render</code></a> function.</p>
-<p>Template rendering can be slow if templates are loaded from disk on every request. To address this, we use <a href="/ref/atr/preload.py"><code>preload</code></a> to load all templates into memory before the server starts serving requests. The <a href="/ref/atr/preload.py:setup_template_preloading"><code>preload.setup_template_preloading</code></a> function registers a startup hook that finds and caches every template file.</p>
-<p>The ATR user interface includes many HTML forms. We use <a href="https://wtforms.readthedocs.io/">WTForms</a> for form handling, accessed through the ATR <a href="/ref/atr/forms.py"><code>forms</code></a> module. The <a href="/ref/atr/forms.py:Typed"><code>forms.Typed</code></a> base class extends the standard <code>QuartForm</code> class in <a href="https://quart-wtf.readthedocs.io/">Quart-WTF</a>. Each form field is created using helper functions such as <a href="/ref/atr/forms.py:s [...]
-<p>In addition to templates, we sometimes need to generate HTML programmatically in Python. For this we use <a href="https://htpy.dev/">htpy</a>, another third party library, for building HTML using Python syntax. The ATR <a href="/ref/atr/htm.py"><code>htm</code></a> module extends htpy with a <a href="/ref/atr/htm.py:Block"><code>Block</code></a> class that makes it easier to build complex HTML structures incrementally. Using htpy means that we get type checking for our HTML generation [...]
-<h3 id="scheduling-and-tasks">Scheduling and tasks</h3>
-<p>Many operations in ATR are too slow to run during an HTTP request, so we run them asynchronously in background worker processes. The task scheduling system in ATR is built from three components: a task queue stored in the SQLite database, a worker manager that spawns and monitors worker processes, and the worker processes themselves that claim and execute tasks.</p>
-<p>The task queue is stored in a table defined by the <a href="/ref/atr/models/sql.py:Task"><code>Task</code></a> model in <a href="/ref/atr/models/sql.py"><code>models.sql</code></a>. Each task has a status, a type, arguments encoded as JSON, and metadata such as when it was added and which user created it. When route handlers need to perform slow operations, they create a new <code>Task</code> row with status <code>QUEUED</code> and commit it to the database.</p>
-<p>The ATR <a href="/ref/atr/manager.py"><code>manager</code></a> module provides the <a href="/ref/atr/manager.py:WorkerManager"><code>WorkerManager</code></a> class, which maintains a pool of worker processes. When the ATR server starts, the manager spawns a configurable number of worker processes and monitors them continuously. The manager checks every few seconds whether workers are still running, whether any tasks have exceeded their time limits, and whether the worker pool needs to [...]
-<p>The ATR <a href="/ref/atr/worker.py"><code>worker</code></a> module implements the workers. Each worker process runs in a loop. It claims the oldest queued task from the database, executes it, records the result, and then claims the next task atomically using an <code>UPDATE ... WHERE</code> statement. After a worker has processed a fixed number of tasks, it exits voluntarily to help to avoid memory leaks. The manager then spawns a fresh worker to replace it. Task execution happens in [...]
-<p>Tasks themselves are defined in the ATR <a href="/ref/atr/tasks/"><code>tasks</code></a> directory. The <a href="/ref/atr/tasks/__init__.py"><code>tasks</code></a> module contains functions for queueing tasks and resolving task types to their handler functions. Task types include operations such as importing keys, generating SBOMs, sending messages, and importing files from SVN. The most common category of task is automated checks on release artifacts. These checks are implemented in [...]
-<h3 id="authentication">Authentication</h3>
-<p>ATR uses ASF OAuth for user login, and then determines what actions each user can perform based on their committee memberships. The ATR <a href="/ref/atr/principal.py"><code>principal</code></a> module handles authorization by checking whether users are members of relevant committees. It queries and caches LDAP to get committee membership information. The <a href="/ref/atr/principal.py:Authorisation"><code>Authorisation</code></a> class provides methods to check whether a user is a me [...]
-<h3 id="api">API</h3>
-<p>The ATR API provides programmatic access to most ATR functionality. API endpoints are defined in <a href="/ref/atr/blueprints/api/api.py"><code>blueprints.api.api</code></a>, and their URL paths are prefixed with <code>/api/</code>. The API uses <a href="https://www.openapis.org/">OpenAPI</a> for documentation, which is automatically generated from the endpoint definitions and served at <code>/api/docs</code>. Users send requests with a <a href="https://en.wikipedia.org/wiki/JSON_Web_ [...]
-<p>API request and response models are defined in <a href="/ref/atr/models/api.py"><code>models.api</code></a> using Pydantic. Each endpoint has an associated request model that validates incoming data, and a response model that validates outgoing data. The API returns JSON in all cases, with appropriate HTTP status codes.</p>
-<h3 id="other-important-interfaces">Other important interfaces</h3>
-<p>The server configuration in <a href="/ref/atr/config.py"><code>config</code></a> determines a lot of global state, and the <a href="/ref/atr/util.py"><code>util</code></a> module contains lots of useful code which is used throughout ATR.</p>
+<h2 id="introduction">Introduction</h2>
+<p>This is a guide for developers of ATR, explaining how to make changes to the ATR source code. For more information about how to contribute those changes back to us, please read the <a href="contribution-guide">contribution guide</a> instead.</p>
diff --git a/atr/docs/developer-guide.md b/atr/docs/developer-guide.md
index a6dd653..8d752c7 100644
--- a/atr/docs/developer-guide.md
+++ b/atr/docs/developer-guide.md
@@ -1,127 +1,23 @@
-# 2. Developer guide
+# 3. Developer guide
-**Up**: [Apache Trusted Releases documentation](.)
+**Up**: [Documentation](.)
-**Prev**: 1. [Introduction to ATR](introduction-to-atr)
+**Prev**: `2.` [User guide](user-guide)
-This is a guide for developers of ATR, explaining how to make changes to the ATR source code. For more information about how to contribute those changes back to us, please read the [contribution guide](contribution-guide) instead.
-
-## Running the server
-
-To develop ATR locally, we manage dependencies using [uv](https://docs.astral.sh/uv/). To run ATR on ASF hardware, we run it in containers managed by Puppet, but since this guide is about development, we focus on using uv.
-
-### Get the source
-
-[Fork the source code](https://github.com/apache/tooling-trusted-releases/fork) of [ATR on GitHub](https://github.com/apache/tooling-trusted-releases), and then [clone your fork locally](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository).
-
-There are lots of files and directories in the root of the ATR Git repository. The most important thing to know is that `atr/` contains the source code. ATR is a Python application based on [ASFQuart](https://github.com/apache/infrastructure-asfquart), which is based on [Quart](https://github.com/pallets/quart). The Quart web framework is an asynchronous version of [Flask](https://github.com/pallets/flask), a very widely used synchronous web framework. In addition to Python, we use small [...]
-
-### Install dependencies
-
-To run ATR locally after cloning the source, you will need to install the following dependencies:
-
-* Any [POSIX](https://en.wikipedia.org/wiki/POSIX) compliant [make](https://frippery.org/make/)
-* [mkcert](https://github.com/FiloSottile/mkcert)
-* [Python 3.13](https://www.python.org/downloads/release/python-3138/)
-* [uv](https://docs.astral.sh/uv/#installation)
-
-You can install Python 3.13 through your package manager or through uv. Here is how to install these dependencies on [Alpine Linux](https://en.wikipedia.org/wiki/Alpine_Linux):
-
-```shell
-apk add curl git make mkcert@testing
-curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/local/bin" sh
-uv python install 3.13
-```
-
-You do not need to have Alpine Linux to develop ATR. It should work in any POSIX style environment.
-
-### Run the server
-
-Then, to run the server:
-
-```shell
-cd tooling-trusted-releases/
-mkdir state
-make certs-local
-make serve-local
-```
-
-The `certs-local` step runs `mkcert localhost.apache.org localhost 127.0.0.1 ::1` to generate a locally trusted TLS certificate. If the certificate is not trusted, you may have to follow the [mkcert guide](https://github.com/FiloSottile/mkcert/blob/master/README.md) to resolve the issue.
-
-ATR requires TLS even for development because login is performed through the actual ASF OAuth server. This way, the development behavior aligns closely with the production behavior. We try to minimize differences between development and production environments.
-
-### Load the site
-
-ATR will then be served on various hosts, but we recommend using only `localhost.apache.org`. This requires adding an entry to your `/etc/hosts` and potentially restarting your DNS server. If you do this, the following link should work:
-
-[`https://localhost.apache.org:8080/`](https://localhost.apache.org:8080/)
-
-If you do not want to change your `/etc/hosts`, you can use `127.0.0.1`. You should not use `localhost`. The following link should work:
-
-[`https://127.0.0.1:8080/`](https://127.0.0.1:8080/)
-
-Pick one or the other, because logging into the site on one host does not log you in to the site on any other host.
-
-## Understanding the code
-
-Once you have the server running, you can test it. At this point, it is useful to understand how the ATR works in general. References to symbols in this section are given without their `atr.` prefix, for brevity, and are linked to the respective source code. You should understand e.g. [`server.app`](/ref/atr/server.py:app) to mean `atr.server.app`.
+**Next**: (none)
-### Hypercorn and ASGI
+**Pages**:
-ATR is an [ASFQuart](https://github.com/apache/infrastructure-asfquart) application, running in [Hypercorn](https://hypercorn.readthedocs.io/en/latest/index.html). The entry point for Hypercorn is [`server.app`](/ref/atr/server.py:app), which is then called with a few standard arguments [as per the ASGI specification](https://asgi.readthedocs.io/en/latest/specs/main.html#overview). We create the `app` object using the following code:
+* `3.1.` [Running the server](running-the-server)
+* `3.2.` [Overview of the code](overview-of-the-code)
+* `3.3.` [Database](database)
+* `3.4.` [Build processes](build-processes)
+* `3.5.` [How to contribute](how-to-contribute)
-```python
-app = create_app(config.get())
-```
+**Sections**:
-The [`server.create_app`](/ref/atr/server.py:create_app) function performs a lot of setup, and if you're interested in how the server works then you can read it and the functions it calls to understand the process further. In general, however, when developing ATR we do not make modifications at the ASFQuart, Quart, and Hypercorn levels very often.
+* [Introduction](#introduction)
-### Routes and database
+## Introduction
-Users request ATR pages over HTTPS, and the ATR server processes those requests in route handlers. Most of those handlers are in [`routes`](/ref/atr/routes/), but not all. What each handler does varies, of course, from handler to handler, but most perform at least one access to the ATR SQLite database.
-
-The path of the SQLite database is configured in [`config.AppConfig.SQLITE_DB_PATH`](/ref/atr/config.py:SQLITE_DB_PATH) by default, and will usually appear as `state/atr.db` with related `shm` and `wal` files. We do not expect ATR to have so many users that we need to scale beyond SQLite.
-
-We use [SQLModel](https://sqlmodel.tiangolo.com/), an ORM utilising [Pydantic](https://docs.pydantic.dev/latest/) and [SQLAlchemy](https://www.sqlalchemy.org/), to create Python models for the ATR database. The core models file is [`models.sql`](/ref/atr/models/sql.py). The most important individual SQLite models in this module are [`Committee`](/ref/atr/models/sql.py:Committee), [`Project`](/ref/atr/models/sql.py:Project), and [`Release`](/ref/atr/models/sql.py:Release).
-
-It is technically possible to interact with SQLite directly, but we do not do that in the ATR source. We use various interfaces in [`db`](/ref/atr/db/__init__.py) for reads, and interfaces in [`storage`](/ref/atr/storage/) for writes. We plan to move the `db` code into `storage` too eventually, because `storage` is designed to have read components and write components. There is also a legacy [`db.interaction`](/ref/atr/db/interaction.py) module which we plan to migrate into `storage`.
-
-These three interfaces, [`routes`](/ref/atr/routes/), [`models.sql`](/ref/atr/models/sql.py), and [`storage`](/ref/atr/storage/), are where the majority of activity happens when developing ATR.
-
-### User interface
-
-ATR provides a web interface for users to interact with the platform, and the implementation of that interface is split across several modules. The web interface uses server-side rendering almost entirely, where HTML is generated on the server and sent to the browser.
-
-The template system in ATR is [Jinja2](https://jinja.palletsprojects.com/), always accessed through the ATR [`template`](/ref/atr/template.py) module. Template files in Jinja2 syntax are stored in [`templates/`](/ref/atr/templates/), and route handlers render them using the asynchronous [`template.render`](/ref/atr/template.py:render) function.
-
-Template rendering can be slow if templates are loaded from disk on every request. To address this, we use [`preload`](/ref/atr/preload.py) to load all templates into memory before the server starts serving requests. The [`preload.setup_template_preloading`](/ref/atr/preload.py:setup_template_preloading) function registers a startup hook that finds and caches every template file.
-
-The ATR user interface includes many HTML forms. We use [WTForms](https://wtforms.readthedocs.io/) for form handling, accessed through the ATR [`forms`](/ref/atr/forms.py) module. The [`forms.Typed`](/ref/atr/forms.py:Typed) base class extends the standard `QuartForm` class in [Quart-WTF](https://quart-wtf.readthedocs.io/). Each form field is created using helper functions such as [`forms.string`](/ref/atr/forms.py:string), [`forms.select`](/ref/atr/forms.py:select), and [`forms.submit`] [...]
-
-In addition to templates, we sometimes need to generate HTML programmatically in Python. For this we use [htpy](https://htpy.dev/), another third party library, for building HTML using Python syntax. The ATR [`htm`](/ref/atr/htm.py) module extends htpy with a [`Block`](/ref/atr/htm.py:Block) class that makes it easier to build complex HTML structures incrementally. Using htpy means that we get type checking for our HTML generation, and can compose HTML elements just like any other Python [...]
-
-### Scheduling and tasks
-
-Many operations in ATR are too slow to run during an HTTP request, so we run them asynchronously in background worker processes. The task scheduling system in ATR is built from three components: a task queue stored in the SQLite database, a worker manager that spawns and monitors worker processes, and the worker processes themselves that claim and execute tasks.
-
-The task queue is stored in a table defined by the [`Task`](/ref/atr/models/sql.py:Task) model in [`models.sql`](/ref/atr/models/sql.py). Each task has a status, a type, arguments encoded as JSON, and metadata such as when it was added and which user created it. When route handlers need to perform slow operations, they create a new `Task` row with status `QUEUED` and commit it to the database.
-
-The ATR [`manager`](/ref/atr/manager.py) module provides the [`WorkerManager`](/ref/atr/manager.py:WorkerManager) class, which maintains a pool of worker processes. When the ATR server starts, the manager spawns a configurable number of worker processes and monitors them continuously. The manager checks every few seconds whether workers are still running, whether any tasks have exceeded their time limits, and whether the worker pool needs to be replenished. If a worker process exits afte [...]
-
-The ATR [`worker`](/ref/atr/worker.py) module implements the workers. Each worker process runs in a loop. It claims the oldest queued task from the database, executes it, records the result, and then claims the next task atomically using an `UPDATE ... WHERE` statement. After a worker has processed a fixed number of tasks, it exits voluntarily to help to avoid memory leaks. The manager then spawns a fresh worker to replace it. Task execution happens in the [`_task_process`](/ref/atr/work [...]
-
-Tasks themselves are defined in the ATR [`tasks`](/ref/atr/tasks/) directory. The [`tasks`](/ref/atr/tasks/__init__.py) module contains functions for queueing tasks and resolving task types to their handler functions. Task types include operations such as importing keys, generating SBOMs, sending messages, and importing files from SVN. The most common category of task is automated checks on release artifacts. These checks are implemented in [`tasks/checks/`](/ref/atr/tasks/checks/), and [...]
-
-### Authentication
-
-ATR uses ASF OAuth for user login, and then determines what actions each user can perform based on their committee memberships. The ATR [`principal`](/ref/atr/principal.py) module handles authorization by checking whether users are members of relevant committees. It queries and caches LDAP to get committee membership information. The [`Authorisation`](/ref/atr/principal.py:Authorisation) class provides methods to check whether a user is a member of a committee or a project participant, w [...]
-
-### API
-
-The ATR API provides programmatic access to most ATR functionality. API endpoints are defined in [`blueprints.api.api`](/ref/atr/blueprints/api/api.py), and their URL paths are prefixed with `/api/`. The API uses [OpenAPI](https://www.openapis.org/) for documentation, which is automatically generated from the endpoint definitions and served at `/api/docs`. Users send requests with a [JWT](https://en.wikipedia.org/wiki/JSON_Web_Token) created from a [PAT](https://en.wikipedia.org/wiki/Per [...]
-
-API request and response models are defined in [`models.api`](/ref/atr/models/api.py) using Pydantic. Each endpoint has an associated request model that validates incoming data, and a response model that validates outgoing data. The API returns JSON in all cases, with appropriate HTTP status codes.
-
-### Other important interfaces
-
-The server configuration in [`config`](/ref/atr/config.py) determines a lot of global state, and the [`util`](/ref/atr/util.py) module contains lots of useful code which is used throughout ATR.
+This is a guide for developers of ATR, explaining how to make changes to the ATR source code. For more information about how to contribute those changes back to us, please read the [contribution guide](contribution-guide) instead.
diff --git a/atr/docs/how-to-contribute.html b/atr/docs/how-to-contribute.html
new file mode 100644
index 0000000..dcebd13
--- /dev/null
+++ b/atr/docs/how-to-contribute.html
@@ -0,0 +1,10 @@
+<h1 id="5-how-to-contribute">3.5. How to contribute</h1>
+<p><strong>Up</strong>: <code>3.</code> <a href="developer-guide">Developer guide</a></p>
+<p><strong>Prev</strong>: <code>3.4.</code> <a href="build-processes">Build processes</a></p>
+<p><strong>Next</strong>: (none)</p>
+<p><strong>Sections</strong>:</p>
+<ul>
+<li><a href="#introduction">Introduction</a></li>
+</ul>
+<h2 id="introduction">Introduction</h2>
+<p>[TODO]</p>
diff --git a/atr/docs/how-to-contribute.md b/atr/docs/how-to-contribute.md
new file mode 100644
index 0000000..dc9a891
--- /dev/null
+++ b/atr/docs/how-to-contribute.md
@@ -0,0 +1,15 @@
+# 3.5. How to contribute
+
+**Up**: `3.` [Developer guide](developer-guide)
+
+**Prev**: `3.4.` [Build processes](build-processes)
+
+**Next**: (none)
+
+**Sections**:
+
+* [Introduction](#introduction)
+
+## Introduction
+
+[TODO]
diff --git a/atr/docs/index.html b/atr/docs/index.html
index dc3be12..7426c6a 100644
--- a/atr/docs/index.html
+++ b/atr/docs/index.html
@@ -1,11 +1,17 @@
-<h1 id="apache-trusted-releases-documentation">Apache Trusted Releases documentation</h1>
-<p>Welcome to the documentation for the <strong>Apache Trusted Releases</strong> (ATR) platform.</p>
+<h1 id="apache-trusted-releases-atr-documentation">Apache Trusted Releases (ATR) documentation</h1>
+<p>Welcome to the documentation for the <strong>ATR</strong> platform.</p>
<p>NOTE: This documentation is a work in progress.</p>
+<h2 id="table-of-contents">Table of contents</h2>
<ul>
-<li>1. <a href="introduction-to-atr">Introduction to ATR</a></li>
-<li>2. <a href="developer-guide">Developer guide</a>
+<li><code>1.</code> <a href="introduction-to-atr">Introduction to ATR</a></li>
+<li><code>2.</code> <a href="user-guide">User guide</a></li>
+<li><code>3.</code> <a href="developer-guide">Developer guide</a>
<ul>
-<li>2.1 <a href="contribution-guide">Contribution guide</a></li>
+<li><code>3.1.</code> <a href="running-the-server">Running the server</a></li>
+<li><code>3.2.</code> <a href="overview-of-the-code">Overview of the code</a></li>
+<li><code>3.3.</code> <a href="database">Database</a></li>
+<li><code>3.4.</code> <a href="build-processes">Build processes</a></li>
+<li><code>3.5.</code> <a href="how-to-contribute">How to contribute</a></li>
</ul>
</li>
</ul>
diff --git a/atr/docs/index.md b/atr/docs/index.md
index e8e6a68..4410aba 100644
--- a/atr/docs/index.md
+++ b/atr/docs/index.md
@@ -1,9 +1,16 @@
-# Apache Trusted Releases documentation
+# Apache Trusted Releases (ATR) documentation
-Welcome to the documentation for the **Apache Trusted Releases** (ATR) platform.
+Welcome to the documentation for the **ATR** platform.
NOTE: This documentation is a work in progress.
-* 1\. [Introduction to ATR](introduction-to-atr)
-* 2\. [Developer guide](developer-guide)
- * 2.1 [Contribution guide](contribution-guide)
+## Table of contents
+
+* `1.` [Introduction to ATR](introduction-to-atr)
+* `2.` [User guide](user-guide)
+* `3.` [Developer guide](developer-guide)
+ * `3.1.` [Running the server](running-the-server)
+ * `3.2.` [Overview of the code](overview-of-the-code)
+ * `3.3.` [Database](database)
+ * `3.4.` [Build processes](build-processes)
+ * `3.5.` [How to contribute](how-to-contribute)
diff --git a/atr/docs/introduction-to-atr.html b/atr/docs/introduction-to-atr.html
index 3a98507..82df722 100644
--- a/atr/docs/introduction-to-atr.html
+++ b/atr/docs/introduction-to-atr.html
@@ -1,6 +1,8 @@
<h1 id="introduction-to-atr">1. Introduction to ATR</h1>
-<p><strong>Up</strong>: <a href=".">Apache Trusted Releases documentation</a></p>
-<p><strong>Next</strong>: 2. <a href="developer-guide">Developer guide</a></p>
+<p><strong>Up</strong>: <a href=".">Documentation</a></p>
+<p><strong>Prev</strong>: (none)</p>
+<p><strong>Next</strong>: <code>2.</code> <a href="user-guide">User guide</a></p>
+<p><strong>Sections</strong>:</p>
<ul>
<li><a href="#what-is-atr">What is ATR?</a></li>
<li><a href="#who-are-atr-users">Who are ATR users?</a></li>
diff --git a/atr/docs/introduction-to-atr.md b/atr/docs/introduction-to-atr.md
index f39eecd..acc5b20 100644
--- a/atr/docs/introduction-to-atr.md
+++ b/atr/docs/introduction-to-atr.md
@@ -1,8 +1,12 @@
# 1. Introduction to ATR
-**Up**: [Apache Trusted Releases documentation](.)
+**Up**: [Documentation](.)
-**Next**: 2. [Developer guide](developer-guide)
+**Prev**: (none)
+
+**Next**: `2.` [User guide](user-guide)
+
+**Sections**:
* [What is ATR?](#what-is-atr)
* [Who are ATR users?](#who-are-atr-users)
diff --git a/atr/docs/developer-guide.html b/atr/docs/overview-of-the-code.html
similarity index 68%
copy from atr/docs/developer-guide.html
copy to atr/docs/overview-of-the-code.html
index dc1c045..e9e1f34 100644
--- a/atr/docs/developer-guide.html
+++ b/atr/docs/overview-of-the-code.html
@@ -1,70 +1,45 @@
-<h1 id="developer-guide">2. Developer guide</h1>
-<p><strong>Up</strong>: <a href=".">Apache Trusted Releases documentation</a></p>
-<p><strong>Prev</strong>: 1. <a href="introduction-to-atr">Introduction to ATR</a></p>
-<p>This is a guide for developers of ATR, explaining how to make changes to the ATR source code. For more information about how to contribute those changes back to us, please read the <a href="contribution-guide">contribution guide</a> instead.</p>
-<h2 id="running-the-server">Running the server</h2>
-<p>To develop ATR locally, we manage dependencies using <a href="https://docs.astral.sh/uv/">uv</a>. To run ATR on ASF hardware, we run it in containers managed by Puppet, but since this guide is about development, we focus on using uv.</p>
-<h3 id="get-the-source">Get the source</h3>
-<p><a href="https://github.com/apache/tooling-trusted-releases/fork">Fork the source code</a> of <a href="https://github.com/apache/tooling-trusted-releases">ATR on GitHub</a>, and then <a href="https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository">clone your fork locally</a>.</p>
-<p>There are lots of files and directories in the root of the ATR Git repository. The most important thing to know is that <code>atr/</code> contains the source code. ATR is a Python application based on <a href="https://github.com/apache/infrastructure-asfquart">ASFQuart</a>, which is based on <a href="https://github.com/pallets/quart">Quart</a>. The Quart web framework is an asynchronous version of <a href="https://github.com/pallets/flask">Flask</a>, a very widely used synchronous web [...]
-<h3 id="install-dependencies">Install dependencies</h3>
-<p>To run ATR locally after cloning the source, you will need to install the following dependencies:</p>
+<h1 id="2-overview-of-the-code">3.2. Overview of the code</h1>
+<p><strong>Up</strong>: <code>3.</code> <a href="developer-guide">Developer guide</a></p>
+<p><strong>Prev</strong>: <code>3.1.</code> <a href="running-the-server">Running the server</a></p>
+<p><strong>Next</strong>: <code>3.3.</code> <a href="database">Database</a></p>
+<p><strong>Sections</strong>:</p>
<ul>
-<li>Any <a href="https://en.wikipedia.org/wiki/POSIX">POSIX</a> compliant <a href="https://frippery.org/make/">make</a></li>
-<li><a href="https://github.com/FiloSottile/mkcert">mkcert</a></li>
-<li><a href="https://www.python.org/downloads/release/python-3138/">Python 3.13</a></li>
-<li><a href="https://docs.astral.sh/uv/#installation">uv</a></li>
+<li><a href="#introduction">Introduction</a></li>
+<li><a href="#hypercorn-and-asgi">Hypercorn and ASGI</a></li>
+<li><a href="#routes-and-database">Routes and database</a></li>
+<li><a href="#user-interface">User interface</a></li>
+<li><a href="#scheduling-and-tasks">Scheduling and tasks</a></li>
+<li><a href="#api">API</a></li>
+<li><a href="#other-important-interfaces">Other important interfaces</a></li>
</ul>
-<p>You can install Python 3.13 through your package manager or through uv. Here is how to install these dependencies on <a href="https://en.wikipedia.org/wiki/Alpine_Linux">Alpine Linux</a>:</p>
-<pre><code class="language-shell">apk add curl git make mkcert@testing
-curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/local/bin" sh
-uv python install 3.13
-</code></pre>
-<p>You do not need to have Alpine Linux to develop ATR. It should work in any POSIX style environment.</p>
-<h3 id="run-the-server">Run the server</h3>
-<p>Then, to run the server:</p>
-<pre><code class="language-shell">cd tooling-trusted-releases/
-mkdir state
-make certs-local
-make serve-local
-</code></pre>
-<p>The <code>certs-local</code> step runs <code>mkcert localhost.apache.org localhost 127.0.0.1 ::1</code> to generate a locally trusted TLS certificate. If the certificate is not trusted, you may have to follow the <a href="https://github.com/FiloSottile/mkcert/blob/master/README.md">mkcert guide</a> to resolve the issue.</p>
-<p>ATR requires TLS even for development because login is performed through the actual ASF OAuth server. This way, the development behavior aligns closely with the production behavior. We try to minimize differences between development and production environments.</p>
-<h3 id="load-the-site">Load the site</h3>
-<p>ATR will then be served on various hosts, but we recommend using only <code>localhost.apache.org</code>. This requires adding an entry to your <code>/etc/hosts</code> and potentially restarting your DNS server. If you do this, the following link should work:</p>
-<p><a href="https://localhost.apache.org:8080/"><code>https://localhost.apache.org:8080/</code></a></p>
-<p>If you do not want to change your <code>/etc/hosts</code>, you can use <code>127.0.0.1</code>. You should not use <code>localhost</code>. The following link should work:</p>
-<p><a href="https://127.0.0.1:8080/"><code>https://127.0.0.1:8080/</code></a></p>
-<p>Pick one or the other, because logging into the site on one host does not log you in to the site on any other host.</p>
-<h2 id="understanding-the-code">Understanding the code</h2>
-<p>Once you have the server running, you can test it. At this point, it is useful to understand how the ATR works in general. References to symbols in this section are given without their <code>atr.</code> prefix, for brevity, and are linked to the respective source code. You should understand e.g. <a href="/ref/atr/server.py:app"><code>server.app</code></a> to mean <code>atr.server.app</code>.</p>
-<h3 id="hypercorn-and-asgi">Hypercorn and ASGI</h3>
+<h2 id="introduction">Introduction</h2>
+<p>This page is a high level view of the ATR code. References to symbols in this section are given without their <code>atr.</code> prefix, for brevity, and are linked to the respective source code. You should understand e.g. <a href="/ref/atr/server.py:app"><code>server.app</code></a> to mean <code>atr.server.app</code>.</p>
+<h2 id="hypercorn-and-asgi">Hypercorn and ASGI</h2>
<p>ATR is an <a href="https://github.com/apache/infrastructure-asfquart">ASFQuart</a> application, running in <a href="https://hypercorn.readthedocs.io/en/latest/index.html">Hypercorn</a>. The entry point for Hypercorn is <a href="/ref/atr/server.py:app"><code>server.app</code></a>, which is then called with a few standard arguments <a href="https://asgi.readthedocs.io/en/latest/specs/main.html#overview">as per the ASGI specification</a>. We create the <code>app</code> object using the f [...]
<pre><code class="language-python">app = create_app(config.get())
</code></pre>
<p>The <a href="/ref/atr/server.py:create_app"><code>server.create_app</code></a> function performs a lot of setup, and if you're interested in how the server works then you can read it and the functions it calls to understand the process further. In general, however, when developing ATR we do not make modifications at the ASFQuart, Quart, and Hypercorn levels very often.</p>
-<h3 id="routes-and-database">Routes and database</h3>
+<h2 id="routes-and-database">Routes and database</h2>
<p>Users request ATR pages over HTTPS, and the ATR server processes those requests in route handlers. Most of those handlers are in <a href="/ref/atr/routes/"><code>routes</code></a>, but not all. What each handler does varies, of course, from handler to handler, but most perform at least one access to the ATR SQLite database.</p>
<p>The path of the SQLite database is configured in <a href="/ref/atr/config.py:SQLITE_DB_PATH"><code>config.AppConfig.SQLITE_DB_PATH</code></a> by default, and will usually appear as <code>state/atr.db</code> with related <code>shm</code> and <code>wal</code> files. We do not expect ATR to have so many users that we need to scale beyond SQLite.</p>
<p>We use <a href="https://sqlmodel.tiangolo.com/">SQLModel</a>, an ORM utilising <a href="https://docs.pydantic.dev/latest/">Pydantic</a> and <a href="https://www.sqlalchemy.org/">SQLAlchemy</a>, to create Python models for the ATR database. The core models file is <a href="/ref/atr/models/sql.py"><code>models.sql</code></a>. The most important individual SQLite models in this module are <a href="/ref/atr/models/sql.py:Committee"><code>Committee</code></a>, <a href="/ref/atr/models/sql. [...]
<p>It is technically possible to interact with SQLite directly, but we do not do that in the ATR source. We use various interfaces in <a href="/ref/atr/db/__init__.py"><code>db</code></a> for reads, and interfaces in <a href="/ref/atr/storage/"><code>storage</code></a> for writes. We plan to move the <code>db</code> code into <code>storage</code> too eventually, because <code>storage</code> is designed to have read components and write components. There is also a legacy <a href="/ref/atr [...]
<p>These three interfaces, <a href="/ref/atr/routes/"><code>routes</code></a>, <a href="/ref/atr/models/sql.py"><code>models.sql</code></a>, and <a href="/ref/atr/storage/"><code>storage</code></a>, are where the majority of activity happens when developing ATR.</p>
-<h3 id="user-interface">User interface</h3>
+<h2 id="user-interface">User interface</h2>
<p>ATR provides a web interface for users to interact with the platform, and the implementation of that interface is split across several modules. The web interface uses server-side rendering almost entirely, where HTML is generated on the server and sent to the browser.</p>
<p>The template system in ATR is <a href="https://jinja.palletsprojects.com/">Jinja2</a>, always accessed through the ATR <a href="/ref/atr/template.py"><code>template</code></a> module. Template files in Jinja2 syntax are stored in <a href="/ref/atr/templates/"><code>templates/</code></a>, and route handlers render them using the asynchronous <a href="/ref/atr/template.py:render"><code>template.render</code></a> function.</p>
<p>Template rendering can be slow if templates are loaded from disk on every request. To address this, we use <a href="/ref/atr/preload.py"><code>preload</code></a> to load all templates into memory before the server starts serving requests. The <a href="/ref/atr/preload.py:setup_template_preloading"><code>preload.setup_template_preloading</code></a> function registers a startup hook that finds and caches every template file.</p>
<p>The ATR user interface includes many HTML forms. We use <a href="https://wtforms.readthedocs.io/">WTForms</a> for form handling, accessed through the ATR <a href="/ref/atr/forms.py"><code>forms</code></a> module. The <a href="/ref/atr/forms.py:Typed"><code>forms.Typed</code></a> base class extends the standard <code>QuartForm</code> class in <a href="https://quart-wtf.readthedocs.io/">Quart-WTF</a>. Each form field is created using helper functions such as <a href="/ref/atr/forms.py:s [...]
<p>In addition to templates, we sometimes need to generate HTML programmatically in Python. For this we use <a href="https://htpy.dev/">htpy</a>, another third party library, for building HTML using Python syntax. The ATR <a href="/ref/atr/htm.py"><code>htm</code></a> module extends htpy with a <a href="/ref/atr/htm.py:Block"><code>Block</code></a> class that makes it easier to build complex HTML structures incrementally. Using htpy means that we get type checking for our HTML generation [...]
-<h3 id="scheduling-and-tasks">Scheduling and tasks</h3>
+<h2 id="scheduling-and-tasks">Scheduling and tasks</h2>
<p>Many operations in ATR are too slow to run during an HTTP request, so we run them asynchronously in background worker processes. The task scheduling system in ATR is built from three components: a task queue stored in the SQLite database, a worker manager that spawns and monitors worker processes, and the worker processes themselves that claim and execute tasks.</p>
<p>The task queue is stored in a table defined by the <a href="/ref/atr/models/sql.py:Task"><code>Task</code></a> model in <a href="/ref/atr/models/sql.py"><code>models.sql</code></a>. Each task has a status, a type, arguments encoded as JSON, and metadata such as when it was added and which user created it. When route handlers need to perform slow operations, they create a new <code>Task</code> row with status <code>QUEUED</code> and commit it to the database.</p>
<p>The ATR <a href="/ref/atr/manager.py"><code>manager</code></a> module provides the <a href="/ref/atr/manager.py:WorkerManager"><code>WorkerManager</code></a> class, which maintains a pool of worker processes. When the ATR server starts, the manager spawns a configurable number of worker processes and monitors them continuously. The manager checks every few seconds whether workers are still running, whether any tasks have exceeded their time limits, and whether the worker pool needs to [...]
<p>The ATR <a href="/ref/atr/worker.py"><code>worker</code></a> module implements the workers. Each worker process runs in a loop. It claims the oldest queued task from the database, executes it, records the result, and then claims the next task atomically using an <code>UPDATE ... WHERE</code> statement. After a worker has processed a fixed number of tasks, it exits voluntarily to help to avoid memory leaks. The manager then spawns a fresh worker to replace it. Task execution happens in [...]
<p>Tasks themselves are defined in the ATR <a href="/ref/atr/tasks/"><code>tasks</code></a> directory. The <a href="/ref/atr/tasks/__init__.py"><code>tasks</code></a> module contains functions for queueing tasks and resolving task types to their handler functions. Task types include operations such as importing keys, generating SBOMs, sending messages, and importing files from SVN. The most common category of task is automated checks on release artifacts. These checks are implemented in [...]
-<h3 id="authentication">Authentication</h3>
-<p>ATR uses ASF OAuth for user login, and then determines what actions each user can perform based on their committee memberships. The ATR <a href="/ref/atr/principal.py"><code>principal</code></a> module handles authorization by checking whether users are members of relevant committees. It queries and caches LDAP to get committee membership information. The <a href="/ref/atr/principal.py:Authorisation"><code>Authorisation</code></a> class provides methods to check whether a user is a me [...]
-<h3 id="api">API</h3>
+<h2 id="api">API</h2>
<p>The ATR API provides programmatic access to most ATR functionality. API endpoints are defined in <a href="/ref/atr/blueprints/api/api.py"><code>blueprints.api.api</code></a>, and their URL paths are prefixed with <code>/api/</code>. The API uses <a href="https://www.openapis.org/">OpenAPI</a> for documentation, which is automatically generated from the endpoint definitions and served at <code>/api/docs</code>. Users send requests with a <a href="https://en.wikipedia.org/wiki/JSON_Web_ [...]
<p>API request and response models are defined in <a href="/ref/atr/models/api.py"><code>models.api</code></a> using Pydantic. Each endpoint has an associated request model that validates incoming data, and a response model that validates outgoing data. The API returns JSON in all cases, with appropriate HTTP status codes.</p>
-<h3 id="other-important-interfaces">Other important interfaces</h3>
+<h2 id="other-important-interfaces">Other important interfaces</h2>
+<p>ATR uses ASF OAuth for user login, and then determines what actions each user can perform based on their committee memberships. The ATR <a href="/ref/atr/principal.py"><code>principal</code></a> module handles authorization by checking whether users are members of relevant committees. It queries and caches LDAP to get committee membership information. The <a href="/ref/atr/principal.py:Authorisation"><code>Authorisation</code></a> class provides methods to check whether a user is a me [...]
<p>The server configuration in <a href="/ref/atr/config.py"><code>config</code></a> determines a lot of global state, and the <a href="/ref/atr/util.py"><code>util</code></a> module contains lots of useful code which is used throughout ATR.</p>
diff --git a/atr/docs/developer-guide.md b/atr/docs/overview-of-the-code.md
similarity index 69%
copy from atr/docs/developer-guide.md
copy to atr/docs/overview-of-the-code.md
index a6dd653..264c19b 100644
--- a/atr/docs/developer-guide.md
+++ b/atr/docs/overview-of-the-code.md
@@ -1,72 +1,26 @@
-# 2. Developer guide
+# 3.2. Overview of the code
-**Up**: [Apache Trusted Releases documentation](.)
+**Up**: `3.` [Developer guide](developer-guide)
-**Prev**: 1. [Introduction to ATR](introduction-to-atr)
+**Prev**: `3.1.` [Running the server](running-the-server)
-This is a guide for developers of ATR, explaining how to make changes to the ATR source code. For more information about how to contribute those changes back to us, please read the [contribution guide](contribution-guide) instead.
+**Next**: `3.3.` [Database](database)
-## Running the server
+**Sections**:
-To develop ATR locally, we manage dependencies using [uv](https://docs.astral.sh/uv/). To run ATR on ASF hardware, we run it in containers managed by Puppet, but since this guide is about development, we focus on using uv.
+* [Introduction](#introduction)
+* [Hypercorn and ASGI](#hypercorn-and-asgi)
+* [Routes and database](#routes-and-database)
+* [User interface](#user-interface)
+* [Scheduling and tasks](#scheduling-and-tasks)
+* [API](#api)
+* [Other important interfaces](#other-important-interfaces)
-### Get the source
+## Introduction
-[Fork the source code](https://github.com/apache/tooling-trusted-releases/fork) of [ATR on GitHub](https://github.com/apache/tooling-trusted-releases), and then [clone your fork locally](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository).
+This page is a high level view of the ATR code. References to symbols in this section are given without their `atr.` prefix, for brevity, and are linked to the respective source code. You should understand e.g. [`server.app`](/ref/atr/server.py:app) to mean `atr.server.app`.
-There are lots of files and directories in the root of the ATR Git repository. The most important thing to know is that `atr/` contains the source code. ATR is a Python application based on [ASFQuart](https://github.com/apache/infrastructure-asfquart), which is based on [Quart](https://github.com/pallets/quart). The Quart web framework is an asynchronous version of [Flask](https://github.com/pallets/flask), a very widely used synchronous web framework. In addition to Python, we use small [...]
-
-### Install dependencies
-
-To run ATR locally after cloning the source, you will need to install the following dependencies:
-
-* Any [POSIX](https://en.wikipedia.org/wiki/POSIX) compliant [make](https://frippery.org/make/)
-* [mkcert](https://github.com/FiloSottile/mkcert)
-* [Python 3.13](https://www.python.org/downloads/release/python-3138/)
-* [uv](https://docs.astral.sh/uv/#installation)
-
-You can install Python 3.13 through your package manager or through uv. Here is how to install these dependencies on [Alpine Linux](https://en.wikipedia.org/wiki/Alpine_Linux):
-
-```shell
-apk add curl git make mkcert@testing
-curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/local/bin" sh
-uv python install 3.13
-```
-
-You do not need to have Alpine Linux to develop ATR. It should work in any POSIX style environment.
-
-### Run the server
-
-Then, to run the server:
-
-```shell
-cd tooling-trusted-releases/
-mkdir state
-make certs-local
-make serve-local
-```
-
-The `certs-local` step runs `mkcert localhost.apache.org localhost 127.0.0.1 ::1` to generate a locally trusted TLS certificate. If the certificate is not trusted, you may have to follow the [mkcert guide](https://github.com/FiloSottile/mkcert/blob/master/README.md) to resolve the issue.
-
-ATR requires TLS even for development because login is performed through the actual ASF OAuth server. This way, the development behavior aligns closely with the production behavior. We try to minimize differences between development and production environments.
-
-### Load the site
-
-ATR will then be served on various hosts, but we recommend using only `localhost.apache.org`. This requires adding an entry to your `/etc/hosts` and potentially restarting your DNS server. If you do this, the following link should work:
-
-[`https://localhost.apache.org:8080/`](https://localhost.apache.org:8080/)
-
-If you do not want to change your `/etc/hosts`, you can use `127.0.0.1`. You should not use `localhost`. The following link should work:
-
-[`https://127.0.0.1:8080/`](https://127.0.0.1:8080/)
-
-Pick one or the other, because logging into the site on one host does not log you in to the site on any other host.
-
-## Understanding the code
-
-Once you have the server running, you can test it. At this point, it is useful to understand how the ATR works in general. References to symbols in this section are given without their `atr.` prefix, for brevity, and are linked to the respective source code. You should understand e.g. [`server.app`](/ref/atr/server.py:app) to mean `atr.server.app`.
-
-### Hypercorn and ASGI
+## Hypercorn and ASGI
ATR is an [ASFQuart](https://github.com/apache/infrastructure-asfquart) application, running in [Hypercorn](https://hypercorn.readthedocs.io/en/latest/index.html). The entry point for Hypercorn is [`server.app`](/ref/atr/server.py:app), which is then called with a few standard arguments [as per the ASGI specification](https://asgi.readthedocs.io/en/latest/specs/main.html#overview). We create the `app` object using the following code:
@@ -76,7 +30,7 @@ app = create_app(config.get())
The [`server.create_app`](/ref/atr/server.py:create_app) function performs a lot of setup, and if you're interested in how the server works then you can read it and the functions it calls to understand the process further. In general, however, when developing ATR we do not make modifications at the ASFQuart, Quart, and Hypercorn levels very often.
-### Routes and database
+## Routes and database
Users request ATR pages over HTTPS, and the ATR server processes those requests in route handlers. Most of those handlers are in [`routes`](/ref/atr/routes/), but not all. What each handler does varies, of course, from handler to handler, but most perform at least one access to the ATR SQLite database.
@@ -88,7 +42,7 @@ It is technically possible to interact with SQLite directly, but we do not do th
These three interfaces, [`routes`](/ref/atr/routes/), [`models.sql`](/ref/atr/models/sql.py), and [`storage`](/ref/atr/storage/), are where the majority of activity happens when developing ATR.
-### User interface
+## User interface
ATR provides a web interface for users to interact with the platform, and the implementation of that interface is split across several modules. The web interface uses server-side rendering almost entirely, where HTML is generated on the server and sent to the browser.
@@ -100,7 +54,7 @@ The ATR user interface includes many HTML forms. We use [WTForms](https://wtform
In addition to templates, we sometimes need to generate HTML programmatically in Python. For this we use [htpy](https://htpy.dev/), another third party library, for building HTML using Python syntax. The ATR [`htm`](/ref/atr/htm.py) module extends htpy with a [`Block`](/ref/atr/htm.py:Block) class that makes it easier to build complex HTML structures incrementally. Using htpy means that we get type checking for our HTML generation, and can compose HTML elements just like any other Python [...]
-### Scheduling and tasks
+## Scheduling and tasks
Many operations in ATR are too slow to run during an HTTP request, so we run them asynchronously in background worker processes. The task scheduling system in ATR is built from three components: a task queue stored in the SQLite database, a worker manager that spawns and monitors worker processes, and the worker processes themselves that claim and execute tasks.
@@ -112,16 +66,14 @@ The ATR [`worker`](/ref/atr/worker.py) module implements the workers. Each worke
Tasks themselves are defined in the ATR [`tasks`](/ref/atr/tasks/) directory. The [`tasks`](/ref/atr/tasks/__init__.py) module contains functions for queueing tasks and resolving task types to their handler functions. Task types include operations such as importing keys, generating SBOMs, sending messages, and importing files from SVN. The most common category of task is automated checks on release artifacts. These checks are implemented in [`tasks/checks/`](/ref/atr/tasks/checks/), and [...]
-### Authentication
-
-ATR uses ASF OAuth for user login, and then determines what actions each user can perform based on their committee memberships. The ATR [`principal`](/ref/atr/principal.py) module handles authorization by checking whether users are members of relevant committees. It queries and caches LDAP to get committee membership information. The [`Authorisation`](/ref/atr/principal.py:Authorisation) class provides methods to check whether a user is a member of a committee or a project participant, w [...]
-
-### API
+## API
The ATR API provides programmatic access to most ATR functionality. API endpoints are defined in [`blueprints.api.api`](/ref/atr/blueprints/api/api.py), and their URL paths are prefixed with `/api/`. The API uses [OpenAPI](https://www.openapis.org/) for documentation, which is automatically generated from the endpoint definitions and served at `/api/docs`. Users send requests with a [JWT](https://en.wikipedia.org/wiki/JSON_Web_Token) created from a [PAT](https://en.wikipedia.org/wiki/Per [...]
API request and response models are defined in [`models.api`](/ref/atr/models/api.py) using Pydantic. Each endpoint has an associated request model that validates incoming data, and a response model that validates outgoing data. The API returns JSON in all cases, with appropriate HTTP status codes.
-### Other important interfaces
+## Other important interfaces
+
+ATR uses ASF OAuth for user login, and then determines what actions each user can perform based on their committee memberships. The ATR [`principal`](/ref/atr/principal.py) module handles authorization by checking whether users are members of relevant committees. It queries and caches LDAP to get committee membership information. The [`Authorisation`](/ref/atr/principal.py:Authorisation) class provides methods to check whether a user is a member of a committee or a project participant, w [...]
The server configuration in [`config`](/ref/atr/config.py) determines a lot of global state, and the [`util`](/ref/atr/util.py) module contains lots of useful code which is used throughout ATR.
diff --git a/atr/docs/running-the-server.html b/atr/docs/running-the-server.html
new file mode 100644
index 0000000..fbf31c1
--- /dev/null
+++ b/atr/docs/running-the-server.html
@@ -0,0 +1,46 @@
+<h1 id="1-running-the-server">3.1. Running the server</h1>
+<p><strong>Up</strong>: <code>3.</code> <a href="developer-guide">Developer guide</a></p>
+<p><strong>Prev</strong>: (none)</p>
+<p><strong>Next</strong>: <code>3.2.</code> <a href="overview-of-the-code">Overview of the code</a></p>
+<p><strong>Sections</strong>:</p>
+<ul>
+<li><a href="#introduction">Introduction</a></li>
+<li><a href="#get-the-source">Get the source</a></li>
+<li><a href="#install-dependencies">Install dependencies</a></li>
+<li><a href="#run-the-server">Run the server</a></li>
+<li><a href="#load-the-site">Load the site</a></li>
+</ul>
+<h2 id="introduction">Introduction</h2>
+<p>To develop ATR locally, we manage dependencies using <a href="https://docs.astral.sh/uv/">uv</a>. To run ATR on ASF hardware, we run it in containers managed by Puppet, but since this guide is about development, we focus on using uv.</p>
+<h2 id="get-the-source">Get the source</h2>
+<p><a href="https://github.com/apache/tooling-trusted-releases/fork">Fork the source code</a> of <a href="https://github.com/apache/tooling-trusted-releases">ATR on GitHub</a>, and then <a href="https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository">clone your fork locally</a>.</p>
+<p>There are lots of files and directories in the root of the ATR Git repository. The most important thing to know is that <code>atr/</code> contains the source code. ATR is a Python application based on <a href="https://github.com/apache/infrastructure-asfquart">ASFQuart</a>, which is based on <a href="https://github.com/pallets/quart">Quart</a>. The Quart web framework is an asynchronous version of <a href="https://github.com/pallets/flask">Flask</a>, a very widely used synchronous web [...]
+<h2 id="install-dependencies">Install dependencies</h2>
+<p>To run ATR locally after cloning the source, you will need to install the following dependencies:</p>
+<ul>
+<li>Any <a href="https://en.wikipedia.org/wiki/POSIX">POSIX</a> compliant <a href="https://frippery.org/make/">make</a></li>
+<li><a href="https://github.com/FiloSottile/mkcert">mkcert</a></li>
+<li><a href="https://www.python.org/downloads/release/python-3138/">Python 3.13</a></li>
+<li><a href="https://docs.astral.sh/uv/#installation">uv</a></li>
+</ul>
+<p>You can install Python 3.13 through your package manager or through uv. Here is how to install these dependencies on <a href="https://en.wikipedia.org/wiki/Alpine_Linux">Alpine Linux</a>:</p>
+<pre><code class="language-shell">apk add curl git make mkcert@testing
+curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/local/bin" sh
+uv python install 3.13
+</code></pre>
+<p>You do not need to have Alpine Linux to develop ATR. It should work in any POSIX style environment.</p>
+<h2 id="run-the-server">Run the server</h2>
+<p>Then, to run the server:</p>
+<pre><code class="language-shell">cd tooling-trusted-releases/
+mkdir state
+make certs-local
+make serve-local
+</code></pre>
+<p>The <code>certs-local</code> step runs <code>mkcert localhost.apache.org localhost 127.0.0.1 ::1</code> to generate a locally trusted TLS certificate. If the certificate is not trusted, you may have to follow the <a href="https://github.com/FiloSottile/mkcert/blob/master/README.md">mkcert guide</a> to resolve the issue.</p>
+<p>ATR requires TLS even for development because login is performed through the actual ASF OAuth server. This way, the development behavior aligns closely with the production behavior. We try to minimize differences between development and production environments.</p>
+<h2 id="load-the-site">Load the site</h2>
+<p>ATR will then be served on various hosts, but we recommend using only <code>localhost.apache.org</code>. This requires adding an entry to your <code>/etc/hosts</code> and potentially restarting your DNS server. If you do this, the following link should work:</p>
+<p><a href="https://localhost.apache.org:8080/"><code>https://localhost.apache.org:8080/</code></a></p>
+<p>If you do not want to change your <code>/etc/hosts</code>, you can use <code>127.0.0.1</code>. You should not use <code>localhost</code>. The following link should work:</p>
+<p><a href="https://127.0.0.1:8080/"><code>https://127.0.0.1:8080/</code></a></p>
+<p>Pick one or the other, because logging into the site on one host does not log you in to the site on any other host.</p>
diff --git a/atr/docs/running-the-server.md b/atr/docs/running-the-server.md
new file mode 100644
index 0000000..5c6226d
--- /dev/null
+++ b/atr/docs/running-the-server.md
@@ -0,0 +1,71 @@
+# 3.1. Running the server
+
+**Up**: `3.` [Developer guide](developer-guide)
+
+**Prev**: (none)
+
+**Next**: `3.2.` [Overview of the code](overview-of-the-code)
+
+**Sections**:
+
+* [Introduction](#introduction)
+* [Get the source](#get-the-source)
+* [Install dependencies](#install-dependencies)
+* [Run the server](#run-the-server)
+* [Load the site](#load-the-site)
+
+## Introduction
+
+To develop ATR locally, we manage dependencies using [uv](https://docs.astral.sh/uv/). To run ATR on ASF hardware, we run it in containers managed by Puppet, but since this guide is about development, we focus on using uv.
+
+## Get the source
+
+[Fork the source code](https://github.com/apache/tooling-trusted-releases/fork) of [ATR on GitHub](https://github.com/apache/tooling-trusted-releases), and then [clone your fork locally](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository).
+
+There are lots of files and directories in the root of the ATR Git repository. The most important thing to know is that `atr/` contains the source code. ATR is a Python application based on [ASFQuart](https://github.com/apache/infrastructure-asfquart), which is based on [Quart](https://github.com/pallets/quart). The Quart web framework is an asynchronous version of [Flask](https://github.com/pallets/flask), a very widely used synchronous web framework. In addition to Python, we use small [...]
+
+## Install dependencies
+
+To run ATR locally after cloning the source, you will need to install the following dependencies:
+
+* Any [POSIX](https://en.wikipedia.org/wiki/POSIX) compliant [make](https://frippery.org/make/)
+* [mkcert](https://github.com/FiloSottile/mkcert)
+* [Python 3.13](https://www.python.org/downloads/release/python-3138/)
+* [uv](https://docs.astral.sh/uv/#installation)
+
+You can install Python 3.13 through your package manager or through uv. Here is how to install these dependencies on [Alpine Linux](https://en.wikipedia.org/wiki/Alpine_Linux):
+
+```shell
+apk add curl git make mkcert@testing
+curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/local/bin" sh
+uv python install 3.13
+```
+
+You do not need to have Alpine Linux to develop ATR. It should work in any POSIX style environment.
+
+## Run the server
+
+Then, to run the server:
+
+```shell
+cd tooling-trusted-releases/
+mkdir state
+make certs-local
+make serve-local
+```
+
+The `certs-local` step runs `mkcert localhost.apache.org localhost 127.0.0.1 ::1` to generate a locally trusted TLS certificate. If the certificate is not trusted, you may have to follow the [mkcert guide](https://github.com/FiloSottile/mkcert/blob/master/README.md) to resolve the issue.
+
+ATR requires TLS even for development because login is performed through the actual ASF OAuth server. This way, the development behavior aligns closely with the production behavior. We try to minimize differences between development and production environments.
+
+## Load the site
+
+ATR will then be served on various hosts, but we recommend using only `localhost.apache.org`. This requires adding an entry to your `/etc/hosts` and potentially restarting your DNS server. If you do this, the following link should work:
+
+[`https://localhost.apache.org:8080/`](https://localhost.apache.org:8080/)
+
+If you do not want to change your `/etc/hosts`, you can use `127.0.0.1`. You should not use `localhost`. The following link should work:
+
+[`https://127.0.0.1:8080/`](https://127.0.0.1:8080/)
+
+Pick one or the other, because logging into the site on one host does not log you in to the site on any other host.
diff --git a/atr/docs/user-guide.html b/atr/docs/user-guide.html
new file mode 100644
index 0000000..c8ee1c9
--- /dev/null
+++ b/atr/docs/user-guide.html
@@ -0,0 +1,10 @@
+<h1 id="user-guide">2. User guide</h1>
+<p><strong>Up</strong>: <a href=".">Documentation</a></p>
+<p><strong>Prev</strong>: <code>1.</code> <a href="introduction-to-atr">Introduction to ATR</a></p>
+<p><strong>Next</strong>: <code>3.</code> <a href="developer-guide">Developer guide</a></p>
+<p><strong>Sections</strong>:</p>
+<ul>
+<li><a href="#introduction">Introduction</a></li>
+</ul>
+<h2 id="introduction">Introduction</h2>
+<p>[TODO]</p>
diff --git a/atr/docs/user-guide.md b/atr/docs/user-guide.md
new file mode 100644
index 0000000..de2e363
--- /dev/null
+++ b/atr/docs/user-guide.md
@@ -0,0 +1,15 @@
+# 2. User guide
+
+**Up**: [Documentation](.)
+
+**Prev**: `1.` [Introduction to ATR](introduction-to-atr)
+
+**Next**: `3.` [Developer guide](developer-guide)
+
+**Sections**:
+
+* [Introduction](#introduction)
+
+## Introduction
+
+[TODO]
diff --git a/atr/static/css/atr.css b/atr/static/css/atr.css
index e8186c4..7a58ea6 100644
--- a/atr/static/css/atr.css
+++ b/atr/static/css/atr.css
@@ -86,7 +86,7 @@ pre {
border: 1px solid #d1d2d3;
}
-p code { color: #aa0033; }
+p code, li code { color: #aa0033; }
a:has(code) {
text-decoration-color: #aa0033;
diff --git a/scripts/build_docs.py b/scripts/build_docs.py
new file mode 100755
index 0000000..c0047cb
--- /dev/null
+++ b/scripts/build_docs.py
@@ -0,0 +1,255 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import pathlib
+import re
+import sys
+from typing import NamedTuple
+
+
+class TocEntry(NamedTuple):
+ number: str
+ title: str
+ path: str
+ level: int
+ parent: str | None
+
+
+class Navigation(NamedTuple):
+ up: tuple[str, str] | None
+ prev: tuple[str, str, str] | None
+ next: tuple[str, str, str] | None
+ children: list[TocEntry]
+
+
+def parse_toc_line(line: str) -> TocEntry | None:
+ pattern = r"^(\s*)\*\s+`([^`]+)`\s+\[([^\]]+)\]\(([^)]+)\)"
+ match = re.match(pattern, line)
+ if not match:
+ return None
+
+ indent, number, title, path = match.groups()
+ level = len(indent) // 2
+ number_clean = number.strip()
+
+ number_no_dots = number_clean.rstrip(".")
+ parts = number_no_dots.split(".")
+
+ if level == 0:
+ parent = None
+ else:
+ parent = ".".join(parts[:-1])
+
+ return TocEntry(number_clean, title, path, level, parent)
+
+
+def parse_toc(index_path: pathlib.Path) -> list[TocEntry]:
+ content = index_path.read_text(encoding="utf-8")
+ lines = content.splitlines()
+
+ in_toc = False
+ entries = []
+
+ for line in lines:
+ if line.strip() == "## Table of contents":
+ in_toc = True
+ continue
+ if in_toc:
+ if line.strip() and (not line.startswith("*")) and (not line.startswith(" ")):
+ break
+ entry = parse_toc_line(line)
+ if entry:
+ entries.append(entry)
+
+ return entries
+
+
+def validate_files(docs_dir: pathlib.Path, entries: list[TocEntry]) -> None:
+ referenced_files = {entry.path + ".md" for entry in entries}
+ referenced_files.add("index.md")
+
+ existing_files = {f.name for f in docs_dir.glob("*.md")}
+
+ missing = referenced_files - existing_files
+ if missing:
+ print(f"Error: TOC references files that don't exist: {missing}", file=sys.stderr)
+ sys.exit(1)
+
+ unreferenced = existing_files - referenced_files
+ if unreferenced:
+ print(f"Error: Markdown files exist that are not in TOC: {unreferenced}", file=sys.stderr)
+ sys.exit(1)
+
+
+def build_navigation(entries: list[TocEntry]) -> dict[str, Navigation]:
+ nav_map = {}
+
+ for i, entry in enumerate(entries):
+ if entry.level == 0:
+ up = ("Documentation", ".")
+ else:
+ parent_entry = next(e for e in entries if e.number.rstrip(".") == entry.parent)
+ up = (f"{parent_entry.number} {parent_entry.title}", parent_entry.path)
+
+ prev_entry = None
+ for j in range(i - 1, -1, -1):
+ if entries[j].level == entry.level:
+ prev_entry = entries[j]
+ break
+
+ next_entry = None
+ for j in range(i + 1, len(entries)):
+ if entries[j].level == entry.level:
+ next_entry = entries[j]
+ break
+
+ if prev_entry:
+ prev = (prev_entry.number, prev_entry.title, prev_entry.path)
+ else:
+ prev = None
+
+ if next_entry:
+ next_nav = (next_entry.number, next_entry.title, next_entry.path)
+ else:
+ next_nav = None
+
+ entry_number_no_dots = entry.number.rstrip(".")
+ children = [e for e in entries if e.parent == entry_number_no_dots]
+
+ nav_map[entry.path] = Navigation(up, prev, next_nav, children)
+
+ return nav_map
+
+
+def extract_h2_headings(content: str) -> list[tuple[str, str]]:
+ lines = content.splitlines()
+ headings = []
+
+ for line in lines:
+ match = re.match(r"^##\s+(.+)$", line)
+ if match:
+ title = match.group(1)
+ anchor = title.lower()
+ anchor = re.sub(r"^\d+\.\s*", "", anchor)
+ anchor = re.sub(r"[^\w\s-]", "", anchor)
+ anchor = re.sub(r"[\s_]+", "-", anchor)
+ anchor = anchor.strip("-")
+ headings.append((title, anchor))
+
+ return headings
+
+
+def generate_navigation_block(entry: TocEntry, nav: Navigation, h2_headings: list[tuple[str, str]]) -> str:
+ blocks = []
+
+ if nav.up is not None:
+ up_text = nav.up[0]
+ up_parts = up_text.split(" ", 1)
+ if (len(up_parts) == 2) and (up_parts[0].rstrip(".").replace(".", "").isdigit()):
+ blocks.append(f"**Up**: `{up_parts[0]}` [{up_parts[1]}]({nav.up[1]})")
+ else:
+ blocks.append(f"**Up**: [{up_text}]({nav.up[1]})")
+ blocks.append("")
+
+ if nav.prev is not None:
+ blocks.append(f"**Prev**: `{nav.prev[0]}` [{nav.prev[1]}]({nav.prev[2]})")
+ else:
+ blocks.append("**Prev**: (none)")
+ blocks.append("")
+
+ if nav.next is not None:
+ blocks.append(f"**Next**: `{nav.next[0]}` [{nav.next[1]}]({nav.next[2]})")
+ else:
+ blocks.append("**Next**: (none)")
+ blocks.append("")
+
+ if nav.children:
+ blocks.append("**Pages**:")
+ blocks.append("")
+ for child in nav.children:
+ indent = " " * (child.level - entry.level - 1)
+ blocks.append(f"{indent}* `{child.number}` [{child.title}]({child.path})")
+ blocks.append("")
+
+ if h2_headings:
+ blocks.append("**Sections**:")
+ blocks.append("")
+ for title, anchor in h2_headings:
+ blocks.append(f"* [{title}](#{anchor})")
+ blocks.append("")
+
+ return "\n".join(blocks)
+
+
+def update_document(docs_dir: pathlib.Path, entry: TocEntry, nav: Navigation) -> None:
+ file_path = docs_dir / f"{entry.path}.md"
+ content = file_path.read_text(encoding="utf-8")
+ lines = content.splitlines()
+
+ h2_headings = extract_h2_headings(content)
+
+ first_h2_index = None
+ for i, line in enumerate(lines):
+ if re.match(r"^##\s+", line):
+ first_h2_index = i
+ break
+
+ if first_h2_index is None:
+ print(f"Warning: {entry.path}.md has no h2 sections", file=sys.stderr)
+ content_lines = lines
+ else:
+ content_lines = lines[first_h2_index:]
+
+ new_heading = f"# {entry.number} {entry.title}"
+ nav_block = generate_navigation_block(entry, nav, h2_headings)
+
+ new_lines = [new_heading, "", nav_block, *content_lines]
+
+ new_content = "\n".join(new_lines)
+ if not new_content.endswith("\n"):
+ new_content += "\n"
+
+ file_path.write_text(new_content, encoding="utf-8")
+
+
+def main() -> None:
+ docs_dir = pathlib.Path("atr/docs")
+ index_path = docs_dir / "index.md"
+
+ if not index_path.exists():
+ print(f"Error: {index_path} not found", file=sys.stderr)
+ sys.exit(1)
+
+ entries = parse_toc(index_path)
+ if not entries:
+ print("Error: No TOC entries found in index.md", file=sys.stderr)
+ sys.exit(1)
+
+ validate_files(docs_dir, entries)
+
+ nav_map = build_navigation(entries)
+
+ for entry in entries:
+ update_document(docs_dir, entry, nav_map[entry.path])
+
+ print(f"Updated {len(entries)} documentation files")
+
+
+if __name__ == "__main__":
+ main()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org