You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ponymail.apache.org by rb...@apache.org on 2026/06/19 17:03:25 UTC
(incubator-ponymail-foal) 01/01: Add comprehensive documentation suite
This is an automated email from the ASF dual-hosted git repository.
rbowen pushed a commit to branch rbowen-docs
in repository https://gitbox.apache.org/repos/asf/incubator-ponymail-foal.git
commit 8c7556c66559edabc5b1e30960c73044cbcbfd47
Author: Rich Bowen <rb...@rcbowen.com>
AuthorDate: Fri Jun 19 13:01:00 2026 -0400
Add comprehensive documentation suite
New docs:
- docs/index.md: Documentation hub linking all guides
- docs/architecture.md: Component diagram, data model, plugin system
- docs/operator_guide.md: Docker quickstart + production deployment
- docs/user_guide.md: Browsing, searching, OAuth, composing replies
- docs/admin_guide.md: Management console, GDPR, audit log, bulk ops
- docs/plugins.md: How to write endpoints and extend the system
- docs/api_client_guide.md: Curl examples for common API operations
- docs/releases.md: Versioning and how to identify deployed version
Also adds AGENTS.md for AI coding agent guidance, and updates
README.md to link to the documentation directory.
Disclosure: This was written with the assistance of our robot overlords
(Claude, if it matters.)
---
AGENTS.md | 191 ++++++++++++++++++++++++++++++++++++++++
README.md | 7 ++
docs/admin_guide.md | 156 +++++++++++++++++++++++++++++++++
docs/api_client_guide.md | 195 +++++++++++++++++++++++++++++++++++++++++
docs/architecture.md | 184 +++++++++++++++++++++++++++++++++++++++
docs/index.md | 47 ++++++++++
docs/operator_guide.md | 220 +++++++++++++++++++++++++++++++++++++++++++++++
docs/plugins.md | 191 ++++++++++++++++++++++++++++++++++++++++
docs/releases.md | 117 +++++++++++++++++++++++++
docs/user_guide.md | 138 +++++++++++++++++++++++++++++
10 files changed, 1446 insertions(+)
diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 0000000..aaa0403
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1,191 @@
+# AGENTS.md
+
+Guidance for AI coding agents working on this repository.
+
+## Project Overview
+
+Apache Pony Mail Foal is a mailing list archive system. It consists of:
+
+- **`server/`** — Python async HTTP API (aiohttp) that queries ElasticSearch
+- **`webui/`** — Static JavaScript/HTML frontend served by a reverse proxy
+- **`tools/`** — Python CLI utilities for archiving, importing, and migrating email
+
+Production deployment: `https://lists.apache.org` (archives all ASF mailing lists).
+
+## Repository Layout
+
+```
+server/
+├── main.py # Entry point, request routing, endpoint discovery
+├── ponymail.yaml.example # Sample configuration
+├── openapi.yaml # OpenAPI 3.0 specification
+├── server_version.py # Auto-generated version from git
+├── endpoints/ # API endpoint handlers (one file per endpoint)
+│ ├── stats.py # Search/list emails
+│ ├── email.py # Fetch single email
+│ ├── thread.py # Fetch email thread
+│ ├── source.py # Raw RFC 2822 source
+│ ├── mbox.py # Mbox download
+│ ├── compose.py # Send email (authenticated)
+│ ├── preferences.py # User prefs + list overview
+│ ├── mgmt.py # Admin GDPR operations
+│ ├── oauth.py # OAuth login flow
+│ ├── pminfo.py # Server activity info
+│ ├── gravatar.py # Avatar caching proxy
+│ └── plain.py # SEO-friendly HTML rendering
+├── plugins/ # Shared server internals
+│ ├── configuration.py # YAML config parsing (source of truth for all config keys)
+│ ├── database.py # ElasticSearch async client pool
+│ ├── messages.py # Email query, threading, trimming logic
+│ ├── session.py # Cookie-based sessions + OAuth credential tracking
+│ ├── aaa.py # Access control (public vs private lists)
+│ ├── defuzzer.py # Date/query parameter normalization
+│ ├── background.py # Periodic index refresh tasks
+│ ├── formdata.py # Request body parsing (form vs JSON)
+│ ├── offloader.py # Thread pool for CPU-bound JSON serialization
+│ ├── auditlog.py # Admin action logging
+│ └── oauth*.py # Provider-specific OAuth token exchange
+└── testendpoints/ # Extra endpoints loaded with --testendpoints
+
+tools/
+├── archiver.py # Pipe-based email archiver (called from /etc/aliases)
+├── import-mbox.py # Bulk mbox file importer
+├── migrate.py # Migration from old PonyMail databases
+├── setup.py # First-time ES index setup
+├── rethread.py # Re-compute threading for existing emails
+├── bulk-edit.py # Batch metadata edits
+├── archiver.yaml # Archiver configuration
+└── plugins/ # Shared archiver internals
+ ├── elastic.py # Synchronous ES client for tools
+ ├── generators.py # Email ID generation (DKIM-based)
+ ├── dkim_id.py # DKIM permalink generation
+ ├── textlib.py # Text normalization, list-ID handling
+ └── mboxo_patch.py # Mbox format quirks
+
+webui/
+├── index.html # Front page (list of lists)
+├── list.html # List view (emails in a list)
+├── thread.html # Thread view
+├── admin.html # Management console
+├── oauth.html # OAuth callback handler
+├── js/
+│ ├── ponymail.js # Built/concatenated JS (from source/)
+│ ├── config.js # Client-side config (OAuth client IDs, API URL)
+│ └── source/ # Individual JS source files (concatenated by build.sh)
+│ ├── build.sh # Concatenation build script
+│ ├── search.js # Search functionality
+│ ├── composer.js # Email compose UI
+│ └── ... # ~24 source files
+├── css/
+└── images/
+
+test/
+├── test_archiver.py # Unit tests for archiver
+├── test_defuzzer.py # Unit tests for date/query parsing
+├── test_msgid.py # Unit tests for message ID handling
+├── itest_integration.py # Integration tests (requires running ES + server)
+└── resources/ # Test fixtures (sample emails)
+
+docs/
+├── index.md # Documentation home page
+├── architecture.md # Component diagram, data model, request lifecycle
+├── operator_guide.md # Install & deploy (Docker + production)
+├── user_guide.md # Browse, search, reply
+├── admin_guide.md # Management console, GDPR, bulk ops
+├── plugins.md # Endpoint & plugin development guide
+├── API.md # HTTP API endpoint reference
+├── api_client_guide.md # Curl examples, integration patterns
+├── search.md # Search query syntax
+├── configuration.md # All ponymail.yaml options
+├── releases.md # Versioning & release process
+└── STYLEGUIDE.md # Code style conventions
+```
+
+## Tech Stack
+
+| Component | Technology |
+|-----------|-----------|
+| Server | Python 3.8+, aiohttp, async/await |
+| Database | ElasticSearch 7.x+ (async client) |
+| Frontend | Vanilla JavaScript (ES2018), jQuery 1.12, Bootstrap |
+| Build (JS) | Shell concatenation (`webui/js/source/build.sh`) |
+| Config | YAML (`ponymail.yaml` for server, `archiver.yaml` for tools) |
+| Tests | pytest (unit + integration) |
+| Linting | pylint (`pylintrc`), black (line length 120), mypy (type checking) |
+| CI | GitHub Actions (linting, type tests, unit tests, integration tests) |
+
+## Code Conventions
+
+### Python (server + tools)
+
+- **PEP 8** with 120-char line limit
+- **Format with** `black -l 120`
+- **Type annotations** required on all functions; checked with `mypy`
+- **Endpoint pattern**: each endpoint is a module in `server/endpoints/` with:
+ - `async def process(server, session, indata) -> dict | Response`
+ - `def register(server) -> plugins.server.Endpoint`
+- **Config access**: via `server.config.<section>.<key>` (never raw YAML dicts)
+- **ES queries**: use `session.database` (async); never construct raw HTTP calls to ES
+- **Dependencies**: must be Apache 2.0 compatible (see comments in `requirements.txt`)
+
+### JavaScript (webui)
+
+- **ECMAScript 2018** (ES9) — no transpilation
+- **Global variables** prefixed with `G_` (e.g. `G_apiURL`, `G_current_json`)
+- **No module bundler** — files in `webui/js/source/` are concatenated by `build.sh` into `ponymail.js`
+- **After editing source files**: commit the source changes first, then run `build.sh` and commit the built `ponymail.js` separately
+- **jQuery 1.12** is available but new code should prefer vanilla DOM APIs
+
+## How to Run Locally
+
+```bash
+# Docker (easiest)
+git clone https://github.com/apache/incubator-ponymail-foal.git
+cd incubator-ponymail-foal
+docker compose build
+docker compose up
+# In another terminal:
+docker compose exec pmfoal bash -c 'cd tools; python3 setup.py --devel'
+docker compose exec pmfoal bash -c 'cd server; python3 main.py --testendpoints'
+# Browse to http://localhost:1080/
+```
+
+## How to Run Tests
+
+```bash
+# Unit tests
+cd test
+pip install -r requirements.txt
+pytest test_archiver.py test_defuzzer.py test_msgid.py
+
+# Integration tests (requires running ES + server)
+pytest itest_integration.py
+```
+
+## API Pattern
+
+All endpoints accept both `.lua` (GET + query params) and `.json` (POST + JSON body) suffixes. The suffix determines how the request body is parsed — both route to the same handler:
+
+```
+GET /api/stats.lua?list=dev&domain=httpd.apache.org → form parsing
+POST /api/stats.json {"list":"dev","domain":"httpd.apache.org"} → JSON parsing
+```
+
+See `docs/API.md` for the full reference and `server/openapi.yaml` for the formal schema.
+
+## Key Design Decisions
+
+- **No external framework** for the frontend — vanilla JS concatenated into a single file
+- **Async everywhere** on the server — aiohttp + async ES client, connection pool
+- **Backward compatibility** with original Lua-based PonyMail API (`.lua` suffix support)
+- **Privacy model**: emails can be `private: true` (hidden from unauthenticated users). OAuth providers marked `authoritative` grant access to private lists. Admin users can hide/delete emails.
+- **Email IDs**: generated from DKIM signatures for collision resistance (see `tools/plugins/dkim_id.py`)
+- **Threading**: stored as `in-reply-to`/`references` in ES; thread tree constructed at query time. Known limitation: threading can be inconsistent (see GitHub issues #244, #304)
+
+## Common Pitfalls
+
+- **ElasticSearch version sensitivity**: The ES client library from 7.14+ has strict server version checking. Pin to `<7.14.0` as in requirements.txt.
+- **`ponymail.js` is generated**: Don't edit it directly. Edit files in `webui/js/source/` then run `build.sh`.
+- **Server version file is generated**: Don't edit `server_version.py` directly. Commit endpoint changes first, then run `server/update_version.sh`.
+- **Config key names don't always match YAML keys**: e.g. YAML `mgmtconsole` → Python `config.ui.mgmt_enabled`, YAML `allow_delete` → Python `config.ui.fully_delete`. Check `plugins/configuration.py` for the mapping.
+- **Two separate config systems**: The server uses `ponymail.yaml` (parsed by `plugins/configuration.py`). The tools use `archiver.yaml` (parsed by `tools/plugins/ponymailconfig.py` via ConfigParser — INI-style, not YAML).
diff --git a/README.md b/README.md
index 72de07f..835b10e 100644
--- a/README.md
+++ b/README.md
@@ -30,6 +30,13 @@ Among other things, Foal sports the following new features:
## Installation Guide
Please see the [installation documentation](INSTALL.md) for setup instructions.
+## Documentation
+Full documentation is available in the [docs/](docs/index.md) directory, including:
+- [Operator Guide](docs/operator_guide.md) — Install and deploy
+- [User Guide](docs/user_guide.md) — Browse, search, reply
+- [API Reference](docs/API.md) — HTTP endpoint specs
+- [Configuration Reference](docs/configuration.md) — All config options
+
### Current setup requirements:
* An operating system, such as:
diff --git a/docs/admin_guide.md b/docs/admin_guide.md
new file mode 100644
index 0000000..6b11dde
--- /dev/null
+++ b/docs/admin_guide.md
@@ -0,0 +1,156 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Admin Guide
+
+## Enabling Administration
+
+Two configuration steps are required in `server/ponymail.yaml`:
+
+1. Enable the management console:
+
+```yaml
+ui:
+ mgmtconsole: true
+```
+
+2. Define admin users (must authenticate via an authoritative OAuth provider):
+
+```yaml
+oauth:
+ authoritative_domains:
+ - googleapis.com
+ admins:
+ - admin@example.org
+ - another-admin@example.org
+```
+
+Restart the server after changes.
+
+---
+
+## Management Console
+
+Admin users see a yellow cog icon in the context menu of each email.
+Clicking it opens the management interface for that message.
+
+### Available Actions
+
+| Action | Effect |
+|--------|--------|
+| **Hide** | Marks the email as hidden. It remains in ES but is not shown to users. Recoverable. |
+| **Unhide** | Restores a previously hidden email. |
+| **Delete** | Permanently removes the email from both `mbox` and `source` indices. Only available when `allow_delete: true`. Irreversible. |
+| **Edit** | Change metadata (e.g. move to different list-ID). |
+
+### GDPR Compliance
+
+For full GDPR compliance (right to erasure), set:
+
+```yaml
+ui:
+ mgmtconsole: true
+ allow_delete: true
+```
+
+When `allow_delete` is `true`:
+- "Delete" fully expunges the email from ElasticSearch (both `mbox` and `source` indices)
+- The deletion is logged in the audit trail
+- The email cannot be recovered
+
+When `allow_delete` is `false` (default):
+- "Delete" merely hides the email (sets a hidden flag)
+- An admin can unhide it later
+- The raw source remains in the `source` index
+
+---
+
+## Audit Log
+
+All admin actions are logged. View the audit log via the API:
+
+```bash
+curl -X POST https://your-instance/api/mgmt.json \
+ -H "Content-Type: application/json" \
+ -H "Cookie: ponymail=your-session-cookie" \
+ -d '{"action": "log", "size": 50, "page": 0}'
+```
+
+Or filter by action type:
+
+```bash
+curl -X POST https://your-instance/api/mgmt.json \
+ -H "Content-Type: application/json" \
+ -H "Cookie: ponymail=your-session-cookie" \
+ -d '{"action": "log", "filter": "delete"}'
+```
+
+---
+
+## Command-Line Tools
+
+### Bulk Edit (`tools/bulk-edit.py`)
+
+Perform batch metadata changes across many emails. Useful for:
+- Moving emails to a different list-ID after a list rename
+- Bulk privacy changes
+
+### Re-threading (`tools/rethread.py`)
+
+Recompute threading metadata for all emails. Run this after:
+- Importing mbox files that arrived out of order
+- Enabling `archiver.threadinfo` after initial import
+
+```bash
+cd tools
+python3 rethread.py
+```
+
+### Re-indexing
+
+If you need to rebuild the ES indices from scratch:
+
+1. Back up your data
+2. Delete the indices: `curl -X DELETE http://localhost:9200/ponymail-*`
+3. Re-run setup: `python3 tools/setup.py`
+4. Re-import all mbox files: `python3 tools/import-mbox.py --source /path/to/archives/`
+
+---
+
+## Monitoring
+
+### Check Server Health
+
+```bash
+curl -s http://localhost:8080/api/pminfo.json | python3 -m json.tool
+```
+
+### Check ElasticSearch Index Sizes
+
+```bash
+curl -s http://localhost:9200/_cat/indices/ponymail-*?v
+```
+
+### Common Issues
+
+| Symptom | Likely Cause |
+|---------|-------------|
+| 404 on all API calls | Server not running or proxy misconfigured |
+| Empty list overview | Background refresh hasn't run yet (wait `refresh_rate` seconds) |
+| Private emails visible without login | `authoritative_domains` not set, or AAA plugin bypassed |
+| "API error occurred" with traceback | Set `ui.traceback: false` in production, check journal for details |
+| Compose/reply silently fails | Check `ui.mailhost` and `ui.sender_domains` config |
diff --git a/docs/api_client_guide.md b/docs/api_client_guide.md
new file mode 100644
index 0000000..bbb0afd
--- /dev/null
+++ b/docs/api_client_guide.md
@@ -0,0 +1,195 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# API Client Guide
+
+Practical examples for consuming the Pony Mail Foal API programmatically.
+For the full endpoint reference, see [API.md](API.md).
+
+---
+
+## Two Access Modes
+
+Every endpoint supports two URL patterns:
+
+| Suffix | Method | Body | Use when |
+|--------|--------|------|----------|
+| `.json` | POST | JSON | Preferred — native Foal protocol |
+| `.lua` | GET | Query params | Legacy compatibility |
+
+All examples below show both modes.
+
+---
+
+## Authentication
+
+Public lists require no authentication. For private lists or write
+operations, include a session cookie:
+
+```
+Cookie: ponymail=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+```
+
+Obtain the cookie by completing an OAuth flow via the web UI, then
+extract it from your browser's DevTools (Network tab → Request Headers).
+
+---
+
+## Common Operations
+
+### List All Mailing Lists
+
+**.json (POST):**
+```bash
+curl -s -X POST https://lists.apache.org/api/preferences.json \
+ -H "Content-Type: application/json" \
+ -d '{}' | jq '.lists | keys'
+```
+
+**.lua (GET):**
+```bash
+curl -s "https://lists.apache.org/api/preferences.lua" | jq '.lists | keys'
+```
+
+Returns domain → list → message count mappings.
+
+---
+
+### Search a List
+
+**.json (POST):**
+```bash
+curl -s -X POST https://lists.apache.org/api/stats.json \
+ -H "Content-Type: application/json" \
+ -d '{
+ "list": "dev",
+ "domain": "httpd.apache.org",
+ "d": "lte=30d",
+ "q": "mod_proxy"
+ }' | jq '.hits, .emails[0].subject'
+```
+
+**.lua (GET):**
+```bash
+curl -s "https://lists.apache.org/api/stats.lua?list=dev&domain=httpd.apache.org&d=lte%3D30d&q=mod_proxy" \
+ | jq '.hits, .emails[0].subject'
+```
+
+#### Useful Parameters
+
+| Parameter | Example | Effect |
+|-----------|---------|--------|
+| `d` | `2026-06` | Specific month |
+| `d` | `lte=7d` | Last 7 days |
+| `d` | `dfr=2026-01-01\|dto=2026-06-30` | Date range |
+| `q` | `+proxy -balancer` | Required/excluded terms |
+| `header_from` | `rbowen@apache.org` | Filter by sender |
+| `header_subject` | `[VOTE]` | Filter by subject |
+| `quick` | `1` | Stats only (faster, no emails/threads) |
+| `emailsOnly` | `1` | Emails only (no thread_struct, participants, word cloud) |
+
+---
+
+### Fetch a Single Email
+
+**.json (POST):**
+```bash
+curl -s -X POST https://lists.apache.org/api/email.json \
+ -H "Content-Type: application/json" \
+ -d '{"id": "rt6hrlhc4cwz0bwzf4lys43cb7lz30h6"}' | jq '.subject, .from, .date'
+```
+
+**.lua (GET):**
+```bash
+curl -s "https://lists.apache.org/api/email.lua?id=rt6hrlhc4cwz0bwzf4lys43cb7lz30h6" \
+ | jq '.subject, .from, .date'
+```
+
+---
+
+### Fetch a Thread
+
+**.json (POST):**
+```bash
+curl -s -X POST https://lists.apache.org/api/thread.json \
+ -H "Content-Type: application/json" \
+ -d '{"id": "rt6hrlhc4cwz0bwzf4lys43cb7lz30h6"}' \
+ | jq '.emails | length'
+```
+
+Use `"find_parent": true` to navigate from any reply up to the thread root.
+
+---
+
+### Fetch Raw Email Source
+
+**.json (POST):**
+```bash
+curl -s -X POST https://lists.apache.org/api/source.json \
+ -H "Content-Type: application/json" \
+ -d '{"id": "rt6hrlhc4cwz0bwzf4lys43cb7lz30h6"}'
+```
+
+Returns `text/plain` — the raw RFC 2822 message.
+
+---
+
+### Download Mbox Archive
+
+**.json (POST):**
+```bash
+curl -s -X POST https://lists.apache.org/api/mbox.json \
+ -H "Content-Type: application/json" \
+ -d '{"list": "dev", "domain": "httpd.apache.org", "d": "2026-06"}' \
+ -o dev-2026-06.mbox
+```
+
+**.lua (GET):**
+```bash
+curl -s "https://lists.apache.org/api/mbox.lua?list=dev&domain=httpd.apache.org&date=2026-06" \
+ -o dev-2026-06.mbox
+```
+
+---
+
+## Rate Limiting
+
+There is no rate limiting on the API. However, be respectful:
+
+- Use `quick=1` or `emailsOnly=1` when you don't need full results
+- Cache responses where appropriate
+- Avoid tight polling loops — use the `since` parameter for change detection
+
+---
+
+## Error Handling
+
+| Status | Meaning |
+|--------|---------|
+| 200 | Success |
+| 400 | Bad request (malformed parameters) |
+| 403 | Forbidden (admin endpoint, insufficient permissions) |
+| 404 | Email/thread not found, or invalid endpoint |
+| 500 | Server error (check `traceback` setting) |
+
+---
+
+## Related
+
+- [API.md](API.md) — Full endpoint reference with response schemas
+- [search.md](search.md) — Search query syntax
+- GitHub issue [#312](https://github.com/apache/incubator-ponymail-foal/issues/312) — API token support (not yet implemented)
diff --git a/docs/architecture.md b/docs/architecture.md
new file mode 100644
index 0000000..226c094
--- /dev/null
+++ b/docs/architecture.md
@@ -0,0 +1,184 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Architecture Overview
+
+## Component Diagram
+
+```
+ ┌─────────────────────┐
+ │ Mailing List MTA │
+ │ (Postfix / ezmlm) │
+ └──────────┬──────────┘
+ │ pipe via /etc/aliases
+ ▼
+ ┌─────────────────────┐
+ │ tools/archiver.py │
+ │ (Email ingestion) │
+ └──────────┬──────────┘
+ │ index
+ ▼
+┌──────────────┐ ┌─────────────────────┐ ┌──────────────┐
+│ Browser │◄───────►│ ElasticSearch 7+ │◄───────►│ tools/*.py │
+│ │ │ (Data store) │ │ (import, │
+└──────┬───────┘ └──────────▲──────────┘ │ migrate, │
+ │ │ │ rethread) │
+ │ HTTP │ async queries └──────────────┘
+ ▼ │
+┌──────────────┐ ┌──────────┴──────────┐
+│ httpd/nginx │────────►│ server/main.py │
+│ (reverse │ proxy │ (aiohttp API) │
+│ proxy) │ /api/ │ │
+└──────┬───────┘ └─────────────────────┘
+ │
+ │ static files
+ ▼
+┌──────────────┐
+│ webui/ │
+│ (JS + HTML) │
+└──────────────┘
+```
+
+## Components
+
+### Email Ingestion (`tools/`)
+
+- **`archiver.py`** — Called by the MTA (via `/etc/aliases`) for each
+ incoming email. Parses the message, generates a DKIM-based permalink
+ ID, stores the parsed email in the `mbox` index and the raw source in
+ the `source` index.
+- **`import-mbox.py`** — Bulk imports existing mbox files into ES.
+- **`migrate.py`** — Migrates databases from the older Lua-based PonyMail.
+- **`rethread.py`** — Recomputes threading metadata for existing emails.
+- **`setup.py`** — First-time ES index creation with correct mappings.
+
+### API Server (`server/`)
+
+An async Python HTTP server built on **aiohttp**. Handles all API
+requests, manages sessions, and queries ElasticSearch.
+
+- Listens on a configurable port (default 8080)
+- Placed behind a reverse proxy (httpd or nginx) that serves the static
+ UI and forwards `/api/` requests to the server
+
+### Web UI (`webui/`)
+
+A static JavaScript/HTML frontend. No build step required for deployment
+— just serve the directory. Source JS files in `webui/js/source/` are
+concatenated into `webui/js/ponymail.js` via `build.sh` during development.
+
+### ElasticSearch
+
+The sole data store. Contains these indices (prefixed by `db_prefix`,
+default `ponymail`):
+
+| Index | Content |
+|-------|---------|
+| `ponymail-mbox` | Parsed email metadata (from, subject, date, body, list-id, threading info) |
+| `ponymail-source` | Raw RFC 2822 email source (the original message as received) |
+| `ponymail-account` | User session/preference data |
+| `ponymail-auditlog` | Admin action audit trail |
+
+---
+
+## Data Model
+
+### Email Document (`mbox` index)
+
+Each email is stored with:
+
+- `mid` — Internal permalink ID (DKIM-based, collision-resistant)
+- `dbid` — SHA3-256 of the raw message source
+- `message-id` — Original Message-ID header
+- `from`, `to`, `cc`, `subject`, `date`, `epoch`
+- `body` — Parsed plain-text body
+- `list` / `list_raw` — List-ID header
+- `in-reply-to`, `references` — Threading headers
+- `private` — Whether the email is on a private list
+- `attachments` — Array of attachment metadata (filename, hash, size)
+- `gravatar` — MD5 of sender address for avatar lookup
+
+### Source Document (`source` index)
+
+- `id` — Same `dbid` as the corresponding mbox entry
+- `source` — The complete raw RFC 2822 message
+
+### Threading
+
+Threading is reconstructed at query time from `in-reply-to` and
+`references` headers. The `thread.json` endpoint walks up/down the
+chain to build a tree structure.
+
+Optional threading metadata (enabled via `archiver.threadinfo` config):
+- `top` — Boolean: is this email the root of a thread?
+- `thread` — ID of the thread root
+- `previous` — ID of the parent message
+
+---
+
+## Request Lifecycle
+
+1. **Client** sends request to reverse proxy (e.g. `GET /api/stats.lua?list=dev&domain=httpd.apache.org`)
+2. **Proxy** forwards `/api/*` to the aiohttp server on port 8080
+3. **`main.py` routing**:
+ - Strips suffix (`.lua` → form parsing, `.json` → JSON parsing)
+ - Looks up handler name in `self.handlers` dict
+ - Acquires a database connection from the async pool
+ - Creates a session object (validates cookie if present)
+4. **Endpoint** `process(server, session, indata)` executes:
+ - Builds ES query via `plugins.defuzzer` (normalizes dates/filters)
+ - Checks access via `plugins.aaa` (private list filtering)
+ - Queries ES via `session.database`
+ - Returns a dict (auto-serialized to JSON) or a custom `aiohttp.web.Response`
+5. **Response** sent back through proxy to client
+
+---
+
+## Server Plugin Architecture
+
+Plugins in `server/plugins/` are shared internal modules — not
+user-installable extensions.
+
+| Plugin | Role |
+|--------|------|
+| `configuration.py` | YAML config parsing (all config keys defined here) |
+| `database.py` | Async ES client, connection pool, index names |
+| `messages.py` | Email retrieval, threading logic, access filtering, trimming |
+| `session.py` | Cookie management, OAuth credential tracking |
+| `aaa.py` | Access control (public vs private list checks) |
+| `defuzzer.py` | Date/query parameter normalization and validation |
+| `background.py` | Periodic tasks (refresh list counts, activity stats) |
+| `formdata.py` | Request body parsing (form-encoded vs JSON) |
+| `offloader.py` | Thread pool executor for CPU-bound JSON serialization |
+| `auditlog.py` | Admin action audit trail |
+| `server.py` | Base classes: `Endpoint`, `StreamingEndpoint`, `BaseServer` |
+| `oauthGeneric.py` | Generic OAuth token exchange |
+| `oauthGoogle.py` | Google-specific OAuth (ID token verification) |
+| `oauthGithub.py` | GitHub OAuth (code → token → user info) |
+
+## Tools Plugin Architecture
+
+Plugins in `tools/plugins/` support the archiver and import tools:
+
+| Plugin | Role |
+|--------|------|
+| `elastic.py` | Synchronous ES client for CLI tools |
+| `generators.py` | Email ID generation strategies |
+| `dkim_id.py` | DKIM-based permalink generation (collision-resistant) |
+| `textlib.py` | Text normalization, List-ID parsing |
+| `mboxo_patch.py` | Workarounds for mbox format edge cases |
+| `ponymailconfig.py` | INI-style config parser for `archiver.yaml` |
diff --git a/docs/index.md b/docs/index.md
new file mode 100644
index 0000000..95b119c
--- /dev/null
+++ b/docs/index.md
@@ -0,0 +1,47 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Apache Pony Mail Foal — Documentation
+
+## Getting Started
+
+- [Operator Guide](operator_guide.md) — Install and run Foal (Docker or production)
+- [Architecture Overview](architecture.md) — How the components fit together
+
+## User Documentation
+
+- [User Guide](user_guide.md) — Browse, search, and reply to mailing list archives
+- [Search Syntax](search.md) — Query language reference
+
+## Administration
+
+- [Admin Guide](admin_guide.md) — Management console, GDPR, bulk operations
+- [Configuration Reference](configuration.md) — All `ponymail.yaml` options
+
+## Developer Documentation
+
+- [API Reference](API.md) — HTTP endpoint specifications
+- [API Client Guide](api_client_guide.md) — Curl examples and integration patterns
+- [Plugin & Endpoint Development](plugins.md) — How to extend Foal
+- [Style Guide](STYLEGUIDE.md) — Code conventions (Python + JavaScript)
+- [Versioning & Releases](releases.md) — How versions are tracked
+
+## Also See
+
+- [INSTALL.md](../INSTALL.md) — Detailed installation (OAuth, archiving, web replies)
+- [DOCKER.md](../DOCKER.md) — Docker-specific setup
+- [MIGRATING.md](../MIGRATING.md) — Migration from legacy Pony Mail
diff --git a/docs/operator_guide.md b/docs/operator_guide.md
new file mode 100644
index 0000000..f9d8737
--- /dev/null
+++ b/docs/operator_guide.md
@@ -0,0 +1,220 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Operator Guide
+
+Two paths to get Foal running: Docker (quick, for development/testing)
+or a production deployment.
+
+---
+
+## Quick Start (Docker)
+
+```bash
+git clone https://github.com/apache/incubator-ponymail-foal.git ponymail-foal
+cd ponymail-foal
+docker compose build
+docker compose up
+```
+
+In a second terminal, set up the database (one-time):
+
+```bash
+docker compose exec pmfoal bash -c 'cd tools; python3 setup.py --devel'
+```
+
+Import some test data:
+
+```bash
+mkdir mbox-testdata && cd mbox-testdata
+wget "https://lists.apache.org/api/mbox.lua?list=dev&domain=community.apache.org"
+cd ..
+docker compose exec pmfoal bash -c 'cd tools; python3 import-mbox.py --source /var/maildata/mbox-testdata/*.mbox'
+```
+
+Start the API server:
+
+```bash
+docker compose exec pmfoal bash -c 'cd server; python3 main.py --testendpoints'
+```
+
+Browse to `http://localhost:1080/`.
+
+For Docker details (env vars, volume mounts, test auth), see [DOCKER.md](../DOCKER.md).
+
+---
+
+## Production Deployment
+
+### Requirements
+
+- Linux (or FreeBSD/macOS)
+- Python 3.8+
+- ElasticSearch 7.x (client pinned to `<7.14` due to strict version checks)
+- A reverse proxy (Apache httpd or nginx)
+- PipEnv or pip for dependency management
+
+### Install
+
+```bash
+git clone https://github.com/apache/incubator-ponymail-foal.git /opt/ponymail-foal
+cd /opt/ponymail-foal/tools
+pip install -r requirements.txt
+python3 setup.py # Interactive — creates ES indices + ponymail.yaml
+
+cd /opt/ponymail-foal/server
+pip install -r requirements.txt
+```
+
+### Configure
+
+Edit `server/ponymail.yaml`. See [configuration.md](configuration.md) for
+all options. At minimum:
+
+```yaml
+server:
+ port: 8080
+ bind: 127.0.0.1
+
+database:
+ dburl: http://localhost:9200/
+
+ui:
+ traceback: false
+```
+
+### Run as a systemd Service
+
+Create `/etc/systemd/system/ponymail.service`:
+
+```ini
+[Unit]
+Description=Apache Pony Mail Foal API Server
+After=network.target elasticsearch.service
+
+[Service]
+Type=simple
+User=ponymail
+Group=ponymail
+WorkingDirectory=/opt/ponymail-foal/server
+ExecStart=/usr/bin/python3 main.py
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target
+```
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl enable --now ponymail
+```
+
+### Reverse Proxy (Apache httpd)
+
+```apache
+<VirtualHost *:443>
+ ServerName lists.example.org
+ DocumentRoot /opt/ponymail-foal/webui/
+ AcceptPathInfo On
+ AllowEncodedSlashes On
+
+ ProxyPass /api/ http://127.0.0.1:8080/api/
+ ProxyPassReverse /api/ http://127.0.0.1:8080/api/
+
+ <Directory /opt/ponymail-foal/webui/>
+ Require all granted
+ Options +MultiViews
+ </Directory>
+</VirtualHost>
+```
+
+### Reverse Proxy (nginx)
+
+```nginx
+server {
+ listen 443 ssl;
+ server_name lists.example.org;
+ root /opt/ponymail-foal/webui/;
+ index index.html;
+
+ location /api/ {
+ proxy_pass http://127.0.0.1:8080/api/;
+ }
+}
+```
+
+---
+
+## Archiving New Emails
+
+Add a pipe alias in `/etc/aliases`:
+
+```
+mylist: "| /opt/ponymail-foal/tools/archiver.py"
+```
+
+Run `newaliases`. For private lists, use `--private`. To override the
+list ID: `--lid mylist@example.org`.
+
+For full archiving options (OAuth, web replies, management console),
+see [INSTALL.md](../INSTALL.md).
+
+---
+
+## Operations
+
+### Health Check
+
+The API server responds to any valid endpoint. A simple check:
+
+```bash
+curl -s http://localhost:8080/api/pminfo.json | jq .
+```
+
+This returns activity data (list counts, processing stats).
+
+### Logs
+
+- **API server**: stdout/stderr (captured by systemd journal)
+- **Archiver**: stderr (captured by MTA's pipe logging)
+- **ES queries**: Enable with `--logger INFO` or `--trace DEBUG` flags on `main.py`
+- **API request log**: Enable with `--apilog INFO` flag
+
+View server logs:
+
+```bash
+journalctl -u ponymail -f
+```
+
+### Background Tasks
+
+The server runs a background refresh task (interval set by
+`tasks.refresh_rate`, default 150 seconds) that updates:
+
+- List message counts
+- Activity statistics (used by `pminfo.json`)
+
+### Importing Existing Archives
+
+```bash
+cd /opt/ponymail-foal/tools
+python3 import-mbox.py --source /path/to/mbox-files/
+```
+
+Use `--lid listname@domain` to override list detection. For migration
+from old PonyMail, see [MIGRATING.md](../MIGRATING.md).
diff --git a/docs/plugins.md b/docs/plugins.md
new file mode 100644
index 0000000..ff3ec08
--- /dev/null
+++ b/docs/plugins.md
@@ -0,0 +1,191 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Plugin & Endpoint Development Guide
+
+## Adding a New API Endpoint
+
+### 1. Create the module
+
+Create a new Python file in `server/endpoints/`:
+
+```python
+# server/endpoints/myendpoint.py
+
+import plugins.server
+import plugins.session
+import typing
+import aiohttp.web
+
+
+async def process(
+ server: plugins.server.BaseServer,
+ session: plugins.session.SessionObject,
+ indata: dict,
+) -> typing.Union[dict, aiohttp.web.Response]:
+ """Handle the request and return a JSON-serializable dict or a Response."""
+
+ name = indata.get("name", "world")
+ return {"greeting": f"Hello, {name}!"}
+
+
+def register(server: plugins.server.BaseServer):
+ """Called once at startup. Return an Endpoint or StreamingEndpoint."""
+ return plugins.server.Endpoint(process)
+```
+
+### 2. That's it
+
+The server auto-discovers endpoints on startup by scanning the
+`server/endpoints/` directory. Any `.py` file with a `register()`
+function is loaded and mapped to `/api/{filename}`.
+
+Your endpoint is now available at:
+- `POST /api/myendpoint.json` (JSON body)
+- `GET /api/myendpoint.lua?name=Rich` (query params)
+
+### Key Conventions
+
+| Concern | Pattern |
+|---------|---------|
+| Return JSON | Return a `dict` — the server serializes it automatically |
+| Return custom response | Return an `aiohttp.web.Response` (e.g. for binary data, redirects) |
+| Access config | `server.config.ui.mailhost`, `server.config.database.max_hits`, etc. |
+| Query ES | Use `session.database` (an async ES client from the pool) |
+| Check auth | `session.credentials` is `None` (anonymous) or has `.authoritative`, `.admin`, `.email` |
+| Error response | Return `aiohttp.web.Response(status=400, text="message")` |
+
+### Streaming Endpoints
+
+For large responses (like mbox downloads), use `StreamingEndpoint`:
+
+```python
+async def process(server, request, session, indata):
+ response = aiohttp.web.StreamResponse()
+ await response.prepare(request)
+ # Write chunks...
+ await response.write(b"data chunk")
+ return response
+
+def register(server):
+ return plugins.server.StreamingEndpoint(process)
+```
+
+Note: streaming endpoints receive the raw `request` object instead of
+parsed `indata`.
+
+---
+
+## Server Plugins (`server/plugins/`)
+
+These are shared internal modules used by endpoints. They are **not**
+user-installable — to extend behavior, add endpoints instead.
+
+### Commonly Used
+
+| Module | What you use it for |
+|--------|-------------------|
+| `plugins.messages` | `get_email()`, `fetch_children()`, `find_parent()`, `query()` |
+| `plugins.aaa` | `can_access_list()` — checks private list permissions |
+| `plugins.session` | `SessionObject` with `.credentials`, `.database` |
+| `plugins.defuzzer` | `defuzz(indata)` — normalizes date/search parameters into ES queries |
+| `plugins.server` | `BaseServer`, `Endpoint`, `StreamingEndpoint` base classes |
+
+### Adding to an Existing Plugin
+
+If you need shared logic used by multiple endpoints, add it to the
+appropriate plugin module. Follow the existing patterns:
+
+- Type annotations on all functions
+- Async where ES queries are involved
+- Use `session.database` for queries (never create your own ES client)
+
+---
+
+## Tools Plugins (`tools/plugins/`)
+
+These support the CLI tools (archiver, importer, migrator):
+
+| Module | Purpose |
+|--------|---------|
+| `elastic.py` | Synchronous ES client wrapper (tools don't use async) |
+| `generators.py` | ID generation strategies for emails |
+| `dkim_id.py` | DKIM-based permalink generation |
+| `textlib.py` | Text normalization, List-ID parsing, character encoding |
+| `mboxo_patch.py` | Workaround for `mboxo` format quirks (unescaped "From " lines) |
+| `ponymailconfig.py` | Reads `archiver.yaml` (INI-style config, separate from server's YAML) |
+
+---
+
+## OAuth Provider Plugins
+
+To add a new OAuth provider, create a module in `server/plugins/`:
+
+```python
+# server/plugins/oauthMyProvider.py
+
+async def process(server, formdata):
+ """Exchange auth code for user info.
+
+ Returns:
+ dict with 'email' and 'name' keys on success,
+ or dict with 'error' key on failure.
+ """
+ code = formdata.get("code")
+ # Exchange code for token, fetch user info...
+ return {"email": "user@example.org", "name": "Jane Doe"}
+```
+
+Then register it in `server/endpoints/oauth.py`'s dispatch logic and
+add a provider entry in `ponymail.yaml`:
+
+```yaml
+oauth:
+ providers:
+ myprovider:
+ name: My OAuth
+ oauth_portal: https://auth.example.org/authorize
+ client_id: your-client-id
+```
+
+---
+
+## Testing
+
+### Unit Tests
+
+```bash
+cd test
+pip install -r requirements.txt
+pytest test_archiver.py test_defuzzer.py test_msgid.py -v
+```
+
+### Integration Tests
+
+Require a running ES instance and server with `--testendpoints`:
+
+```bash
+pytest itest_integration.py -v
+```
+
+### Linting & Type Checking
+
+```bash
+black -l 120 --check server/ tools/
+mypy server/
+pylint server/ tools/
+```
diff --git a/docs/releases.md b/docs/releases.md
new file mode 100644
index 0000000..a1ef86a
--- /dev/null
+++ b/docs/releases.md
@@ -0,0 +1,117 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Versioning & Release Process
+
+## Current State
+
+Pony Mail Foal does not yet have formal tagged releases. The project
+runs from `master` (the ASF production instance at `lists.apache.org`
+tracks a deployed commit).
+
+---
+
+## How Versions Are Tracked
+
+Two version identifiers exist:
+
+### Server Version (`server_version.py`)
+
+Auto-generated from the git commit SHA:
+
+```bash
+cd server
+./update_version.sh
+```
+
+This writes `server_version.py` with a short hash like `PONYMAIL_SERVER_VERSION = "a3f8c2d"`.
+
+**When to regenerate:** After committing changes to `server/endpoints/`,
+`server/plugins/`, or `server/main.py`.
+
+### WebUI Version (in `ponymail.js`)
+
+Generated by the JS build script:
+
+```bash
+cd webui/js/source
+./build.sh
+```
+
+The version is embedded in the concatenated `ponymail.js` output.
+
+**When to regenerate:** After committing changes to any file in
+`webui/js/source/`.
+
+### Foal Version
+
+A static string (`PONYMAIL_FOAL_VERSION = "0.1.0"`) in `server/main.py`.
+Updated manually for milestone releases.
+
+---
+
+## Identifying the Running Version
+
+The `/api/preferences.json` endpoint returns version information for
+authenticated users:
+
+```json
+{
+ "versions": {
+ "foal": "0.1.0",
+ "server": "a3f8c2d"
+ }
+}
+```
+
+Admin users additionally see `elasticsearch_engine` and
+`elasticsearch_library` versions.
+
+---
+
+## Release History
+
+| Version | Date | Notes |
+|---------|------|-------|
+| 0.1.0 (Foal) | ongoing | Next-generation Python rewrite, no formal release |
+| 1.0.1 (legacy) | — | Last planned version of old Lua-based Pony Mail |
+
+The original Lua-based Pony Mail had releases at `0.9.x` beta level.
+Foal superseded it without a formal release — the ASF deployment
+transitioned directly from Lua to Foal on `lists.apache.org`.
+
+---
+
+## Future: Git Tags and Releases
+
+The project does not currently use git tags for releases
+([#297](https://github.com/apache/incubator-ponymail-foal/issues/297)).
+
+When formal releases are established, the expected process would be:
+
+1. Update `PONYMAIL_FOAL_VERSION` in `server/main.py`
+2. Regenerate `server_version.py` and `ponymail.js`
+3. Tag the commit: `git tag v{version}`
+4. Create a GitHub release with changelog
+
+---
+
+## Deployment (ASF Production)
+
+The ASF infrastructure team deploys from a specific commit on `master`.
+There is no automated CI/CD pipeline to production — deployment is
+manual, coordinated with INFRA.
diff --git a/docs/user_guide.md b/docs/user_guide.md
new file mode 100644
index 0000000..4b3d217
--- /dev/null
+++ b/docs/user_guide.md
@@ -0,0 +1,138 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# User Guide
+
+## Browsing the Archive
+
+### Front Page
+
+The front page shows all available mailing list domains and their lists
+with message counts. Click a domain to expand, then click a list name
+to view its archive.
+
+### List View
+
+Once you select a list, you see the current month's emails. The sidebar
+shows:
+
+- **Calendar** — click any month to browse historical archives
+- **Participants** — top contributors for the selected period
+- **Word Cloud** — common terms (if enabled by the administrator)
+
+### Display Modes
+
+Three display modes are available (toggle in the header):
+
+| Mode | Description |
+|------|-------------|
+| **Threaded** | Groups emails by conversation thread. Click a thread to expand. |
+| **Flat** | Shows all emails in chronological order. |
+| **Tree** | Hierarchical indented view showing reply relationships. |
+
+### Reading an Email
+
+Click any email subject to open it. The view shows:
+
+- Full message body (with quoted text collapsed by default)
+- Sender, date, and list information
+- Attachments (downloadable)
+- Navigation: "Find parent email" link, permalink
+
+---
+
+## Searching
+
+Use the search box at the top of any page. Supports:
+
+- Free text: `budget meeting`
+- Required terms: `+budget`
+- Excluded terms: `-spam`
+- Exact phrases: `"quarterly review"`
+
+For full search syntax including date ranges and header filters, see
+[search.md](search.md).
+
+---
+
+## Authentication
+
+### Logging In
+
+Click the profile icon (top right) to log in via OAuth. Available
+providers depend on the deployment (commonly Google, GitHub, or a
+custom provider like ASF OAuth).
+
+### What Login Enables
+
+| Feature | Anonymous | Logged In (authoritative) | Admin |
+|---------|-----------|--------------------------|-------|
+| Browse public lists | ✓ | ✓ | ✓ |
+| Browse private lists | ✗ | ✓ | ✓ |
+| Compose replies | ✗ | ✓ | ✓ |
+| Management console | ✗ | ✗ | ✓ |
+
+"Authoritative" means the OAuth provider is listed in the server's
+`authoritative_domains` configuration.
+
+### Logging Out
+
+Click the profile icon again and select "Log out". Your session cookie
+is invalidated.
+
+---
+
+## Composing Replies
+
+When logged in via an authoritative OAuth provider:
+
+1. Open an email you want to reply to
+2. Click the **Reply** button below the message
+3. Compose your reply in the text box
+4. Click **Send**
+
+Restrictions:
+
+- You can only reply to lists whose domain matches the server's
+ `sender_domains` configuration
+- The email is sent from your OAuth-verified email address
+- The `In-Reply-To` and `References` headers are set automatically
+
+---
+
+## Permalinks
+
+Every email has a stable permalink URL of the form:
+
+```
+https://your-instance/thread/{permalink_id}
+```
+
+Use the "Permalink" link in the email view to copy it. These URLs are
+stable and suitable for sharing.
+
+**Note:** If an email requires authentication (private list), visitors
+who are not logged in will see an error. They must log in first via the
+profile icon.
+
+---
+
+## Mbox Download
+
+You can download a month's archive in mbox format by clicking the
+download icon in the list view header. This produces a standard mbox
+file suitable for import into other mail clients or tools.