You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by sb...@apache.org on 2026/10/02 14:24:21 UTC
(tooling-trusted-releases) branch sbp updated (38822e1c -> c0e1397f)
This is an automated email from the ASF dual-hosted git repository.
sbp pushed a change to branch sbp
in repository https://gitbox.apache.org/repos/asf/tooling-trusted-releases.git
discard 38822e1c Improve and expand the user guide to uploading files
discard 04965b8d #1666 - pull key signing requirements into a single place and document clearly
discard 74644274 Document file handling constraints in the developer guide
add de2abf80 #1666 - pull key signing requirements into a single place and document clearly
new c0e1397f Document file handling constraints in the developer guide
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (38822e1c)
\
N -- N -- N refs/heads/sbp (c0e1397f)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
atr/docs/checks.md | 13 +++----------
atr/docs/uploading-files.md | 36 +++++++++++++++++-------------------
2 files changed, 20 insertions(+), 29 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org
(tooling-trusted-releases) 01/01: Document file handling constraints in the developer guide
Posted by sb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sbp pushed a commit to branch sbp
in repository https://gitbox.apache.org/repos/asf/tooling-trusted-releases.git
commit c0e1397f7add265fa7242b00e06fe5ac565c0010
Author: Sean B. Palmer <se...@miscoranda.com>
AuthorDate: Fri Oct 2 14:45:30 2026 +0100
Document file handling constraints in the developer guide
---
atr/docs/checks.md | 6 ++--
atr/docs/developer-guide.md | 2 ++
atr/docs/file-handling.md | 72 +++++++++++++++++++++++++++++++++++++++++
atr/docs/index.md | 1 +
atr/docs/input-validation.md | 26 ++-------------
atr/docs/resource-management.md | 8 +++--
atr/docs/sbom-architecture.md | 2 +-
atr/docs/uploading-files.md | 17 ++++++++++
8 files changed, 104 insertions(+), 30 deletions(-)
diff --git a/atr/docs/checks.md b/atr/docs/checks.md
index cbb84125..5505449b 100644
--- a/atr/docs/checks.md
+++ b/atr/docs/checks.md
@@ -24,7 +24,7 @@ Checks are recorded against the release revision that you upload, and the result
## How ATR selects checks
-When you create or update a draft revision, ATR scans the files in the revision directory and schedules checks based on file suffix and project policy. A signature file with the `.asc` suffix triggers signature verification. A checksum file with the `.sha256` or `.sha512` suffix triggers hash verification. Archive files with the `.tar.gz`, `.tgz`, or `.zip` suffix trigger archive integrity, archive structure, and license checks. Files with the `.cdx.json` suffix trigger SBOM analysis. AT [...]
+When you create or update a draft revision, ATR scans the files in the revision directory and schedules checks based on file suffix and project policy. A signature file with the `.asc` suffix triggers signature verification. A checksum file with the `.sha256` or `.sha512` suffix triggers hash verification. Archive files with the `.tar.gz`, `.tgz`, or `.zip` suffix trigger archive structure and license checks. Files with the `.cdx.json` suffix trigger SBOM analysis. ATR also runs a path a [...]
Project policy tells ATR which artifacts are source artifacts and which are binary artifacts, and it supplies exclusion patterns for license related checks. Project policy also controls the license check mode for source artifacts. You can choose lightweight checks, Apache RAT, or both for source artifacts. Binary artifacts are not scanned by RAT, and always rely on the lightweight checks.
@@ -60,9 +60,7 @@ The checker key is `atr.tasks.checks.signature.check`, and the uploader concern
### Archive integrity checks
-ATR checks that each `.tar.gz`, `.tgz`, or `.zip` archive can be read in full. For tar based archives it reads all members using the tar reader. For zip archives it lists members and verifies that the zip structure is valid. Archives that are corrupted or contain too many members fail this check.
-
-The checker key for tar based archives is `atr.tasks.checks.targz.integrity`, and the checker key for zip archives is `atr.tasks.checks.zipformat.integrity`.
+ATR validates archive integrity during quarantine extraction, before accepting the proposed revision. Corrupt archives and those that exceed extraction limits fail validation. This is separate from the checks recorded against a revision. See [File handling](file-handling) for the formats covered and failure behavior.
### Archive structure checks
diff --git a/atr/docs/developer-guide.md b/atr/docs/developer-guide.md
index cb765fd5..df0a4f56 100644
--- a/atr/docs/developer-guide.md
+++ b/atr/docs/developer-guide.md
@@ -29,6 +29,7 @@
* `7.19.` [ASF modules](asf-modules)
* `7.20.` [Resource management](resource-management)
* `7.21.` [SBOM architecture](sbom-architecture)
+* `7.22.` [File handling](file-handling)
**Sections**:
@@ -46,5 +47,6 @@ ATR is security-critical infrastructure for the Apache Software Foundation. Befo
* [Authentication security](authentication-security) - How users authenticate to ATR via ASF OAuth and API tokens
* [Authorization security](authorization-security) - The role-based access control model and LDAP integration
* [Input validation](input-validation) - Data validation patterns and injection prevention
+* [File handling](file-handling) - Upload limits, archive validation and downloads
For reporting security vulnerabilities, see [SECURITY.md](https://github.com/apache/tooling-trusted-releases/blob/main/SECURITY.md) in the repository root.
diff --git a/atr/docs/file-handling.md b/atr/docs/file-handling.md
new file mode 100644
index 00000000..e6a8a6be
--- /dev/null
+++ b/atr/docs/file-handling.md
@@ -0,0 +1,72 @@
+# 7.22. File handling
+
+**Up**: `7.` [Developer guide](developer-guide)
+
+**Prev**: `7.21.` [SBOM architecture](sbom-architecture)
+
+**Next**: (none)
+
+**Sections**:
+
+* [Overview](#overview)
+* [Uploads and limits](#uploads-and-limits)
+* [Format checks](#format-checks)
+* [Archive quarantine](#archive-quarantine)
+* [Downloads](#downloads)
+
+## Overview
+
+ATR validates release uploads in temporary storage before creating a revision. Archives that need validation stay in quarantine until a worker extracts them successfully. The previous revision remains available while this happens.
+
+## Uploads and limits
+
+| Route | Content | Default size limit |
+| --- | --- | --- |
+| Browser upload | Release files in a multipart form | 512 MiB per request |
+| API `release/store` | One file as the raw request body | 512 MiB per request |
+| API `release/upload` | One file, base64 encoded in JSON | 512 MiB per request |
+| rsync | Files and directory trees | Files over 2 GB are skipped |
+| SVN import | Files from the committee's `dist/dev` area | No byte limit on the import |
+| OpenPGP key | One ASCII armored public key | 1 MiB |
+| KEYS file | ASCII armored public keys | 10 MiB |
+| Admin catalogue import | Projects, releases, and artifacts CSV files | 512 MiB per request |
+
+`MAX_CONTENT_LENGTH` in [`config`](/ref/atr/config.py) sets the HTTP request limit, enforced by [`body`](/ref/atr/body.py). Form fields and encoding count towards it, so multipart and base64 uploads have less room for file content. There is no limit on the total size of a release.
+
+[`ssh`](/ref/atr/ssh.py) sets the rsync limit, and [`shared.keys`](/ref/atr/shared/keys.py) defines the key limits. Key imports reject private key material. Keys and catalogue CSV files are parsed separately from release uploads.
+
+## Format checks
+
+Release upload paths must satisfy the [path rules](input-validation#file-names). Before creating a revision, [`revision`](/ref/atr/storage/writers/revision.py) validates the whole proposed tree, including files carried over from the previous revision.
+
+[`detection`](/ref/atr/detection.py) rejects symlinks and checks file contents against the extensions below. Empty files, unrecognized contents and format mismatches fail these checks. Files with other extensions are not inspected at this stage.
+
+| Expected format | Extensions |
+| --- | --- |
+| ZIP | `.apk`, `.jar`, `.nar`, `.nbm`, `.vsix`, `.war`, `.whl`, `.zip` |
+| gzip | `.pack.gz`, `.tar.gz`, `.tgz` |
+| bzip2, xz, tar | `.tar.bz2`, `.tar.xz`, `.tar` |
+| Debian, RPM, Windows executable, PDF | `.deb`, `.rpm`, `.exe`, `.pdf` |
+
+## Archive quarantine
+
+Quarantine applies to `.tar.gz`, `.tgz`, `.zip`, `.tar.bz2`, `.tar.xz`, `.jar`, `.war`, `.apk`, `.nar`, and `.whl` archives. ATR can reuse validation for identical content previously accepted in the same release with the same suffix, and `.tgz` and `.tar.gz` count as one suffix.
+
+If any archive needs validation, the whole proposed revision stays outside download paths while a [`quarantine`](/ref/atr/tasks/quarantine.py) worker extracts its archives. [`archives`](/ref/atr/archives.py) sets these default extraction limits:
+
+* 2 GiB per file and per archive, configured by `MAX_EXTRACT_SIZE`
+* 100,000 files per archive
+* A compression ratio of 100
+* A path depth of 32
+
+Absolute paths and hard links are rejected. Symlinks inside archives must stay within the extraction root.
+
+Once all archives pass validation, the worker creates the revision and starts the release [checks](checks). Those checks can block a vote, but the files remain downloadable. An extraction error marks the submission failed and removes the quarantined files. A worker interruption can leave the quarantine pending, which blocks starting a vote. See [Resource management](resource-management) for worker limits.
+
+## Downloads
+
+Individual files are public, except in embargoed releases, which are hidden from users without access. [`download`](/ref/atr/get/download.py) serves local files as `application/octet-stream` attachments. [`safe.StatePath`](/ref/atr/models/safe.py) keeps resolved paths inside managed storage. The frontend proxy must supply `X-Content-Type-Options: nosniff`, as noted in [`server`](/ref/atr/server.py). Published downloads redirect to the ASF download services.
+
+Committers can also download an unreleased revision as a ZIP, with no size or file count limit. File previews require a committer login and show the first 512 KiB as escaped text or a hex dump.
+
+ATR does not scan files for malware.
diff --git a/atr/docs/index.md b/atr/docs/index.md
index e586e70b..86b46c3d 100644
--- a/atr/docs/index.md
+++ b/atr/docs/index.md
@@ -48,3 +48,4 @@ NOTE: This documentation is a work in progress.
* `7.19.` [ASF modules](asf-modules)
* `7.20.` [Resource management](resource-management)
* `7.21.` [SBOM architecture](sbom-architecture)
+ * `7.22.` [File handling](file-handling)
diff --git a/atr/docs/input-validation.md b/atr/docs/input-validation.md
index 5ab17893..a1c823e9 100644
--- a/atr/docs/input-validation.md
+++ b/atr/docs/input-validation.md
@@ -29,7 +29,7 @@ Input validation is critical for ATR's security posture. As a system that handle
ATR employs multiple layers of validation:
1. **Transport layer**: HTTPS required, enforced by httpd
-2. **Request layer**: Size limits enforced by httpd (`MAX_CONTENT_LENGTH`)
+2. **Request layer**: ATR enforces the request body limit set by `MAX_CONTENT_LENGTH`
3. **Form layer**: Pydantic models validate structure and types
4. **Application layer**: Business logic validation in route handlers, interaction helpers, and storage writers
5. **Database layer**: SQLAlchemy ORM with parameterized queries, plus constraints
@@ -154,11 +154,7 @@ Committee and project names are validated against the set of known committees an
### File names
-File names in uploads are sanitized to prevent path traversal:
-
-* Directory separators (`/`, `\`) and the path token `..` are rejected or stripped
-* Null bytes are rejected
-* Only expected extensions are permitted per upload type
+Release upload paths use [`safe.RelPath`](/ref/atr/models/safe.py), which rejects absolute paths, directory traversal, null bytes and most dotfiles. These rules apply to every upload method, including rsync and SVN imports. For file content checks, see [File handling](file-handling).
## Data integrity validation
@@ -314,23 +310,7 @@ Never mark user-controlled data as safe without proper sanitization.
## File upload security
-File uploads are handled with several security measures:
-
-### Size limits
-
-Maximum upload size is enforced at the httpd layer via `MAX_CONTENT_LENGTH`. This prevents denial-of-service attacks via large uploads.
-
-### Extension validation
-
-Each upload type has an allowlist of permitted file extensions. Files with unexpected extensions are rejected.
-
-### Storage location
-
-Uploaded files are stored outside the application in configured directories (e.g., `state/unfinished/`). They are not directly accessible via HTTP.
-
-### File handling
-
-Files are processed via [`quart.datastructures.FileStorage`](/ref/atr/form.py) (quart_request) and validated before being written to disk. Empty files (where the browser sends a file input with no selection) are filtered out.
+See [File handling](file-handling) for upload limits, format checks, archive quarantine and downloads.
## Injection prevention
diff --git a/atr/docs/resource-management.md b/atr/docs/resource-management.md
index 307f16da..3edce546 100644
--- a/atr/docs/resource-management.md
+++ b/atr/docs/resource-management.md
@@ -43,7 +43,7 @@ A claim recorded before creation times were kept identifies its claimant by proc
## Heavy operations
-Extraction ([`archives`](/ref/atr/archives.py)) allows 2 GB of file content (total and per file), 100,000 regular files, 100 to 1 expansion per member, depth 32, no absolute paths or hard links, and confines symlinks to the root. Quarantine extracts `.tar.gz`, `.tgz`, `.zip`, `.tar.bz2`, `.tar.xz`, `.jar`, `.war`, `.apk`, `.nar`, and `.whl` archives (plain `.tar`, `.deb`, `.rpm`, etc. are excluded). Caching is per release and content hash, but SBOM generation extracts again and concurren [...]
+Archive extraction ([`archives`](/ref/atr/archives.py)) allows up to 2 GiB of extracted file content and 100,000 files per archive by default. See [File handling](file-handling) for the formats and remaining extraction limits. Extracted content is cached per release and content hash. Concurrent validations can extract the same archive, and SBOM generation extracts it again.
RAT checks use capped JVM resources (64 MB heap, 32 MB metaspace), and are killed at 300s. Lightweight license checks allow 1 MB per license, and 4 KB per source file. SBOM tasks are divided into generation (extract, syft, 300s wait), quality score (sbomqs, 300s), tool score (untimed CycloneDX validator), OSV scan (1000 per batch at 60s, then a detail request per new vulnerability), augmentation (30s requests), and conversion (local). Augment and OSV rebuild whole revisions on change. Co [...]
@@ -51,7 +51,11 @@ Signature and hash checks read whole artifacts; KEYS import parses keys, consult
## Requests and responses
-Uploads (512 MB max, 3600s to receive on the three upload routes, 60s elsewhere) build a revision before answering unless newly detected archives require quarantine; then revision creation is deferred to a worker pending validation. They perform a hard link clone, validation, hash every file, and compare inodes. API uploads base64 decode entire payloads in memory first, except on the store endpoint, which streams the raw body to temporary files. Moves, deletions, hash generation, and SBO [...]
+HTTP uploads allow 512 MiB per request by default. ATR allows one hour to receive bodies on the browser and two API upload routes, and 60 seconds elsewhere. Uploads build a revision before answering unless archives need quarantine, which defers revision creation to a worker. Creating a revision involves cloning the previous files with hard links, validating the new tree, hashing files and comparing inodes. JSON API uploads decode the entire base64 payload in memory, and the raw store end [...]
+
+Moves, deletions, hash generation, and SBOM draft actions also build revisions during requests. ZIP downloads list every file before streaming. Announcing checks publication during the request and queues local file cleanup as a background task.
+
+Vote tabulation fetches whole mail threads, fetching up to 100 messages at once, then buffers and sorts them. A 10,000 message limit applies after fetching. Request handlers do not have the CPU or memory limits applied to workers.
Other heavy work is queued and pages poll. Recording a distribution means checking an external registry, which may not list the package yet. When that happens, the standard API endpoint returns an error and gives up, while the workflow endpoint stores the distribution as pending, reports success to the caller, and leaves a scheduled task to retry the check later. Slow bodies get a logged 408. After a handler returns, transmission has 60s before the connection closes. Handlers are unbound [...]
diff --git a/atr/docs/sbom-architecture.md b/atr/docs/sbom-architecture.md
index efdc88f1..44af10c0 100644
--- a/atr/docs/sbom-architecture.md
+++ b/atr/docs/sbom-architecture.md
@@ -4,7 +4,7 @@
**Prev**: `7.20.` [Resource management](resource-management)
-**Next**: (none)
+**Next**: `7.22.` [File handling](file-handling)
**Sections**:
diff --git a/atr/docs/uploading-files.md b/atr/docs/uploading-files.md
index 959cf3b0..6f591a90 100644
--- a/atr/docs/uploading-files.md
+++ b/atr/docs/uploading-files.md
@@ -9,6 +9,7 @@
**Sections**:
* [Overview](#overview)
+* [File handling](#file-handling)
* [Browser](#browser)
* [rsync](#rsync)
* [SVN import](#svn-import)
@@ -31,6 +32,22 @@ The upload page for a release, reached with the "Upload files" button on its com
the browser, SVN, rsync, and GitHub workflow routes, and shows the exact commands and paths for
that release.
+## File handling
+
+Upload release artifacts, their signatures and checksums, and related documentation and SBOMs. Common artifact formats include `.tar.gz`, `.tgz` and `.zip`. ATR checks the contents of selected formats but does not reject every unknown extension. An accepted upload can still fail the [release checks](checks).
+
+| Upload method | Size limit |
+| --- | --- |
+| Browser and HTTP API | 512 MiB per request by default, including form fields and encoding overhead |
+| rsync | Files over 2 GB are skipped |
+| SVN import | No explicit byte limit on the imported files |
+
+For selected archive formats, ATR holds the whole proposed revision in quarantine while it extracts and validates the archives. The default unpacked size limit is 2 GiB for each file and for the total content of each archive. The previous revision remains available. If an archive fails extraction, ATR records the error and removes the quarantined upload. Correct the files and upload again. Dismissing the failure only acknowledges it.
+
+Files already in a revision remain downloadable while release checks run or report problems. Blockers prevent starting a vote. Local files are served as attachments, and published downloads go to ASF distribution services. ATR does not scan files for malware. Verify the [signatures](signing-artifacts) and checksums before using a download.
+
+See [File handling in the developer guide](file-handling) for the format lists, extraction limits, download controls and separate limits for key imports.
+
## Browser
Use the "File upload" form on the upload page to select one or more files. If your project has a
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org