You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tooling.apache.org by "andrewmusselman (via GitHub)" <gi...@apache.org> on 2026/06/16 23:19:38 UTC
[I] Verify find() bookmark pagination on a >10k-document namespace; add skip-based fallback if needed (tooling-gofannon)
andrewmusselman opened a new issue, #48:
URL: https://github.com/apache/tooling-gofannon/issues/48
## Background
`CouchDBService.find()` (`services/database_service/couchdb.py`) historically defaulted to `limit=10000`. Callers that span a whole namespace — `list_keys`, `get_all`, `list_namespaces` in `data_store_service.py` — called it without an explicit limit and were silently truncated at 10,000 rows. Consequences on any namespace with >10k docs:
- `clear_namespace` deleted only the first 10,000 docs and left the rest as orphans.
- Every `list_keys`-based read (ASVS discovery file scope, component detection, audit file reads) saw at most 10,000 files.
This affects any repository with more than 10,000 files. It surfaced when an OpenOffice audit logged `Cleared 10000 existing files` — a suspiciously round number that turned out to be the cap, not a real count.
## Fix applied
- `couchdb.find()`: `limit=None` now means "return all matching rows," fetched via CouchDB Mango **bookmark pagination** (10,000-row pages until a short page or a repeated/absent bookmark). Explicit integer limits are unchanged — they still issue a single bounded request (e.g. `run_registry`'s `limit=100`).
- `data_store_service.py`: `list_keys`, `get_all`, `list_namespaces` now pass `limit=None`.
- `base.find()`: accepts `limit=None` as unbounded, so non-CouchDB backends (memory, etc.) are safe.
Tested in isolation: pagination returns all rows at 44,850 and 130,000 with no duplicates.
## Why this issue exists (the open risk)
The bookmark pagination reads the continuation token off the query result object as `result.bookmark`. This matches python-cloudant's documented behavior, but **has not yet been verified against the actual deployed CouchDB client on a real large namespace.** If the client surfaces the bookmark under a different attribute (or not at all), the pagination loop stops after the first page and degrades to the old capped behavior — no worse than before, but not fixed. The code is defensive: it stops rather than looping forever if the bookmark is missing or unchanged.
## Verification owed (do this on the first >10k-file run)
OpenOffice (9,938 files) is under the cap and **cannot** exercise this. Use a repository known to exceed 10,000 files.
Either:
1. Run a normal audit and watch the log for the file/cleared counts, **or**
2. Direct check from the gofannon environment:
```python
ns = data_store.use_namespace("files:apache/<big-repo>")
print(len(ns.list_keys()))
```
**Pass:** the count reflects the true number (e.g. `47213`).
**Fail:** the count freezes at exactly `10000` for a repo known to be larger.
## Fallback if verification fails
Switch `couchdb.find()`'s `limit=None` path from bookmark pagination to **skip-based pagination** (`"skip": offset`, incrementing by page size until a short page). Skip is universally supported by all CouchDB clients, so it cannot have the surfacing problem. Trade-off: slightly slower on very large namespaces (the server scans past skipped rows each page), but correct everywhere. This is a localized change to the one pagination loop.
## Cleanup note (operational, not code)
This fix prevents future orphans but does not retroactively remove orphans left by past truncated clears. Any namespace previously cleared while holding >10k docs still has stale documents. After deploying, those namespaces should be cleaned once — either a manual CouchDB wipe of the affected `files:*` namespaces, or a `clearCache=true` run (now that `clear` paginates, it will delete all of them).
## Deploy paths
- `services/database_service/couchdb.py`
- `services/database_service/base.py`
- `services/data_store_service.py`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tooling.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org