(magpie-site) 01/01: Serve and advertise the Markdown twins

[jm...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

justinmclean pushed a commit to branch markdown-negotiation
in repository https://gitbox.apache.org/repos/asf/magpie-site.git

commit 79eabd3996d1dabebd1ea5d97e46a866574e88e2
Author: Justin Mclean <ju...@classsoftware.com>
AuthorDate: Sat Sep 5 03:42:59 2026 +0000

    Serve and advertise the Markdown twins
    
    The previous change writes a .md twin beside every docs page; this one
    makes them findable without knowing the convention.
    
    - public/.htaccess: text/markdown on .md files and the two llms files, and
      X-Robots-Tag: noindex on the twins, which are alternates of the HTML
      page, not pages
    - src/components/JsonLd.astro: <link rel="alternate" type="text/markdown">
      on docs pages, pointing at the twin, so it is discoverable from the HTML.
      DocsLayout.astro needs to pass docMarkdownPath(entry) through
    
    No Accept-header negotiation, as on iggy.apache.org: the sibling file needs
    no server control. The is-agentic scan reported negotiation without Vary on
    this site; PR.md has the curl to confirm whether the server does it on its
    own, and the one-line Vary header to add only if it does.
    
    PR.md carries the wiring steps and the curl checks for all three branches.
    
    Generated-by: Claude (Fable 5.1)
---
 PR.md                       | 69 +++++++++++++++++++++++++++++++++++++++++++++
 public/.htaccess            | 10 +++++++
 src/components/JsonLd.astro | 11 +++++++-
 3 files changed, 89 insertions(+), 1 deletion(-)

diff --git a/PR.md b/PR.md
new file mode 100644
index 0000000..ff34160
--- /dev/null
+++ b/PR.md
@@ -0,0 +1,69 @@
+# feat(site): agent discoverability — llms.txt, Markdown twins, sitemap, 404, structured data
+
+Ports apache/iggy-website #77 and #79 to the Astro site. Nothing visible to a
+person changes; every addition is a file agents read or a header they see.
+
+## What the scan found (is-agentic, 59/100, 5 Sep)
+
+- unknown paths answer 200 with the app shell (critical)
+- the scanner reported Accept negotiation without `Vary: Accept`; unverified, see below
+- no `llms.txt`; developer resources not discoverable by name
+- phantom "API" surface: the scanner inferred an HTTP API and marked it blocked
+
+## Files
+
+| Path | Does |
+|---|---|
+| `public/.htaccess` | `ErrorDocument 404 /404.html`; `text/markdown` type on `.md` and the llms files |
+| `public/robots.txt` | allow all, points at the sitemap |
+| `src/lib/site.ts` | site constants, static route list, docs → markdown helpers |
+| `src/pages/llms.txt.ts` | llmstxt.org index from the docs collection; says plainly there is no API on this domain |
+| `src/pages/llms-full.txt.ts` | every docs page concatenated |
+| `src/pages/docs/[...slug].md.ts` | Markdown twin of every docs page, same path with `.md` |
+| `src/components/JsonLd.astro` | Organization / WebSite / SoftwareSourceCode JSON-LD, `rel=alternate` to the twin, `rel=llms-txt` |
+| `src/pages/404.astro` | recovery links |
+
+## Wiring (needs the checkout)
+
+1. `npm install` (branch 1 adds `@astrojs/sitemap` to package.json and wires
+   it in astro.config.mjs; the lockfile needs regenerating).
+2. `DOCS_COLLECTION` in `src/lib/site.ts` assumes the synced collection is
+   named `docs` and entries carry `title` / `description` frontmatter. Match to
+   `src/content.config.ts`.
+3. `docHtmlPath` / `docSourceUrl` assume the URL scheme `/docs/<id>/` and the
+   upstream layout `apache/magpie/docs/<id>.md`. If the sync rewrites names
+   (readme → index), adjust the two regexes.
+4. Add `<JsonLd markdownPath={...} />` to the base layout `<head>`; on docs
+   pages pass `docMarkdownPath(entry)`, elsewhere omit it.
+5. If a 404 page already exists, merge the link list into it and delete mine.
+6. No workflow change: Astro copies `public/` into `dist/`, and build.yml
+   pushes `dist/` to `publish`. Confirm `.htaccess` is in `dist/` after a build.
+7. **Check the catch-all.** The scan got a 200 for `/apache/magpie`. Search
+   `.htaccess`, `.asf.yaml` and the workflow for `RewriteRule`, `FallbackResource`
+   or a copy of `index.html` to `404.html`; whichever is producing the 200 has to
+   go, or `ErrorDocument` never fires.
+
+## Verify after `npm run build` (or on staging)
+
+```sh
+curl -s -o /dev/null -w "%{http_code}\n" https://magpie.apache.org/no-such-page      # 404
+curl -sI https://magpie.apache.org/docs/modes.md | grep -i '^content-type'            # text/markdown
+curl -sI -H "Accept: text/markdown" https://magpie.apache.org/docs/modes/ | grep -i '^content-type'  # see note
+curl -s https://magpie.apache.org/llms.txt | head -20
+curl -s https://magpie.apache.org/docs/modes/ | grep -o '<link rel="alternate"[^>]*>'
+grep -c '"@type":"SoftwareSourceCode"' dist/index.html                                # 1
+```
+
+Then rescan: `npx is-agentic magpie.apache.org`.
+
+## Deliberately not done
+
+- No Accept-header negotiation, same as Iggy: a static export on ASF
+  infrastructure has no server-side control worth relying on, and the `.md`
+  sibling needs none. If the curl above already returns text/markdown, the
+  server is negotiating on its own (MultiViews); then, and only then, add
+  `Header merge Vary "Accept"` to `.htaccess` so caches key the variants apart.
+
+- No OpenAPI spec and no JSON error bodies: there is no API. `llms.txt` states
+  that, which is what removes the scanner's phantom API surface.
+- No ASF postal address in the JSON-LD (dropped from Iggy for the same reason).
diff --git a/public/.htaccess b/public/.htaccess
index 6d39488..05c74ff 100644
--- a/public/.htaccess
+++ b/public/.htaccess
@@ -24,6 +24,16 @@
 # with its own bare "Not Found" body and the page's recovery links are lost.
 ErrorDocument 404 /404.html
 
+# Root discovery files and the .md twins carry the Markdown media type. The
+# twins are alternates of the HTML page, not pages, so they stay out of
+# search results.
 <Files "llms.txt">
   Header set Content-Type "text/markdown; charset=utf-8"
 </Files>
+<Files "llms-full.txt">
+  Header set Content-Type "text/markdown; charset=utf-8"
+</Files>
+<FilesMatch "\.md$">
+  Header set Content-Type "text/markdown; charset=utf-8"
+  Header set X-Robots-Tag "noindex"
+</FilesMatch>
diff --git a/src/components/JsonLd.astro b/src/components/JsonLd.astro
index 4b00a77..fd8d421 100644
--- a/src/components/JsonLd.astro
+++ b/src/components/JsonLd.astro
@@ -19,10 +19,16 @@
  */
 
 // Organization / WebSite / SoftwareSourceCode structured data for the <head>.
-// Drop into BaseLayout.astro's <head> once.
+// Drop into BaseLayout.astro's <head> once; DocsLayout.astro passes
+// `markdownPath` through so the <link rel="alternate"> lands on docs pages.
 
 import { SITE_URL, SITE_NAME, SITE_DESCRIPTION, SOURCE_REPO, DEV_LIST } from "../lib/site";
 
+interface Props {
+  /** Path of this page's Markdown twin, e.g. "/docs/modes.md". Omit on non-docs pages. */
+  markdownPath?: string;
+}
+const { markdownPath } = Astro.props;
 
 const org = {
   "@type": "Organization",
@@ -67,4 +73,7 @@ const json = JSON.stringify(payload).replace(/</g, "\\u003c");
 ---
 
 <script type="application/ld+json" set:html={json} />
+{markdownPath && (
+  <link rel="alternate" type="text/markdown" href={`${SITE_URL}${markdownPath}`} />
+)}
 <link rel="llms-txt" href={`${SITE_URL}/llms.txt`} />