[I] [Docs] Refactor or replace website/scripts/sync-docs.js with Docusaurus-native content loading [mahout]

["bearomorphism (via GitHub)" <gi...@apache.org>]

bearomorphism opened a new issue, #1308:
URL: https://github.com/apache/mahout/issues/1308

   ### What
   
   `website/scripts/sync-docs.js` is a Node script that copies `/docs/` to `website/docs/` at build time, transforming markdown along the way. It was introduced in #914 (the Jekyll → Docusaurus migration) and has barely been touched since (only #1040 for versioning wiring). While working on #1305 we found multiple issues with both the script and its underlying design.
   
   **Concrete problems in the script today**
   
   - `transformLinks()` strips `.md` from internal links, defeating Docusaurus's static link resolution. This is the direct cause of the 404s in #1301; #1305 reverts the strip, but the regex-based approach still won't see `<url>` autolinks, reference-style `[text][ref]` definitions, or links inside HTML/JSX.
   - `parseFrontmatter` is hand-rolled YAML parsing (regex + `split(':')`). Fine for `title: Foo` but silently misparses multi-line values, lists, nested maps, or quoted strings containing `:`. `generateFrontmatter` only quotes when the value contains `:` or `#`, leaving other YAML edge cases untested.
   - `transformFrontmatter` auto-generates titles via `s/[_-]/ /g` + `\b\w` capitalize, which mangles acronyms (`qdp` → `Qdp`, not `QDP`).
   - No tests on any of it.
   
   **Bigger architectural concern**
   
   The whole *"sync source into a gitignored build dir, then run Docusaurus on the build dir"* model is fighting the framework. Docusaurus has first-class support for content from arbitrary paths (`docs.path` in the classic preset, multi-instance plugin, monorepo presets). Pointing Docusaurus directly at `/docs/` would eliminate the sync step and let us use Docusaurus's link resolution, frontmatter parsing, and authoring tools natively — which is exactly the loop that broke for #1301 (a sync-time transform stripped the very signal Docusaurus needed to resolve links correctly).
   
   ### Why
   
   - New contributors hit silent broken links (#1301 → #1305 review thread).
   - The sync layer is a non-obvious layer of indirection: edits in `/docs/` only show up after `npm run sync`. Contributors who edit `website/docs/` directly lose their work on the next sync (the dir is gitignored).
   - It's a place where bugs hide that don't show up in code review — the `.md` strip was merged without anyone noticing the implication, and lay there for ~4 months until #1301.
   - ADR-grade discussion is probably warranted before changing the model — there may be reasons for the current setup that aren't visible from the script alone (versioning workflow, blog/docs split, etc.).
   
   ### How
   
   Two reasonable directions, in increasing scope:
   
   1. **Patch in place.** Replace the ad-hoc YAML parser with `gray-matter` (already transitively pulled in via Docusaurus); replace the regex link transform with `unified` + `remark-parse` for any transforms still needed; add a small test suite. Low-risk, modest payoff.
   
   2. **Adopt Docusaurus-native content loading.** Configure the classic preset's `docs.path` (and `blog.path`) to read directly from `/docs/` and `/docs/blog/`. Drop `sync-docs.js`, drop `website/docs/` and `website/blog/` (currently both gitignored mirrors), drop the prebuild hook. This eliminates the class of bugs that #1301 / #1305 came from but probably warrants an ADR — versioning still snapshots into `website/versioned_docs/`, and the interaction with that needs a quick design pass.
   
   #1305 fixes the immediate symptom. This issue is about deciding which of the two directions (or a hybrid) to take next.
   
   ### Related
   
   - #1301 (How to Contribute stale links — the user-facing symptom)
   - #1305 (the immediate fix; reverts the `.md`-strip in `transformLinks`)
   - #914 (origin commit that introduced `sync-docs.js`)
   - #948 (compounded the bug by switching some links to `../...` form)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Docs] Refactor or replace website/scripts/sync-docs.js with Docusaurus-native content loading [mahout]

["ryankert01 (via GitHub)" <gi...@apache.org>]

ryankert01 commented on issue #1308:
URL: https://github.com/apache/mahout/issues/1308#issuecomment-4414827309

   Giving some context: the sync is due to a design decision at https://github.com/apache/mahout/discussions/882
   From a slack discussion of we should be somewhat framework agnostic and some more reasons.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Docs] Refactor or replace website/scripts/sync-docs.js with Docusaurus-native content loading [mahout]

["bearomorphism (via GitHub)" <gi...@apache.org>]

bearomorphism commented on issue #1308:
URL: https://github.com/apache/mahout/issues/1308#issuecomment-4414772964

   cc @ryankert01 the first author of `sync-docs.js`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org