You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mahout.apache.org by "400Ping (via GitHub)" <gi...@apache.org> on 2026/02/03 14:10:19 UTC
[I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
400Ping opened a new issue, #1012:
URL: https://github.com/apache/mahout/issues/1012
### Summary
Implement an automated API documentation pipeline that generates and publishes API reference documentation from the Python (Qumat, QDP) and Rust (qdp-core) codebases, integrated into the project's Docusaurus website and CI.
Current state:
- QuMat API is maintained by hand and can drift from code.
- QDP API has a placeholder "TODO: Add API reference documentation for QDP".
- Rust (qdp-core) has extensive doc comments but no published rustdoc in the website.
Goals:
1. Generate API reference from source for Python (Qumat + QDP) and Rust (qdp-core).
2. Integrate generated docs into the existing Docusaurus site.
3. Automate the pipeline in CI so doc builds run on changes.
4. Define conventions (docstrings, public API) for future contributors.
Deliverables:
- Python API doc pipeline (Sphinx or pdoc) for qumat and QDP.
- QDP API reference page populated (replace TODO).
- QuMat API reference either generated or explicitly linked.
- Rust (qdp-core) rustdoc built and linked from the website.
- CI job(s) that build Python API docs and rustdoc and fail on errors.
- Short contribution guide on docstring style and how to update API docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen commented on issue #1012:
URL: https://github.com/apache/mahout/issues/1012#issuecomment-3864895949
Hi, @400Ping .
I’m interested in this GSoC proposal and would like to get involved.
I’ve contributed **15+ PRs** to Apache Mahout and have been working closely with the QDP and QuMat codebases (Python, Rust, and CUDA...), so I’m fairly familiar with the current API surfaces, documentation gaps, and the existing website/CI setup.
I’ve started thinking about possible approaches (e.g., Python API docs via Sphinx or pdoc, rustdoc for qdp-core, and integrating or linking them from the Docusaurus site), and I’d really appreciate mentors’ thoughts on scope, tooling choices, and how best to integrate this with the current website.
Happy to discuss further or start with a small PoC if that’s helpful.
Thanks! 🙌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen commented on issue #1012:
URL: https://github.com/apache/mahout/issues/1012#issuecomment-4583019706
I opened the roadmap follow-up issues and linked them below. I cannot edit the original issue body directly, so adding the linked roadmap as a comment.
# GSoC 2026 — Automated API Documentation Pipeline
**Scope:** `qumat`, `qumat_qdp`, `qdp-core`, Docusaurus website, docs CI
**Goal:** Generate Python and Rust API references directly from source code and publish them under a unified Docusaurus `API Reference` section, with CI validation, local developer commands, and release/tag-based versioning.
**Source planning issue:** viiccwen/mahout#3
---
## Track A — Public API audit
> Establish the public API boundary and documentation conventions before building the generator.
| ID | Issue | What to do | Week |
|----|-------|------------|------|
| A1 | [#1344](https://github.com/apache/mahout/issues/1344) `[docs] Audit public Python APIs for missing docstrings` | Audit public `qumat` / `qumat_qdp` APIs and identify public APIs with missing docstrings. | Week 1 |
| A2 | [#1345](https://github.com/apache/mahout/issues/1345) `[docs] Refresh insufficient or outdated Python API docstrings` | Review existing public API docstrings, identify stale or insufficient descriptions, and define/align the docstring convention for follow-up implementation. | Week 1 |
---
## Track B — Python API documentation pipeline
> Build the Sphinx-based Python API reference pipeline and generate Docusaurus-compatible Markdown in reviewable stages.
| ID | Issue | What to do | Week |
|----|-------|------------|------|
| B1 | [#1346](https://github.com/apache/mahout/issues/1346) `[feature] Add Sphinx Markdown foundation for Python API docs` | Set up Sphinx with Markdown output; configure autodoc and type hints; add a local `make docs-python` entry point; produce the first end-to-end Python API build under `docs/api/python/`. | Week 2 |
| B2 | [#1347](https://github.com/apache/mahout/issues/1347) `[feature] Generate Python API pages from public module discovery` | Discover public modules automatically; generate Sphinx stubs and module lists; enforce one-module-per-page output for the Python API reference. | Week 3 |
| B3 | [#1348](https://github.com/apache/mahout/issues/1348) `[docs] Polish generated Python API Markdown for Docusaurus readability` | Post-process generated Markdown where needed, including long signatures and formatting that affects Docusaurus readability, without changing the documented API surface. | Week 3 |
---
## Track C — Rust API documentation pipeline
> Convert Rust API docs from rustdoc comments into website-ready Markdown without making CUDA/GPU availability a hard requirement.
| ID | Issue | What to do | Week |
|----|-------|------------|------|
| C1 | [#1349](https://github.com/apache/mahout/issues/1349) `[feature] Plan Rust API surface and rustdoc conversion path` | Review the Rust API surface starting with `qdp-core`; define published crates/modules; validate rustdoc JSON and `rustdoc-md` feasibility; document how the workflow should behave when CUDA or nightly rustdoc tooling is unavailable. | Week 4 |
| C2 | [#1350](https://github.com/apache/mahout/issues/1350) `[feature] Generate Rust API Markdown for Docusaurus` | Implement the Rust API doc generation script and package list; post-process Markdown for Docusaurus; add a local `make docs-rust` entry point; output generated docs to `docs/api/rust/`. | Week 5 |
---
## Track D — Website integration
> Publish Python and Rust API references under one Docusaurus API section with generated navigation.
| ID | Issue | What to do | Week |
|----|-------|------------|------|
| D1 | [#1351](https://github.com/apache/mahout/issues/1351) `[feature] Add unified API Reference section to the website` | Add an API Reference landing page and integrate the generated Python/Rust API docs into the Docusaurus docs structure with a consistent top-level entry point. | Week 6 |
| D2 | [#1352](https://github.com/apache/mahout/issues/1352) `[feature] Generate Docusaurus API sidebars from Python and Rust doc outputs` | Add a website-side sidebar generation script; wire it into local website build/start commands; provide graceful fallback behavior when generated API outputs do not exist yet. | Week 6 |
---
## Track E — CI, previews, and versioning
> Keep API docs reproducible for contributors and releases, while separating low-risk validation from higher-risk workflows that require write or deploy permissions.
| ID | Issue | What to do | Week |
|----|-------|------------|------|
| E1 | [#1353](https://github.com/apache/mahout/issues/1353) `[ci] Add API docs build validation workflow` | Add a GitHub Actions workflow that installs docs dependencies, runs the Python API docs build, runs the Rust API docs build where supported, and validates the Docusaurus website build. Keep this workflow read-only and focused on validation. | Week 7 |
| E2 | [#1354](https://github.com/apache/mahout/issues/1354) `[ci] Upload API docs preview artifacts for pull requests` | Upload generated API docs / website preview artifacts for PR review, with API-related path filters. Avoid branch pushes or deploy permissions in the first preview iteration. | Week 7 |
| E3 | [#1355](https://github.com/apache/mahout/issues/1355) `[feature] Add local docs versioning command for release validation` | Add a local command/script for generating versioned docs from a supplied version string, so release documentation can be tested before enabling tag automation. | Week 8 |
| E4 | [#1356](https://github.com/apache/mahout/issues/1356) `[ci] Generate versioned API docs from release tags` | Add a tag-triggered workflow for versioned API documentation after the local versioning flow and ASF deployment/write-back policy are agreed. Keep permissions and generated-file write-back behavior explicit for review. | Week 8 |
---
## Track F — UX iteration, hardening, and contributor docs
> Use feedback to refine the generated docs and make the workflow maintainable after GSoC.
| ID | Issue | What to do | Week |
|----|-------|------------|------|
| F1 | [#1357](https://github.com/apache/mahout/issues/1357) `[docs] Iterate on API docs UX with user feedback` | Collect feedback from ML practitioners, quant traders, and contributors; evaluate usability/navigation; iterate on structure and wording. | Week 9 |
| F2 | [#1358](https://github.com/apache/mahout/issues/1358) `[feature] Polish and harden the API documentation pipeline` | Improve consistency across Python and Rust docs; handle edge cases and error states; address mentor feedback. | Week 10 |
| F3 | [#1359](https://github.com/apache/mahout/issues/1359) `[docs] Document the API docs contributor workflow` | Write contributor guide for building, previewing, versioning, and troubleshooting API docs. | Week 11 |
| F4 | [#1360](https://github.com/apache/mahout/issues/1360) `[docs] Final validation, demo, and report for API docs pipeline` | Reserve buffer for unexpected issues; validate final goals; prepare final report and demo. | Week 12 |
---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen commented on issue #1012:
URL: https://github.com/apache/mahout/issues/1012#issuecomment-3942310130
@shiavm006 It's a GSoC proposal, everyone can raise a early proposal to apply. : )
You can trace comments above to know more detail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on issue #1012:
URL: https://github.com/apache/mahout/issues/1012#issuecomment-3865189908
@Vanessamae23 Sweet, please introduce yourself with the community on the dev mailing list! Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on issue #1012:
URL: https://github.com/apache/mahout/issues/1012#issuecomment-3902000353
@Vanessamae23 Any updates?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen commented on issue #1012:
URL: https://github.com/apache/mahout/issues/1012#issuecomment-3869978002
> Sounds great, email me([jiekaichang@apache.org](mailto:jiekaichang@apache.org)) and let's have a chat. I will love to have a proposal and a PoC for what you going to do.
Hello @400Ping,
I've already sent you the preview link and the first draft of the proposal via email.
Really looking forward to your feedback and happy to discuss further anytime. Thanks a lot! 🙌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "Vanessamae23 (via GitHub)" <gi...@apache.org>.
Vanessamae23 commented on issue #1012:
URL: https://github.com/apache/mahout/issues/1012#issuecomment-3902092701
> [@Vanessamae23](https://github.com/Vanessamae23) Sweet, please introduce yourself with the community on the dev mailing list! Thanks!
> [@Vanessamae23](https://github.com/Vanessamae23) Any updates?
Yup will do thankss!! I will send over a short draft proposal on the current implementation but if @viiccwen or anyone else wants to take the feature, feel free to =)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "shiavm006 (via GitHub)" <gi...@apache.org>.
shiavm006 commented on issue #1012:
URL: https://github.com/apache/mahout/issues/1012#issuecomment-3941601991
hey @viiccwen are u already working on this ? then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on issue #1012:
URL: https://github.com/apache/mahout/issues/1012#issuecomment-3902443979
> > [@Vanessamae23](https://github.com/Vanessamae23) Sweet, please introduce yourself with the community on the dev mailing list! Thanks!
>
> > [@Vanessamae23](https://github.com/Vanessamae23) Any updates?
>
> Yup will do thankss!! I will send over a short draft proposal on the current implementation but if @viiccwen or anyone else wants to take the feature, feel free to =)
Hi, I replied to your email. Btw do you need any help with the workflow or something? If you need help we can have a sync online and I can try to help you within my abilities.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on issue #1012:
URL: https://github.com/apache/mahout/issues/1012#issuecomment-3865066922
> Hi everyone! I drafted a rough sketch to address this (https://issues.apache.org/jira/browse/GSOC-313) although i am not sure if i am getting it right
>
> This script currently handles the Python API generation using pdoc and rustdoc for the Rust with a automated CI and a bot to deploy latest updates if there is, just a very drafted version here : https://github.com/Vanessamae23/mahout/pull/1
>
> With samples here,
> https://vanessamae23.github.io/mahout/api/rust/
> https://vanessamae23.github.io/mahout/api/python
>
> To run and see how it looks like in the web version: run start the docusaurus
>
> <img width="1470" height="843" alt="Image" src="https://github.com/user-attachments/assets/9f07b6f1-d2c6-430f-b8b4-b4a3aa53cd32" />
>
> <img width="1447" height="821" alt="Image" src="https://github.com/user-attachments/assets/a6f9e449-8549-482b-9f66-509e68505e8b" />
>
> I’d love to get some feedback on this approach and i am interested to take part in the gsoc but if it does not work or someone else has a better approach, sounds great too.. learnt along the way as well
Email me your proposal first and show me the pros and cons of different approaches. Also this needs to be implemented during summer not right now, but if best you can show me a simple PoC for this and record a video and a doc for step by step guide of your PoC will be best.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen closed issue #1012: [Feature] Automated API Documentation Pipeline for Qumat & QDP
URL: https://github.com/apache/mahout/issues/1012
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "Vanessamae23 (via GitHub)" <gi...@apache.org>.
Vanessamae23 commented on issue #1012:
URL: https://github.com/apache/mahout/issues/1012#issuecomment-3864850948
Hi everyone! I've put together a initial PR to address this although i am not sure if i am getting it right.
This script currently handles the Python API generation using pdoc and rustdoc for the Rust with a automated CI and a bot to deploy latest updates if there is, just a very drafted version here : https://github.com/Vanessamae23/mahout/pull/1
With samples here,
https://vanessamae23.github.io/mahout/api/rust/
https://vanessamae23.github.io/mahout/api/python
To run and see how it looks like in the web version: run start the docusaurus
<img width="1470" height="843" alt="Image" src="https://github.com/user-attachments/assets/9f07b6f1-d2c6-430f-b8b4-b4a3aa53cd32" />
<img width="1447" height="821" alt="Image" src="https://github.com/user-attachments/assets/a6f9e449-8549-482b-9f66-509e68505e8b" />
I’d love to get some feedback on this approach but if it does not work or someone else has a better approach, sounds great too.. learnt along the way as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen commented on issue #1012:
URL: https://github.com/apache/mahout/issues/1012#issuecomment-5306267952
Closing as superseded by #1361, which is the active API-documentation roadmap with concrete implementation tracks and remaining handoff work in #1359 and #1360.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on issue #1012:
URL: https://github.com/apache/mahout/issues/1012#issuecomment-3858280809
Nice, we need this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Automated API Documentation Pipeline for Qumat & QDP [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on issue #1012:
URL: https://github.com/apache/mahout/issues/1012#issuecomment-3865071101
> Hi, @400Ping .
>
> I’m interested in this GSoC proposal and would like to get involved.
>
> I’ve contributed **15+ PRs** to Apache Mahout and have been working closely with the QDP and QuMat codebases (Python, Rust, and CUDA...), so I’m fairly familiar with the current API surfaces, documentation gaps, and the existing website/CI setup.
>
> I’ve started thinking about possible approaches (e.g., Python API docs via Sphinx or pdoc, rustdoc for qdp-core, and integrating or linking them from the Docusaurus site), and I’d really appreciate mentors’ thoughts on scope, tooling choices, and how best to integrate this with the current website.
>
> Happy to discuss further or start with a small PoC if that’s helpful.
>
> Thanks! 🙌
Sounds great, email me(jiekaichang@apache.org) and let's have a chat. I would love to have a proposal and a PoC for what you going to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org