You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mahout.apache.org by "rich7420 (via GitHub)" <gi...@apache.org> on 2026/02/03 08:39:30 UTC
[PR] [QDP] File and streaming Quantum Data Loader [mahout]
rich7420 opened a new pull request, #1010:
URL: https://github.com/apache/mahout/pull/1010
### Purpose of PR
<!-- Describe what this PR does. -->
Adds file-backed and streaming Parquet data sources to the Quantum Data Loader. Users can call `.source_file(path)` or `.source_file(path, streaming=True)` and iterate in batches. PyO3 loader bindings are restored so `QuantumDataLoader` works with synthetic, file, and streaming sources.
- **qdp-core:** `DataSource::InMemory` and `DataSource::Streaming`; `new_from_file` (full read, supports .parquet/.arrow/.npy/.pt/.pb etc.) and `new_from_file_streaming` (Parquet, chunked read). Shared `path_extension_lower`, `take_batch_from_source`, named constants, first-chunk buffer reuse. Basis encoding fix: state vector allocated as Float64 to match kernel, then converted to engine precision.
- **qdp-python (Rust):** `PyQuantumLoader`, `create_synthetic_loader`, `create_file_loader`, `create_streaming_file_loader` (Linux only). `batch_limit=None` → `usize::MAX`; path from str or Path; file/streaming build in `py.detach()`.
- **loader.py:** `source_file(path, streaming=False)`, `_create_iterator()`. Streaming requires `.parquet`.
- **Tests:** New loader tests (mutual exclusion, batch count, extension, streaming). DLPack and bindings tests updated for current error messages.
### Related Issues or PRs
<!-- Add links to related issues or PRs. -->
<!-- - Closes #123 -->
<!-- - Related to #123 -->
Related to #969
### Changes Made
<!-- Please mark one with an "x" -->
- [ ] Bug fix
- [x] New feature
- [ ] Refactoring
- [ ] Documentation
- [ ] Test
- [ ] CI/CD pipeline
- [ ] Other
### Breaking Changes
<!-- Does this PR introduce a breaking change? -->
- [ ] Yes
- [x] No
### Checklist
<!-- Please mark each item with an "x" when complete -->
<!-- If not all items are complete, please open this as a **Draft PR**.
Once all requirements are met, mark as ready for review. -->
- [x] Added or updated unit tests for all changes
- [ ] Added or updated documentation for all changes
- [x] Successfully built and ran all unit tests or manual tests locally
- [ ] PR title follows "MAHOUT-XXX: Brief Description" format (if related to an issue)
- [ ] Code follows ASF guidelines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] File and streaming Quantum Data Loader [mahout]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming commented on code in PR #1010:
URL: https://github.com/apache/mahout/pull/1010#discussion_r2762758510
##########
qdp/qdp-python/qumat_qdp/loader.py:
##########
Review Comment:
I think both _source_type string and three boolean flags track the same information. Maybe we could consider to keep only booleans since this would eliminate ~10 lines of validation logic in `_create_iterator()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] File and streaming Quantum Data Loader [mahout]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming commented on PR #1010:
URL: https://github.com/apache/mahout/pull/1010#issuecomment-3864631401
Thanks for the update!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] File and streaming Quantum Data Loader [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #1010:
URL: https://github.com/apache/mahout/pull/1010#issuecomment-3861778013
Overall LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] File and streaming Quantum Data Loader [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #1010:
URL: https://github.com/apache/mahout/pull/1010#issuecomment-3846006001
Please resolve conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] File and streaming Quantum Data Loader [mahout]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming commented on PR #1010:
URL: https://github.com/apache/mahout/pull/1010#issuecomment-3861684300
Overall looks nice! Please help resolve conflict and I think we are good to merge~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] File and streaming Quantum Data Loader [mahout]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on PR #1010:
URL: https://github.com/apache/mahout/pull/1010#issuecomment-3865984791
thanks for the review!
@guan404ming , @400Ping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] File and streaming Quantum Data Loader [mahout]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming merged PR #1010:
URL: https://github.com/apache/mahout/pull/1010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] File and streaming Quantum Data Loader [mahout]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on code in PR #1010:
URL: https://github.com/apache/mahout/pull/1010#discussion_r2767505011
##########
qdp/qdp-python/qumat_qdp/loader.py:
##########
Review Comment:
nice catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] File and streaming Quantum Data Loader [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #1010:
URL: https://github.com/apache/mahout/pull/1010#issuecomment-3861724388
@rich7420 Please solve conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org