You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "machichima (via GitHub)" <gi...@apache.org> on 2025/12/24 11:24:43 UTC
[PR] [Housekeeping] Clean structure and make commands [mahout]
machichima opened a new pull request, #756:
URL: https://github.com/apache/mahout/pull/756
### Purpose of PR
Improve development experience by introducing make commands to install, run tests, and do profiling.
Also move `benchmark/` into `qdp-python` and update `pyproject.toml` to install dependencies needed with different `--group` args
### Related Issues or PRs
Closes #735
### Changes Made
<!-- Please mark one with an "x" -->
- [ ] Bug fix
- [ ] New feature
- [x] Refactoring
- [x] Documentation
- [ ] Test
- [ ] CI/CD pipeline
- [ ] Other
### Breaking Changes
<!-- Does this PR introduce a breaking change? -->
- [ ] Yes
- [ ] No
### Checklist
<!-- Please mark each item with an "x" when complete -->
<!-- If not all items are complete, please open this as a **Draft PR**.
Once all requirements are met, mark as ready for review. -->
- [ ] Added or updated unit tests for all changes
- [ ] Added or updated documentation for all changes
- [ ] Successfully built and ran all unit tests or manual tests locally
- [ ] PR title follows "MAHOUT-XXX: Brief Description" format (if related to an issue)
- [ ] Code follows ASF guidelines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "machichima (via GitHub)" <gi...@apache.org>.
machichima commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645577023
##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+ @echo "Available targets:"
+ @echo " make install - Install the mahout python package"
+ @echo " make build - Build qdp-core"
+ @echo " make build_nvtx_profile - Build qdp-core with observability features"
+ @echo " make run_nvtx_profile - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+ @echo " make unit_test - Run all unit tests (Python + Rust)"
+ @echo " make unit_test_python - Run Python unit tests only"
+ @echo " make unit_test_rust - Run Rust unit tests only"
+ @echo " make e2e_test - Run all e2e tests under qdp-python/benchmark"
+ @echo " make clean - Clean build artifacts"
+
+install:
+ @echo "Installing mahout python package..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop
+
+build:
+ @echo "Building qdp-core..."
+ cargo build -p qdp-core
+
+build_nvtx_profile:
+ @echo "Building qdp-core with observability features..."
+ cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+ @echo "Running Python unit tests..."
+ cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+ @echo "Running Rust unit tests..."
+ cargo test --workspace
+
+e2e_test:
+ @echo "Running e2e benchmark tests..."
+ cd qdp-python/benchmark && uv run python benchmark_e2e.py
+ cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+ $(eval EXAMPLE ?= nvtx_profile)
+ @echo "Building example '$(EXAMPLE)' with observability features..."
+ cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+ @echo "Running '$(EXAMPLE)' with nsys profiling..."
+ nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+ @echo "Showing profiling statistics..."
+ nsys stats --force-export=true report.nsys-rep
Review Comment:
Could you please show me how to profile benchmarks directly? The documentation we had profiling the example, that's why I only put example here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645526466
##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+ @echo "Available targets:"
+ @echo " make install - Install the mahout python package"
+ @echo " make build - Build qdp-core"
+ @echo " make build_nvtx_profile - Build qdp-core with observability features"
+ @echo " make run_nvtx_profile - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+ @echo " make unit_test - Run all unit tests (Python + Rust)"
+ @echo " make unit_test_python - Run Python unit tests only"
+ @echo " make unit_test_rust - Run Rust unit tests only"
+ @echo " make e2e_test - Run all e2e tests under qdp-python/benchmark"
+ @echo " make clean - Clean build artifacts"
+
+install:
+ @echo "Installing mahout python package..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop
+
+build:
+ @echo "Building qdp-core..."
+ cargo build -p qdp-core
+
+build_nvtx_profile:
+ @echo "Building qdp-core with observability features..."
+ cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+ @echo "Running Python unit tests..."
+ cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+ @echo "Running Rust unit tests..."
+ cargo test --workspace
+
+e2e_test:
+ @echo "Running e2e benchmark tests..."
+ cd qdp-python/benchmark && uv run python benchmark_e2e.py
+ cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+ $(eval EXAMPLE ?= nvtx_profile)
+ @echo "Building example '$(EXAMPLE)' with observability features..."
+ cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+ @echo "Running '$(EXAMPLE)' with nsys profiling..."
+ nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+ @echo "Showing profiling statistics..."
+ nsys stats --force-export=true report.nsys-rep
Review Comment:
(optional) In this, we can only profile /example? Is this a big limitation?
I think it's also unnessary for make, but not sure if others think otherwise.
cc @rich7420 I'm not familiar with this. I never use /example to profile XD. I almost profile benchmarks everytime directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645499351
##########
qdp/DEVELOPMENT.md:
##########
@@ -71,22 +97,36 @@ uv sync --group dev
uv run maturin develop
```
+Alternatively, you can directly run the following command from the `qdp/` directory:
+
+```bash
+make install
+```
+
## Test
There are two types of tests in mahout qdp: unit tests and e2e tests (benchmark tests).
### Unit Tests
-You can simply follow the instructions in [test docs](./docs/test/README.md) to run unit tests.
+You can use the following make commands from the `qdp/` directory:
+
+```bash
+make unit_test # Run all unit tests (Python + Rust)
+make unit_test_python # Run Python tests only
+make unit_test_rust # Run Rust tests only
Review Comment:
nits
```suggestion
make test # Run all unit tests (Python + Rust)
make test_python # Run Python tests only
make test_rust # Run Rust tests only
```
##########
qdp/DEVELOPMENT.md:
##########
@@ -105,6 +145,12 @@ python benchmark_dataloader_throughput.py
python benchmark_e2e.py
```
+Or use the make command from the `qdp/` directory:
+
+```bash
+make e2e_test
Review Comment:
Our benchmarks are not stable right now, we might use different benchmark command. so I don't think we need this in make.
eg. tryout different qubits`python e2e.py --qubits 100`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3689702575
Thanks @machichima 's contribution! I tested locally and these commands works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "machichima (via GitHub)" <gi...@apache.org>.
machichima commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3689772251
Thanks @ryankert01 for review! I updated based on your review:
1. unit_test_... -> test_...
2. make e2e_test -> make benchmark
3. Add `make install_profile`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645566518
##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+ @echo "Available targets:"
+ @echo " make install - Install the mahout python package"
+ @echo " make build - Build qdp-core"
+ @echo " make build_nvtx_profile - Build qdp-core with observability features"
+ @echo " make run_nvtx_profile - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+ @echo " make unit_test - Run all unit tests (Python + Rust)"
+ @echo " make unit_test_python - Run Python unit tests only"
+ @echo " make unit_test_rust - Run Rust unit tests only"
+ @echo " make e2e_test - Run all e2e tests under qdp-python/benchmark"
+ @echo " make clean - Clean build artifacts"
+
+install:
+ @echo "Installing mahout python package..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop
+
+build:
+ @echo "Building qdp-core..."
+ cargo build -p qdp-core
+
+build_nvtx_profile:
+ @echo "Building qdp-core with observability features..."
+ cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+ @echo "Running Python unit tests..."
+ cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+ @echo "Running Rust unit tests..."
+ cargo test --workspace
+
+e2e_test:
+ @echo "Running e2e benchmark tests..."
+ cd qdp-python/benchmark && uv run python benchmark_e2e.py
+ cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
Review Comment:
```suggestion
benchmark: install_benchmark
@echo "Running e2e benchmark tests..."
uv run python qdp-python/benchmark/benchmark_e2e.py
uv run python qdp-python/benchmark/benchmark_dataloader_throughput.py
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP][Housekeeping] Clean structure and make commands [mahout]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3691379022
thanks for the update!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645605210
##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+ @echo "Available targets:"
+ @echo " make install - Install the mahout python package"
+ @echo " make build - Build qdp-core"
+ @echo " make build_nvtx_profile - Build qdp-core with observability features"
+ @echo " make run_nvtx_profile - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+ @echo " make unit_test - Run all unit tests (Python + Rust)"
+ @echo " make unit_test_python - Run Python unit tests only"
+ @echo " make unit_test_rust - Run Rust unit tests only"
+ @echo " make e2e_test - Run all e2e tests under qdp-python/benchmark"
+ @echo " make clean - Clean build artifacts"
+
+install:
+ @echo "Installing mahout python package..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop
+
+build:
+ @echo "Building qdp-core..."
+ cargo build -p qdp-core
+
+build_nvtx_profile:
+ @echo "Building qdp-core with observability features..."
+ cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+ @echo "Running Python unit tests..."
+ cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+ @echo "Running Rust unit tests..."
+ cargo test --workspace
+
+e2e_test:
+ @echo "Running e2e benchmark tests..."
+ cd qdp-python/benchmark && uv run python benchmark_e2e.py
+ cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+ $(eval EXAMPLE ?= nvtx_profile)
+ @echo "Building example '$(EXAMPLE)' with observability features..."
+ cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+ @echo "Running '$(EXAMPLE)' with nsys profiling..."
+ nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+ @echo "Showing profiling statistics..."
+ nsys stats --force-export=true report.nsys-rep
Review Comment:
I use something like this.
```
nsys profile python some_benchmark.py
```
Could you implement a make install or something that makes it profile-able?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP][Housekeeping] Clean structure and make commands [mahout]
Posted by "machichima (via GitHub)" <gi...@apache.org>.
machichima commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3691338744
> @machichima plz resolve the conflict.
Done! Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645526466
##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+ @echo "Available targets:"
+ @echo " make install - Install the mahout python package"
+ @echo " make build - Build qdp-core"
+ @echo " make build_nvtx_profile - Build qdp-core with observability features"
+ @echo " make run_nvtx_profile - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+ @echo " make unit_test - Run all unit tests (Python + Rust)"
+ @echo " make unit_test_python - Run Python unit tests only"
+ @echo " make unit_test_rust - Run Rust unit tests only"
+ @echo " make e2e_test - Run all e2e tests under qdp-python/benchmark"
+ @echo " make clean - Clean build artifacts"
+
+install:
+ @echo "Installing mahout python package..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop
+
+build:
+ @echo "Building qdp-core..."
+ cargo build -p qdp-core
+
+build_nvtx_profile:
+ @echo "Building qdp-core with observability features..."
+ cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+ @echo "Running Python unit tests..."
+ cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+ @echo "Running Rust unit tests..."
+ cargo test --workspace
+
+e2e_test:
+ @echo "Running e2e benchmark tests..."
+ cd qdp-python/benchmark && uv run python benchmark_e2e.py
+ cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+ $(eval EXAMPLE ?= nvtx_profile)
+ @echo "Building example '$(EXAMPLE)' with observability features..."
+ cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+ @echo "Running '$(EXAMPLE)' with nsys profiling..."
+ nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+ @echo "Showing profiling statistics..."
+ nsys stats --force-export=true report.nsys-rep
Review Comment:
(optional) In this, we can only profile /example? Is this a big limitation?
I think it's also unnessary for make, but not sure if others think otherwise.
cc @rich7420 I'm not familiar with this. I never use /example to profile XD. I profile benchmarks directly everytime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645605210
##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+ @echo "Available targets:"
+ @echo " make install - Install the mahout python package"
+ @echo " make build - Build qdp-core"
+ @echo " make build_nvtx_profile - Build qdp-core with observability features"
+ @echo " make run_nvtx_profile - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+ @echo " make unit_test - Run all unit tests (Python + Rust)"
+ @echo " make unit_test_python - Run Python unit tests only"
+ @echo " make unit_test_rust - Run Rust unit tests only"
+ @echo " make e2e_test - Run all e2e tests under qdp-python/benchmark"
+ @echo " make clean - Clean build artifacts"
+
+install:
+ @echo "Installing mahout python package..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop
+
+build:
+ @echo "Building qdp-core..."
+ cargo build -p qdp-core
+
+build_nvtx_profile:
+ @echo "Building qdp-core with observability features..."
+ cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+ @echo "Running Python unit tests..."
+ cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+ @echo "Running Rust unit tests..."
+ cargo test --workspace
+
+e2e_test:
+ @echo "Running e2e benchmark tests..."
+ cd qdp-python/benchmark && uv run python benchmark_e2e.py
+ cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+ $(eval EXAMPLE ?= nvtx_profile)
+ @echo "Building example '$(EXAMPLE)' with observability features..."
+ cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+ @echo "Running '$(EXAMPLE)' with nsys profiling..."
+ nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+ @echo "Showing profiling statistics..."
+ nsys stats --force-export=true report.nsys-rep
Review Comment:
I use something like this.
```
nsys profile python some_benchmark.py
```
Could you implement a make install or something that makes it profile-able?
It should be build like ` cargo build -p qdp-core --example $(EXAMPLE) --features observability --release` as in L46 to make profile-able.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645566518
##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+ @echo "Available targets:"
+ @echo " make install - Install the mahout python package"
+ @echo " make build - Build qdp-core"
+ @echo " make build_nvtx_profile - Build qdp-core with observability features"
+ @echo " make run_nvtx_profile - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+ @echo " make unit_test - Run all unit tests (Python + Rust)"
+ @echo " make unit_test_python - Run Python unit tests only"
+ @echo " make unit_test_rust - Run Rust unit tests only"
+ @echo " make e2e_test - Run all e2e tests under qdp-python/benchmark"
+ @echo " make clean - Clean build artifacts"
+
+install:
+ @echo "Installing mahout python package..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop
+
+build:
+ @echo "Building qdp-core..."
+ cargo build -p qdp-core
+
+build_nvtx_profile:
+ @echo "Building qdp-core with observability features..."
+ cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+ @echo "Running Python unit tests..."
+ cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+ @echo "Running Rust unit tests..."
+ cargo test --workspace
+
+e2e_test:
+ @echo "Running e2e benchmark tests..."
+ cd qdp-python/benchmark && uv run python benchmark_e2e.py
+ cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
Review Comment:
```suggestion
install_benchmark:
cd qdp/qdp-python && uv sync --group benchmark
benchmark: install_benchmark
@echo "Running e2e benchmark tests..."
uv run python qdp-python/benchmark/benchmark_e2e.py
uv run python qdp-python/benchmark/benchmark_dataloader_throughput.py
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP][Housekeeping] Clean structure and make commands [mahout]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3690125462
@machichima thanks for the patch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "machichima (via GitHub)" <gi...@apache.org>.
machichima commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3689536013
cc @ryankert01
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645566518
##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+ @echo "Available targets:"
+ @echo " make install - Install the mahout python package"
+ @echo " make build - Build qdp-core"
+ @echo " make build_nvtx_profile - Build qdp-core with observability features"
+ @echo " make run_nvtx_profile - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+ @echo " make unit_test - Run all unit tests (Python + Rust)"
+ @echo " make unit_test_python - Run Python unit tests only"
+ @echo " make unit_test_rust - Run Rust unit tests only"
+ @echo " make e2e_test - Run all e2e tests under qdp-python/benchmark"
+ @echo " make clean - Clean build artifacts"
+
+install:
+ @echo "Installing mahout python package..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop
+
+build:
+ @echo "Building qdp-core..."
+ cargo build -p qdp-core
+
+build_nvtx_profile:
+ @echo "Building qdp-core with observability features..."
+ cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+ @echo "Running Python unit tests..."
+ cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+ @echo "Running Rust unit tests..."
+ cargo test --workspace
+
+e2e_test:
+ @echo "Running e2e benchmark tests..."
+ cd qdp-python/benchmark && uv run python benchmark_e2e.py
+ cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
Review Comment:
```suggestion
install_benchmark:
uv sync --group benchmark
benchmark: install_benchmark
@echo "Running e2e benchmark tests..."
uv run python qdp-python/benchmark/benchmark_e2e.py
uv run python qdp-python/benchmark/benchmark_dataloader_throughput.py
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645526466
##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+ @echo "Available targets:"
+ @echo " make install - Install the mahout python package"
+ @echo " make build - Build qdp-core"
+ @echo " make build_nvtx_profile - Build qdp-core with observability features"
+ @echo " make run_nvtx_profile - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+ @echo " make unit_test - Run all unit tests (Python + Rust)"
+ @echo " make unit_test_python - Run Python unit tests only"
+ @echo " make unit_test_rust - Run Rust unit tests only"
+ @echo " make e2e_test - Run all e2e tests under qdp-python/benchmark"
+ @echo " make clean - Clean build artifacts"
+
+install:
+ @echo "Installing mahout python package..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop
+
+build:
+ @echo "Building qdp-core..."
+ cargo build -p qdp-core
+
+build_nvtx_profile:
+ @echo "Building qdp-core with observability features..."
+ cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+ @echo "Running Python unit tests..."
+ cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+ @echo "Running Rust unit tests..."
+ cargo test --workspace
+
+e2e_test:
+ @echo "Running e2e benchmark tests..."
+ cd qdp-python/benchmark && uv run python benchmark_e2e.py
+ cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+ $(eval EXAMPLE ?= nvtx_profile)
+ @echo "Building example '$(EXAMPLE)' with observability features..."
+ cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+ @echo "Running '$(EXAMPLE)' with nsys profiling..."
+ nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+ @echo "Showing profiling statistics..."
+ nsys stats --force-export=true report.nsys-rep
Review Comment:
(optional) In this, we can only profile /example? Is this a big limitation?
I think it's also unnessary for make, but not sure if others think otherwise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP][Housekeeping] Clean structure and make commands [mahout]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3690714017
@machichima plz resolve the conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP][Housekeeping] Clean structure and make commands [mahout]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming merged PR #756:
URL: https://github.com/apache/mahout/pull/756
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP][Housekeeping] Clean structure and make commands [mahout]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645956521
##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+ @echo "Available targets:"
+ @echo " make install - Install the mahout python package"
+ @echo " make build - Build qdp-core"
+ @echo " make build_nvtx_profile - Build qdp-core with observability features"
+ @echo " make run_nvtx_profile - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+ @echo " make unit_test - Run all unit tests (Python + Rust)"
+ @echo " make unit_test_python - Run Python unit tests only"
+ @echo " make unit_test_rust - Run Rust unit tests only"
+ @echo " make e2e_test - Run all e2e tests under qdp-python/benchmark"
+ @echo " make clean - Clean build artifacts"
+
+install:
+ @echo "Installing mahout python package..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop
+
+build:
+ @echo "Building qdp-core..."
+ cargo build -p qdp-core
+
+build_nvtx_profile:
+ @echo "Building qdp-core with observability features..."
+ cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+ @echo "Running Python unit tests..."
+ cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+ @echo "Running Rust unit tests..."
+ cargo test --workspace
+
+e2e_test:
+ @echo "Running e2e benchmark tests..."
+ cd qdp-python/benchmark && uv run python benchmark_e2e.py
+ cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+ $(eval EXAMPLE ?= nvtx_profile)
+ @echo "Building example '$(EXAMPLE)' with observability features..."
+ cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+ @echo "Running '$(EXAMPLE)' with nsys profiling..."
+ nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+ @echo "Showing profiling statistics..."
+ nsys stats --force-export=true report.nsys-rep
Review Comment:
@ryankert01
Actually I did profile example before you told me this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP][Housekeeping] Clean structure and make commands [mahout]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3691225955
Nice one!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645605210
##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+ @echo "Available targets:"
+ @echo " make install - Install the mahout python package"
+ @echo " make build - Build qdp-core"
+ @echo " make build_nvtx_profile - Build qdp-core with observability features"
+ @echo " make run_nvtx_profile - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+ @echo " make unit_test - Run all unit tests (Python + Rust)"
+ @echo " make unit_test_python - Run Python unit tests only"
+ @echo " make unit_test_rust - Run Rust unit tests only"
+ @echo " make e2e_test - Run all e2e tests under qdp-python/benchmark"
+ @echo " make clean - Clean build artifacts"
+
+install:
+ @echo "Installing mahout python package..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop
+
+build:
+ @echo "Building qdp-core..."
+ cargo build -p qdp-core
+
+build_nvtx_profile:
+ @echo "Building qdp-core with observability features..."
+ cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+ @echo "Running Python unit tests..."
+ cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+ @echo "Running Rust unit tests..."
+ cargo test --workspace
+
+e2e_test:
+ @echo "Running e2e benchmark tests..."
+ cd qdp-python/benchmark && uv run python benchmark_e2e.py
+ cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+ $(eval EXAMPLE ?= nvtx_profile)
+ @echo "Building example '$(EXAMPLE)' with observability features..."
+ cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+ @echo "Running '$(EXAMPLE)' with nsys profiling..."
+ nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+ @echo "Showing profiling statistics..."
+ nsys stats --force-export=true report.nsys-rep
Review Comment:
I use something like this, and it's because I also want to profile every parts of the code.
```
nsys profile python some_benchmark.py
```
Could you implement a make install or something that makes it profile-able?
It should be build like ` cargo build -p qdp-core --example $(EXAMPLE) --features observability --release` as in L46 to make profile-able.
##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+ @echo "Available targets:"
+ @echo " make install - Install the mahout python package"
+ @echo " make build - Build qdp-core"
+ @echo " make build_nvtx_profile - Build qdp-core with observability features"
+ @echo " make run_nvtx_profile - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+ @echo " make unit_test - Run all unit tests (Python + Rust)"
+ @echo " make unit_test_python - Run Python unit tests only"
+ @echo " make unit_test_rust - Run Rust unit tests only"
+ @echo " make e2e_test - Run all e2e tests under qdp-python/benchmark"
+ @echo " make clean - Clean build artifacts"
+
+install:
+ @echo "Installing mahout python package..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop
+
+build:
+ @echo "Building qdp-core..."
+ cargo build -p qdp-core
+
+build_nvtx_profile:
+ @echo "Building qdp-core with observability features..."
+ cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+ @echo "Running Python unit tests..."
+ cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+ @echo "Running Rust unit tests..."
+ cargo test --workspace
+
+e2e_test:
+ @echo "Running e2e benchmark tests..."
+ cd qdp-python/benchmark && uv run python benchmark_e2e.py
+ cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+ $(eval EXAMPLE ?= nvtx_profile)
+ @echo "Building example '$(EXAMPLE)' with observability features..."
+ cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+ @echo "Running '$(EXAMPLE)' with nsys profiling..."
+ nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+ @echo "Showing profiling statistics..."
+ nsys stats --force-export=true report.nsys-rep
Review Comment:
I use something like this, and it's because I also want to profile every parts of the code.
```
nsys profile python some_benchmark.py
```
Could you implement a make install or something that makes it profile-able?
It should be build like `cargo build -p qdp-core --example $(EXAMPLE) --features observability --release` as in L46 to make profile-able.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Housekeeping] Clean structure and make commands [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645566518
##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+ @echo "Available targets:"
+ @echo " make install - Install the mahout python package"
+ @echo " make build - Build qdp-core"
+ @echo " make build_nvtx_profile - Build qdp-core with observability features"
+ @echo " make run_nvtx_profile - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+ @echo " make unit_test - Run all unit tests (Python + Rust)"
+ @echo " make unit_test_python - Run Python unit tests only"
+ @echo " make unit_test_rust - Run Rust unit tests only"
+ @echo " make e2e_test - Run all e2e tests under qdp-python/benchmark"
+ @echo " make clean - Clean build artifacts"
+
+install:
+ @echo "Installing mahout python package..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop
+
+build:
+ @echo "Building qdp-core..."
+ cargo build -p qdp-core
+
+build_nvtx_profile:
+ @echo "Building qdp-core with observability features..."
+ cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+ @echo "Running Python unit tests..."
+ cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+ @echo "Running Rust unit tests..."
+ cargo test --workspace
+
+e2e_test:
+ @echo "Running e2e benchmark tests..."
+ cd qdp-python/benchmark && uv run python benchmark_e2e.py
+ cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
Review Comment:
```suggestion
install_benchmark:
cd qdp/qdp-python && uv sync --group benchmark
benchmark: install install_benchmark
@echo "Running e2e benchmark tests..."
uv run python qdp-python/benchmark/benchmark_e2e.py
uv run python qdp-python/benchmark/benchmark_dataloader_throughput.py
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org