You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "machichima (via GitHub)" <gi...@apache.org> on 2025/12/24 11:24:43 UTC

[PR] [Housekeeping] Clean structure and make commands [mahout]

machichima opened a new pull request, #756:
URL: https://github.com/apache/mahout/pull/756

   ### Purpose of PR
   
   Improve development experience by introducing make commands to install, run tests, and do profiling. 
   
   Also move `benchmark/` into `qdp-python` and update `pyproject.toml` to install dependencies needed with different `--group` args
   
   ### Related Issues or PRs
   
   Closes #735
   
   ### Changes Made
   <!-- Please mark one with an "x"   -->
   - [ ] Bug fix
   - [ ] New feature
   - [x] Refactoring
   - [x] Documentation
   - [ ] Test
   - [ ] CI/CD pipeline
   - [ ] Other
   
   ### Breaking Changes
   <!-- Does this PR introduce a breaking change? -->
   - [ ] Yes
   - [ ] No
   
   ### Checklist
   <!-- Please mark each item with an "x" when complete -->
   <!-- If not all items are complete, please open this as a **Draft PR**.
   Once all requirements are met, mark as ready for review. -->
   
   - [ ] Added or updated unit tests for all changes
   - [ ] Added or updated documentation for all changes
   - [ ] Successfully built and ran all unit tests or manual tests locally
   - [ ] PR title follows "MAHOUT-XXX: Brief Description" format (if related to an issue)
   - [ ] Code follows ASF guidelines
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "machichima (via GitHub)" <gi...@apache.org>.
machichima commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645577023


##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+	@echo "Available targets:"
+	@echo "  make install             - Install the mahout python package"
+	@echo "  make build               - Build qdp-core"
+	@echo "  make build_nvtx_profile  - Build qdp-core with observability features"
+	@echo "  make run_nvtx_profile    - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+	@echo "  make unit_test           - Run all unit tests (Python + Rust)"
+	@echo "  make unit_test_python    - Run Python unit tests only"
+	@echo "  make unit_test_rust      - Run Rust unit tests only"
+	@echo "  make e2e_test            - Run all e2e tests under qdp-python/benchmark"
+	@echo "  make clean               - Clean build artifacts"
+
+install:
+	@echo "Installing mahout python package..."
+	cd qdp-python && uv sync --group dev
+	cd qdp-python && uv run maturin develop
+
+build:
+	@echo "Building qdp-core..."
+	cargo build -p qdp-core
+
+build_nvtx_profile:
+	@echo "Building qdp-core with observability features..."
+	cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+	@echo "Running Python unit tests..."
+	cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+	@echo "Running Rust unit tests..."
+	cargo test --workspace
+
+e2e_test:
+	@echo "Running e2e benchmark tests..."
+	cd qdp-python/benchmark && uv run python benchmark_e2e.py
+	cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+	$(eval EXAMPLE ?= nvtx_profile)
+	@echo "Building example '$(EXAMPLE)' with observability features..."
+	cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+	@echo "Running '$(EXAMPLE)' with nsys profiling..."
+	nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+	@echo "Showing profiling statistics..."
+	nsys stats --force-export=true report.nsys-rep

Review Comment:
   Could you please show me how to profile benchmarks directly? The documentation we had profiling the example, that's why I only put example here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645526466


##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+	@echo "Available targets:"
+	@echo "  make install             - Install the mahout python package"
+	@echo "  make build               - Build qdp-core"
+	@echo "  make build_nvtx_profile  - Build qdp-core with observability features"
+	@echo "  make run_nvtx_profile    - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+	@echo "  make unit_test           - Run all unit tests (Python + Rust)"
+	@echo "  make unit_test_python    - Run Python unit tests only"
+	@echo "  make unit_test_rust      - Run Rust unit tests only"
+	@echo "  make e2e_test            - Run all e2e tests under qdp-python/benchmark"
+	@echo "  make clean               - Clean build artifacts"
+
+install:
+	@echo "Installing mahout python package..."
+	cd qdp-python && uv sync --group dev
+	cd qdp-python && uv run maturin develop
+
+build:
+	@echo "Building qdp-core..."
+	cargo build -p qdp-core
+
+build_nvtx_profile:
+	@echo "Building qdp-core with observability features..."
+	cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+	@echo "Running Python unit tests..."
+	cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+	@echo "Running Rust unit tests..."
+	cargo test --workspace
+
+e2e_test:
+	@echo "Running e2e benchmark tests..."
+	cd qdp-python/benchmark && uv run python benchmark_e2e.py
+	cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+	$(eval EXAMPLE ?= nvtx_profile)
+	@echo "Building example '$(EXAMPLE)' with observability features..."
+	cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+	@echo "Running '$(EXAMPLE)' with nsys profiling..."
+	nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+	@echo "Showing profiling statistics..."
+	nsys stats --force-export=true report.nsys-rep

Review Comment:
   (optional) In this, we can only profile /example? Is this a big limitation?
   I think it's also unnessary for make, but not sure if others think otherwise.
   cc @rich7420 I'm not familiar with this. I never use /example to profile XD. I almost profile benchmarks everytime directly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645499351


##########
qdp/DEVELOPMENT.md:
##########
@@ -71,22 +97,36 @@ uv sync --group dev
 uv run maturin develop
 ```
 
+Alternatively, you can directly run the following command from the `qdp/` directory:
+
+```bash
+make install
+```
+
 ## Test
 
 There are two types of tests in mahout qdp: unit tests and e2e tests (benchmark tests).
 
 ### Unit Tests
 
-You can simply follow the instructions in [test docs](./docs/test/README.md) to run unit tests.
+You can use the following make commands from the `qdp/` directory:
+
+```bash
+make unit_test        # Run all unit tests (Python + Rust)
+make unit_test_python # Run Python tests only
+make unit_test_rust   # Run Rust tests only

Review Comment:
   nits
   ```suggestion
   make test        # Run all unit tests (Python + Rust)
   make test_python # Run Python tests only
   make test_rust   # Run Rust tests only
   ```



##########
qdp/DEVELOPMENT.md:
##########
@@ -105,6 +145,12 @@ python benchmark_dataloader_throughput.py
 python benchmark_e2e.py
 ```
 
+Or use the make command from the `qdp/` directory:
+
+```bash
+make e2e_test

Review Comment:
   Our benchmarks are not stable right now, we might use different benchmark command. so I don't think we need this in make.
   eg. tryout different qubits`python e2e.py --qubits 100`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3689702575

   Thanks @machichima 's contribution! I tested locally and these commands works.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "machichima (via GitHub)" <gi...@apache.org>.
machichima commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3689772251

   Thanks @ryankert01 for review! I updated based on your review:
   1. unit_test_... -> test_...
   2. make e2e_test -> make benchmark
   3. Add `make install_profile`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645566518


##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+	@echo "Available targets:"
+	@echo "  make install             - Install the mahout python package"
+	@echo "  make build               - Build qdp-core"
+	@echo "  make build_nvtx_profile  - Build qdp-core with observability features"
+	@echo "  make run_nvtx_profile    - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+	@echo "  make unit_test           - Run all unit tests (Python + Rust)"
+	@echo "  make unit_test_python    - Run Python unit tests only"
+	@echo "  make unit_test_rust      - Run Rust unit tests only"
+	@echo "  make e2e_test            - Run all e2e tests under qdp-python/benchmark"
+	@echo "  make clean               - Clean build artifacts"
+
+install:
+	@echo "Installing mahout python package..."
+	cd qdp-python && uv sync --group dev
+	cd qdp-python && uv run maturin develop
+
+build:
+	@echo "Building qdp-core..."
+	cargo build -p qdp-core
+
+build_nvtx_profile:
+	@echo "Building qdp-core with observability features..."
+	cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+	@echo "Running Python unit tests..."
+	cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+	@echo "Running Rust unit tests..."
+	cargo test --workspace
+
+e2e_test:
+	@echo "Running e2e benchmark tests..."
+	cd qdp-python/benchmark && uv run python benchmark_e2e.py
+	cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py

Review Comment:
   ```suggestion
   benchmark: install_benchmark
   	@echo "Running e2e benchmark tests..."
   	uv run python qdp-python/benchmark/benchmark_e2e.py
   	uv run python qdp-python/benchmark/benchmark_dataloader_throughput.py
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [QDP][Housekeeping] Clean structure and make commands [mahout]

Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3691379022

   thanks for the update!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645605210


##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+	@echo "Available targets:"
+	@echo "  make install             - Install the mahout python package"
+	@echo "  make build               - Build qdp-core"
+	@echo "  make build_nvtx_profile  - Build qdp-core with observability features"
+	@echo "  make run_nvtx_profile    - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+	@echo "  make unit_test           - Run all unit tests (Python + Rust)"
+	@echo "  make unit_test_python    - Run Python unit tests only"
+	@echo "  make unit_test_rust      - Run Rust unit tests only"
+	@echo "  make e2e_test            - Run all e2e tests under qdp-python/benchmark"
+	@echo "  make clean               - Clean build artifacts"
+
+install:
+	@echo "Installing mahout python package..."
+	cd qdp-python && uv sync --group dev
+	cd qdp-python && uv run maturin develop
+
+build:
+	@echo "Building qdp-core..."
+	cargo build -p qdp-core
+
+build_nvtx_profile:
+	@echo "Building qdp-core with observability features..."
+	cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+	@echo "Running Python unit tests..."
+	cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+	@echo "Running Rust unit tests..."
+	cargo test --workspace
+
+e2e_test:
+	@echo "Running e2e benchmark tests..."
+	cd qdp-python/benchmark && uv run python benchmark_e2e.py
+	cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+	$(eval EXAMPLE ?= nvtx_profile)
+	@echo "Building example '$(EXAMPLE)' with observability features..."
+	cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+	@echo "Running '$(EXAMPLE)' with nsys profiling..."
+	nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+	@echo "Showing profiling statistics..."
+	nsys stats --force-export=true report.nsys-rep

Review Comment:
   I use something like this. 
   ```
   nsys profile python some_benchmark.py
   ```
   Could you implement a make install or something that makes it profile-able?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [QDP][Housekeeping] Clean structure and make commands [mahout]

Posted by "machichima (via GitHub)" <gi...@apache.org>.
machichima commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3691338744

   > @machichima plz resolve the conflict.
   Done! Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645526466


##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+	@echo "Available targets:"
+	@echo "  make install             - Install the mahout python package"
+	@echo "  make build               - Build qdp-core"
+	@echo "  make build_nvtx_profile  - Build qdp-core with observability features"
+	@echo "  make run_nvtx_profile    - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+	@echo "  make unit_test           - Run all unit tests (Python + Rust)"
+	@echo "  make unit_test_python    - Run Python unit tests only"
+	@echo "  make unit_test_rust      - Run Rust unit tests only"
+	@echo "  make e2e_test            - Run all e2e tests under qdp-python/benchmark"
+	@echo "  make clean               - Clean build artifacts"
+
+install:
+	@echo "Installing mahout python package..."
+	cd qdp-python && uv sync --group dev
+	cd qdp-python && uv run maturin develop
+
+build:
+	@echo "Building qdp-core..."
+	cargo build -p qdp-core
+
+build_nvtx_profile:
+	@echo "Building qdp-core with observability features..."
+	cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+	@echo "Running Python unit tests..."
+	cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+	@echo "Running Rust unit tests..."
+	cargo test --workspace
+
+e2e_test:
+	@echo "Running e2e benchmark tests..."
+	cd qdp-python/benchmark && uv run python benchmark_e2e.py
+	cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+	$(eval EXAMPLE ?= nvtx_profile)
+	@echo "Building example '$(EXAMPLE)' with observability features..."
+	cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+	@echo "Running '$(EXAMPLE)' with nsys profiling..."
+	nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+	@echo "Showing profiling statistics..."
+	nsys stats --force-export=true report.nsys-rep

Review Comment:
   (optional) In this, we can only profile /example? Is this a big limitation?
   I think it's also unnessary for make, but not sure if others think otherwise.
   cc @rich7420 I'm not familiar with this. I never use /example to profile XD. I profile benchmarks directly everytime.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645605210


##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+	@echo "Available targets:"
+	@echo "  make install             - Install the mahout python package"
+	@echo "  make build               - Build qdp-core"
+	@echo "  make build_nvtx_profile  - Build qdp-core with observability features"
+	@echo "  make run_nvtx_profile    - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+	@echo "  make unit_test           - Run all unit tests (Python + Rust)"
+	@echo "  make unit_test_python    - Run Python unit tests only"
+	@echo "  make unit_test_rust      - Run Rust unit tests only"
+	@echo "  make e2e_test            - Run all e2e tests under qdp-python/benchmark"
+	@echo "  make clean               - Clean build artifacts"
+
+install:
+	@echo "Installing mahout python package..."
+	cd qdp-python && uv sync --group dev
+	cd qdp-python && uv run maturin develop
+
+build:
+	@echo "Building qdp-core..."
+	cargo build -p qdp-core
+
+build_nvtx_profile:
+	@echo "Building qdp-core with observability features..."
+	cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+	@echo "Running Python unit tests..."
+	cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+	@echo "Running Rust unit tests..."
+	cargo test --workspace
+
+e2e_test:
+	@echo "Running e2e benchmark tests..."
+	cd qdp-python/benchmark && uv run python benchmark_e2e.py
+	cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+	$(eval EXAMPLE ?= nvtx_profile)
+	@echo "Building example '$(EXAMPLE)' with observability features..."
+	cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+	@echo "Running '$(EXAMPLE)' with nsys profiling..."
+	nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+	@echo "Showing profiling statistics..."
+	nsys stats --force-export=true report.nsys-rep

Review Comment:
   I use something like this. 
   ```
   nsys profile python some_benchmark.py
   ```
   Could you implement a make install or something that makes it profile-able?
   It should be build like `	cargo build -p qdp-core --example $(EXAMPLE) --features observability --release` as in L46 to make profile-able.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645566518


##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+	@echo "Available targets:"
+	@echo "  make install             - Install the mahout python package"
+	@echo "  make build               - Build qdp-core"
+	@echo "  make build_nvtx_profile  - Build qdp-core with observability features"
+	@echo "  make run_nvtx_profile    - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+	@echo "  make unit_test           - Run all unit tests (Python + Rust)"
+	@echo "  make unit_test_python    - Run Python unit tests only"
+	@echo "  make unit_test_rust      - Run Rust unit tests only"
+	@echo "  make e2e_test            - Run all e2e tests under qdp-python/benchmark"
+	@echo "  make clean               - Clean build artifacts"
+
+install:
+	@echo "Installing mahout python package..."
+	cd qdp-python && uv sync --group dev
+	cd qdp-python && uv run maturin develop
+
+build:
+	@echo "Building qdp-core..."
+	cargo build -p qdp-core
+
+build_nvtx_profile:
+	@echo "Building qdp-core with observability features..."
+	cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+	@echo "Running Python unit tests..."
+	cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+	@echo "Running Rust unit tests..."
+	cargo test --workspace
+
+e2e_test:
+	@echo "Running e2e benchmark tests..."
+	cd qdp-python/benchmark && uv run python benchmark_e2e.py
+	cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py

Review Comment:
   ```suggestion
   install_benchmark:
   	cd qdp/qdp-python && uv sync --group benchmark
   benchmark: install_benchmark
   	@echo "Running e2e benchmark tests..."
   	uv run python qdp-python/benchmark/benchmark_e2e.py
   	uv run python qdp-python/benchmark/benchmark_dataloader_throughput.py
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [QDP][Housekeeping] Clean structure and make commands [mahout]

Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3690125462

   @machichima thanks for the patch


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "machichima (via GitHub)" <gi...@apache.org>.
machichima commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3689536013

   cc @ryankert01


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645566518


##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+	@echo "Available targets:"
+	@echo "  make install             - Install the mahout python package"
+	@echo "  make build               - Build qdp-core"
+	@echo "  make build_nvtx_profile  - Build qdp-core with observability features"
+	@echo "  make run_nvtx_profile    - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+	@echo "  make unit_test           - Run all unit tests (Python + Rust)"
+	@echo "  make unit_test_python    - Run Python unit tests only"
+	@echo "  make unit_test_rust      - Run Rust unit tests only"
+	@echo "  make e2e_test            - Run all e2e tests under qdp-python/benchmark"
+	@echo "  make clean               - Clean build artifacts"
+
+install:
+	@echo "Installing mahout python package..."
+	cd qdp-python && uv sync --group dev
+	cd qdp-python && uv run maturin develop
+
+build:
+	@echo "Building qdp-core..."
+	cargo build -p qdp-core
+
+build_nvtx_profile:
+	@echo "Building qdp-core with observability features..."
+	cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+	@echo "Running Python unit tests..."
+	cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+	@echo "Running Rust unit tests..."
+	cargo test --workspace
+
+e2e_test:
+	@echo "Running e2e benchmark tests..."
+	cd qdp-python/benchmark && uv run python benchmark_e2e.py
+	cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py

Review Comment:
   ```suggestion
   install_benchmark:
   	uv sync --group benchmark
   benchmark: install_benchmark
   	@echo "Running e2e benchmark tests..."
   	uv run python qdp-python/benchmark/benchmark_e2e.py
   	uv run python qdp-python/benchmark/benchmark_dataloader_throughput.py
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645526466


##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+	@echo "Available targets:"
+	@echo "  make install             - Install the mahout python package"
+	@echo "  make build               - Build qdp-core"
+	@echo "  make build_nvtx_profile  - Build qdp-core with observability features"
+	@echo "  make run_nvtx_profile    - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+	@echo "  make unit_test           - Run all unit tests (Python + Rust)"
+	@echo "  make unit_test_python    - Run Python unit tests only"
+	@echo "  make unit_test_rust      - Run Rust unit tests only"
+	@echo "  make e2e_test            - Run all e2e tests under qdp-python/benchmark"
+	@echo "  make clean               - Clean build artifacts"
+
+install:
+	@echo "Installing mahout python package..."
+	cd qdp-python && uv sync --group dev
+	cd qdp-python && uv run maturin develop
+
+build:
+	@echo "Building qdp-core..."
+	cargo build -p qdp-core
+
+build_nvtx_profile:
+	@echo "Building qdp-core with observability features..."
+	cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+	@echo "Running Python unit tests..."
+	cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+	@echo "Running Rust unit tests..."
+	cargo test --workspace
+
+e2e_test:
+	@echo "Running e2e benchmark tests..."
+	cd qdp-python/benchmark && uv run python benchmark_e2e.py
+	cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+	$(eval EXAMPLE ?= nvtx_profile)
+	@echo "Building example '$(EXAMPLE)' with observability features..."
+	cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+	@echo "Running '$(EXAMPLE)' with nsys profiling..."
+	nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+	@echo "Showing profiling statistics..."
+	nsys stats --force-export=true report.nsys-rep

Review Comment:
   (optional) In this, we can only profile /example? Is this a big limitation?
   I think it's also unnessary for make, but not sure if others think otherwise.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [QDP][Housekeeping] Clean structure and make commands [mahout]

Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3690714017

   @machichima plz resolve the conflict.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [QDP][Housekeeping] Clean structure and make commands [mahout]

Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming merged PR #756:
URL: https://github.com/apache/mahout/pull/756


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [QDP][Housekeeping] Clean structure and make commands [mahout]

Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645956521


##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+	@echo "Available targets:"
+	@echo "  make install             - Install the mahout python package"
+	@echo "  make build               - Build qdp-core"
+	@echo "  make build_nvtx_profile  - Build qdp-core with observability features"
+	@echo "  make run_nvtx_profile    - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+	@echo "  make unit_test           - Run all unit tests (Python + Rust)"
+	@echo "  make unit_test_python    - Run Python unit tests only"
+	@echo "  make unit_test_rust      - Run Rust unit tests only"
+	@echo "  make e2e_test            - Run all e2e tests under qdp-python/benchmark"
+	@echo "  make clean               - Clean build artifacts"
+
+install:
+	@echo "Installing mahout python package..."
+	cd qdp-python && uv sync --group dev
+	cd qdp-python && uv run maturin develop
+
+build:
+	@echo "Building qdp-core..."
+	cargo build -p qdp-core
+
+build_nvtx_profile:
+	@echo "Building qdp-core with observability features..."
+	cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+	@echo "Running Python unit tests..."
+	cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+	@echo "Running Rust unit tests..."
+	cargo test --workspace
+
+e2e_test:
+	@echo "Running e2e benchmark tests..."
+	cd qdp-python/benchmark && uv run python benchmark_e2e.py
+	cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+	$(eval EXAMPLE ?= nvtx_profile)
+	@echo "Building example '$(EXAMPLE)' with observability features..."
+	cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+	@echo "Running '$(EXAMPLE)' with nsys profiling..."
+	nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+	@echo "Showing profiling statistics..."
+	nsys stats --force-export=true report.nsys-rep

Review Comment:
   @ryankert01 
   Actually I did profile example before you told me this way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [QDP][Housekeeping] Clean structure and make commands [mahout]

Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming commented on PR #756:
URL: https://github.com/apache/mahout/pull/756#issuecomment-3691225955

   Nice one!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645605210


##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+	@echo "Available targets:"
+	@echo "  make install             - Install the mahout python package"
+	@echo "  make build               - Build qdp-core"
+	@echo "  make build_nvtx_profile  - Build qdp-core with observability features"
+	@echo "  make run_nvtx_profile    - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+	@echo "  make unit_test           - Run all unit tests (Python + Rust)"
+	@echo "  make unit_test_python    - Run Python unit tests only"
+	@echo "  make unit_test_rust      - Run Rust unit tests only"
+	@echo "  make e2e_test            - Run all e2e tests under qdp-python/benchmark"
+	@echo "  make clean               - Clean build artifacts"
+
+install:
+	@echo "Installing mahout python package..."
+	cd qdp-python && uv sync --group dev
+	cd qdp-python && uv run maturin develop
+
+build:
+	@echo "Building qdp-core..."
+	cargo build -p qdp-core
+
+build_nvtx_profile:
+	@echo "Building qdp-core with observability features..."
+	cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+	@echo "Running Python unit tests..."
+	cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+	@echo "Running Rust unit tests..."
+	cargo test --workspace
+
+e2e_test:
+	@echo "Running e2e benchmark tests..."
+	cd qdp-python/benchmark && uv run python benchmark_e2e.py
+	cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+	$(eval EXAMPLE ?= nvtx_profile)
+	@echo "Building example '$(EXAMPLE)' with observability features..."
+	cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+	@echo "Running '$(EXAMPLE)' with nsys profiling..."
+	nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+	@echo "Showing profiling statistics..."
+	nsys stats --force-export=true report.nsys-rep

Review Comment:
   I use something like this, and it's because I also want to profile every parts of the code.
   ```
   nsys profile python some_benchmark.py
   ```
   Could you implement a make install or something that makes it profile-able?
   It should be build like `	cargo build -p qdp-core --example $(EXAMPLE) --features observability --release` as in L46 to make profile-able.



##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+	@echo "Available targets:"
+	@echo "  make install             - Install the mahout python package"
+	@echo "  make build               - Build qdp-core"
+	@echo "  make build_nvtx_profile  - Build qdp-core with observability features"
+	@echo "  make run_nvtx_profile    - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+	@echo "  make unit_test           - Run all unit tests (Python + Rust)"
+	@echo "  make unit_test_python    - Run Python unit tests only"
+	@echo "  make unit_test_rust      - Run Rust unit tests only"
+	@echo "  make e2e_test            - Run all e2e tests under qdp-python/benchmark"
+	@echo "  make clean               - Clean build artifacts"
+
+install:
+	@echo "Installing mahout python package..."
+	cd qdp-python && uv sync --group dev
+	cd qdp-python && uv run maturin develop
+
+build:
+	@echo "Building qdp-core..."
+	cargo build -p qdp-core
+
+build_nvtx_profile:
+	@echo "Building qdp-core with observability features..."
+	cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+	@echo "Running Python unit tests..."
+	cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+	@echo "Running Rust unit tests..."
+	cargo test --workspace
+
+e2e_test:
+	@echo "Running e2e benchmark tests..."
+	cd qdp-python/benchmark && uv run python benchmark_e2e.py
+	cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+	$(eval EXAMPLE ?= nvtx_profile)
+	@echo "Building example '$(EXAMPLE)' with observability features..."
+	cargo build -p qdp-core --example $(EXAMPLE) --features observability --release
+	@echo "Running '$(EXAMPLE)' with nsys profiling..."
+	nsys profile --trace=cuda,nvtx --force-overwrite=true -o report ./target/release/examples/$(EXAMPLE)
+	@echo "Showing profiling statistics..."
+	nsys stats --force-export=true report.nsys-rep

Review Comment:
   I use something like this, and it's because I also want to profile every parts of the code.
   ```
   nsys profile python some_benchmark.py
   ```
   Could you implement a make install or something that makes it profile-able?
   It should be build like `cargo build -p qdp-core --example $(EXAMPLE) --features observability --release` as in L46 to make profile-able.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Housekeeping] Clean structure and make commands [mahout]

Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #756:
URL: https://github.com/apache/mahout/pull/756#discussion_r2645566518


##########
qdp/Makefile:
##########
@@ -0,0 +1,59 @@
+.PHONY: install build build_nvtx_profile run_nvtx_profile unit_test unit_test_python unit_test_rust e2e_test clean help
+
+help:
+	@echo "Available targets:"
+	@echo "  make install             - Install the mahout python package"
+	@echo "  make build               - Build qdp-core"
+	@echo "  make build_nvtx_profile  - Build qdp-core with observability features"
+	@echo "  make run_nvtx_profile    - Build example, run with nsys, and show stats (EXAMPLE=nvtx_profile)"
+	@echo "  make unit_test           - Run all unit tests (Python + Rust)"
+	@echo "  make unit_test_python    - Run Python unit tests only"
+	@echo "  make unit_test_rust      - Run Rust unit tests only"
+	@echo "  make e2e_test            - Run all e2e tests under qdp-python/benchmark"
+	@echo "  make clean               - Clean build artifacts"
+
+install:
+	@echo "Installing mahout python package..."
+	cd qdp-python && uv sync --group dev
+	cd qdp-python && uv run maturin develop
+
+build:
+	@echo "Building qdp-core..."
+	cargo build -p qdp-core
+
+build_nvtx_profile:
+	@echo "Building qdp-core with observability features..."
+	cargo build -p qdp-core --features observability --release
+
+unit_test: unit_test_python unit_test_rust
+
+unit_test_python:
+	@echo "Running Python unit tests..."
+	cd qdp-python && uv run pytest tests/
+
+unit_test_rust:
+	@echo "Running Rust unit tests..."
+	cargo test --workspace
+
+e2e_test:
+	@echo "Running e2e benchmark tests..."
+	cd qdp-python/benchmark && uv run python benchmark_e2e.py
+	cd qdp-python/benchmark && uv run python benchmark_dataloader_throughput.py

Review Comment:
   ```suggestion
   install_benchmark:
   	cd qdp/qdp-python && uv sync --group benchmark
   benchmark: install install_benchmark
   	@echo "Running e2e benchmark tests..."
   	uv run python qdp-python/benchmark/benchmark_e2e.py
   	uv run python qdp-python/benchmark/benchmark_dataloader_throughput.py
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org