[I] gpu_ptr_encoding test fails during make test_rust but passes 5/5 times in isolation — test-ordering or GPU-state-pollution issue [mahout]

["andrewmusselman (via GitHub)" <gi...@apache.org>]

andrewmusselman opened a new issue, #1332:
URL: https://github.com/apache/mahout/issues/1332

   ### Summary
    
   `test_encode_from_gpu_ptr_f32_with_stream_non_default_success` (in `qdp/qdp-core/tests/gpu_ptr_encoding.rs:795`) fails reproducibly during full `make test_rust` runs, but passes 5/5 times when run in isolation. Suggests a test-isolation issue (GPU state pollution from preceding tests, or a synchronization gap exposed only under specific test ordering) rather than a defect in the f32-non-default-stream code path itself.
    
   ### Failure observed
    
   ```
   thread 'test_encode_from_gpu_ptr_f32_with_stream_non_default_success' panicked at
   qdp-core/tests/gpu_ptr_encoding.rs:795:14:
   encode_from_gpu_ptr_f32_with_stream (non-default stream):
       InvalidInput("Input data (f32) has zero or non-finite norm
                     (contains NaN, Inf, or all zeros)")
   ```
    
   The other 63 tests in `gpu_ptr_encoding` (including `test_encode_from_gpu_ptr_f32_with_stream_success` on the default stream) pass.
    
   ### Reproducer
    
   ```bash
   git checkout mahout-qumat-0.6.0-RC2
   export CXX=g++-13 QDP_CUDA_ARCH_LIST=61+PTX
    
   # Fails consistently as part of the full Rust suite
   make test_rust    # 1 failure: test_encode_from_gpu_ptr_f32_with_stream_non_default_success
    
   # Passes consistently in isolation
   cd qdp
   for i in 1 2 3 4 5; do
     cargo test -p qdp-core --test gpu_ptr_encoding -- \
       test_encode_from_gpu_ptr_f32_with_stream_non_default_success --exact 2>&1 | tail -1
   done
   # test result: ok. 1 passed; 0 failed (×5)
   ```
    
   ### Likely causes (worth investigating in order)
    
   1. **Test ordering / state pollution.** A preceding test in `gpu_ptr_encoding` (or in another test binary linked into the same process if cargo-test reuses one) leaves GPU memory or stream state in a configuration that breaks the non-default-stream input setup. Try running `gpu_ptr_encoding` with `--test-threads=1` and bisecting via `--skip` to find the polluting test.
   2. **Missing synchronization in the test harness.** The test creates an input buffer on a user-provided stream, then calls the encoder which reads the norm — possibly on a different stream — without a `cudaStreamSynchronize` or event ordering between the two. Under high GPU load (full suite), the encoder reads before the buffer is populated.
   3. **Encoder bug with non-default stream.** Less likely given isolation passes, but if (1) and (2) rule out, the encoder may be assuming default-stream semantics in some f32 path.
   ### Environment
    
   - OS: Ubuntu 24.04
   - CUDA Toolkit: 12.4
   - GPU: NVIDIA GeForce GTX 1060 with Max-Q Design (sm_61)
   - Rust: stable, compiled with `CXX=g++-13` (CUDA 12.4 rejects gcc 14)
   - Arch list: `QDP_CUDA_ARCH_LIST=61+PTX`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org