[I] [Feature] Pipeline file load respects PipelineConfig.dtype [mahout]

["0lai0 (via GitHub)" <gi...@apache.org>]

0lai0 opened a new issue, #1341:
URL: https://github.com/apache/mahout/issues/1341

   ### What
   
   Wire `PipelineConfig.dtype` through `read_file_by_extension` and the producer layer so that f32-configured pipelines actually reach the f32 encoder kernel when loading from file sources.
   
   ### Why
   
   `read_file_by_extension` always returns `Vec<f64>` (`pipeline_runner.rs`). `InMemoryProducer` and `StreamingProducer` populate `BatchData::F64` even when `config.dtype == Float32`, so the f32 encoder is never reached from file sources. The user's dtype choice is silently ignored.
   
   ### How
   
   - `read_file_by_extension` (and streaming path) selects the `f32` vs `f64` reader based on `PipelineConfig.dtype`
   - `InMemoryProducer` / `StreamingProducer` use the matching `BatchData` variant
   - Python API (`QuantumDataLoader.source_file`) unchanged at the surface
   - Integration test: f32 Parquet + `dtype=f32` config → `encode_batch_f32_for_pipeline` path is exercised
   
   **Acceptance criteria:**
   - [ ] Integration test: f32 Parquet + `dtype=f32` config → `encode_batch_f32_for_pipeline` path is used
   - [ ] Existing f64 file tests still pass
   - [ ] No silent f64 kernel when user requested f32 file pipeline
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Pipeline file load respects PipelineConfig.dtype [mahout]

["rich7420 (via GitHub)" <gi...@apache.org>]

rich7420 closed issue #1341: [Feature] Pipeline file load respects PipelineConfig.dtype
URL: https://github.com/apache/mahout/issues/1341


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@mahout.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org