You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "400Ping (via GitHub)" <gi...@apache.org> on 2026/01/12 16:48:37 UTC
[PR] [QDP] PyTorch input format support [mahout]
400Ping opened a new pull request, #815:
URL: https://github.com/apache/mahout/pull/815
### Purpose of PR
As title
### Related Issues or PRs
Closes #726 #727 Related #752
### Changes Made
<!-- Please mark one with an "x" -->
- [ ] Bug fix
- [x] New feature
- [x] Refactoring
- [ ] Documentation
- [ ] Test
- [ ] CI/CD pipeline
- [ ] Other
### Breaking Changes
<!-- Does this PR introduce a breaking change? -->
- [ ] Yes
- [x] No
### Checklist
<!-- Please mark each item with an "x" when complete -->
<!-- If not all items are complete, please open this as a **Draft PR**.
Once all requirements are met, mark as ready for review. -->
- [ ] Added or updated unit tests for all changes
- [ ] Added or updated documentation for all changes
- [ ] Successfully built and ran all unit tests or manual tests locally
- [ ] PR title follows "MAHOUT-XXX: Brief Description" format (if related to an issue)
- [ ] Code follows ASF guidelines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on code in PR #815:
URL: https://github.com/apache/mahout/pull/815#discussion_r2689407477
##########
qdp/qdp-core/src/readers/torch.rs:
##########
@@ -0,0 +1,165 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! PyTorch tensor reader implementation.
+//!
+//! Supports `.pt`/`.pth` files containing a single tensor saved with `torch.save`.
+//! The tensor must be 1D or 2D and will be converted to `float64`.
+//! Requires the `pytorch` feature to be enabled.
+
+use std::path::Path;
+
+use crate::error::{MahoutError, Result};
+use crate::reader::DataReader;
+
+/// Reader for PyTorch `.pt`/`.pth` tensor files.
+pub struct TorchReader {
+ path: std::path::PathBuf,
+ read: bool,
+}
+
+impl TorchReader {
+ /// Create a new PyTorch reader.
+ ///
+ /// # Arguments
+ /// * `path` - Path to the `.pt`/`.pth` file
+ pub fn new<P: AsRef<Path>>(path: P) -> Result<Self> {
+ let path = path.as_ref();
+
+ match path.try_exists() {
+ Ok(false) => {
+ return Err(MahoutError::Io(format!(
+ "PyTorch file not found: {}",
+ path.display()
+ )));
+ }
+ Err(e) => {
+ return Err(MahoutError::Io(format!(
+ "Failed to check if PyTorch file exists at {}: {}",
+ path.display(),
+ e
+ )));
+ }
+ Ok(true) => {}
+ }
+
+ Ok(Self {
+ path: path.to_path_buf(),
+ read: false,
+ })
+ }
+}
+
+impl DataReader for TorchReader {
+ fn read_batch(&mut self) -> Result<(Vec<f64>, usize, usize)> {
+ if self.read {
+ return Err(MahoutError::InvalidInput(
+ "Reader already consumed".to_string(),
+ ));
+ }
+ self.read = true;
+
+ #[cfg(feature = "pytorch")]
+ {
+ return read_torch_tensor(&self.path);
+ }
+
+ #[cfg(not(feature = "pytorch"))]
+ {
+ return Err(MahoutError::NotImplemented(
+ "PyTorch reader requires the 'pytorch' feature".to_string(),
+ ));
+ }
+ }
+
+ fn get_sample_size(&self) -> Option<usize> {
+ None
+ }
+
+ fn get_num_samples(&self) -> Option<usize> {
+ None
+ }
+}
+
+#[cfg(feature = "pytorch")]
+fn read_torch_tensor(path: &Path) -> Result<(Vec<f64>, usize, usize)> {
+ use tch::{Device, Kind, Tensor};
+
+ let tensor = Tensor::load(path).map_err(|e| {
+ MahoutError::Io(format!(
+ "Failed to load PyTorch tensor from {}: {}",
+ path.display(),
+ e
+ ))
+ })?;
+
+ let tensor = tensor.to_device(Device::Cpu).to_kind(Kind::Double).contiguous();
+ let sizes = tensor.size();
+ let (num_samples, sample_size) = parse_shape(&sizes)?;
Review Comment:
Done in [4ed3346](https://github.com/apache/mahout/pull/815/commits/4ed33461bc84fe548e2d45e52adf2820c4865d04)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #815:
URL: https://github.com/apache/mahout/pull/815#issuecomment-3740222304
locally tested, should pass CI by now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen commented on code in PR #815:
URL: https://github.com/apache/mahout/pull/815#discussion_r2683320042
##########
qdp/qdp-core/tests/torch_io.rs:
##########
@@ -0,0 +1,62 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#[cfg(feature = "pytorch")]
+mod pytorch_tests {
+ use qdp_core::io::read_torch_batch;
+ use qdp_core::reader::DataReader;
+ use qdp_core::readers::TorchReader;
+ use std::fs;
+ use tch::Tensor;
+
+ #[test]
+ fn test_torch_reader_basic_1d() {
+ let temp_path = "/tmp/test_torch_basic_1d.pt";
Review Comment:
Does this work in Windows?
##########
qdp/qdp-core/src/readers/torch.rs:
##########
@@ -0,0 +1,165 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! PyTorch tensor reader implementation.
+//!
+//! Supports `.pt`/`.pth` files containing a single tensor saved with `torch.save`.
+//! The tensor must be 1D or 2D and will be converted to `float64`.
+//! Requires the `pytorch` feature to be enabled.
+
+use std::path::Path;
+
+use crate::error::{MahoutError, Result};
+use crate::reader::DataReader;
+
+/// Reader for PyTorch `.pt`/`.pth` tensor files.
+pub struct TorchReader {
+ path: std::path::PathBuf,
+ read: bool,
+}
+
+impl TorchReader {
+ /// Create a new PyTorch reader.
+ ///
+ /// # Arguments
+ /// * `path` - Path to the `.pt`/`.pth` file
+ pub fn new<P: AsRef<Path>>(path: P) -> Result<Self> {
+ let path = path.as_ref();
+
+ match path.try_exists() {
+ Ok(false) => {
+ return Err(MahoutError::Io(format!(
+ "PyTorch file not found: {}",
+ path.display()
+ )));
+ }
+ Err(e) => {
+ return Err(MahoutError::Io(format!(
+ "Failed to check if PyTorch file exists at {}: {}",
+ path.display(),
+ e
+ )));
+ }
+ Ok(true) => {}
+ }
+
+ Ok(Self {
+ path: path.to_path_buf(),
+ read: false,
+ })
+ }
+}
+
+impl DataReader for TorchReader {
+ fn read_batch(&mut self) -> Result<(Vec<f64>, usize, usize)> {
+ if self.read {
+ return Err(MahoutError::InvalidInput(
+ "Reader already consumed".to_string(),
+ ));
+ }
+ self.read = true;
+
+ #[cfg(feature = "pytorch")]
+ {
+ return read_torch_tensor(&self.path);
+ }
+
+ #[cfg(not(feature = "pytorch"))]
+ {
+ return Err(MahoutError::NotImplemented(
+ "PyTorch reader requires the 'pytorch' feature".to_string(),
+ ));
+ }
+ }
+
+ fn get_sample_size(&self) -> Option<usize> {
+ None
+ }
+
+ fn get_num_samples(&self) -> Option<usize> {
+ None
+ }
+}
+
+#[cfg(feature = "pytorch")]
+fn read_torch_tensor(path: &Path) -> Result<(Vec<f64>, usize, usize)> {
+ use tch::{Device, Kind, Tensor};
+
+ let tensor = Tensor::load(path).map_err(|e| {
+ MahoutError::Io(format!(
+ "Failed to load PyTorch tensor from {}: {}",
+ path.display(),
+ e
+ ))
+ })?;
+
+ let tensor = tensor.to_device(Device::Cpu).to_kind(Kind::Double).contiguous();
+ let sizes = tensor.size();
+ let (num_samples, sample_size) = parse_shape(&sizes)?;
Review Comment:
I think we should validate tensor shape/size first, then do a bit expensive transform with device & type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen commented on code in PR #815:
URL: https://github.com/apache/mahout/pull/815#discussion_r2683370495
##########
qdp/qdp-core/tests/torch_io.rs:
##########
@@ -0,0 +1,62 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#[cfg(feature = "pytorch")]
+mod pytorch_tests {
+ use qdp_core::io::read_torch_batch;
+ use qdp_core::reader::DataReader;
+ use qdp_core::readers::TorchReader;
+ use std::fs;
+ use tch::Tensor;
+
+ #[test]
+ fn test_torch_reader_basic_1d() {
+ let temp_path = "/tmp/test_torch_basic_1d.pt";
Review Comment:
> our PoC is focus on linux
got it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #815:
URL: https://github.com/apache/mahout/pull/815#issuecomment-3758313452
> plz rebase and solve the precommit error, thanks!
Let me look into it, I tested pre-commit locally and it passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 merged PR #815:
URL: https://github.com/apache/mahout/pull/815
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #815:
URL: https://github.com/apache/mahout/pull/815#issuecomment-3741632068
Or should I open a quick fix pr to add this in the pre-commit ci?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming commented on PR #815:
URL: https://github.com/apache/mahout/pull/815#issuecomment-3762607922
cc @ryankert01
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #815:
URL: https://github.com/apache/mahout/pull/815#issuecomment-3745464597
The current CI can't pass because it doesn't have libtorch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #815:
URL: https://github.com/apache/mahout/pull/815#issuecomment-3741630636
I included the ci fix in the pr, because the current ci doesn't have PyTorch related installed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "CheyuWu (via GitHub)" <gi...@apache.org>.
CheyuWu commented on code in PR #815:
URL: https://github.com/apache/mahout/pull/815#discussion_r2688773399
##########
qdp/qdp-core/src/readers/torch.rs:
##########
@@ -0,0 +1,175 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! PyTorch tensor reader implementation.
+//!
+//! Supports `.pt`/`.pth` files containing a single tensor saved with `torch.save`.
+//! The tensor must be 1D or 2D and will be converted to `float64`.
+//! Requires the `pytorch` feature to be enabled.
+
+use std::path::Path;
+
+use crate::error::{MahoutError, Result};
+use crate::reader::DataReader;
+
+/// Reader for PyTorch `.pt`/`.pth` tensor files.
+pub struct TorchReader {
+ path: std::path::PathBuf,
+ read: bool,
Review Comment:
I think we can add
```rust
num_samples: usize,
sample_size: usize,
```
So, `get_sample_size` and `get_num_samples` can access them directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #815:
URL: https://github.com/apache/mahout/pull/815#issuecomment-3745460494
> locally tested, should pass CI by now
Need to add the CI check in this PR first https://github.com/apache/mahout/pull/822
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming commented on PR #815:
URL: https://github.com/apache/mahout/pull/815#issuecomment-3756055070
plz rebase and solve the precommit error, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #815:
URL: https://github.com/apache/mahout/pull/815#discussion_r2683354800
##########
qdp/qdp-core/tests/torch_io.rs:
##########
@@ -0,0 +1,62 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#[cfg(feature = "pytorch")]
+mod pytorch_tests {
+ use qdp_core::io::read_torch_batch;
+ use qdp_core::reader::DataReader;
+ use qdp_core::readers::TorchReader;
+ use std::fs;
+ use tch::Tensor;
+
+ #[test]
+ fn test_torch_reader_basic_1d() {
+ let temp_path = "/tmp/test_torch_basic_1d.pt";
Review Comment:
Todo: We should document that it is recommended to run the test on Linux in the test doc later when we add docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "ryankert01 (via GitHub)" <gi...@apache.org>.
ryankert01 commented on code in PR #815:
URL: https://github.com/apache/mahout/pull/815#discussion_r2683354800
##########
qdp/qdp-core/tests/torch_io.rs:
##########
@@ -0,0 +1,62 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#[cfg(feature = "pytorch")]
+mod pytorch_tests {
+ use qdp_core::io::read_torch_batch;
+ use qdp_core::reader::DataReader;
+ use qdp_core::readers::TorchReader;
+ use std::fs;
+ use tch::Tensor;
+
+ #[test]
+ fn test_torch_reader_basic_1d() {
+ let temp_path = "/tmp/test_torch_basic_1d.pt";
Review Comment:
We should document that it is recommended to run the test on Linux in the test doc if later we added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on code in PR #815:
URL: https://github.com/apache/mahout/pull/815#discussion_r2683335627
##########
qdp/qdp-core/tests/torch_io.rs:
##########
@@ -0,0 +1,62 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#[cfg(feature = "pytorch")]
+mod pytorch_tests {
+ use qdp_core::io::read_torch_batch;
+ use qdp_core::reader::DataReader;
+ use qdp_core::readers::TorchReader;
+ use std::fs;
+ use tch::Tensor;
+
+ #[test]
+ fn test_torch_reader_basic_1d() {
+ let temp_path = "/tmp/test_torch_basic_1d.pt";
Review Comment:
our PoC is focus on linux
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #815:
URL: https://github.com/apache/mahout/pull/815#issuecomment-3758323670
Results:
```
$ uv run pre-commit run --all-files -v
Built qumat @ file:///home/jay/work/mahout
Installed 94 packages in 25ms
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/astral-sh/ruff-pre-commit.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/lucas-c/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
check yaml...............................................................Passed
- hook id: check-yaml
- duration: 0.04s
fix end of file..........................................................Passed
- hook id: end-of-file-fixer
- duration: 0.04s
trim trailing whitespace.................................................Passed
- hook id: trailing-whitespace
- duration: 0.04s
ruff linting.............................................................Passed
- hook id: ruff
- duration: 0.01s
All checks passed!
ruff formatting..........................................................Passed
- hook id: ruff-format
- duration: 0.01s
34 files left unchanged
mypy.....................................................................Passed
- hook id: mypy
- duration: 0.73s
Success: no issues found in 34 source files
check license headers (Python)...........................................Passed
- hook id: insert-license
- duration: 0.03s
check license headers (Rust).............................................Passed
- hook id: insert-license
- duration: 0.03s
check license headers (CUDA).............................................Passed
- hook id: insert-license
- duration: 0.02s
fmt......................................................................Passed
- hook id: fmt
- duration: 0.14s
clippy...................................................................Passed
- hook id: clippy
- duration: 2.87s
Compiling pyo3-build-config v0.27.2
Checking qdp-kernels v0.1.0 (/home/jay/work/mahout/qdp/qdp-kernels)
Checking qdp-core v0.1.0 (/home/jay/work/mahout/qdp/qdp-core)
Compiling pyo3-ffi v0.27.2
Compiling pyo3-macros-backend v0.27.2
Compiling pyo3 v0.27.2
Compiling numpy v0.27.1
Compiling pyo3-macros v0.27.2
Checking qdp-python v0.1.0 (/home/jay/work/mahout/qdp/qdp-python)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.79s
(qdp) jay@Lab1405a:~/work/mahout/qdp$ pre-commit run --all-files
check yaml...............................................................Passed
fix end of file..........................................................Passed
trim trailing whitespace.................................................Passed
ruff linting.............................................................Passed
ruff formatting..........................................................Passed
mypy.....................................................................Passed
check license headers (Python)...........................................Passed
check license headers (Rust).............................................Passed
check license headers (CUDA).............................................Passed
fmt......................................................................Passed
clippy...................................................................Passed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [QDP] PyTorch input format support [mahout]
Posted by "400Ping (via GitHub)" <gi...@apache.org>.
400Ping commented on PR #815:
URL: https://github.com/apache/mahout/pull/815#issuecomment-3758332370
updated cargo.lock, CI should pass by now in here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mahout.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org