You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "vacu9708 (via GitHub)" <gi...@apache.org> on 2025/05/26 15:56:21 UTC
[I] [Bug][meta_schedule] Tutorial `e2e_opt_model.py` fails [tvm]
vacu9708 opened a new issue, #18018:
URL: https://github.com/apache/tvm/issues/18018
Thanks for participating in the TVM community! We use https://discuss.tvm.ai for any general usage questions and discussions. The issue tracker is used for actionable items such as feature proposals discussion, roadmaps, and bug tracking. You are always welcomed to post on the forum first :smile_cat:
Issues that are inactive for a period of time may get closed. We adopt this policy so that we won't lose track of actionable issues that may fall at the bottom of the pile. Feel free to reopen a new one if you feel there is an additional problem that needs attention when an old one gets closed.
### Expected behavior
What you were expecting
-> The tutorial code `e2e_opt_model.py` should work.
### Actual behavior
What actually happened
```
File "/home/ysy/Documents/open_source/tvm/source/src/tir/transforms/inject_software_pipeline.cc", line 1143, in tvm::tir::software_pipeline::PipelineInjector::VisitStmt_(tvm::tir::ForNode const*)
InternalError: Check failed: pipeline_stages.size() == original_order.size() (3 vs. 4) : PrimFunc "main" has original order ["", "", "", ""], but pipeline annotation is [0, 0, 3] with different size
```
### Environment
Any environment details, such as: Operating System, TVM version, etc
- Ubuntu 22.04, Intel i7 13650hx, RTX 4060
- **commit:** 2d964b4133aac2f92e4185b3f095df4eb3bf3a90 (0.21.0 dev)
### Steps to reproduce
Preferably a minimal script to cause the issue to occur.
-> Execute `e2e_opt_model.py`
### My analysis
**Tracing back the bug**
The error occurs at `inject_software_pipeline.cc:1133` during the post process `VerifyGPUCode`
```cpp
auto pipeline_stages =
Downcast<Array<Integer>>(op->annotations.at(attr::software_pipeline_stage));
CHECK_EQ(pipeline_stages.size(), original_order.size())
```
As described in the error message, pipeline_stages.size() is 3 while original_order.size() is 4
There are 4 blocks, while the annotation `software_pipeline_stage` has 3 elements.
Before `VerifyGPUCode`, `RewriteReduction` is executed, which adds an extra block called `init`.
This changes original_order.size() from 3 to 4.
This seems to cause the bug.
**Potential solution**
In my opinion, `VerifyGPUCode` just validates the normal state and the problem lies with `RewriteReduction`.
`RewriteReduction` should update the size of the annotations after adding the extra block `init`, shouldn't it?
I attempted to but failed to make this modification due to the complexity of the optimization algorithm.
Is there any expert who can manage to do this? I'd appreciate your expertise to resolve this bug.
### Triage
Please refer to the list of label tags [here](https://github.com/apache/tvm/wiki/Issue-Triage-Labels) to find the relevant tags and add them below in a bullet format (example below).
* tune:meta_schedule
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][meta_schedule] Tutorial `e2e_opt_model.py` fails [tvm]
Posted by "w1049 (via GitHub)" <gi...@apache.org>.
w1049 commented on issue #18018:
URL: https://github.com/apache/tvm/issues/18018#issuecomment-2959951987
Same issue here. After downgrading to Release v0.20.0, the bug no longer occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][meta_schedule] Tutorial `e2e_opt_model.py` fails [tvm]
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen closed issue #18018: [Bug][meta_schedule] Tutorial `e2e_opt_model.py` fails
URL: https://github.com/apache/tvm/issues/18018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][meta_schedule] Tutorial `e2e_opt_model.py` fails [tvm]
Posted by "w1049 (via GitHub)" <gi...@apache.org>.
w1049 commented on issue #18018:
URL: https://github.com/apache/tvm/issues/18018#issuecomment-3022872503
In `verify_gpu_code.cc`, errors don't terminate the program but instead return `false`:
```cpp
try { /* ... */ } catch (const dmlc::Error& e) {
return false;
}
```
Before commit #95d1268, `InternalError` inherited from `::dmlc::Error`:
```cpp
/*!
* \brief Base error type for TVM. Wraps a string message.
*/
class Error : public ::dmlc::Error { // for backwards compatibility
public:
/*!
* \brief Construct an error.
* \param s The message to be displayed with the error.
*/
explicit Error(const std::string& s) : ::dmlc::Error(s) {}
};
/*!
* \brief Error type for errors from CHECK, ICHECK, and LOG(FATAL).
* Contains a backtrace of where it occurred.
*/
class InternalError : public Error {
// ...
};
```
After the commit, the error type changed to `ffi::Error`, which doesn't inherit from `dmlc::Error`:
```cpp
using ffi::EnvErrorAlreadySet;
using ffi::Error;
/*!
* \brief Error type for errors from CHECK, ICHECK, and LOG(FATAL).
* Contains a backtrace of where it occurred.
*/
class InternalError : public Error {
// ...
};
/*!
* \brief Managed reference to ErrorObj
* \sa Error Object
*/
class Error : public ObjectRef, public std::exception {
// ...
};
```
I believe this is the source of the bug, as the error handling in `verify_gpu_code.cc` expects `dmlc::Error` but now receives `ffi::Error`. Modifying the `catch()` block in `src/meta_schedule/postproc/verify_gpu_code.cc:191` will fix this issue.
Additionally, I've found other parts of the codebase that still depend on `dmlc::Error`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug][meta_schedule] Tutorial `e2e_opt_model.py` fails [tvm]
Posted by "w1049 (via GitHub)" <gi...@apache.org>.
w1049 commented on issue #18018:
URL: https://github.com/apache/tvm/issues/18018#issuecomment-3022528516
I came back to check this issue and now I believe the problem is caused by the error handling mechanism. The issue was introduced by 95d1268 (FFI refactoring). Before this commit, CHECK_EQ would also report errors but wouldn't cause the entire program to terminate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org