You are viewing a plain text version of this content. The canonical link for it is here.
Posted to discuss-archive@tvm.apache.org by ading via TVM Discuss <no...@discuss.tvm.ai> on 2020/04/24 14:19:27 UTC
[TVM Discuss] [Questions] Autotvm error: cudaErrorCudartUnloading:
initialization error' error_no=2
Hello!
I added a judgment about tensorcore (“nvcc.have_tensorcore(tvm.gpu(0).compute_version)”) in conv2d schedule, like this:
"""schedule optimized for NHWC direct conv2d"""
pad_data, kernel = s[Conv].op.input_tensors
s[pad_data].compute_inline()
test_tensorcore = nvcc.have_tensorcore(tvm.gpu(0).compute_version)
if isinstance(kernel.op, tvm.te.ComputeOp) and 'dilate' in kernel.op.tag:
s[kernel].compute_inline()
But when I used autotvm to tune conv2d, the program reported the following error:
DEBUG:autotvm:No: 336 GFLOPS: 0.00/0.00 result: MeasureResult(costs=(TVMError('Traceback (most recent call last):\n [bt] (6) /home/acc/4test_tvm/test_tvm/build/libtvm.so(TVMFuncCall+0x95) [0x7fcef17af612]\n [bt] (5) /home/acc/4test_tvm/test_tvm/build/libtvm.so(tvm::runtime::PackedFunc::CallPacked(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const+0x30) [0x7fcef0d2ce6a]\n [bt] (4) /home/acc/4test_tvm/test_tvm/build/libtvm.so(std::function<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const+0x5a) [0x7fcef0d2d4b6]\n [bt] (3) /home/acc/4test_tvm/test_tvm/build/libtvm.so(+0x25e7d70) [0x7fcef17b0d70]\n [bt] (2) /home/acc/4test_tvm/test_tvm/build/libtvm.so(+0x25e7531) [0x7fcef17b0531]\n [bt] (1) /home/acc/4test_tvm/test_tvm/build/libtvm.so(tvm::runtime::CUDADeviceAPI::GetAttr(DLContext, tvm::runtime::DeviceAttrKind, tvm::runtime::TVMRetValue*)+0x3b2) [0x7fcef1841650]\n [bt] (0) /home/acc/4test_tvm/test_tvm/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x4a) [0x7fcef0bbb4fe]\n File "/home/acc/4test_tvm/test_tvm/src/runtime/cuda/cuda_device_api.cc", line 68\nCUDA: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: initialization error',),), error_no=2, all_cost=0.017934799194335938, timestamp=1587735735.6513684) [('tile_n', 4), ('tile_c', 8), ('num_thread_n', 16), ('num_thread_c', 4), ('vthread_n', 1), ('vthread_c', 1), ('step', 64)],None,997
If I delete the judgment about tensorcore, the autotvm can run normally.
Does autotvm not support adding judgment about tensorcore in schedule?
---
[Visit Topic](https://discuss.tvm.ai/t/autotvm-error-cudaerrorcudartunloading-initialization-error-error-no-2/6487/1) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.ai/email/unsubscribe/8a6ca9ce21181d83033559b2665f3dcbfd52bbd92f155b8b4b82f16601155c80).
[TVM Discuss] [Questions] Autotvm error: cudaErrorCudartUnloading:
initialization error' error_no=2
Posted by "Cody H. Yu via TVM Discuss" <no...@discuss.tvm.ai>.
That's a good finding. Maybe we can pass this parameter via `LocalBuilder` (https://github.com/apache/incubator-tvm/blob/96873076ebe895f967a04886cbcc95d209ee3980/python/tvm/autotvm/measure/measure_methods.py#L93). @merrymercy do you have any suggestions?
---
[Visit Topic](https://discuss.tvm.ai/t/autotvm-error-cudaerrorcudartunloading-initialization-error-error-no-2/6487/6) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.ai/email/unsubscribe/1a9ee4f4a5843a2092bfebd82c1cfe6e90f4bf5d9c5293297bf85c2f2cdea8d0).
[TVM Discuss] [Questions] Autotvm error: cudaErrorCudartUnloading:
initialization error' error_no=2
Posted by ading via TVM Discuss <no...@discuss.tvm.ai>.
Hello!
After debugging I have some new discoveries
In the https://github.com/apache/incubator-tvm/blob/1014fefa54b5f0a359501b6d19ea3b5a52d6dca6/python/tvm/autotvm/measure/local_executor.py#L130
Parameter "do_fork" will affect the success of autotvm when the schedule increases the acquisition of CUDA version
I add "self.do_fork=False" in line 145:
def submit(self, func, *args, **kwargs):
self.do_fork = False
if not self.do_fork:
return LocalFutureNoFork(func(*args, **kwargs))
queue = Queue(2)
process = Process(target=call_with_timeout,
args=(queue, self.timeout, func, args, kwargs))
process.start()
return LocalFuture(process, queue)
Through the above modification, autotvm can run normally.
Can I directly control this parameter to run autotvm? Just like the following settings:
tuning_option = {
'log_filename': log_file,
'tuner': 'ga',
'n_trial': 2000,
'early_stopping': 600,
'measure_option': autotvm.measure_option(
builder=autotvm.LocalBuilder(timeout=10),
runner=autotvm.LocalRunner(number=20, repeat=3, timeout=4, min_repeat_ms=150)
),
}
I will be very grateful if you can help me.
---
[Visit Topic](https://discuss.tvm.ai/t/autotvm-error-cudaerrorcudartunloading-initialization-error-error-no-2/6487/4) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.ai/email/unsubscribe/d4f5906023b70a926d295def3701a3fca5dbff4e7b69f87e3b127d13ae9ba2c6).
[TVM Discuss] [Questions] Autotvm error: cudaErrorCudartUnloading:
initialization error' error_no=2
Posted by ading via TVM Discuss <no...@discuss.tvm.ai>.
Hello!
After debugging I have some new discoveries
In the https://github.com/apache/incubator-tvm/blob/1014fefa54b5f0a359501b6d19ea3b5a52d6dca6/python/tvm/autotvm/measure/local_executor.py#L130
Parameter "do_fork" will affect the success of autotvm when the schedule increases the acquisition of CUDA version
I add "self.do_fork=False" in line 145:
def submit(self, func, *args, **kwargs):
self.do_fork = False
if not self.do_fork:
return LocalFutureNoFork(func(*args, **kwargs))
queue = Queue(2)
process = Process(target=call_with_timeout,
args=(queue, self.timeout, func, args, kwargs))
process.start()
return LocalFuture(process, queue)
Through the above modification, autotvm can run normally.
Can I directly control this parameter to run autotvm? Just like the following settings:
tuning_option = {
'log_filename': log_file,
'tuner': 'ga',
'n_trial': 2000,
'early_stopping': 600,
'measure_option': autotvm.measure_option(
builder=autotvm.LocalBuilder(timeout=10),
runner=autotvm.LocalRunner(number=20, repeat=3, timeout=4, min_repeat_ms=150)
),
}
I will be very grateful if you can help me.
---
[Visit Topic](https://discuss.tvm.ai/t/autotvm-error-cudaerrorcudartunloading-initialization-error-error-no-2/6487/5) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.ai/email/unsubscribe/65614c53accf4bddfb3af1d333540a55b43c58fa9a1713ca0a52310d97dfc8a4).
[TVM Discuss] [Questions] Autotvm error: cudaErrorCudartUnloading:
initialization error' error_no=2
Posted by ading via TVM Discuss <no...@discuss.tvm.ai>.
Thank you for your reply!
I tried to build the model directly without running AutoTVM and it can run normally. But when I used autotvm to tune conv2d, the program always reported this error even if I set n_trial to 5000.
To reproduce the problem, I modified the code: [incubator-tvm](https://github.com/apache/incubator-tvm)/[tutorials](https://github.com/apache/incubator-tvm/tree/master/tutorials)/[autotvm](https://github.com/apache/incubator-tvm/tree/master/tutorials/autotvm)/ **tune_conv2d_cuda.py**
Compared with the source program, I made the following modifications:
1. add "from tvm.contrib import nvcc" in line 50;
2. add "test_tensorcore=nvcc.have_tensorcore(tvm.gpu(0).compute_version)" in line 90;
3. modify n_trial=2000 in line 200.
If I delete the "test_tensorcore=nvcc.have_tensorcore(tvm.gpu(0).compute_version)", the autotvm can run normally.
---
[Visit Topic](https://discuss.tvm.ai/t/autotvm-error-cudaerrorcudartunloading-initialization-error-error-no-2/6487/3) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.ai/email/unsubscribe/74ce82a3930b24297e3b58e69a2bb279452f56fdce29fe8f6c974a1fdfdb4e37).
[TVM Discuss] [Questions] Autotvm error: cudaErrorCudartUnloading:
initialization error' error_no=2
Posted by "Cody H. Yu via TVM Discuss" <no...@discuss.tvm.ai>.
I think this is nothing to do with AutoTVM yet. Did you try to build the model directly without running AutoTVM? If you encounter the same error without running AutoTVM, then it means you change results in errors in the generated CUDA code.
If you didn't encounter the error without running AutoTVM, then it means the default config is fine, but certain configs in that schedule still result in errors in generated code. At least from your error log we can make sure that config No: 336 is indeed problematic with your change.
---
[Visit Topic](https://discuss.tvm.ai/t/autotvm-error-cudaerrorcudartunloading-initialization-error-error-no-2/6487/2) to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.tvm.ai/email/unsubscribe/d970d35378aa5dc6d312f9821ac437d598552eab7dae42a3874a1bce30fa70c0).