You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/02/14 14:49:22 UTC
[GitHub] [tvm] pfk-beta opened a new issue #10240: [Bug] Some tuning tasks causes device reboot / RPC server termination
pfk-beta opened a new issue #10240:
URL: https://github.com/apache/tvm/issues/10240
Hello,
I'm using heavily RPC tracker and server with TVM. I encounter many times issue, when tuning task cause device reboot or rpc_android_app restart. After that, whole tuning process need to be restarted. Tuning process waits for something - when I turn RPC_APP again, it breaks.
Error from tuning process:
```
[Task 4/16] Current/Best: 0.29/ 0.96 GFLOPS | Progress: (10/10) | 89.11 s Done.
[Task 5/16] Current/Best: 0.00/ 0.00 GFLOPS | Progress: (0/10) | 0.00 sTraceback (most recent call last):
File "tests/android_auto.py", line 180, in <module>
tune_and_evaluate(tuning_option)
File "tests/android_auto.py", line 142, in tune_and_evaluate
tune_tasks(tasks, **tuning_opt)
File "tests/android_auto.py", line 120, in tune_tasks
autotvm.callback.log_to_file(tmp_log_file),
File "/workspace/python/tvm/autotvm/tuner/xgboost_tuner.py", line 105, in tune
super(XGBTuner, self).tune(*args, **kwargs)
File "/workspace/python/tvm/autotvm/tuner/tuner.py", line 112, in tune
measure_batch = create_measure_batch(self.task, measure_option)
File "/workspace/python/tvm/autotvm/measure/measure.py", line 282, in create_measure_batch
attach_objects = runner.set_task(task)
File "/workspace/python/tvm/autotvm/measure/measure_methods.py", line 326, in set_task
"set_task - Cannot get remote devices from the tracker. "
RuntimeError: set_task - Cannot get remote devices from the tracker. Please check the status of tracker by 'python -m tvm.exec.query_rpc_tracker --port [THE PORT YOU USE]' and make sure you have free devices on the queue status.
```
Ouptput from query_rpc_tracker:
```
Tracker address 0.0.0.0:9190
Server List
----------------------------
server-address key
----------------------------
----------------------------
Queue Status
-------------------------------
key total free pending
-------------------------------
android 0 0 1
-------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi closed issue #10240: [Bug] Some tuning tasks causes device reboot / RPC server termination
Posted by GitBox <gi...@apache.org>.
masahi closed issue #10240:
URL: https://github.com/apache/tvm/issues/10240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi commented on issue #10240: [Bug] Some tuning tasks causes device reboot / RPC server termination
Posted by GitBox <gi...@apache.org>.
masahi commented on issue #10240:
URL: https://github.com/apache/tvm/issues/10240#issuecomment-1042298125
https://github.com/apache/tvm/pull/10255
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] pfk-beta commented on issue #10240: [Bug] Some tuning tasks causes device reboot / RPC server termination
Posted by GitBox <gi...@apache.org>.
pfk-beta commented on issue #10240:
URL: https://github.com/apache/tvm/issues/10240#issuecomment-1040223745
Hello,
I think that method `check_remote` may be working incorrecly: https://github.com/apache/tvm/blob/main/python/tvm/autotvm/measure/measure_methods.py#L781
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] pfk-beta commented on issue #10240: [Bug] Some tuning tasks causes device reboot / RPC server termination
Posted by GitBox <gi...@apache.org>.
pfk-beta commented on issue #10240:
URL: https://github.com/apache/tvm/issues/10240#issuecomment-1040223745
Hello,
I think that method `check_remote` may be working incorrecly: https://github.com/apache/tvm/blob/main/python/tvm/autotvm/measure/measure_methods.py#L781
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org