[GitHub] [tvm] PENGUINLIONG opened a new pull request #7887: Protect child process enumeration in AutoTVM

[GitBox <gi...@apache.org>]

PENGUINLIONG opened a new pull request #7887:
URL: https://github.com/apache/tvm/pull/7887


   The previous version doesn't protect child thread enumeration within the `try` scope, while it is interrupting the tuning process frequently with Python 3.8.3 on Windows 10 20H2.
   
   ```py
   [Task  1/ 1]  Current/Best:    0.00/   0.00 GFLOPS | Progress: (0/500) | 0.00 s
   [Task  1/ 1]  Current/Best:    4.12/ 139.89 GFLOPS | Progress: (8/500) | 92.42 s---------------------------------------------------------------------------
   NoSuchProcess                             Traceback (most recent call last)
   ~\Repositories\tvm\test_and_tune\topi_nhcw4c.py in <module>
       448         ),
       449     }
   --> 450     tune_and_evaluate(tuning_option)
   
   ~\Repositories\tvm\test_and_tune\topi_nhcw4c.py in tune_and_evaluate(tuning_opt)
       317     print(f"Tuning...{ith}th task", shape, device_key)
       318     tasks.reverse()
   --> 319     tune_tasks(tasks, **tuning_opt)
       320 
       321 
   
   ~\Repositories\tvm\test_and_tune\tune.py in tune_tasks(tasks, measure_option, tuner, n_trial, early_stopping, log_filename, use_transfer_learning)
       219         # do tuning
       220         tsk_trial = min(n_trial, len(tsk.config_space))
   --> 221         tuner_obj.tune(
       222             n_trial=tsk_trial,
       223             early_stopping=early_stopping,
   
   ~\Repositories\tvm\python\tvm\autotvm\tuner\xgboost_tuner.py in tune(self, *args, **kwargs)
       101 
       102     def tune(self, *args, **kwargs):  # pylint: disable=arguments-differ
   --> 103         super(XGBTuner, self).tune(*args, **kwargs)
       104 
       105         # manually close pool to avoid multiprocessing issues
   
   ~\Repositories\tvm\python\tvm\autotvm\tuner\tuner.py in tune(self, n_trial, measure_option, early_stopping, callbacks, si_prefix)
       153             inputs = [MeasureInput(self.task.target, self.task, config) for config in configs]
       154             #with self.remote_lock.value_lock:
   --> 155             results = measure_batch(inputs)
       156 
       157             # keep best config
   
   ~\Repositories\tvm\python\tvm\autotvm\measure\measure.py in measure_batch(measure_inputs)
       263 
       264     def measure_batch(measure_inputs):
   --> 265         build_results = builder.build(measure_inputs)
       266         results = runner.run(measure_inputs, build_results)
       267         return results
   
   ~\Repositories\tvm\python\tvm\autotvm\measure\measure_methods.py in build(self, measure_inputs)
       111 
       112             for future in futures:
   --> 113                 res = future.get()
       114 
       115                 if isinstance(res, Exception):
   
   ~\Repositories\tvm\python\tvm\autotvm\measure\local_executor.py in get(self, timeout)
        98             raise executor.TimeoutError()
        99         if self._process.is_alive():
   --> 100             kill_child_processes(self._process.pid)
       101             self._process.terminate()
       102         self._process.join()
   
   ~\Repositories\tvm\python\tvm\autotvm\measure\local_executor.py in kill_child_processes(parent_pid, sig)
        40     except psutil.NoSuchProcess:
        41         return
   ---> 42     children = parent.children(recursive=True)
        43     for process in children:
        44         try:
   
   ~\Anaconda3\lib\site-packages\psutil\__init__.py in wrapper(self, *args, **kwargs)
       289     def wrapper(self, *args, **kwargs):
       290         if not self.is_running():
   --> 291             raise NoSuchProcess(self.pid, self._name)
       292         return fun(self, *args, **kwargs)
       293     return wrapper
   
   NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=32736)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] tqchen merged pull request #7887: Protect child process enumeration in AutoTVM

[GitBox <gi...@apache.org>]

tqchen merged pull request #7887:
URL: https://github.com/apache/tvm/pull/7887


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org