You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/11/23 19:34:13 UTC
[GitHub] [incubator-tvm] tqchen opened a new pull request #6959: [CONTRIB] PopenPoolExecutor
tqchen opened a new pull request #6959:
URL: https://github.com/apache/incubator-tvm/pull/6959
PopenPoolExecutor implements a ProcessPoolExecutor backed by popen.
- Only handles invoking functions in tvm namespace.
- Unlike multiprocessing, does not require __main__ block,
which means it can directly run on jupyter notebook.
- Come with timeout and fault tolerant support to timeout
long running jobs, and restart the process when an error happens.
Recommended usage: it is recommended to create a pool and reuse
it in a long running job(e.g. autotuning) so that the processes
are reused when possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen edited a comment on pull request #6959: [CONTRIB] PopenPoolExecutor
Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on pull request #6959:
URL: https://github.com/apache/incubator-tvm/pull/6959#issuecomment-732405908
In that case the function will need to be registered at the startup time when tvm is imported (since the popen worker also import tvm during startup). Otherwise it won't be available in the subprocess. Closures can still be passed via cloudpickle. To make registeration of any place work we will need to use fork (note that multiprocessing + spawn only works when registeration happens in global scope as well). We could support an additional closures for registeration during pool creation, if there is really a need to do so.
PopenPool is not intended to serve as a general purpose pool, but could be used to solve the particular problem of tir compilation where we can control the behavior inside the tvm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen edited a comment on pull request #6959: [CONTRIB] PopenPoolExecutor
Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on pull request #6959:
URL: https://github.com/apache/incubator-tvm/pull/6959#issuecomment-732405908
In that case the function will need to be registered at the startup time when tvm is imported (since the popen worker also import tvm during startup). Otherwise it won't be available in the subprocess. Closures can still be passed via cloudpickle. To make registeration of any place work we will need to use fork (note that multiprocessing + spawn only works when registeration happens in global scope as well). We could support an additional closures for registeration during pool creation, if there is really a need to do so.
This is not a general purpose pool, but could be used to solve the particular problem of tir compilation when we most of the control the behavior inside the tvm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on pull request #6959: [CONTRIB] PopenPoolExecutor
Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #6959:
URL: https://github.com/apache/incubator-tvm/pull/6959#issuecomment-732391402
Additional note: the system overhead of the popen pool and multiprocess.Pool is around 1e-4 sec/item. Which means they can be used to perform heavy duty tasks like compilation, but are not intended for fine grained parallelism. parallel_for in c++ should be used in those cases
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen edited a comment on pull request #6959: [CONTRIB] PopenPoolExecutor
Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on pull request #6959:
URL: https://github.com/apache/incubator-tvm/pull/6959#issuecomment-732405908
In that case the function will need to be registered at the startup time when tvm is imported (since the popen worker also import tvm during startup). Otherwise it won't be available in the subprocess. Closures can still be passed via cloudpickle. To make registeration of any place work we will need to use fork (note that multiprocessing + spawn only works when registeration happens in global scope as well). We could support an additional closures of for registeration during pool creation, if there is really a need to do so.
This is not a general purpose pool, but could be used to solve the particular problem of tir compilation when we most of the control the behavior inside the tvm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen edited a comment on pull request #6959: [CONTRIB] PopenPoolExecutor
Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on pull request #6959:
URL: https://github.com/apache/incubator-tvm/pull/6959#issuecomment-732405908
In that case the function will need to be registered at the startup time when tvm is imported (since the popen worker also import tvm during startup). Otherwise it won't be available in the subprocess. Closures can still be passed via cloudpickle. To make registeration of any place work we will need to use fork (note that multiprocessing + spawn only works when registeration happens in global scope as well). We could support an additional closures for registeration during pool creation, if there is really a need to do so.
PopenPool is not intended to serve as a general purpose pool, but could be used to solve the particular problem of tir compilation when we most of the control the behavior inside the tvm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen edited a comment on pull request #6959: [CONTRIB] PopenPoolExecutor
Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on pull request #6959:
URL: https://github.com/apache/incubator-tvm/pull/6959#issuecomment-732405908
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on pull request #6959: [CONTRIB] PopenPoolExecutor
Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #6959:
URL: https://github.com/apache/incubator-tvm/pull/6959#issuecomment-732405908
In that case the function will need to be registered at the startup time of libtvm (since the popen worker also import tvm during startup). Otherwise it won't be available in the subprocess. Closures can still be passed via cloudpickle. To make registeration of any place work we will need to use fork (since multiprocessing + spawn only works with registeration in global scope as well).
Again this is not a general purpose pool, but could be used to solve the particular problem of tir compilation when we most of the control the behavior inside the tvm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen edited a comment on pull request #6959: [CONTRIB] PopenPoolExecutor
Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on pull request #6959:
URL: https://github.com/apache/incubator-tvm/pull/6959#issuecomment-732405908
In that case the function will need to be registered at the startup time of libtvm (since the popen worker also import tvm during startup). Otherwise it won't be available in the subprocess. Closures can still be passed via cloudpickle. To make registeration of any place work we will need to use fork (since multiprocessing + spawn only works with registeration in global scope as well).
This is not a general purpose pool, but could be used to solve the particular problem of tir compilation when we most of the control the behavior inside the tvm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on pull request #6959: [CONTRIB] PopenPoolExecutor
Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #6959:
URL: https://github.com/apache/incubator-tvm/pull/6959#issuecomment-732382694
cc @merrymercy @tkonolige @junrushao1994
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen edited a comment on pull request #6959: [CONTRIB] PopenPoolExecutor
Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on pull request #6959:
URL: https://github.com/apache/incubator-tvm/pull/6959#issuecomment-732405908
In that case the function will need to be registered at the startup time when tvm is imported (since the popen worker also import tvm during startup). Otherwise it won't be available in the subprocess. Closures can still be passed via cloudpickle. To make registeration of any place work we will need to use fork (note that multiprocessing + spawn only works when registeration happens in global scope as well).
This is not a general purpose pool, but could be used to solve the particular problem of tir compilation when we most of the control the behavior inside the tvm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen edited a comment on pull request #6959: [CONTRIB] PopenPoolExecutor
Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on pull request #6959:
URL: https://github.com/apache/incubator-tvm/pull/6959#issuecomment-732405908
In that case the function will need to be registered at the startup time of libtvm (since the popen worker also import tvm during startup). Otherwise it won't be available in the subprocess. Closures can still be passed via cloudpickle. To make registeration of any place work we will need to use fork (note that multiprocessing + spawn only works when registeration happens in global scope as well).
This is not a general purpose pool, but could be used to solve the particular problem of tir compilation when we most of the control the behavior inside the tvm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tkonolige commented on pull request #6959: [CONTRIB] PopenPoolExecutor
Posted by GitBox <gi...@apache.org>.
tkonolige commented on pull request #6959:
URL: https://github.com/apache/incubator-tvm/pull/6959#issuecomment-732394788
How does this work when a user registers a function? Will the registered function be available in the subprocess?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org