You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2020/10/08 00:10:07 UTC
[GitHub] [incubator-tvm] tkonolige opened a new issue #6650: Ansor support for multiprocessing spawn
tkonolige opened a new issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650
When using multiprocessing in python, the spawn start method (default on windows and macOS) causes errors with ansor. These errors are mostly related to serialization of `Object`s which do not expose all data via `VisitAttrs`.
There are a couple ways we can fix this. @merrymercy suggests:
"1. Fix VisitAttrs. I think this won’t work even if we fix all VisitAttrs because there are pointers in MeasureInput. The json serialization format automatically generated by TVM will be wrong for this case.
2. Use the custom c++ json serializer (https://github.com/tkonolige/incubator-tvm/blob/d5cca8cd32c21f53df40a1be497c382dc473f089/src/auto_scheduler/measure_record.cc#L229-L234) to serialize all arguments of pool.map. I think this is the fasted way with minimal change.
3. move the whole measure.py to c++. This is the most elegant solution in the long term. It can make Ansor fully independent of python and its future changes. But we heavily rely on some things only provided by python or tvm python API, so this solution needs some work."
I believe 3. is the best way forward on this as it is a much more reliable solution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] merrymercy edited a comment on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
merrymercy edited a comment on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-706448409
The issue is not about RPC. It is about the argument passing in `multiprocessing.Pool.map`. Some TVMObjects cannot be correctly serialized, so we cannot pass them as arguments of `multiprocessing.Pool.map`
The issue exists in both autotvm and ansor, because the "fork trick" to avoid serialization is used in both of them.
There are three parts that are accelerated by multi-processing/multi-threading in autotvm/ansor: feature extraction, build and benchmark. The last one (benchmark) is the least important one because most people only use one machine for tuning. The benchmark is serial in most cases. The first two items (feature extraction and build) are more important because all people have mulit-core CPUs.
In autotvm, we have this issue for all three parts.
In ansor, the feature extraction is already implemented by c++ multi-threading. So we have this issue for build and benchmark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
tqchen commented on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-719100141
close by #6671
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen closed issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
tqchen closed issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tkonolige edited a comment on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
tkonolige edited a comment on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-706271838
These lines are the issue: https://github.com/apache/incubator-tvm/blob/master/python/tvm/auto_scheduler/measure.py#L63-L66. The globals are not propagated to the child process when using spawn. The inputs cannot current be passed as arguments to pool.map because they are not serializable. A lot of the auto_scheduling data structures do not expose all of their members via visitattrs, so they do no serialize correctly (some also contain pointers, so they might not serialize period).
@tqchen Not sure what you mean by starting a separate RPC server?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
tqchen commented on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-706319603
There are two factors here, the cost of build and cost of benchmarking. In terms of benchmarking, we do not need multi-processing and can multiplex things through RPC, e.g. try to obtain a few sessions and run the benchmarking.
I guess the main problem comes from compilation side. Switching to multi-threading in C++ might help, although we might still enjoy some features that python provide (e.g. easy plugin of a command line tool for compilation), PackedFunc can still be used.
Regardless, I would recommend go with option2 and then think about path towards option3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tqchen commented on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
tqchen commented on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-706208023
Would be great to confirm if this is caused by RPC itself being created via multi-processing and whether starting a separate RPC server would fix the problem.
Multiprocessing is generally something we need to deal with very carefully especially when it comes to passing state through multi-processing. There are also other problems, like certain drivers themselves are not fork safe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] merrymercy edited a comment on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
merrymercy edited a comment on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-706448409
The issue is not about RPC. It is about the argument passing in `multiprocessing.Pool.map`. Some TVMObjects cannot be correctly serialized, so we cannot pass them as arguments of `multiprocessing.Pool.map`
The issue exists in both autotvm and ansor, because the "fork trick" to avoid the serialization are used in both of them.
There are three parts that are accelerated by multi-processing/multi-threading in autotvm/ansor: feature extraction, build and benchmark. The last one (benchmark) is the least important one because most people only use one machine for tuning. The benchmark is serial in most cases. The first two items (feature extraction and build) are more important because all people have mulit-core CPUs.
In autotvm, we have this issue for all three parts.
In ansor, the feature extraction is already implemented by c++ multi-threading. So we have this issue for build and benchmark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] merrymercy edited a comment on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
merrymercy edited a comment on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-706448409
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] merrymercy commented on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
merrymercy commented on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-706448409
The issue is not about RPC. It is about the argument passing in `multiprocessing.Pool.map`. Some TVMObjects cannot be correctly serialized, so we cannot pass them as arguments of `multiprocessing.Pool.map`
The issue exists in both autotvm and ansor, because I used the fork trick to avoid the serialization in them.
There are three parts that are accelerated by multi-processing/multi-threading in autotvm/ansor: feature extraction, build and benchmark. The last one (benchmark) is the least important one because most people only use one machine for tuning. The benchmark is serial in most cases. The first two items (feature extraction and build) are more important because all people have mulit-core CPUs.
In autotvm, we have this issue for all three parts.
In ansor, the feature extraction is already implemented by c++ multi-threading. So we have this issue for build and benchmark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] comaniac commented on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
comaniac commented on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-705264104
also cc @jcf94 @FrozenGene
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tkonolige commented on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
tkonolige commented on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-706271838
These lines are the issue: https://github.com/apache/incubator-tvm/blob/master/python/tvm/auto_scheduler/measure.py#L63-L66. The globals are not propagated to the child process when using spawn. The inputs cannot current be passed as arguments to pool.map because they are not serializable. A lot of the auto_scheduling data structures do not expose all of their members via visitattrs, so they do no serialize correctly (some also contain pointers, so they might not serialize period).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] merrymercy edited a comment on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
merrymercy edited a comment on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-706448409
The issue is not about RPC. It is about the argument passing in `multiprocessing.Pool.map`. Some TVMObjects cannot be correctly serialized, so we cannot pass them as arguments of `multiprocessing.Pool.map`
The issue exists in both autotvm and ansor, because the "fork trick" to avoid serialization is used in both of them.
There are three parts that are accelerated by multi-processing/multi-threading in autotvm/ansor: feature extraction, build and benchmark. The last one (benchmark) is the least important one because most people only use one machine for tuning. The benchmark is serial in most cases. The first two items (feature extraction and build) are more important because all people have mulit-core CPUs.
In autotvm, we have this issue for all three parts, because all of them are paralleled by `multiprocessing.Pool.map`.
In ansor, the feature extraction part is already moved to c++ multi-threading. So we have this issue for build and benchmark. I think the issue can be solved by moving all parts to c++ multi-threading.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tkonolige edited a comment on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
tkonolige edited a comment on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-706295726
I'm also seeing a similar issue with `python3 -m tvm.driver.tvmc tune --target llvm --output autotuner_records.json resnet50-v2-7.onnx`. It looks like something in autotvm is now having issues with spawn.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] tkonolige commented on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
tkonolige commented on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-706295726
I'm also seeing a similar issue with `python3 -m tvm.driver.tvmc tune --target llvm --output autotuner_records.json ~/Downloads/resnet50-v2-7.onnx`. It looks like something in autotvm is now having issues with spawn.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] merrymercy edited a comment on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
merrymercy edited a comment on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-706448409
The issue is not about RPC. It is about the argument passing in `multiprocessing.Pool.map`. Some TVMObjects cannot be correctly serialized, so we cannot pass them as arguments of `multiprocessing.Pool.map`
The issue exists in both autotvm and ansor, because the "fork trick" to avoid serialization is used in both of them.
There are three parts that are accelerated by multi-processing/multi-threading in autotvm/ansor: feature extraction, build and benchmark. The last one (benchmark) is the least important one because most people only use one machine for tuning. The benchmark is serial in most cases. The first two items (feature extraction and build) are more important because all people have mulit-core CPUs.
In autotvm, we have this issue for all three parts, because all of them are paralleled by `multiprocessing.Pool.map`.
In ansor, the feature extraction part is already moved to c++ multi-threading. So we only have this issue for build and benchmark. I think the issue can be solved by moving all parts to c++ multi-threading.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-tvm] merrymercy edited a comment on issue #6650: Ansor support for multiprocessing spawn
Posted by GitBox <gi...@apache.org>.
merrymercy edited a comment on issue #6650:
URL: https://github.com/apache/incubator-tvm/issues/6650#issuecomment-706448409
The issue is not about RPC. It is about the argument passing in `multiprocessing.Pool.map`. Some TVMObjects cannot be correctly serialized, so we cannot pass them as arguments of `multiprocessing.Pool.map`
The issue exists in both autotvm and ansor, because the "fork trick" to avoid serialization is used in both of them.
There are three parts that are accelerated by multi-processing/multi-threading in autotvm/ansor: feature extraction, build and benchmark. The last one (benchmark) is the least important one because most people only use one machine for tuning. The benchmark is serial in most cases. The first two items (feature extraction and build) are more important because all people have mulit-core CPUs.
In autotvm, we have this issue for all three parts, because all of them are paralleled by `multiprocessing.Pool.map`.
In ansor, the feature extraction part is already moved to c++ multi-threading. So we only have this issue for build and benchmark. I think the issue can be solved by moving all parts to c++ multi-threading, which is more reliable and portable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org