You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/04/20 06:32:11 UTC
[GitHub] [tvm] zxybazh opened a new pull request, #11074: [MetaSchedule] Reduce Local Builder Memory Usage
zxybazh opened a new pull request, #11074:
URL: https://github.com/apache/tvm/pull/11074
This PR introduce a fix to restart Popen pool in every usage. It's a current work around to avoid significant memory usage by Popen workers reused during tuning following #10874. Considering the time cost of postprocessors and evolutionary search, current implementation will not increase the overhead by a large margin in practice.
CC: @tqchen @junrushao1994 @shingjan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi merged pull request #11074: [MetaSchedule] Reduce Local Builder Memory Usage
Posted by GitBox <gi...@apache.org>.
masahi merged PR #11074:
URL: https://github.com/apache/tvm/pull/11074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] shingjan commented on a diff in pull request #11074: [MetaSchedule] Reduce Local Builder Memory Usage
Posted by GitBox <gi...@apache.org>.
shingjan commented on code in PR #11074:
URL: https://github.com/apache/tvm/pull/11074#discussion_r854697996
##########
python/tvm/meta_schedule/builder/local_builder.py:
##########
@@ -149,8 +146,14 @@ def build(self, build_inputs: List[BuilderInput]) -> List[BuilderResult]:
results: List[BuilderResult] = []
map_result: MapResult
+ pool = PopenPoolExecutor(
Review Comment:
can we add some comment here just to show why we use `PopenPool` here differently than how we use it in RPCRunner and LocalRunner?
##########
python/tvm/meta_schedule/builder/local_builder.py:
##########
@@ -181,15 +184,22 @@ def build(self, build_inputs: List[BuilderInput]) -> List[BuilderResult]:
)
else:
raise ValueError("Unreachable: unexpected result: {map_result}")
+ del pool
return results
def _sanity_check(self) -> None:
def _check(f_build, f_export) -> None:
get_global_func_with_default_on_worker(name=f_build, default=None)
get_global_func_with_default_on_worker(name=f_export, default=None)
- value = self.pool.submit(_check, self.f_build, self.f_export)
+ pool = PopenPoolExecutor(
Review Comment:
same above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] zxybazh commented on pull request #11074: [MetaSchedule] Reduce Local Builder Memory Usage
Posted by GitBox <gi...@apache.org>.
zxybazh commented on PR #11074:
URL: https://github.com/apache/tvm/pull/11074#issuecomment-1104583008
> @zxybazh Thanks you for effort! Would you mind also doing this for RPCRunner and LocalRunner? Thanks a lot!
Thanks for the suggestion. I would like to keep RPCRunner and LocalRunner the same because:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao1994 commented on pull request #11074: [MetaSchedule] Reduce Local Builder Memory Usage
Posted by GitBox <gi...@apache.org>.
junrushao1994 commented on PR #11074:
URL: https://github.com/apache/tvm/pull/11074#issuecomment-1104042577
@zxybazh Thanks you for effort! Would you mind also doing this for RPCRunner and LocalRunner? Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org