You are viewing a plain text version of this content. The canonical link for it is here.
Posted to devnull@infra.apache.org by "larroy (GitHub)" <gi...@apache.org> on 2018/08/21 16:05:03 UTC
[GitHub] [incubator-mxnet] larroy opened pull request #12276: [Don't
merge][Review] A solution to prevent zombie containers locally and in CI
## Description ##
This PR adds mechanisms on the build scripts to cleanup docker containers when there is a cancellation, either from the user by sending SIGINT / SIGTERM or when the process inside the container ends with an error.
It also propagates the environment variables from Jenkins which are used by the process tree killer to identify runaway processes so processes inside the container are killed when the Jenkins job is stopped.
With this patch we catch SIGINT and SIGTERM and install a handler on atexit which cleans (stops) docker containers which were created during execution of the build.
We also switch to python docker API for better management of running containers.
@aaronmarkham @marcoabreu @lebeg @KellenSunderland
## Checklist ##
### Essentials ###
Please feel free to remove inapplicable items for your PR.
- [x] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant [JIRA issue](https://issues.apache.org/jira/projects/MXNET/issues) created (except PRs with tiny changes)
- [x] Changes are complete (i.e. I finished coding on this PR)
- [x] All changes have test coverage:
- Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
- Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
- Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
- [x] Code is well-documented:
- For user-facing API changes, API doc string has been updated.
- For new C++ functions in header files, their functionalities and arguments are documented.
- For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
- Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
- [x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] larroy commented on pull request #12276:
[Don't merge][Review] A solution to prevent zombie containers locally and in
CI
Posted by "larroy (GitHub)" <gi...@apache.org>.
This works now as expected with the space, can be cut and pasted no problem.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
e unused
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Where is docker_client used?
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Ok cool, I'll give it a shot again.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
I'd suggest 0.2.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Commented code needed?
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] larroy commented on pull request #12276:
[Don't merge][Review] A solution to prevent zombie containers locally and in
CI
Posted by "larroy (GitHub)" <gi...@apache.org>.
Why?
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Do we return 0 on success?
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Where do we return 0?
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
redundant ()
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
arguments should be consistently lower case.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] larroy commented on issue #12276: [Don't
merge][Review] A solution to prevent zombie containers locally and in CI
Posted by "larroy (GitHub)" <gi...@apache.org>.
Made the change smaller with @KellenSunderland in a different PR: https://github.com/apache/incubator-mxnet/pull/12336
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Nit: but I'd suggest 0.2. We can jump straight to 1.0 whenever we like but until then I'd just make minor bumps.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Would it be possible to put this in a helper, and describe why it's needed?
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Great change! Not specific to zombie containers, should be in its own PR.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] larroy commented on pull request #12276:
[Don't merge][Review] A solution to prevent zombie containers locally and in
CI
Posted by "larroy (GitHub)" <gi...@apache.org>.
because it's getting close to 1.0
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] zhreshold commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "zhreshold (GitHub)" <gi...@apache.org>.
why bumping to 0.8 directly?
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Is the cleanup param optional? It's not filled here.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Please no commented code.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] larroy commented on pull request #12276:
[Don't merge][Review] A solution to prevent zombie containers locally and in
CI
Posted by "larroy (GitHub)" <gi...@apache.org>.
⚡️
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Can you give a bit more detail about what this change does? What's its relation to zombie containers?
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] larroy commented on issue #12276: [Don't
merge][Review] A solution to prevent zombie containers locally and in CI
Posted by "larroy (GitHub)" <gi...@apache.org>.
Depends on PR https://github.com/apache/incubator-mxnet/pull/12296
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Nit: indentation off, this applies to several of the refactors. Might be worth checking it out with pylint / pycharm.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] larroy closed pull request #12276: [Don't
merge][Review] A solution to prevent zombie containers locally and in CI
Posted by "larroy (GitHub)" <gi...@apache.org>.
[ pull request closed by larroy ]
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Would you be open to not putting these on different lines with \ ? I've had to copy and paste these and each time I have to paste it into a text editor, removed the \ and newlines.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] zhreshold commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "zhreshold (GitHub)" <gi...@apache.org>.
ok
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
See above comment on this.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Cool but should be in a different PR. Reverting your zombie containers fix would mean we'd have to re-implement your retry refactor and vice-versa.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] larroy commented on pull request #12276:
[Don't merge][Review] A solution to prevent zombie containers locally and in
CI
Posted by "larroy (GitHub)" <gi...@apache.org>.
because it's getting close to 1.0, by features and maturity.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Really like this
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Return val.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
This could remove useful files from a developers desktop silently. I suspect this could surprise many users. Could you potentially lose work if for example you have a new file you haven't checked in that you want tested.
This change should also be in its own PR so that we can review it effectively. I believe we could address docker zombies without making this change.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
where is interactive used?
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] larroy commented on issue #12276: [Don't
merge][Review] A solution to prevent zombie containers locally and in CI
Posted by "larroy (GitHub)" <gi...@apache.org>.
@mxnet-label-bot: [pr-awaiting-review]
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] aaronmarkham commented on issue #12276:
[Don't merge][Review] A solution to prevent zombie containers locally and in
CI
Posted by "aaronmarkham (GitHub)" <gi...@apache.org>.
Can you provide any usage/testing info?
What would be a good way to try this out locally: break/bite something and turn it into a zombie, and then kill it?
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Nit: indentation off.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
Maybe a constant instead of a function?
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] KellenSunderland commented on pull request
#12276: [Don't merge][Review] A solution to prevent zombie containers locally
and in CI
Posted by "KellenSunderland (GitHub)" <gi...@apache.org>.
I'd make this change in a different PR.
[ Full content available at: https://github.com/apache/incubator-mxnet/pull/12276 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org