[GitHub] olga-gorun opened a new issue #11139: MXNet prediction on docker image mxnet/python is much slower than outside docker container

[GitBox <gi...@apache.org>]

olga-gorun opened a new issue #11139: MXNet prediction on docker image mxnet/python is much slower than outside docker container
URL: https://github.com/apache/incubator-mxnet/issues/11139
 
 
   I experience significant performance decrease when running the test inside docker container built on mxnet/python:1.1.0 docker image. 
   
   The tests were done on AWS EC2 c5.xlarge  ubuntu 16.06 w/o mxnet/python:1.1.0 docker image, Docker version 18.03.1-ce, build 9ee9f40. Python version - 3.5, mxnet - 1.1.0, python-opencv - 3.3.0 (without docker, 3.4.1 - within docker)
   
   These tests run prediction (only CPU is used) of pre-trained resnet-152 model on 1000 images read from file system n several ways:
   
   - sequentially one by one
   - in parallel
   - sequentially in bulks
   - bulks in parallel
   
   In every case time got for pure OS is much better than time got for run within docker container with the factor from 2 till 5.
   
   Docker container is is run with default properties, only volume is used.
   
   `docker run -it -v /home/ubuntu/data:/data mxnet/python:1.1.0 bash`
   
   Images are copied to a different directory inside docker container to avoid possible influence of access to mounted FS. 
   
   How this difference can be explained? What are best practices of mxnet/python docker? Are there OS-level, docker-level properties that can improve situation?
   
   See used scripts and diagnoze results in gist https://gist.github.com/olga-gorun/895bd09c39aae363121b3836764c0d4e
   See results table in https://docs.google.com/spreadsheets/d/1w1ONvA5Q-atKOJcv3qc6s-rzt6WSsFQ3PgWwRv2Hs0Y/edit?usp=sharing
   (RAM and CPU usage was always taken from external OS, not inside docker container)
    
    
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services