[GitHub] Vikas89 opened a new pull request #13602: Fix for import mxnet taking long time if multiple process launched

[GitBox <gi...@apache.org>]

Vikas89 opened a new pull request #13602: Fix for import mxnet taking long time if multiple process launched
URL: https://github.com/apache/incubator-mxnet/pull/13602
 
 
   In case there are many cores(72 cores as in c5.18xl), doing import mxnet in multiple processes take very long time. Details here: #12255
   
   One of the reason we have OMP tuning code which iterates to find OMP tune overhead. We are reducing this iteration count to reduce the overehead of tuning code.
   Also, We added an environment variable where users can set the number of cores that should be used to determine tuning.
   Reducing number of cores for tuning also helps in reduction of number of iterations.
   
   Successfully ran operator tuning unit test.(OMP_TUNING*)
   Ran below test code on c5.18xl with 72 cores to see if the import mxnet is faster
   
   ```
   import os
   import time
   import multiprocessing
   from os import getpid
   
   def mxnet_worker():
       print("before import: pid:{}".format(getpid()))
       st_time = time.time()
       import mxnet
       end_time = time.time()
       print("after import: pid:{} time:{}".format(getpid(), end_time - st_time))
   
   read_process = [multiprocessing.Process(target=mxnet_worker) for i in range(30)]
   from threading import Thread
   i=0
   #while i<32:
   #    t1 = Thread(target=mxnet_worker)
   #    t1.start()
   #    i=i+1
   for p in read_process:
       p.daemon = True
   #    time.sleep(3)
       p.start()
   time.sleep(100000)
   
   ```
   Loading 30 mxnet processes takes 41 seconds with no change in num_cores for tuning.
   
   Ideally num_cores should be 3
   When I run above code, after setting the environ variable export `MXNET_USE_NUM_CORES_OPERATOR_TUNING=3`
   the 30 mxnet processes finishs loading in 2.2 seconds.
   
   ## Description ##
   (Brief description on what this PR is about)
   
   ## Checklist ##
   ### Essentials ###
   Please feel free to remove inapplicable items for your PR.
   - [ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant [JIRA issue](https://issues.apache.org/jira/projects/MXNET/issues) created (except PRs with tiny changes)
   - [ ] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments are documented. 
   - For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
   - Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
   - [ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [ ] Feature1, tests, (and when applicable, API doc)
   - [ ] Feature2, tests, (and when applicable, API doc)
   
   ## Comments ##
   - If this change is a backward incompatible change, why must this change be made.
   - Interesting edge cases to note here
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services