[GitHub] samskalicky edited a comment on issue #12255: Pretty high cpu load when import mxnet

[GitBox <gi...@apache.org>]

samskalicky edited a comment on issue #12255: Pretty high cpu load when import mxnet
URL: https://github.com/apache/incubator-mxnet/issues/12255#issuecomment-443385116
 
 
   I added a 2 second delay between launching each process in the script and found that most processes complete the import between 1.7-1.9 seconds. This 2 second delay prevents processes from competing for resources at the same time. There appears to be some bad contention going on thats causing the large delay. Adding the 2 second delay prevented large exponential increase in import delay. Heres the data for 1-36 processes with this 2 second delay using the pip wheel:
   
   1: 2.002468
   2: 4.005214
   3: 6.007431
   4: 8.617790
   5: 10.012702
   6: 12.015214
   7: 14.018058
   8: 16.020579
   9: 18.022354
   10: 20.025505
   11: 22.027436
   12: 24.030987
   13: 26.033214
   14: 28.035729
   15: 30.037884
   16: 32.041215
   17: 34.043598
   18: 36.044175
   19: 38.048719
   20: 40.051221
   21: 42.053396
   22: 44.055702
   23: 46.058514
   24: 48.060241
   25: 50.063950
   26: 52.727346
   27: 54.068652
   28: 56.071813
   29: 58.073548
   30: 60.076997
   31: 62.079071
   32: 64.081306
   33: 66.716197
   34: 68.085373
   35: 70.088400
   36: 72.092169
   
   So while the 12 seconds for 6 processes isnt ideal, it much better than the 785 that was found earlier without the 2 second delay. 
   
   So short-term workaround is to measure about how long it takes to "import mxnet" in a single process and then add an appropriately longer (~2second) delay between launching each process to avoid contention. The delay length might need to be tuned for each use case to avoid contention. 
   
   We'll continue debugging and trace the OpenMP problem and try to resolve it.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services