You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/12/01 01:07:23 UTC
[GitHub] samskalicky commented on issue #12255: Pretty high cpu load when
import mxnet
samskalicky commented on issue #12255: Pretty high cpu load when import mxnet
URL: https://github.com/apache/incubator-mxnet/issues/12255#issuecomment-443385116
I added a 2 second delay between launching each process in the script and found that most processes complete the import between 1.7-1.9 seconds. This 2 second delay prevents processes from competing for resources at the same time. There appears to be some bad contention going on thats causing the large delay. Adding the 2 second delay prevented large exponential increase in import delay. Heres the data for 1-36 processes with this 2 second delay using the pip wheel:
1: 2.002468
2: 4.005214
3: 6.007431
4: 8.617790
5: 10.012702
6: 12.015214
7: 14.018058
8: 16.020579
9: 18.022354
10: 20.025505
11: 22.027436
12: 24.030987
13: 26.033214
14: 28.035729
15: 30.037884
16: 32.041215
17: 34.043598
18: 36.044175
19: 38.048719
20: 40.051221
21: 42.053396
22: 44.055702
23: 46.058514
24: 48.060241
25: 50.063950
26: 52.727346
27: 54.068652
28: 56.071813
29: 58.073548
30: 60.076997
31: 62.079071
32: 64.081306
33: 66.716197
34: 68.085373
35: 70.088400
36: 72.092169
So while the 12 seconds for 6 processes isnt ideal, it much better than the 785 that was found earlier without the 2 second delay.
So short-term workaround is to add a ~2second delay between launching each process to avoid contention.
We'll continue debugging and trace the OpenMP problem and try to resolve it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services