You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/12/17 02:57:04 UTC

[GitHub] [incubator-mxnet] mseth10 opened a new issue #19688: [RFC] Large MXNet source files causing CI build failures

mseth10 opened a new issue #19688:
URL: https://github.com/apache/incubator-mxnet/issues/19688


   ## Problem statement
   MXNet CI is running OOM [1] while building MXNet binaries for unix-cpu and unix-gpu stages. This is an intermittent failure and the work around is to re-trigger CI a few times. The issue is caused due to some of the numpy .cc files being too large causing gcc to use too much memory. The issue was not pronounced with gcc7, but with the recent update to use gcc8 [2] for CI builds, we have started to see this OOM error.
   
   The fix is to refactor the numpy .cc files into smaller files so that the objects created during compilation don't use much memory. Here is the list of the largest objects (>10MB in size) generated currently on Mac CPU build:
   ```
    11M	./operator/numpy/linalg/np_norm_backward.cc.o
    11M	./operator/numpy/np_kron.cc.o
    11M	./operator/numpy/random/np_location_scale_op.cc.o
    12M	./operator/numpy/np_insert_op_slice.cc.o
    12M	./operator/numpy/np_insert_op_tensor.cc.o
    13M	./operator/numpy/np_elemwise_broadcast_op_extended_sec.cc.o
    13M	./operator/numpy/np_elemwise_unary_op_basic.cc.o
    13M	./operator/numpy/np_percentile_op.cc.o
    14M	./operator/numpy/np_matrix_op.cc.o
    14M	./operator/numpy/np_moments_op.cc.o
    14M	./operator/numpy/np_where_op.cc.o
    15M	./operator/numpy/np_einsum_op.cc.o
    16M	./operator/numpy/np_elemwise_broadcast_op_extended.cc.o
    21M	./operator/numpy/np_broadcast_reduce_op_value.cc.o
    22M	./operator/numpy/linalg/np_norm_forward.cc.o
    24M	./operator/numpy/np_elemwise_broadcast_op.cc.o
    34M	./operator/numpy/np_elemwise_broadcast_logic_op.cc.o
   ```
   The corresponding cc files to above objects contains more than 210 operator registrations, and to refactor those into smaller files will need a considerable time and effort from the community. With 5 operators per day, that's more than 40 days of developers effort.
   
   ## Proposed solutions
   Option 1: We keep using gcc8 for CI builds and start working on refactoring these numpy .cc files. This would mean the community will have to face the CI failures for 40 days (could be less if more community members contribute).
   
   Option 2: We go back to using gcc7 for CI builds, potentially solving the CI problem immediately, while we work on refactoring the numpy files. Reverting to gcc7 would take 2 days and then refactoring would take another 40 days.
   
   I personally would prefer Option 2 for the reason that it saves contributors time in getting their PRs merged quickly, as well as saves on the CI resources. Would like to request community feedback on the same.
   
   Going forward we also need to add a check to MXNet CI for build time memory usage. Any ideas for the same would be highly appreciated.
   
   ## References
   - [1] CI failure: https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-19588/12/pipeline/
   - [2] CI builds gcc update commit: https://github.com/apache/incubator-mxnet/commit/afc76b0f82839dfe00f7cab04be6de3df94564fe


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org

[GitHub] [incubator-mxnet] leezu commented on issue #19688: [RFC] Large MXNet source files causing CI build failures

Posted by GitBox <gi...@apache.org>.

leezu commented on issue #19688:
URL: https://github.com/apache/incubator-mxnet/issues/19688#issuecomment-747520044


   How do you arrive at the time estimates? Adding a `CC=gcc-7` statement in runtime-functions.sh shouldn't take you 2 days. It should take less than 30 minutes... I'm also a bit worried about a time estimate of 40 days for splitting `.cc` files into multiple files.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org

[GitHub] [incubator-mxnet] wkcn commented on issue #19688: [RFC] Large MXNet source files causing CI build failures

Posted by GitBox <gi...@apache.org>.

wkcn commented on issue #19688:
URL: https://github.com/apache/incubator-mxnet/issues/19688#issuecomment-747178434


   There is a related issue: https://github.com/apache/incubator-mxnet/issues/18501
   I also prefer Option 2 to solve the CI problem immediately : )


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org

[GitHub] [incubator-mxnet] samskalicky commented on issue #19688: [RFC] Large MXNet source files causing CI build failures

Posted by GitBox <gi...@apache.org>.

samskalicky commented on issue #19688:
URL: https://github.com/apache/incubator-mxnet/issues/19688#issuecomment-747637931


   > How do you arrive at the time estimates? Adding a `CC=gcc-7` statement in runtime-functions.sh shouldn't take you 2 days. It should take less than 30 minutes... I'm also a bit worried about a time estimate of 40 days for splitting `.cc` files into multiple files.
   
   Effort will certainly vary depending on who does the work, their familiarity, and how long it takes them to ramp up. If we get people familiar with the code it could be shorter. If we get people unfamiliar, 5/day seems reasonable. If its someone new to MXNet (and possibly not an expert in C++) it could take longer. 
   
   Until we get actual people to volunteer to do the work, and ask them how long they will need to do the work, we cant really give hard estimates. So these will have to do for now. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org

[GitHub] [incubator-mxnet] kpuatamazon edited a comment on issue #19688: [RFC] Large MXNet source files causing CI build failures

Posted by GitBox <gi...@apache.org>.

kpuatamazon edited a comment on issue #19688:
URL: https://github.com/apache/incubator-mxnet/issues/19688#issuecomment-749052168

A related problem is excessive code generation. Take `np.delete` for example.

https://github.com/apache/incubator-mxnet/blob/16e2b15f6e334ca88f29b9c14e55547df2c136fc/src/operator/numpy/np_delete_op-inl.h#L337-L355

That's:
- MSHADOW_TYPE_SWITCH: 8 types on CPU and 7 types on GPU.
- MXNET_NDIM_SWITCH cases 1 through 5.
- MSHADOW_TYPE_SWITCH: 8 types on CPU and 7 types on GPU.
- MXNET_ASSIGN_REQ_SWITCH: 2 cases

That's 8 * 5 * 8 * 2 = 640 ways on CPU and 7 * 5 * 7 * 2 = 490 ways on GPU.

This problem operates on a single axis. It reduces to: size of outer loop (i.e. the product of dimensions before the axis), the size of the axis in question, and the size of the data after the axis (i.e. the product of dimensions after the axis). After this simplification, there's no ndim dispatch. Supports arbitrary dimensionality with a factor of 5 reduction in compilation to 128 cases.

In the common case where the types are the same and output is kWriteTo, a loop over memory copies is much faster. If we're just copying PODs, then the size of the data type can be folded into the size of the data to copy. So all 8 cases of identical input and output types with kWriteTo can be folded into one compilation, reducing 8 on CPU or 7 on GPU to 1.

On CPU there are 121 cases: one for the normal copying operation and 120 for some combination of type conversion and/or kAddTo. On GPU there are 91 cases.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org

[GitHub] [incubator-mxnet] access2rohit edited a comment on issue #19688: [RFC] Large MXNet source files causing CI build failures

Posted by GitBox <gi...@apache.org>.

access2rohit edited a comment on issue #19688:
URL: https://github.com/apache/incubator-mxnet/issues/19688#issuecomment-747230107


   Thank you @mseth10 for looking into this !!
   I think Option 2 helps us unblock CI thus unblocking all open source contributors immediately and gives us more time to take the corrective measure to refactor files to reduce memory utilization by gcc.
   
   Option1 may keep us blocked for long and has the uncertainty of how long will it take.
   
   IMHO Option 2 is the way forward :) 
   @leezu wdyt ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org

[GitHub] [incubator-mxnet] kshitij12345 commented on issue #19688: [RFC] Large MXNet source files causing CI build failures

Posted by GitBox <gi...@apache.org>.

kshitij12345 commented on issue #19688:
URL: https://github.com/apache/incubator-mxnet/issues/19688#issuecomment-747233877


   A tangential question:
   
   Should we raise a bug report with `gcc` community as it seems that `gcc8` is using more resources to handle the same files than `gcc7` and it is a regression between those version.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org

[GitHub] [incubator-mxnet] kpuatamazon commented on issue #19688: [RFC] Large MXNet source files causing CI build failures

Posted by GitBox <gi...@apache.org>.

kpuatamazon commented on issue #19688:
URL: https://github.com/apache/incubator-mxnet/issues/19688#issuecomment-749052168

A related problem is excessive code generation. Take `np.delete` for example.

https://github.com/apache/incubator-mxnet/blob/16e2b15f6e334ca88f29b9c14e55547df2c136fc/src/operator/numpy/np_delete_op-inl.h#L337-L355

That's:
- MSHADOW_TYPE_SWITCH: 8 types on CPU and 7 types on GPU.
- MXNET_NDIM_SWITCH cases 1 through 5.
- MSHADOW_TYPE_SWITCH: 8 types on CPU and 7 types on GPU.
- MXNET_ASSIGN_REQ_SWITCH: 2 cases

That's 8*5*8*2 = 640 ways on CPU and 7*5*7*2 = 490 ways on GPU.

By the way, in the common case where the types are the same and output is kWriteTo, a loop over memory copies is much faster.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org

[GitHub] [incubator-mxnet] kpuatamazon edited a comment on issue #19688: [RFC] Large MXNet source files causing CI build failures

Posted by GitBox <gi...@apache.org>.

kpuatamazon edited a comment on issue #19688:
URL: https://github.com/apache/incubator-mxnet/issues/19688#issuecomment-749052168

A related problem is excessive code generation. Take `np.delete` for example.

https://github.com/apache/incubator-mxnet/blob/16e2b15f6e334ca88f29b9c14e55547df2c136fc/src/operator/numpy/np_delete_op-inl.h#L337-L355

That's:
- MSHADOW_TYPE_SWITCH: 8 types on CPU and 7 types on GPU.
- MXNET_NDIM_SWITCH cases 1 through 5.
- MSHADOW_TYPE_SWITCH: 8 types on CPU and 7 types on GPU.
- MXNET_ASSIGN_REQ_SWITCH: 2 cases

That's 8 * 5 * 8 * 2 = 640 ways on CPU and 7 * 5 * 7 * 2 = 490 ways on GPU.

In the common case where the types are the same and output is kWriteTo, a loop over memory copies is much faster. If we're just copying POD data, then the size of the data type can be folded into the size of the data to copy. So all 8 cases of identical input and output types with kWriteTo can be folded into one compilation, reducing 8 to 1.

So 121 cases: one for the normal copying operation and 120 for some combination of type conversion and/or kAddTo.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org

[GitHub] [incubator-mxnet] kpuatamazon edited a comment on issue #19688: [RFC] Large MXNet source files causing CI build failures

Posted by GitBox <gi...@apache.org>.

kpuatamazon edited a comment on issue #19688:
URL: https://github.com/apache/incubator-mxnet/issues/19688#issuecomment-749052168


   A related problem is excessive code generation.  Take `np.delete` for example.  
   
   https://github.com/apache/incubator-mxnet/blob/16e2b15f6e334ca88f29b9c14e55547df2c136fc/src/operator/numpy/np_delete_op-inl.h#L337-L355
   
   That's:
   - MSHADOW_TYPE_SWITCH: 8 types on CPU and 7 types on GPU.
   - MXNET_NDIM_SWITCH cases 1 through 5.
   - MSHADOW_TYPE_SWITCH: 8 types on CPU and 7 types on GPU.
   - MXNET_ASSIGN_REQ_SWITCH: 2 cases
   
   That's 8 * 5 * 8 * 2 = 640 ways on CPU and 7 * 5 * 7 * 2 = 490 ways on GPU. 
   
   This problem operates on a single axis. It reduces to: size of outer loop (i.e. the product of dimensions before the axis), the size of the axis in question, and the size of the data after the axis (i.e. the product of dimensions after the axis).  After this simplification, there's no ndim dispatch.  Supports arbitrary dimensionality with a factor of 5 reduction in compilation to 128 cases.  
   
   By the way, in the common case where the types are the same and output is kWriteTo, a loop over memory copies is much faster.  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org

[GitHub] [incubator-mxnet] access2rohit commented on issue #19688: [RFC] Large MXNet source files causing CI build failures

Posted by GitBox <gi...@apache.org>.

access2rohit commented on issue #19688:
URL: https://github.com/apache/incubator-mxnet/issues/19688#issuecomment-747230107


   Thank you @mseth10 for looking into this !!
   I think Option 2 helps us unblock CI thus unblocking all open source contributors immediately and gives us more time to take the corrective measure to refactor files to reduce memory utilization by gcc.
   
   Option1 may keep us blocked for long and has the uncertainty of how long will it take.
   
   IMHO Option 2 is the way forward :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org