[GitHub] [incubator-mxnet] houjs opened a new issue #18806: memory access error when inference a splitted pretrained model

[GitBox <gi...@apache.org>]

houjs opened a new issue #18806:
URL: https://github.com/apache/incubator-mxnet/issues/18806


   ## Description
   
   Hi, 
   
   we developed a python program to cut a pretrained object classification model to two parts, top net and bottom net to meeting our application.
   
   The model cutting is based on the model json file. We will re-write the json file and parameter files accordingly into two parts.
   
   We use ssd_512_mobilenet1.0_voc and yolo3_mobilenet1.0_voc to verify.   
   
   We have no issue about the yolo3_mobilenet1.0_voc.  It means, in mxnet environment, we intialize the two nets and the top net output feed into the input of the bottom net. The result is the same as the net without splitting.
   
   However, when we splitting the ssd_512_mobilenet1.0_voc, we encounterred the memory access error, when bottomnet inference with the input of the top net. We have checked the splitted net with netron, it seems no problem. 
   
   We cannot locate the root cause and we would like seek some help to see if it is due to the limitation of mxnet or any limitation we need to take care when splitting a model because our way of inference is unusual.
   
   We have written the test code in the attachment (just run: python splitter_test.py).
   
   
    
   
   
   ### Error Message
     File "splitter_test.py", line 113, in <module>
       class_IDs, scores, bounding_boxes = Bottomnet(*testdata)
     File "C:\Users\junsong.hou\Anaconda3\envs\npuc0100\lib\site-packages\mxnet\gluon\block.py", line 548, in __call__
       out = self.forward(*args)
     File "C:\Users\junsong.hou\Anaconda3\envs\npuc0100\lib\site-packages\mxnet\gluon\block.py", line 1098, in forward
       return self._call_cached_op(x, *args)
     File "C:\Users\junsong.hou\Anaconda3\envs\npuc0100\lib\site-packages\mxnet\gluon\block.py", line 805, in _call_cached_op
       self._build_cache(*args)
     File "C:\Users\junsong.hou\Anaconda3\envs\npuc0100\lib\site-packages\mxnet\gluon\block.py", line 793, in _build_cache
       self._cached_op = ndarray.CachedOp(out, flags)
     File "C:\Users\junsong.hou\Anaconda3\envs\npuc0100\lib\site-packages\mxnet\_ctypes\ndarray.py", line 116, in __init__
       ctypes.byref(self.handle)))
   OSError: exception: access violation reading 0xFFFFFFFFFFFFFFFF
   
   ## To Reproduce
   run: python splitter_test.py
   [cutting_ssd.zip](https://github.com/apache/incubator-mxnet/files/4986237/cutting_ssd.zip)
   
   Regards!
   JS


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] szha commented on issue #18806: memory access error when inference a splitted pretrained model

[GitBox <gi...@apache.org>]

szha commented on issue #18806:
URL: https://github.com/apache/incubator-mxnet/issues/18806#issuecomment-665334794


   @houjs thanks for reporting the issue. Besides the json and params, I think what could really help debugging is how you split the networks to generate them. It might be easier to debug that way.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org