[GitHub] [incubator-tvm] m3at opened a new issue #6624: [Relay] Module mutated in-place

[GitBox <gi...@apache.org>]

m3at opened a new issue #6624:
URL: https://github.com/apache/incubator-tvm/issues/6624


   Following [discussion on the forum](https://discuss.tvm.apache.org/t/unable-to-build-relay-function-twice/7987) I'm opening this issue for what appear to be a bug when running `relay.build`, with potentially `nn.pad` being mutated in-place.
   
   In practice the issue can be side-stepped by making a `deepcopy` of the module before building. This issue is to find which pass is potentially mutating the module in-place.
   
   Steps to reproduce:
   
   ```sh
   # Obtain the model
   python3 -m pip install geffnet
   wget "https://github.com/rwightman/gen-efficientnet-pytorch/blob/master/onnx_export.py"
   python3 onnx_export.py ./b4.onnx --model="tf_efficientnet_b4_ns" --img-size=380
   ```
   
   ```python
   import numpy as np
   import onnx
   import tvm
   from tvm import relay
   from tvm.contrib import graph_runtime
   
   # Prepare parameters
   input_shape = [1, 3, 380, 380]
   example_input = np.random.randn(*input_shape).astype(np.float32)
   target = "llvm -mcpu=core-avx2"
   ctx = tvm.cpu(0)
   
   # Get model from ONNX
   onnx_model = onnx.load("./b4.onnx")
   mod, params = relay.frontend.from_onnx(
       onnx_model, {"input0": input_shape},
   )
   
   # Build module
   with tvm.transform.PassContext(opt_level=3):
       graph_module = relay.build(mod, target=target, target_host=target, params=params)
   
   # Run, no issue
   runtime_module = graph_runtime.GraphModule(graph_module['default'](ctx))
   runtime_module.set_input(key="input0", value=tvm.nd.array(example_input))
   runtime_module.run()
   tvm_output = runtime_module.get_output(0).asnumpy()
   
   # Build again, or use autotvm.task.extract_from_program
   # Error (see below)
   with tvm.transform.PassContext(opt_level=3):
       graph_module = relay.build(mod, target=target, target_host=target, params=params
   ```
   
   Relevant part of the error:
   ```
     %45 = multiply(%43, %44);
     %46 = nn.pad(%45, pad_width=[[0, 0], [0, 0], [0, 1], [0, 1], [0, 0]]) an internal invariant was violated while typechecking your program [05:19:37] ../src/relay/op/nn/pad.cc:125: Check failed: data->shape.size() == param->pad_width.size(): There should be as many pad width pairs as shape dimensions but the shape has 4 dimensions and there are 5 pad width pairs.
   ; ;
     %47 = nn.conv2d(%46, meta[relay.Constant][38], strides=[2, 2], padding=[0, 0, 0, 0], groups=144, kernel_size=[3, 3]);
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-tvm] aseyboldt commented on issue #6624: [Relay] Module mutated in-place

[GitBox <gi...@apache.org>]

aseyboldt commented on issue #6624:
URL: https://github.com/apache/incubator-tvm/issues/6624#issuecomment-711944589


   I ran into what I think is the same issue: https://discuss.tvm.apache.org/t/incorrect-gradient-with-relay-transform-gradient/8105
   This can lead to silently incorrect results in the gradients.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-tvm] electriclilies commented on issue #6624: [Relay] Module mutated in-place

[GitBox <gi...@apache.org>]

electriclilies commented on issue #6624:
URL: https://github.com/apache/incubator-tvm/issues/6624#issuecomment-708048388


   I am getting the exact same issue as you. It seems like the problem is being caused by the AlterOpLayout pass. 
   
   For me, disabling the pass by changing 
   `with tvm.transform.PassContext(opt_level=3):` to `with tvm.transform.PassContext(opt_level=3, disabled_pass=["AlterOpLayout"):` makes the problem go away. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org