You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2022/02/18 00:35:32 UTC

[GitHub] [tvm] driazati opened a new issue #10300: [CI Problem] Flaky tests in `tests/python/contrib/test_ethosu`

driazati opened a new issue #10300:
URL: https://github.com/apache/tvm/issues/10300


   ### Test(s)
   
   - `tests/python/contrib/test_ethosu/test_codegen.py::test_tflite_pack[ifm_shapes1-1-ethos-u65-256]`
   - `tests/python/contrib/test_ethosu/test_codegen.py::test_tflite_transpose_convolution[False-ifm_shape3-ofm_shape3-kernel_shape3-VALID-ethos-u55-128]`
   - `tests/python/contrib/test_ethosu/test_codegen.py::test_tflite_expand_dims[ifm_shape0-0-ethos-u55-64]`
   - `tests/python/contrib/test_ethosu/test_codegen.py::test_tflite_leaky_relu[0.2-ifm_shape1-ethos-u65-256]`
   - `tests/python/contrib/test_ethosu/test_lookup_table.py::test_tflite_lut_activations[ethos-u55-128]`
   - `tests/python/contrib/test_ethosu/test_codegen.py::test_tflite_resize2d_bilinear[ifm_shape0-size0-False-ethos-u65-256]`
   - `tests/python/contrib/test_ethosu/test_codegen.py::test_tflite_sigmoid[ethos-u65-256]`
   
   ### Jenkins Links
   
   - Failed on `main`: https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/2543/pipeline/273
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] lhutton1 edited a comment on issue #10300: [Flaky Test] `tests/python/contrib/test_ethosu`

Posted by GitBox <gi...@apache.org>.
lhutton1 edited a comment on issue #10300:
URL: https://github.com/apache/tvm/issues/10300#issuecomment-1056957983


   After #10214 we witnessed a flaky test failure in #10445:
   ```
   E           RuntimeError: Subprocess failed: make -f /workspace/tests/python/relay/aot/corstone300.mk build_dir=/tmp/tmpel9x370d/test/build CFLAGS='-DTVM_RUNTIME_ALLOC_ALIGNMENT_BYTES=8 ' TVM_ROOT=/workspace/tests/python/relay/aot/../../../.. AOT_TEST_ROOT=/workspace/tests/python/relay/aot CODEGEN_ROOT=/tmp/tmpel9x370d/test/codegen STANDALONE_CRT_DIR=/workspace/build/standalone_crt FVP_DIR=/opt/arm/FVP_Corstone_SSE-300/models/Linux64_GCC-6.4/ run
   [2022-03-02T11:44:54.304Z] E           stdout:
   [2022-03-02T11:44:54.304Z] E           
   [2022-03-02T11:44:54.304Z] E           --------------------------------------------------------------------------------2022-03-02 11:29:32: Execute (/tmp/tmpel9x370d/test/build): make -f /workspace/tests/python/relay/aot/corstone300.mk build_dir=/tmp/tmpel9x370d/test/build CFLAGS='-DTVM_RUNTIME_ALLOC_ALIGNMENT_BYTES=8 ' TVM_ROOT=/workspace/tests/python/relay/aot/../../../.. AOT_TEST_ROOT=/workspace/tests/python/relay/aot CODEGEN_ROOT=/tmp/tmpel9x370d/test/codegen STANDALONE_CRT_DIR=/workspace/build/standalone_crt FVP_DIR=/opt/arm/FVP_Corstone_SSE-300/models/Linux64_GCC-6.4/ run
   [2022-03-02T11:44:54.304Z] E           --------------------------------------------------------------------------------
   [2022-03-02T11:44:54.304Z] E           find: '/tmp/tmpel9x370d/test/codegen/host/src/*.cc': No such file or directory
   [2022-03-02T11:44:54.304Z] E           /opt/arm/FVP_Corstone_SSE-300/models/Linux64_GCC-6.4//FVP_Corstone_SSE-300_Ethos-U55 -C cpu0.CFGDTCMSZ=15 \
   [2022-03-02T11:44:54.304Z] E           -C cpu0.CFGITCMSZ=15 -C mps3_board.uart0.out_file=\"-\" -C mps3_board.uart0.shutdown_tag=\"EXITTHESIM\" \
   [2022-03-02T11:44:54.304Z] E           -C mps3_board.visualisation.disable-visualisation=1 -C mps3_board.telnetterminal0.start_telnet=0 \
   [2022-03-02T11:44:54.304Z] E           -C mps3_board.telnetterminal1.start_telnet=0 -C mps3_board.telnetterminal2.start_telnet=0 -C mps3_board.telnetterminal5.start_telnet=0 \
   [2022-03-02T11:44:54.304Z] E           -C ethosu.extra_args="--fast" \
   [2022-03-02T11:44:54.304Z] E           -C ethosu.num_macs=256 /tmp/tmpel9x370d/test/build/aot_test_runner
   [2022-03-02T11:44:54.304Z] E           telnetterminal0: Listening for serial connection on port 5000
   [2022-03-02T11:44:54.304Z] E           telnetterminal1: Listening for serial connection on port 5001
   [2022-03-02T11:44:54.304Z] E           telnetterminal2: Listening for serial connection on port 5002
   [2022-03-02T11:44:54.304Z] E           telnetterminal5: Listening for serial connection on port 5003
   [2022-03-02T11:44:54.304Z] E           
   [2022-03-02T11:44:54.304Z] E           Stopping simulation...
   [2022-03-02T11:44:54.304Z] E           
   [2022-03-02T11:44:54.304Z] E           
   [2022-03-02T11:44:54.304Z] E           Info: /OSCI/SystemC: Simulation stopped by user.
   [2022-03-02T11:44:54.304Z] E           Terminated
   [2022-03-02T11:44:54.304Z] E           /workspace/tests/python/relay/aot/corstone300.mk:129: recipe for target 'run' failed
   [2022-03-02T11:44:54.304Z] E           make: *** [run] Error 143
   ```
   (https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] manupa-arm commented on issue #10300: [Flaky Test] `tests/python/contrib/test_ethosu`

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on issue #10300:
URL: https://github.com/apache/tvm/issues/10300#issuecomment-1050742233


   We are unable to reproduce these as of today and we are not seeing these in our nightly downstream builds either.
   
   I think landing this : #10214 would really help.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] manupa-arm commented on issue #10300: [Flaky Test] `tests/python/contrib/test_ethosu`

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on issue #10300:
URL: https://github.com/apache/tvm/issues/10300#issuecomment-1057159415


   It seems like model library format extraction seems missing (or the temp directory itself goes missing) and that happens intermittently. Any thoughts why would this happen in the CI ? 
   
   @areusch @leandron @driazati @grant-arm @Mousius 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] lhutton1 commented on issue #10300: [Flaky Test] `tests/python/contrib/test_ethosu`

Posted by GitBox <gi...@apache.org>.
lhutton1 commented on issue #10300:
URL: https://github.com/apache/tvm/issues/10300#issuecomment-1046650588


   Thanks for raising this @driazati, we'll take a look today


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] manupa-arm commented on issue #10300: [Flaky Test] `tests/python/contrib/test_ethosu`

Posted by GitBox <gi...@apache.org>.
manupa-arm commented on issue #10300:
URL: https://github.com/apache/tvm/issues/10300#issuecomment-1051989970


   @ekalda @lhutton1 , so it seems the flaky ness happens to come from the run, until we know better, I would suggest adding a retry to the FVP run. WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] lhutton1 commented on issue #10300: [Flaky Test] `tests/python/contrib/test_ethosu`

Posted by GitBox <gi...@apache.org>.
lhutton1 commented on issue #10300:
URL: https://github.com/apache/tvm/issues/10300#issuecomment-1056957983


   After #10214 we witnessed a flaky test failure in #10445:
   ```
   [](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1028)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1029)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1030)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1031)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1032)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1033)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1034)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1035)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1036)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pip
 eline/#step-525-log-1037)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1038)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1039)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1040)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1041)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1042)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1043)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1044)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1045)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1046)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tv
 m/detail/PR-10445/1/pipeline/#step-525-log-1047)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1048)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1049)[](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/#step-525-log-1050)[2022-03-02T11:44:54.304Z] E           RuntimeError: Subprocess failed: make -f /workspace/tests/python/relay/aot/corstone300.mk build_dir=/tmp/tmpel9x370d/test/build CFLAGS='-DTVM_RUNTIME_ALLOC_ALIGNMENT_BYTES=8 ' TVM_ROOT=/workspace/tests/python/relay/aot/../../../.. AOT_TEST_ROOT=/workspace/tests/python/relay/aot CODEGEN_ROOT=/tmp/tmpel9x370d/test/codegen STANDALONE_CRT_DIR=/workspace/build/standalone_crt FVP_DIR=/opt/arm/FVP_Corstone_SSE-300/models/Linux64_GCC-6.4/ run
   [2022-03-02T11:44:54.304Z] E           stdout:
   [2022-03-02T11:44:54.304Z] E           
   [2022-03-02T11:44:54.304Z] E           --------------------------------------------------------------------------------2022-03-02 11:29:32: Execute (/tmp/tmpel9x370d/test/build): make -f /workspace/tests/python/relay/aot/corstone300.mk build_dir=/tmp/tmpel9x370d/test/build CFLAGS='-DTVM_RUNTIME_ALLOC_ALIGNMENT_BYTES=8 ' TVM_ROOT=/workspace/tests/python/relay/aot/../../../.. AOT_TEST_ROOT=/workspace/tests/python/relay/aot CODEGEN_ROOT=/tmp/tmpel9x370d/test/codegen STANDALONE_CRT_DIR=/workspace/build/standalone_crt FVP_DIR=/opt/arm/FVP_Corstone_SSE-300/models/Linux64_GCC-6.4/ run
   [2022-03-02T11:44:54.304Z] E           --------------------------------------------------------------------------------
   [2022-03-02T11:44:54.304Z] E           find: '/tmp/tmpel9x370d/test/codegen/host/src/*.cc': No such file or directory
   [2022-03-02T11:44:54.304Z] E           /opt/arm/FVP_Corstone_SSE-300/models/Linux64_GCC-6.4//FVP_Corstone_SSE-300_Ethos-U55 -C cpu0.CFGDTCMSZ=15 \
   [2022-03-02T11:44:54.304Z] E           -C cpu0.CFGITCMSZ=15 -C mps3_board.uart0.out_file=\"-\" -C mps3_board.uart0.shutdown_tag=\"EXITTHESIM\" \
   [2022-03-02T11:44:54.304Z] E           -C mps3_board.visualisation.disable-visualisation=1 -C mps3_board.telnetterminal0.start_telnet=0 \
   [2022-03-02T11:44:54.304Z] E           -C mps3_board.telnetterminal1.start_telnet=0 -C mps3_board.telnetterminal2.start_telnet=0 -C mps3_board.telnetterminal5.start_telnet=0 \
   [2022-03-02T11:44:54.304Z] E           -C ethosu.extra_args="--fast" \
   [2022-03-02T11:44:54.304Z] E           -C ethosu.num_macs=256 /tmp/tmpel9x370d/test/build/aot_test_runner
   [2022-03-02T11:44:54.304Z] E           telnetterminal0: Listening for serial connection on port 5000
   [2022-03-02T11:44:54.304Z] E           telnetterminal1: Listening for serial connection on port 5001
   [2022-03-02T11:44:54.304Z] E           telnetterminal2: Listening for serial connection on port 5002
   [2022-03-02T11:44:54.304Z] E           telnetterminal5: Listening for serial connection on port 5003
   [2022-03-02T11:44:54.304Z] E           
   [2022-03-02T11:44:54.304Z] E           Stopping simulation...
   [2022-03-02T11:44:54.304Z] E           
   [2022-03-02T11:44:54.304Z] E           
   [2022-03-02T11:44:54.304Z] E           Info: /OSCI/SystemC: Simulation stopped by user.
   [2022-03-02T11:44:54.304Z] E           Terminated
   [2022-03-02T11:44:54.304Z] E           /workspace/tests/python/relay/aot/corstone300.mk:129: recipe for target 'run' failed
   [2022-03-02T11:44:54.304Z] E           make: *** [run] Error 143
   ```
   (https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10445/1/pipeline/)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] lhutton1 commented on issue #10300: [Flaky Test] `tests/python/contrib/test_ethosu`

Posted by GitBox <gi...@apache.org>.
lhutton1 commented on issue #10300:
URL: https://github.com/apache/tvm/issues/10300#issuecomment-1054325979


   Thanks @manupa-arm I think that would be a good temporary solution until we can figure out the cause of the failures, I've attempted this in #10408


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org