[GitHub] [incubator-nuttx] xiaoxiang781216 edited a comment on issue #3216: CI: functional tests using sim

[GitBox <gi...@apache.org>]

xiaoxiang781216 edited a comment on issue #3216:
URL: https://github.com/apache/incubator-nuttx/issues/3216#issuecomment-809130782


   > > Testing on real hardware is much more complex to setup and maintain.
   > 
   > What I meant by "external infrastructure" is something that's already setup. But I agree that we should worry about this later.
   > 
   > > I would suggest aiming for sim as a first step
   > 
   > For ostest, I think a simple (p)expect script should be enough. We also have this [apache/incubator-nuttx-testing#70](https://github.com/apache/incubator-nuttx-testing/pull/70) which is still under review.
   
   LTP generate a very huge program list and then exceed the maximum command line length. We just made a workaround to split the list into small ones. @anchao please upstream the patch.
   
   
   
   > > > I have a PoC that I started validating the next release against that uses LabGrid (https://labgrid.readthedocs.io/en/latest/index.html) to run the sim and qemu tests. That is pytest driven for the validating. It is really basic, but since I can now grab the build artifacts from CI it is actually fairly quick to run.
   > > 
   > > 
   > > Does this mean that the test job would not run here? I'm not familiar with LabGrid.
   > > In my mind I would add a "test" step in CI, running some predefined tests after each sim build of interest.
   > 
   > For things that do not require actual hardware it can just be run as a regular pytest. What is really nice is that you can also use it to interact with real hardware including via openocd. So I could see a future where some of it happens in the normal CI run and some of it just is part of a nightly run or something that calls out to a physical test rack.
   > 
   > > > One thing that I noticed doing this is it would be really nice if nsh would report the return code for the last command much like bash can with `echo $?`.
   > > 
   > > 
   > > If not already, maybe we can make the sim exit with nsh's exit status, so that one can simply run nuttx sim binary and have the test run. Maybe it is also possible to pipe commands to nuttx binary, like: `echo "ostest; poweroff $?" | nuttx/nuttx`. This can be useful for more involved tests which require a series of commands, such as putting a net interface app, etc.
   > 
   > I mean what I have setup is that I just start the nuttx binary and interact with it like it is a shell. What I was getting at is it would be be nice if in the shell you could query the exit code rather than having to write a special parser for every command.
   > `nsh> ostest; echo "pass: $?"`
   > 
   
   nsh already support $?, we have been used this feature to check the test result more than half year. We have fixed serveral tests inside apps forget to return the error, @ttnie please upstream the related patch.
   
   > > > If there is interest I could put this up on the testing repo or something? I was going to wait until I ironed some of the issues out on this release, but I guess there is no real reason to wait.
   > > 
   > > 
   > > Whatever brings us closer to this being enabled is welcomed IMHO
   > 
   > Alright I'll put something up this week that at least does some of the basic execution.
   
   Yes, it will be great if you can setup the basic test infrastructure. Our test team can integrate more test cases into the infrasturcture.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org