You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nuttx.apache.org by GitBox <gi...@apache.org> on 2023/01/01 12:59:33 UTC
[GitHub] [nuttx] acassis opened a new issue, #8017: Create a Software Qualification Testing for Release new versions
acassis opened a new issue, #8017:
URL: https://github.com/apache/nuttx/issues/8017
Currently we don't have any kind of Qualification Testing to confirm that a regression wasn't included in our code base.
A good starting point is the "ostest" (at apps/testing/ostest) and the SIM CI test, but these tools alone are not enough to detect regressions.
We need some test to exercise many NuttX features and drivers. The recent email in the mailing list about the Rotating Log not working as expected is an example of basic test that we are failing to execute.
Please share ideas how to cold create some automated tests to help to exercise more NuttX features and drivers as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nuttx] acassis commented on issue #8017: Create a Software Qualification Testing for Release new versions
Posted by GitBox <gi...@apache.org>.
acassis commented on issue #8017:
URL: https://github.com/apache/nuttx/issues/8017#issuecomment-1380385052
Thank you @xiaoxiang781216 !!!
Do you have a listing with procedures to test when certificating a new chip on your company? Maybe we could use it as base to create this automated tests. I think using some wiki page to let more people bring tests suggestions will be very good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nuttx] xiaoxiang781216 commented on issue #8017: Create a Software Qualification Testing for Release new versions
Posted by GitBox <gi...@apache.org>.
xiaoxiang781216 commented on issue #8017:
URL: https://github.com/apache/nuttx/issues/8017#issuecomment-1370000157
So, this is a check list for manual testing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nuttx] acassis commented on issue #8017: Create a Software Qualification Testing for Release new versions
Posted by GitBox <gi...@apache.org>.
acassis commented on issue #8017:
URL: https://github.com/apache/nuttx/issues/8017#issuecomment-1378843463
The initial idea is to start testing manually and then we need to create some automatic testing solution for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nuttx] xiaoxiang781216 commented on issue #8017: Create a Software Qualification Testing for Release new versions
Posted by GitBox <gi...@apache.org>.
xiaoxiang781216 commented on issue #8017:
URL: https://github.com/apache/nuttx/issues/8017#issuecomment-1378869385
The simple automation is added by @yamt : https://github.com/apache/nuttx/pull/5382
@ttnie add more test: https://github.com/apache/nuttx/pull/7109 and https://github.com/apache/nuttx/pull/7743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nuttx] acassis commented on issue #8017: Create a Software Qualification Testing for Release new versions
Posted by GitBox <gi...@apache.org>.
acassis commented on issue #8017:
URL: https://github.com/apache/nuttx/issues/8017#issuecomment-1369745316
Hi @btashton very nice! I think we could create a listing of features to test for each peripheral to guarantee the driver is working. Recently I discovered that SSD1306 wasn't working on ESP32 because the I2C driver doesn't support NOSTOP feature. My idea is something like this:
Qualification Test
==============
1) GPIO
- Check if GPIO pins can work as Output, Input
- Check if GPIO Input can generate interrupts
- Check if GPIO can work in Open Drain mode
2) UART
- Check if the NSH console can start on selected UART (case MCU has 2 or more UARTs, test on others too)
- Check if UART can exchange data with other boards/devices, to exercise higher throughput, different baudrates and parity modes, etc
- Check if RS485 (Modbus) works on that UART ports
3) I2C
- Check if I2C can send a byte (to some I2C DAC that receives only a byte by time)
- Check if I2C can send block of bytes (to some I2C Display like SSD1306)
- Check if I2C can receive a byte per transaction (some sensor)
- Check if I2C can receive a block of bytes
...
It is not easy to automate these tests, maybe a FPGA board that use some standard connector pinout (i.e. like Raspberry Pi pinout) should be necessary to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nuttx] fjpanag commented on issue #8017: Create a Software Qualification Testing for Release new versions
Posted by "fjpanag (via GitHub)" <gi...@apache.org>.
fjpanag commented on issue #8017:
URL: https://github.com/apache/nuttx/issues/8017#issuecomment-1410399666
Some thoughts on enhancing testing of NuttX:
# Expect
When we want to test the external interfaces of the system (e.g. console, protocols etc), we are making heavy use of `expect` scripts:
https://linux.die.net/man/1/expect
https://likegeeks.com/expect-command/
We are running the application in the sim and we have the script to interact with the system through NSH. Practically any manual interaction can be scripted this way, easily.
We have our CI to run these scripts, ensuring that at least the basic functionality of the system and its interfaces are OK.
Another way that we are using it, is by testing protocols.
We have sim to use one virtual serial port, and we have `expect` to interact with the system through this port,
exercising all system commands.
I am planning to use `expect` also on lower-level tests, like device drivers.
I am thinking of having it sending dummy data to mocked devices, and evaluate the system's behavior.
All in all, I believe that at least all (most?) NSH commands, and NSH-triggered functionality can be easily
tested with `expect`.
_I can provide more information on how test the system with `expect`, but unfortunately I cannot share my existing scripts verbatim, as they refer to proprietary projects._
# Code Coverage
Code coverage data can be a useful metric to evaluate the coverage of the tests.
NuttX in sim supports `gcov`, and we are already using it in all our CI tests.
The simulator starts and executes its tests.
Either using `expect`, or by any other means of input, or by using purpose-build applications (like os-test).
Upon finishing the coverage data are created automatically.
At the end of the CI pipeline, `lcov` is used to merge all reports into a single one.
This single report provides the overall metrics of the tests.
I believe that it would be nice to have this on NuttX.
Especially since it requires practically 0 effort (everything is already there, enabled with Kconfigs).
At least it would be an indication on what is tested or what may need improvement.
We may even add a "rule" that each release must at least increase the coverage of the tests (even marginally).
_ I can provide my `lcov` scripts that combine the result of the various individual tests and generate the reports._
# Tests Output
One of the greatest issues with testing NuttX in an automated way is that the output of the tests
(os-test, mm-test, fs-test etc), is inconsistent.
Every test has its own output format.
If I recall correctly, one test uses the word "error" for a non-erroneous state (i.g. expected error).
Parsing of the output is very hard, adding tests can be tedious.
It happened to me at least once to have the test fail, but my CI script fail to recognize the issue.
Standardization of the output for all tests will be greatly beneficial.
_Note that all sim tests are currently broken for us, due to a regression. Something have been changed recently, the outputs differ and all tests fail. I am still investigating._
# Networking
The most neglected part of NuttX (at least IMO) is networking.
There is no automated testing I can do there.
And as it seems, it is quite a problematic part of NuttX.
Its a few months that I struggle with networking, and still it is totally unstable for me.
I also see various regressions occasionally, for example the latest master hardfaults immediately when I try to use `sendfile`.
Unfortunately, I never managed to have the networking run on the simulator (let alone within CI).
So no tests can be written for this for the moment.
I believe that this is the part of NuttX that requires the most attention, testing-wise.
Even basic HTTP, ftpc, ftpd etc communications would be a good indication that the system is still functioning properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nuttx] btashton commented on issue #8017: Create a Software Qualification Testing for Release new versions
Posted by GitBox <gi...@apache.org>.
btashton commented on issue #8017:
URL: https://github.com/apache/nuttx/issues/8017#issuecomment-1369106318
It is not automated, but last time I cut a release I created this which uncovered some issues that we then resolved in further release candidates.
https://docs.google.com/spreadsheets/d/1qMQV_CSN5Ka13_wr73QNo2Uh-NiSumjcwINuc0BkyIs/edit?usp=sharing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@nuttx.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org