You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nuttx.apache.org by yfliu2008 <yf...@qq.com.INVALID> on 2024/08/08 03:10:32 UTC

Re:Help us to create a distributed board testing farm

Alan,


Just want to add that besides a testing farm, we can also leverage the cheap QEMU more in our CI.


For example, the "rv-virt" target now supports most configurations that are used on other riscv devices, so it can be a good representative for riscv devices in general. On the other hand, almost all recent failures on k230 devices can be reproduced on QEMU, if we had "ostest" based smoke testing coverage in our CI, those fake k230 issues can be prevented earlier.



Regards,
Yanfeng



                       
Original
                       
                     

From:"Alan C. Assis"< acassis@gmail.com &gt;;

Date:2024/7/19 3:14

To:"dev"< dev@nuttx.apache.org &gt;;

Subject:Help us to create a distributed board testing farm


Hi Everyone,

We are creating a testing farm for NuttX to be integrated on our CI.

As you know, NuttX supports more than 300 boards and we need to test all
boards everytime someone modifies our code base.

So, let's to map each person have each board, please include your name in
the front of each board you have (case there is already some else name
there, put your name in the next column)

https://docs.google.com/spreadsheets/d/1nDMquG-gjfQ0c9jIfv_HztGeq1ysfCeJzOEts3X3Eks/edit?gid=0#gid=0

Thank you very much!

BR,

Alan

Re: Help us to create a distributed board testing farm

Posted by "Lee, Lup Yuen" <lu...@appkaki.com>.
Hi Yanfeng and Alan: I have been running Daily OSTest for NuttX QEMU
RISC-V, hosted on GitHub Actions:

(1) OSTest for 32-bit QEMU RISC-V runs OK on GitHub Actions:

32-bit Flat Build (rv-virt:nsh):
https://github.com/lupyuen/nuttx-riscv64/actions/workflows/qemu-riscv-nsh.yml
32-bit Kernel Build (rv-virt:knsh32):
https://github.com/lupyuen/nuttx-riscv64/actions/workflows/qemu-riscv-knsh32.yml

(2) But OSTest for 64-bit QEMU RISC-V fails on GitHub Actions (wonder why):

64-bit Flat Build (rv-virt:nsh64) crashes with: "fpu_test: Started task
FPU#1  / riscv_exception: Illegal instruction"
https://github.com/lupyuen/nuttx-riscv64/blob/main/.github/workflows/qemu-riscv-nsh64.yml#L2

64-bit Kernel Build (rv-virt:knsh64) hangs at: "ostest_main: Started
user_main"
https://github.com/lupyuen/nuttx-riscv64/blob/main/.github/workflows/qemu-riscv-knsh64.yml#L2

(3) That's why I run a script on my Home Computer to download the 64-bit
Daily Builds and run OSTest locally:

64-bit Flat Build (rv-virt:nsh64):
https://github.com/lupyuen/nuttx-riscv64/releases/tag/qemu-riscv-nsh64-2024-08-08
64-bit Kernel Build (rv-virt:knsh64):
https://github.com/lupyuen/nuttx-riscv64/releases/tag/qemu-riscv-knsh64-2024-08-08

So we might still need a Local Computer to run some of the QEMU RISC-V
tests.

Lup

On Thu, Aug 8, 2024 at 11:10 AM yfliu2008 <yf...@qq.com.invalid> wrote:

> Alan,
>
>
> Just want to add that besides a testing farm, we can also leverage the
> cheap QEMU more in our CI.
>
>
> For example, the "rv-virt" target now supports most configurations that
> are used on other riscv devices, so it can be a good representative for
> riscv devices in general. On the other hand, almost all recent failures on
> k230 devices can be reproduced on QEMU, if we had "ostest" based smoke
> testing coverage in our CI, those fake k230 issues can be prevented earlier.
>
>
>
> Regards,
> Yanfeng
>
>
>
>
> Original
>
>
>
> From:"Alan C. Assis"< acassis@gmail.com &gt;;
>
> Date:2024/7/19 3:14
>
> To:"dev"< dev@nuttx.apache.org &gt;;
>
> Subject:Help us to create a distributed board testing farm
>
>
> Hi Everyone,
>
> We are creating a testing farm for NuttX to be integrated on our CI.
>
> As you know, NuttX supports more than 300 boards and we need to test all
> boards everytime someone modifies our code base.
>
> So, let's to map each person have each board, please include your name in
> the front of each board you have (case there is already some else name
> there, put your name in the next column)
>
>
> https://docs.google.com/spreadsheets/d/1nDMquG-gjfQ0c9jIfv_HztGeq1ysfCeJzOEts3X3Eks/edit?gid=0#gid=0
>
> Thank you very much!
>
> BR,
>
> Alan

Re: Help us to create a distributed board testing farm

Posted by Xiang Xiao <xi...@gmail.com>.
Yanfeng,
There is a basic automation test running on sim/qemu after the ci build
finish:
https://github.com/apache/nuttx/blob/master/tools/ci/cirun.sh
you can add more tests and boards as needed.

On Thu, Aug 8, 2024 at 11:10 AM yfliu2008 <yf...@qq.com.invalid> wrote:

> Alan,
>
>
> Just want to add that besides a testing farm, we can also leverage the
> cheap QEMU more in our CI.
>
>
> For example, the "rv-virt" target now supports most configurations that
> are used on other riscv devices, so it can be a good representative for
> riscv devices in general. On the other hand, almost all recent failures on
> k230 devices can be reproduced on QEMU, if we had "ostest" based smoke
> testing coverage in our CI, those fake k230 issues can be prevented earlier.
>
>
>
> Regards,
> Yanfeng
>
>
>
>
> Original
>
>
>
> From:"Alan C. Assis"< acassis@gmail.com &gt;;
>
> Date:2024/7/19 3:14
>
> To:"dev"< dev@nuttx.apache.org &gt;;
>
> Subject:Help us to create a distributed board testing farm
>
>
> Hi Everyone,
>
> We are creating a testing farm for NuttX to be integrated on our CI.
>
> As you know, NuttX supports more than 300 boards and we need to test all
> boards everytime someone modifies our code base.
>
> So, let's to map each person have each board, please include your name in
> the front of each board you have (case there is already some else name
> there, put your name in the next column)
>
>
> https://docs.google.com/spreadsheets/d/1nDMquG-gjfQ0c9jIfv_HztGeq1ysfCeJzOEts3X3Eks/edit?gid=0#gid=0
>
> Thank you very much!
>
> BR,
>
> Alan

Re: Help us to create a distributed board testing farm

Posted by "Alan C. Assis" <ac...@gmail.com>.
Yes, I think by default ostest should only print SUCCESS, FAIL or SKIPPED.
And in case of failure it could print the error message to quick
understanding of the issue.

For hardware testing something we can use is a 16-ports USB HUB with
individual port power controlled by uhubctl ( https://github.com/mvp/uhubctl
).

I got this "low" cost USB HUB
https://aliexpress.com/item/1005006147378255.html and uhubctl was able to
turn ON/OFF individually all the ports.
According to the uhubctl project some versions of this HUB will not work,
but this model I found on Aliexpress worked fine.

So initially we will need to use the USB HUB with Raspberry Pi board, but
later we could port uhubctl to run on NuttX.

BR,

Alan

On Fri, Aug 9, 2024 at 10:00 AM Nathan Hartman <ha...@gmail.com>
wrote:

> I like Tomek's idea of testing Tiers, with Tier0 being "critical tests"
> that MUST pass, on all architectures and boards.
>
> I pike Alin's idea of an automatically updated test status report form,
> which can be viewed on GitHub.
>
> I recommend to list ALL supported boards in the report form, with "Need
> Test Hardware" (or something like that) shown for boards that are not
> included in any HW test cluster. The idea is to help find volunteers to
> test those boards.
>
> What happens if multiple people have the same board in their test clusters?
> And what happens if tests pass for one person but fail for another, on the
> same board model? We might need to be able to show: Pass, Fail, Need Test
> Hardware, or a percentage, like 2/3 (passing on 2 instances out of 3). That
> might indicate bugs that are sensitive to timing.
>
> Cheers,
> Nathan
>
> On Fri, Aug 9, 2024 at 7:23 AM Tomek CEDRO <to...@cedro.info> wrote:
>
> > On Fri, Aug 9, 2024 at 3:42 AM Nathan Hartman <ha...@gmail.com>
> > wrote:
> > > On Thu, Aug 8, 2024 at 8:58 PM Gregory Nutt <sp...@gmail.com>
> wrote:
> > > > On 8/8/2024 6:48 PM, Nathan Hartman wrote:
> > > > > A dedicated "citest" program is a good idea!
> > > > I think that ostest could meet all of the needs of a "citest"
> program.
> > > > I just needs more control of the verbosity and format of the output.
> > It
> > > > already meets 95% of the requirement.
> > >
> > > Ok so perhaps it could use a command line arg to instruct it how much
> > > output to produce. Many unix programs have the convention of --quiet
> for
> > no
> > > output, --verbose for full output, and by default only necessary
> > messages.
> > > But a CI test might need different output altogether, since it needs to
> > be
> > > compared somehow. So, maybe we need a --ci argument that puts the
> output
> > in
> > > a format suitable for automatic CI testing. My thinking is to provide
> one
> > > mode for CI and another (more user friendly) mode for manual testing. I
> > > think that's needed because if the CI tests fail, then we would likely
> > want
> > > to run it manually and see what isn't working.
> >
> > True `ostest --ci`  would keep everything in one place and make things
> > easier to maintain + provide output easy to parse by ci/test
> > frameworks that we need :-)
> >
> > Also good idea to mark all tests PASS, FAIL, SKIP as this would
> > provide proof on what was done, what was skipped, what failed, so
> > called tests accountability artifacts. The optional `--verbose` switch
> > that would provide problem details when needed :-)
> >
> > I think most test there would have to pass in order to merge code to
> > upstream or make a release. Some tests might be skipped for various
> > reasons thus skip_list (short one) seems more reasonable than
> > test_list (long one). On the other hand when initial non verbose
> > testing has some FAIL results it may be necessary to launch only given
> > set of verbose test to gather fail reason details. Also it may be
> > desired to group tests in tiers - for instance Tier0 group is critical
> > and all test must pass cannot be skipped, Tier1 group may contain
> > tests that can be skipper, Tier2 tests are optional, etc.
> >
> > Thus the idea of test code names using unique OID like mechanism so
> > each code would tell exactly what the test is for. Maybe we could use
> > existing OID base codes, maybe we need to create our own TID (Test ID)
> > tree. If there are OID identifiers already defined that could cover
> > our test areas we would use OID to mark what is tested and Test Number
> > to mark the test number (i.e. OID=0.1.2.3 T=1). Assuming that existing
> > OID/TID/OID+T won't change in future to keep results and tools
> > coherent. I am not that familiar with OID database though here may be
> > a starting point: http://www.oid-info.com/
> >
> > If there are already well adopted existing open-source test analysis
> > frameworks / standards out there (other big Apache projects?) we may
> > just use them as reference for test organization and automated
> > analysis :-)
> >
> > Have a good day folks! :-)
> > Tomek
> >
> > --
> > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >
>

Re: Help us to create a distributed board testing farm

Posted by raiden00pl <ra...@gmail.com>.
One of the ideas that came to mind when working with the x86 architecture
and
the ACRN hypervisor was that this setup could be a powerful tool for
automating
complex tests for NuttX. On a multicore x86 machine with a hypervisor, we
can
compile different NuttX images, run them simultaneously, and test various
scenarios. The output from these NuttX images can be captured on the Linux
host,
analyzed, and the test results reported to the user/web.

We can implement 3 types of test scenarios:

1. Self-testing NuttX images for testing various configuration options
(fuzze
   Kconfig options with ostest etc.)
2. NuttX image that communicates with Linux host on the same machine
3. NuttX image that communicate with other NuttX images over shared memory

Moreover, this can be combined with building, flashing, and running test
images on external hardware connected via USB.

I know this can be achieved with the ACRN hypervisor, but if any other
hypervisor can run many post-launched VMs from a Linux host (Service VM in
ACRN)
this also should work.

pt., 9 sie 2024 o 20:30 Tomek CEDRO <to...@cedro.info> napisał(a):

> On Fri, Aug 9, 2024 at 3:00 PM Nathan Hartman <ha...@gmail.com>
> wrote:
> > I like Tomek's idea of testing Tiers, with Tier0 being "critical tests"
> > that MUST pass, on all architectures and boards.
> >
> > I pike Alin's idea of an automatically updated test status report form,
> > which can be viewed on GitHub.
> >
> > I recommend to list ALL supported boards in the report form, with "Need
> > Test Hardware" (or something like that) shown for boards that are not
> > included in any HW test cluster. The idea is to help find volunteers to
> > test those boards.
> >
> > What happens if multiple people have the same board in their test
> clusters?
> > And what happens if tests pass for one person but fail for another, on
> the
> > same board model? We might need to be able to show: Pass, Fail, Need Test
> > Hardware, or a percentage, like 2/3 (passing on 2 instances out of 3).
> That
> > might indicate bugs that are sensitive to timing.
>
> Thanks Nathan :-) I am sure we can create something useful and helpful
> and I really like all those constructive "Think Before You Do"^TM
> discussions here on the NuttX Dev mailing list :-) :-)
>
> For CI testing on the GitHub I guess we are limited to build and
> simulator / qemu only. Distributed testing would be perfect so anyone
> could just launch a script with some real world boards attached then
> gathered results could be uploaded into a central point with total
> summary tests analysis. That way we could cover almost any hardware
> supported by NuttX because "someone somewhere should have that board"
> :-) Also it would be prone to single point of failure (except the
> central point). What a great idea! :-)
>
> But we should measure strengths for goals, with ENOTIME and ENOCASH
> throwing all around, lets start from something simple that just works
> does the job we need and allow extensions in future :-) Some design
> considerations right now could make that project elegant too :-)
>
> Alan mentioned USB Hub - I have popular and not that expensive
> (~50EUR) iTec USB 3.0 16-port 10W max with external power supply and
> manual power switch at every port (PN:U3CHARGEHUB16). Did not have
> time to use it yet.. and did not know that single hub ports also could
> be controlled from a PC (I know FreeBSD USB stack can reset and power
> off given device but that device first needs to be connected to a
> port). Good hint thanks Alan :-)
>
> Have a good weekend folks! :-)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>

Re: Help us to create a distributed board testing farm

Posted by Tomek CEDRO <to...@cedro.info>.
On Fri, Aug 9, 2024 at 3:00 PM Nathan Hartman <ha...@gmail.com> wrote:
> I like Tomek's idea of testing Tiers, with Tier0 being "critical tests"
> that MUST pass, on all architectures and boards.
>
> I pike Alin's idea of an automatically updated test status report form,
> which can be viewed on GitHub.
>
> I recommend to list ALL supported boards in the report form, with "Need
> Test Hardware" (or something like that) shown for boards that are not
> included in any HW test cluster. The idea is to help find volunteers to
> test those boards.
>
> What happens if multiple people have the same board in their test clusters?
> And what happens if tests pass for one person but fail for another, on the
> same board model? We might need to be able to show: Pass, Fail, Need Test
> Hardware, or a percentage, like 2/3 (passing on 2 instances out of 3). That
> might indicate bugs that are sensitive to timing.

Thanks Nathan :-) I am sure we can create something useful and helpful
and I really like all those constructive "Think Before You Do"^TM
discussions here on the NuttX Dev mailing list :-) :-)

For CI testing on the GitHub I guess we are limited to build and
simulator / qemu only. Distributed testing would be perfect so anyone
could just launch a script with some real world boards attached then
gathered results could be uploaded into a central point with total
summary tests analysis. That way we could cover almost any hardware
supported by NuttX because "someone somewhere should have that board"
:-) Also it would be prone to single point of failure (except the
central point). What a great idea! :-)

But we should measure strengths for goals, with ENOTIME and ENOCASH
throwing all around, lets start from something simple that just works
does the job we need and allow extensions in future :-) Some design
considerations right now could make that project elegant too :-)

Alan mentioned USB Hub - I have popular and not that expensive
(~50EUR) iTec USB 3.0 16-port 10W max with external power supply and
manual power switch at every port (PN:U3CHARGEHUB16). Did not have
time to use it yet.. and did not know that single hub ports also could
be controlled from a PC (I know FreeBSD USB stack can reset and power
off given device but that device first needs to be connected to a
port). Good hint thanks Alan :-)

Have a good weekend folks! :-)

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: Help us to create a distributed board testing farm

Posted by Nathan Hartman <ha...@gmail.com>.
I like Tomek's idea of testing Tiers, with Tier0 being "critical tests"
that MUST pass, on all architectures and boards.

I pike Alin's idea of an automatically updated test status report form,
which can be viewed on GitHub.

I recommend to list ALL supported boards in the report form, with "Need
Test Hardware" (or something like that) shown for boards that are not
included in any HW test cluster. The idea is to help find volunteers to
test those boards.

What happens if multiple people have the same board in their test clusters?
And what happens if tests pass for one person but fail for another, on the
same board model? We might need to be able to show: Pass, Fail, Need Test
Hardware, or a percentage, like 2/3 (passing on 2 instances out of 3). That
might indicate bugs that are sensitive to timing.

Cheers,
Nathan

On Fri, Aug 9, 2024 at 7:23 AM Tomek CEDRO <to...@cedro.info> wrote:

> On Fri, Aug 9, 2024 at 3:42 AM Nathan Hartman <ha...@gmail.com>
> wrote:
> > On Thu, Aug 8, 2024 at 8:58 PM Gregory Nutt <sp...@gmail.com> wrote:
> > > On 8/8/2024 6:48 PM, Nathan Hartman wrote:
> > > > A dedicated "citest" program is a good idea!
> > > I think that ostest could meet all of the needs of a "citest" program.
> > > I just needs more control of the verbosity and format of the output.
> It
> > > already meets 95% of the requirement.
> >
> > Ok so perhaps it could use a command line arg to instruct it how much
> > output to produce. Many unix programs have the convention of --quiet for
> no
> > output, --verbose for full output, and by default only necessary
> messages.
> > But a CI test might need different output altogether, since it needs to
> be
> > compared somehow. So, maybe we need a --ci argument that puts the output
> in
> > a format suitable for automatic CI testing. My thinking is to provide one
> > mode for CI and another (more user friendly) mode for manual testing. I
> > think that's needed because if the CI tests fail, then we would likely
> want
> > to run it manually and see what isn't working.
>
> True `ostest --ci`  would keep everything in one place and make things
> easier to maintain + provide output easy to parse by ci/test
> frameworks that we need :-)
>
> Also good idea to mark all tests PASS, FAIL, SKIP as this would
> provide proof on what was done, what was skipped, what failed, so
> called tests accountability artifacts. The optional `--verbose` switch
> that would provide problem details when needed :-)
>
> I think most test there would have to pass in order to merge code to
> upstream or make a release. Some tests might be skipped for various
> reasons thus skip_list (short one) seems more reasonable than
> test_list (long one). On the other hand when initial non verbose
> testing has some FAIL results it may be necessary to launch only given
> set of verbose test to gather fail reason details. Also it may be
> desired to group tests in tiers - for instance Tier0 group is critical
> and all test must pass cannot be skipped, Tier1 group may contain
> tests that can be skipper, Tier2 tests are optional, etc.
>
> Thus the idea of test code names using unique OID like mechanism so
> each code would tell exactly what the test is for. Maybe we could use
> existing OID base codes, maybe we need to create our own TID (Test ID)
> tree. If there are OID identifiers already defined that could cover
> our test areas we would use OID to mark what is tested and Test Number
> to mark the test number (i.e. OID=0.1.2.3 T=1). Assuming that existing
> OID/TID/OID+T won't change in future to keep results and tools
> coherent. I am not that familiar with OID database though here may be
> a starting point: http://www.oid-info.com/
>
> If there are already well adopted existing open-source test analysis
> frameworks / standards out there (other big Apache projects?) we may
> just use them as reference for test organization and automated
> analysis :-)
>
> Have a good day folks! :-)
> Tomek
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>

Re: Help us to create a distributed board testing farm

Posted by Tomek CEDRO <to...@cedro.info>.
On Fri, Aug 9, 2024 at 3:42 AM Nathan Hartman <ha...@gmail.com> wrote:
> On Thu, Aug 8, 2024 at 8:58 PM Gregory Nutt <sp...@gmail.com> wrote:
> > On 8/8/2024 6:48 PM, Nathan Hartman wrote:
> > > A dedicated "citest" program is a good idea!
> > I think that ostest could meet all of the needs of a "citest" program.
> > I just needs more control of the verbosity and format of the output.  It
> > already meets 95% of the requirement.
>
> Ok so perhaps it could use a command line arg to instruct it how much
> output to produce. Many unix programs have the convention of --quiet for no
> output, --verbose for full output, and by default only necessary messages.
> But a CI test might need different output altogether, since it needs to be
> compared somehow. So, maybe we need a --ci argument that puts the output in
> a format suitable for automatic CI testing. My thinking is to provide one
> mode for CI and another (more user friendly) mode for manual testing. I
> think that's needed because if the CI tests fail, then we would likely want
> to run it manually and see what isn't working.

True `ostest --ci`  would keep everything in one place and make things
easier to maintain + provide output easy to parse by ci/test
frameworks that we need :-)

Also good idea to mark all tests PASS, FAIL, SKIP as this would
provide proof on what was done, what was skipped, what failed, so
called tests accountability artifacts. The optional `--verbose` switch
that would provide problem details when needed :-)

I think most test there would have to pass in order to merge code to
upstream or make a release. Some tests might be skipped for various
reasons thus skip_list (short one) seems more reasonable than
test_list (long one). On the other hand when initial non verbose
testing has some FAIL results it may be necessary to launch only given
set of verbose test to gather fail reason details. Also it may be
desired to group tests in tiers - for instance Tier0 group is critical
and all test must pass cannot be skipped, Tier1 group may contain
tests that can be skipper, Tier2 tests are optional, etc.

Thus the idea of test code names using unique OID like mechanism so
each code would tell exactly what the test is for. Maybe we could use
existing OID base codes, maybe we need to create our own TID (Test ID)
tree. If there are OID identifiers already defined that could cover
our test areas we would use OID to mark what is tested and Test Number
to mark the test number (i.e. OID=0.1.2.3 T=1). Assuming that existing
OID/TID/OID+T won't change in future to keep results and tools
coherent. I am not that familiar with OID database though here may be
a starting point: http://www.oid-info.com/

If there are already well adopted existing open-source test analysis
frameworks / standards out there (other big Apache projects?) we may
just use them as reference for test organization and automated
analysis :-)

Have a good day folks! :-)
Tomek

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Re: Help us to create a distributed board testing farm

Posted by Alin Jerpelea <je...@gmail.com>.
HI all ,

ostest is a nice first step on validating our RTOS but I think that having
a distributed testing farm on real HW will help us scale and diversify the
testing

My proposal is
1. have a build machine that builds every day all configs and provides the
binaries for download so that they can be validated by the test clusters
2. each test cluster, configured for available HW in that cluster,
downloads the corresponding binaries, flashes, tests and reports back the
status for the tested boards on github
3. on our main site we link the github status report form (table with all
boards/configs and the status for the last build)

What do you think about this approach?

Best regards
Alin

On Fri, Aug 9, 2024 at 3:42 AM Nathan Hartman <ha...@gmail.com>
wrote:

> On Thu, Aug 8, 2024 at 8:58 PM Gregory Nutt <sp...@gmail.com> wrote:
>
> >
> > On 8/8/2024 6:48 PM, Nathan Hartman wrote:
> > > A dedicated "citest" program is a good idea!
> > I think that ostest could meet all of the needs of a "citest" program.
> > I just needs more control of the verbosity and format of the output.  It
> > already meets 95% of the requirement.
> >
>
> Ok so perhaps it could use a command line arg to instruct it how much
> output to produce. Many unix programs have the convention of --quiet for no
> output, --verbose for full output, and by default only necessary messages.
> But a CI test might need different output altogether, since it needs to be
> compared somehow. So, maybe we need a --ci argument that puts the output in
> a format suitable for automatic CI testing. My thinking is to provide one
> mode for CI and another (more user friendly) mode for manual testing. I
> think that's needed because if the CI tests fail, then we would likely want
> to run it manually and see what isn't working.
>
> Just a thought.
>
> Cheers
> Nathan
>

Re: Help us to create a distributed board testing farm

Posted by Nathan Hartman <ha...@gmail.com>.
On Thu, Aug 8, 2024 at 8:58 PM Gregory Nutt <sp...@gmail.com> wrote:

>
> On 8/8/2024 6:48 PM, Nathan Hartman wrote:
> > A dedicated "citest" program is a good idea!
> I think that ostest could meet all of the needs of a "citest" program.
> I just needs more control of the verbosity and format of the output.  It
> already meets 95% of the requirement.
>

Ok so perhaps it could use a command line arg to instruct it how much
output to produce. Many unix programs have the convention of --quiet for no
output, --verbose for full output, and by default only necessary messages.
But a CI test might need different output altogether, since it needs to be
compared somehow. So, maybe we need a --ci argument that puts the output in
a format suitable for automatic CI testing. My thinking is to provide one
mode for CI and another (more user friendly) mode for manual testing. I
think that's needed because if the CI tests fail, then we would likely want
to run it manually and see what isn't working.

Just a thought.

Cheers
Nathan

Re: Help us to create a distributed board testing farm

Posted by Gregory Nutt <sp...@gmail.com>.
On 8/8/2024 6:48 PM, Nathan Hartman wrote:
> A dedicated "citest" program is a good idea!
I think that ostest could meet all of the needs of a "citest" program.  
I just needs more control of the verbosity and format of the output.  It 
already meets 95% of the requirement.

Re: Help us to create a distributed board testing farm

Posted by Nathan Hartman <ha...@gmail.com>.
A dedicated "citest" program is a good idea!

The tests would need to be designed so that when a test passes, certain
variables would have certain values, which can be tested to produce the
PASS/FAIL result.

It will be tricky to design these tests, but it will be extremely valuable
for verifying the OS across many CPU architectures. Having all tests PASS
as a prerequisite for releases will give us a high confidence in NuttX.

I think "citest" should not print any output, except in case of failed
test, and a summary of number of PASSED, FAILED, and SKIPPED tests, which
is printed at the end. But there should be a command line arg "--verbose"
that causes it to report each PASS, FAIL, and SKIP as it happens.

A test is SKIPPED when the architecture does not support the feature being
tested, or when the OS is Kconfigured and built without support for a
feature. The test will not run, but will report SKIPPED.

Nathan

On Thu, Aug 8, 2024 at 7:15 AM Tomek CEDRO <to...@cedro.info> wrote:

> On Thu, Aug 8, 2024, 06:02 Gregory Nutt <sp...@gmail.com> wrote:
>
> >
> > On 8/7/2024 9:10 PM, yfliu2008 wrote:
> > > ... if we had "ostest" based smoke testing coverage in our CI, those
> > fake k230 issues can be prevented earlier.
> >
> > ostest would probably need some work to be useful in this case: ostest
> > would need some kind of standard reporting in each test so that we would
> > know when a test case passed or failed.
> >
> > Right now, many tests do not report any test results.  That is because
> > they run relatively complex, multi-threaded scenarios that are not easy
> > to assess.  You have to understand what the test is doing in order to
> > interpret the test output.  Such tests would need to report failures in
> > a way that can be captured as a simple PASS or FAIL.
> >
>
> how about new application citest dedicated just for that purpose ?
>
> it can reuse existing code parts from other places and by default just
> print out something easy to parse i.e:
>
> [TESTX.Y.Z] [OK] Brief test description.
> [TESTX.Y.Z][FA] Brief test description (error code).
>
> X.Y.Z could be code tree based test numbers i.e:
>
> X=1 for kernel
>
> Y=1 for IPC
>
> Z=1 as test number that does something
>
> configuration could include list of excluded tests and verbosity for
> instance when test fails do we want to see the output why (or just include
> error code in test brief).
>
> ? :-)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>
> >
>

Re: Help us to create a distributed board testing farm

Posted by Tomek CEDRO <to...@cedro.info>.
On Thu, Aug 8, 2024, 06:02 Gregory Nutt <sp...@gmail.com> wrote:

>
> On 8/7/2024 9:10 PM, yfliu2008 wrote:
> > ... if we had "ostest" based smoke testing coverage in our CI, those
> fake k230 issues can be prevented earlier.
>
> ostest would probably need some work to be useful in this case: ostest
> would need some kind of standard reporting in each test so that we would
> know when a test case passed or failed.
>
> Right now, many tests do not report any test results.  That is because
> they run relatively complex, multi-threaded scenarios that are not easy
> to assess.  You have to understand what the test is doing in order to
> interpret the test output.  Such tests would need to report failures in
> a way that can be captured as a simple PASS or FAIL.
>

how about new application citest dedicated just for that purpose ?

it can reuse existing code parts from other places and by default just
print out something easy to parse i.e:

[TESTX.Y.Z] [OK] Brief test description.
[TESTX.Y.Z][FA] Brief test description (error code).

X.Y.Z could be code tree based test numbers i.e:

X=1 for kernel

Y=1 for IPC

Z=1 as test number that does something

configuration could include list of excluded tests and verbosity for
instance when test fails do we want to see the output why (or just include
error code in test brief).

? :-)

--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

>

Re: Help us to create a distributed board testing farm

Posted by Gregory Nutt <sp...@gmail.com>.
On 8/7/2024 9:10 PM, yfliu2008 wrote:
> ... if we had "ostest" based smoke testing coverage in our CI, those fake k230 issues can be prevented earlier.

ostest would probably need some work to be useful in this case: ostest 
would need some kind of standard reporting in each test so that we would 
know when a test case passed or failed.

Right now, many tests do not report any test results.  That is because 
they run relatively complex, multi-threaded scenarios that are not easy 
to assess.  You have to understand what the test is doing in order to 
interpret the test output.  Such tests would need to report failures in 
a way that can be captured as a simple PASS or FAIL.