You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nuttx.apache.org by Matteo Golin <ma...@gmail.com> on 2025/12/21 19:36:29 UTC
OS Test re-factor proposal
Hello everyone,
I wanted to get some feedback on my proposal for re-factor of OS test: https://github.com/apache/nuttx-apps/issues/3258
Since it's the holidays and I have some time on my hands, I figured that OS test could use some improvements, especially
since it is the primary test used to check for regressions/correct functionality of kernel logic on NuttX.
My proposal basically aims to use the Unity test framework to organize all of the existing tests in OS test so that
there is a logical code structure, and also logical console output which makes it much clearer to tell if a test has
passed/failed. The goal is that a) this should make reviewing test results much easier in PRs and b) the improved code
readability/extendability should make it easier for contributors to identify gaps in the testing and also add their own
suggested tests.
I have used the Unity support on NuttX for embedded testing for rocketry and really enjoyed it; I think it would be a
huge improvement for NuttX.
If this refactor is accepted by the community, once it is done I plan on completing more documentation for the OS test
on the NuttX website. There is currently a lack of information about all of the test suites that exist for the OS and
what exactly is being verified.
Please let me know what you think!
--
Matteo Golin
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
Yes, I believe so. It is currently integrated in nuttx-apps and it doesn't
look like it depends on any of the license switches.
Matteo
On Sun, Dec 21, 2025 at 6:07 PM Tomek CEDRO <to...@cedro.info> wrote:
> Cmocka tests seems to be mostly used and developed by Xiaomi. Maybe
> they use it internally only for runtime tests aisde from ostest? No
> clue too, I haven't played that yet :-)
>
> Last question about Unity - it uses MIT license so there is no
> conflict with Apache and we can include that code safely in the NuttX
> Apps code base without special switches?
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>
> On Sun, Dec 21, 2025 at 11:44 PM Matteo Golin <ma...@gmail.com>
> wrote:
> >
> > It also appears that "testsuites" (
> >
> https://github.com/apache/nuttx-apps/tree/c4da1b5b9f3c6ab24014f919320bad8a5ee845d7/testing/testsuites
> )
> > uses cmocka. I've never actually seen these tests run for PRs but they
> look
> > quite comprehensive? Maybe it is better to be consistent and use cmocka
> > instead of Unity, although I still think Unity is much more lightweight.
> >
> > How come we haven't added more to the testsuites? Such as the new hrtimer
> > tests? Are these for something specific only? I might be out of the loop.
> >
> > Matteo
> >
> > On Sun, Dec 21, 2025 at 5:34 PM Matteo Golin <ma...@gmail.com>
> wrote:
> >
> > > Okay, after a limited time looking at the cmocka documentation
> > > <https://api.cmocka.org/index.html#main-features> to compare it
> against
> > > Unity, I have determined the following:
> > >
> > > - cmocka is more geared towards mocking interfaces/C objects
> > > - cmocka seems to be less light-weight, with a lot of abstractions for
> > > mocking and the ability to output test results in a bunch of formats,
> > > including XML and JUnit
> > >
> > > I think Unity is probably the framework to move forward with in this
> case
> > > since it's lighter-weight, and I don't think our test cases in OS test
> > > would really benefit from the mocking features of cmocka. It seems more
> > > geared towards a simplistic approach to unit testing, and I think it
> would
> > > also be more flexible. I agree though that whatever proof of concept I
> put
> > > forward should have some memory usage comparison with writing the
> test-case
> > > without the Unity tools. If there is too significant overhead we might
> need
> > > a different solution.
> > >
> > > By the way, for anyone interested, the Unity framework has this basic
> > > webpage with some information: https://www.throwtheswitch.org/unity
> > > Most of their docs are on the GitHub page though:
> > > https://github.com/ThrowTheSwitch/Unity
> > >
> > > Matteo
> > >
> > > On Sun, Dec 21, 2025 at 4:43 PM Matteo Golin <ma...@gmail.com>
> > > wrote:
> > >
> > >> I don't think Unity should take much extra memory to run, since most
> of
> > >> the test infrastructure is very simple assertions/console output
> macros,
> > >> but I agree that I will need to compare the memory usage. I don't
> want to
> > >> compromise any flexibility of OS test since it is meant to run on all
> of
> > >> the boards (and new, low memory devices for board port tests).
> > >>
> > >> Matteo
> > >>
> > >> On Sun, Dec 21, 2025, 4:32 PM Alan C. Assis <ac...@gmail.com>
> wrote:
> > >>
> > >>> Hi Matteo,
> > >>> that is a great idea! This is something that I always missed on
> ostest:
> > >>> just say which test it is executing and if it passed or failed (just
> like
> > >>> the CONFIG_TESTING_LTP does).
> > >>>
> > >>> Although initially using Unity seems cool, I think it could require
> more
> > >>> memory to get it running inside a microcontroller with few KBs of
> > >>> RAM/Flash
> > >>> where ostest currently can run (search for ostest board profile
> inside
> > >>> boards to see some of those).
> > >>>
> > >>> Maybe since we know the expected result, we just need to include a
> simple
> > >>> comparison test and print the result as FAILED or PASSED (of course,
> some
> > >>> tests are more complex).
> > >>>
> > >>> Or maybe Unity will not require much memory as I'm thinking. So, it
> is
> > >>> important to get these numbers to make the right decision.
> > >>>
> > >>> BR,
> > >>>
> > >>> Alan
> > >>>
> > >>> On Sun, Dec 21, 2025 at 4:38 PM Matteo Golin <matteo.golin@gmail.com
> >
> > >>> wrote:
> > >>>
> > >>> > Hello everyone,
> > >>> >
> > >>> > I wanted to get some feedback on my proposal for re-factor of OS
> test:
> > >>> > https://github.com/apache/nuttx-apps/issues/3258
> > >>> >
> > >>> > Since it's the holidays and I have some time on my hands, I figured
> > >>> that
> > >>> > OS test could use some improvements, especially
> > >>> > since it is the primary test used to check for regressions/correct
> > >>> > functionality of kernel logic on NuttX.
> > >>> >
> > >>> > My proposal basically aims to use the Unity test framework to
> organize
> > >>> all
> > >>> > of the existing tests in OS test so that
> > >>> > there is a logical code structure, and also logical console output
> > >>> which
> > >>> > makes it much clearer to tell if a test has
> > >>> > passed/failed. The goal is that a) this should make reviewing test
> > >>> results
> > >>> > much easier in PRs and b) the improved code
> > >>> > readability/extendability should make it easier for contributors to
> > >>> > identify gaps in the testing and also add their own
> > >>> > suggested tests.
> > >>> >
> > >>> > I have used the Unity support on NuttX for embedded testing for
> > >>> rocketry
> > >>> > and really enjoyed it; I think it would be a
> > >>> > huge improvement for NuttX.
> > >>> >
> > >>> > If this refactor is accepted by the community, once it is done I
> plan
> > >>> on
> > >>> > completing more documentation for the OS test
> > >>> > on the NuttX website. There is currently a lack of information
> about
> > >>> all
> > >>> > of the test suites that exist for the OS and
> > >>> > what exactly is being verified.
> > >>> >
> > >>> > Please let me know what you think!
> > >>> >
> > >>> > --
> > >>> > Matteo Golin
> > >>> >
> > >>>
> > >>
>
Re: OS Test re-factor proposal
Posted by Tomek CEDRO <to...@cedro.info>.
Cmocka tests seems to be mostly used and developed by Xiaomi. Maybe
they use it internally only for runtime tests aisde from ostest? No
clue too, I haven't played that yet :-)
Last question about Unity - it uses MIT license so there is no
conflict with Apache and we can include that code safely in the NuttX
Apps code base without special switches?
--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
On Sun, Dec 21, 2025 at 11:44 PM Matteo Golin <ma...@gmail.com> wrote:
>
> It also appears that "testsuites" (
> https://github.com/apache/nuttx-apps/tree/c4da1b5b9f3c6ab24014f919320bad8a5ee845d7/testing/testsuites)
> uses cmocka. I've never actually seen these tests run for PRs but they look
> quite comprehensive? Maybe it is better to be consistent and use cmocka
> instead of Unity, although I still think Unity is much more lightweight.
>
> How come we haven't added more to the testsuites? Such as the new hrtimer
> tests? Are these for something specific only? I might be out of the loop.
>
> Matteo
>
> On Sun, Dec 21, 2025 at 5:34 PM Matteo Golin <ma...@gmail.com> wrote:
>
> > Okay, after a limited time looking at the cmocka documentation
> > <https://api.cmocka.org/index.html#main-features> to compare it against
> > Unity, I have determined the following:
> >
> > - cmocka is more geared towards mocking interfaces/C objects
> > - cmocka seems to be less light-weight, with a lot of abstractions for
> > mocking and the ability to output test results in a bunch of formats,
> > including XML and JUnit
> >
> > I think Unity is probably the framework to move forward with in this case
> > since it's lighter-weight, and I don't think our test cases in OS test
> > would really benefit from the mocking features of cmocka. It seems more
> > geared towards a simplistic approach to unit testing, and I think it would
> > also be more flexible. I agree though that whatever proof of concept I put
> > forward should have some memory usage comparison with writing the test-case
> > without the Unity tools. If there is too significant overhead we might need
> > a different solution.
> >
> > By the way, for anyone interested, the Unity framework has this basic
> > webpage with some information: https://www.throwtheswitch.org/unity
> > Most of their docs are on the GitHub page though:
> > https://github.com/ThrowTheSwitch/Unity
> >
> > Matteo
> >
> > On Sun, Dec 21, 2025 at 4:43 PM Matteo Golin <ma...@gmail.com>
> > wrote:
> >
> >> I don't think Unity should take much extra memory to run, since most of
> >> the test infrastructure is very simple assertions/console output macros,
> >> but I agree that I will need to compare the memory usage. I don't want to
> >> compromise any flexibility of OS test since it is meant to run on all of
> >> the boards (and new, low memory devices for board port tests).
> >>
> >> Matteo
> >>
> >> On Sun, Dec 21, 2025, 4:32 PM Alan C. Assis <ac...@gmail.com> wrote:
> >>
> >>> Hi Matteo,
> >>> that is a great idea! This is something that I always missed on ostest:
> >>> just say which test it is executing and if it passed or failed (just like
> >>> the CONFIG_TESTING_LTP does).
> >>>
> >>> Although initially using Unity seems cool, I think it could require more
> >>> memory to get it running inside a microcontroller with few KBs of
> >>> RAM/Flash
> >>> where ostest currently can run (search for ostest board profile inside
> >>> boards to see some of those).
> >>>
> >>> Maybe since we know the expected result, we just need to include a simple
> >>> comparison test and print the result as FAILED or PASSED (of course, some
> >>> tests are more complex).
> >>>
> >>> Or maybe Unity will not require much memory as I'm thinking. So, it is
> >>> important to get these numbers to make the right decision.
> >>>
> >>> BR,
> >>>
> >>> Alan
> >>>
> >>> On Sun, Dec 21, 2025 at 4:38 PM Matteo Golin <ma...@gmail.com>
> >>> wrote:
> >>>
> >>> > Hello everyone,
> >>> >
> >>> > I wanted to get some feedback on my proposal for re-factor of OS test:
> >>> > https://github.com/apache/nuttx-apps/issues/3258
> >>> >
> >>> > Since it's the holidays and I have some time on my hands, I figured
> >>> that
> >>> > OS test could use some improvements, especially
> >>> > since it is the primary test used to check for regressions/correct
> >>> > functionality of kernel logic on NuttX.
> >>> >
> >>> > My proposal basically aims to use the Unity test framework to organize
> >>> all
> >>> > of the existing tests in OS test so that
> >>> > there is a logical code structure, and also logical console output
> >>> which
> >>> > makes it much clearer to tell if a test has
> >>> > passed/failed. The goal is that a) this should make reviewing test
> >>> results
> >>> > much easier in PRs and b) the improved code
> >>> > readability/extendability should make it easier for contributors to
> >>> > identify gaps in the testing and also add their own
> >>> > suggested tests.
> >>> >
> >>> > I have used the Unity support on NuttX for embedded testing for
> >>> rocketry
> >>> > and really enjoyed it; I think it would be a
> >>> > huge improvement for NuttX.
> >>> >
> >>> > If this refactor is accepted by the community, once it is done I plan
> >>> on
> >>> > completing more documentation for the OS test
> >>> > on the NuttX website. There is currently a lack of information about
> >>> all
> >>> > of the test suites that exist for the OS and
> >>> > what exactly is being verified.
> >>> >
> >>> > Please let me know what you think!
> >>> >
> >>> > --
> >>> > Matteo Golin
> >>> >
> >>>
> >>
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
It also appears that "testsuites" (
https://github.com/apache/nuttx-apps/tree/c4da1b5b9f3c6ab24014f919320bad8a5ee845d7/testing/testsuites)
uses cmocka. I've never actually seen these tests run for PRs but they look
quite comprehensive? Maybe it is better to be consistent and use cmocka
instead of Unity, although I still think Unity is much more lightweight.
How come we haven't added more to the testsuites? Such as the new hrtimer
tests? Are these for something specific only? I might be out of the loop.
Matteo
On Sun, Dec 21, 2025 at 5:34 PM Matteo Golin <ma...@gmail.com> wrote:
> Okay, after a limited time looking at the cmocka documentation
> <https://api.cmocka.org/index.html#main-features> to compare it against
> Unity, I have determined the following:
>
> - cmocka is more geared towards mocking interfaces/C objects
> - cmocka seems to be less light-weight, with a lot of abstractions for
> mocking and the ability to output test results in a bunch of formats,
> including XML and JUnit
>
> I think Unity is probably the framework to move forward with in this case
> since it's lighter-weight, and I don't think our test cases in OS test
> would really benefit from the mocking features of cmocka. It seems more
> geared towards a simplistic approach to unit testing, and I think it would
> also be more flexible. I agree though that whatever proof of concept I put
> forward should have some memory usage comparison with writing the test-case
> without the Unity tools. If there is too significant overhead we might need
> a different solution.
>
> By the way, for anyone interested, the Unity framework has this basic
> webpage with some information: https://www.throwtheswitch.org/unity
> Most of their docs are on the GitHub page though:
> https://github.com/ThrowTheSwitch/Unity
>
> Matteo
>
> On Sun, Dec 21, 2025 at 4:43 PM Matteo Golin <ma...@gmail.com>
> wrote:
>
>> I don't think Unity should take much extra memory to run, since most of
>> the test infrastructure is very simple assertions/console output macros,
>> but I agree that I will need to compare the memory usage. I don't want to
>> compromise any flexibility of OS test since it is meant to run on all of
>> the boards (and new, low memory devices for board port tests).
>>
>> Matteo
>>
>> On Sun, Dec 21, 2025, 4:32 PM Alan C. Assis <ac...@gmail.com> wrote:
>>
>>> Hi Matteo,
>>> that is a great idea! This is something that I always missed on ostest:
>>> just say which test it is executing and if it passed or failed (just like
>>> the CONFIG_TESTING_LTP does).
>>>
>>> Although initially using Unity seems cool, I think it could require more
>>> memory to get it running inside a microcontroller with few KBs of
>>> RAM/Flash
>>> where ostest currently can run (search for ostest board profile inside
>>> boards to see some of those).
>>>
>>> Maybe since we know the expected result, we just need to include a simple
>>> comparison test and print the result as FAILED or PASSED (of course, some
>>> tests are more complex).
>>>
>>> Or maybe Unity will not require much memory as I'm thinking. So, it is
>>> important to get these numbers to make the right decision.
>>>
>>> BR,
>>>
>>> Alan
>>>
>>> On Sun, Dec 21, 2025 at 4:38 PM Matteo Golin <ma...@gmail.com>
>>> wrote:
>>>
>>> > Hello everyone,
>>> >
>>> > I wanted to get some feedback on my proposal for re-factor of OS test:
>>> > https://github.com/apache/nuttx-apps/issues/3258
>>> >
>>> > Since it's the holidays and I have some time on my hands, I figured
>>> that
>>> > OS test could use some improvements, especially
>>> > since it is the primary test used to check for regressions/correct
>>> > functionality of kernel logic on NuttX.
>>> >
>>> > My proposal basically aims to use the Unity test framework to organize
>>> all
>>> > of the existing tests in OS test so that
>>> > there is a logical code structure, and also logical console output
>>> which
>>> > makes it much clearer to tell if a test has
>>> > passed/failed. The goal is that a) this should make reviewing test
>>> results
>>> > much easier in PRs and b) the improved code
>>> > readability/extendability should make it easier for contributors to
>>> > identify gaps in the testing and also add their own
>>> > suggested tests.
>>> >
>>> > I have used the Unity support on NuttX for embedded testing for
>>> rocketry
>>> > and really enjoyed it; I think it would be a
>>> > huge improvement for NuttX.
>>> >
>>> > If this refactor is accepted by the community, once it is done I plan
>>> on
>>> > completing more documentation for the OS test
>>> > on the NuttX website. There is currently a lack of information about
>>> all
>>> > of the test suites that exist for the OS and
>>> > what exactly is being verified.
>>> >
>>> > Please let me know what you think!
>>> >
>>> > --
>>> > Matteo Golin
>>> >
>>>
>>
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
Okay, after a limited time looking at the cmocka documentation
<https://api.cmocka.org/index.html#main-features> to compare it against
Unity, I have determined the following:
- cmocka is more geared towards mocking interfaces/C objects
- cmocka seems to be less light-weight, with a lot of abstractions for
mocking and the ability to output test results in a bunch of formats,
including XML and JUnit
I think Unity is probably the framework to move forward with in this case
since it's lighter-weight, and I don't think our test cases in OS test
would really benefit from the mocking features of cmocka. It seems more
geared towards a simplistic approach to unit testing, and I think it would
also be more flexible. I agree though that whatever proof of concept I put
forward should have some memory usage comparison with writing the test-case
without the Unity tools. If there is too significant overhead we might need
a different solution.
By the way, for anyone interested, the Unity framework has this basic
webpage with some information: https://www.throwtheswitch.org/unity
Most of their docs are on the GitHub page though:
https://github.com/ThrowTheSwitch/Unity
Matteo
On Sun, Dec 21, 2025 at 4:43 PM Matteo Golin <ma...@gmail.com> wrote:
> I don't think Unity should take much extra memory to run, since most of
> the test infrastructure is very simple assertions/console output macros,
> but I agree that I will need to compare the memory usage. I don't want to
> compromise any flexibility of OS test since it is meant to run on all of
> the boards (and new, low memory devices for board port tests).
>
> Matteo
>
> On Sun, Dec 21, 2025, 4:32 PM Alan C. Assis <ac...@gmail.com> wrote:
>
>> Hi Matteo,
>> that is a great idea! This is something that I always missed on ostest:
>> just say which test it is executing and if it passed or failed (just like
>> the CONFIG_TESTING_LTP does).
>>
>> Although initially using Unity seems cool, I think it could require more
>> memory to get it running inside a microcontroller with few KBs of
>> RAM/Flash
>> where ostest currently can run (search for ostest board profile inside
>> boards to see some of those).
>>
>> Maybe since we know the expected result, we just need to include a simple
>> comparison test and print the result as FAILED or PASSED (of course, some
>> tests are more complex).
>>
>> Or maybe Unity will not require much memory as I'm thinking. So, it is
>> important to get these numbers to make the right decision.
>>
>> BR,
>>
>> Alan
>>
>> On Sun, Dec 21, 2025 at 4:38 PM Matteo Golin <ma...@gmail.com>
>> wrote:
>>
>> > Hello everyone,
>> >
>> > I wanted to get some feedback on my proposal for re-factor of OS test:
>> > https://github.com/apache/nuttx-apps/issues/3258
>> >
>> > Since it's the holidays and I have some time on my hands, I figured that
>> > OS test could use some improvements, especially
>> > since it is the primary test used to check for regressions/correct
>> > functionality of kernel logic on NuttX.
>> >
>> > My proposal basically aims to use the Unity test framework to organize
>> all
>> > of the existing tests in OS test so that
>> > there is a logical code structure, and also logical console output which
>> > makes it much clearer to tell if a test has
>> > passed/failed. The goal is that a) this should make reviewing test
>> results
>> > much easier in PRs and b) the improved code
>> > readability/extendability should make it easier for contributors to
>> > identify gaps in the testing and also add their own
>> > suggested tests.
>> >
>> > I have used the Unity support on NuttX for embedded testing for rocketry
>> > and really enjoyed it; I think it would be a
>> > huge improvement for NuttX.
>> >
>> > If this refactor is accepted by the community, once it is done I plan on
>> > completing more documentation for the OS test
>> > on the NuttX website. There is currently a lack of information about all
>> > of the test suites that exist for the OS and
>> > what exactly is being verified.
>> >
>> > Please let me know what you think!
>> >
>> > --
>> > Matteo Golin
>> >
>>
>
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
I don't think Unity should take much extra memory to run, since most of the
test infrastructure is very simple assertions/console output macros, but I
agree that I will need to compare the memory usage. I don't want to
compromise any flexibility of OS test since it is meant to run on all of
the boards (and new, low memory devices for board port tests).
Matteo
On Sun, Dec 21, 2025, 4:32 PM Alan C. Assis <ac...@gmail.com> wrote:
> Hi Matteo,
> that is a great idea! This is something that I always missed on ostest:
> just say which test it is executing and if it passed or failed (just like
> the CONFIG_TESTING_LTP does).
>
> Although initially using Unity seems cool, I think it could require more
> memory to get it running inside a microcontroller with few KBs of RAM/Flash
> where ostest currently can run (search for ostest board profile inside
> boards to see some of those).
>
> Maybe since we know the expected result, we just need to include a simple
> comparison test and print the result as FAILED or PASSED (of course, some
> tests are more complex).
>
> Or maybe Unity will not require much memory as I'm thinking. So, it is
> important to get these numbers to make the right decision.
>
> BR,
>
> Alan
>
> On Sun, Dec 21, 2025 at 4:38 PM Matteo Golin <ma...@gmail.com>
> wrote:
>
> > Hello everyone,
> >
> > I wanted to get some feedback on my proposal for re-factor of OS test:
> > https://github.com/apache/nuttx-apps/issues/3258
> >
> > Since it's the holidays and I have some time on my hands, I figured that
> > OS test could use some improvements, especially
> > since it is the primary test used to check for regressions/correct
> > functionality of kernel logic on NuttX.
> >
> > My proposal basically aims to use the Unity test framework to organize
> all
> > of the existing tests in OS test so that
> > there is a logical code structure, and also logical console output which
> > makes it much clearer to tell if a test has
> > passed/failed. The goal is that a) this should make reviewing test
> results
> > much easier in PRs and b) the improved code
> > readability/extendability should make it easier for contributors to
> > identify gaps in the testing and also add their own
> > suggested tests.
> >
> > I have used the Unity support on NuttX for embedded testing for rocketry
> > and really enjoyed it; I think it would be a
> > huge improvement for NuttX.
> >
> > If this refactor is accepted by the community, once it is done I plan on
> > completing more documentation for the OS test
> > on the NuttX website. There is currently a lack of information about all
> > of the test suites that exist for the OS and
> > what exactly is being verified.
> >
> > Please let me know what you think!
> >
> > --
> > Matteo Golin
> >
>
Re: OS Test re-factor proposal
Posted by "Alan C. Assis" <ac...@gmail.com>.
Hi Matteo,
that is a great idea! This is something that I always missed on ostest:
just say which test it is executing and if it passed or failed (just like
the CONFIG_TESTING_LTP does).
Although initially using Unity seems cool, I think it could require more
memory to get it running inside a microcontroller with few KBs of RAM/Flash
where ostest currently can run (search for ostest board profile inside
boards to see some of those).
Maybe since we know the expected result, we just need to include a simple
comparison test and print the result as FAILED or PASSED (of course, some
tests are more complex).
Or maybe Unity will not require much memory as I'm thinking. So, it is
important to get these numbers to make the right decision.
BR,
Alan
On Sun, Dec 21, 2025 at 4:38 PM Matteo Golin <ma...@gmail.com> wrote:
> Hello everyone,
>
> I wanted to get some feedback on my proposal for re-factor of OS test:
> https://github.com/apache/nuttx-apps/issues/3258
>
> Since it's the holidays and I have some time on my hands, I figured that
> OS test could use some improvements, especially
> since it is the primary test used to check for regressions/correct
> functionality of kernel logic on NuttX.
>
> My proposal basically aims to use the Unity test framework to organize all
> of the existing tests in OS test so that
> there is a logical code structure, and also logical console output which
> makes it much clearer to tell if a test has
> passed/failed. The goal is that a) this should make reviewing test results
> much easier in PRs and b) the improved code
> readability/extendability should make it easier for contributors to
> identify gaps in the testing and also add their own
> suggested tests.
>
> I have used the Unity support on NuttX for embedded testing for rocketry
> and really enjoyed it; I think it would be a
> huge improvement for NuttX.
>
> If this refactor is accepted by the community, once it is done I plan on
> completing more documentation for the OS test
> on the NuttX website. There is currently a lack of information about all
> of the test suites that exist for the OS and
> what exactly is being verified.
>
> Please let me know what you think!
>
> --
> Matteo Golin
>
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
I'm actually not familiar with cmocka so I will take the time to compare
the two.
After looking at Unity docs, it is possible to ignore tests (skip), but I
don't think there is support for selectively running specific suites/test
cases. However, it's a C based framework so it should be possible within OS
test to use something like a command line argument to select the suites to
be run. Test cases selection would just have to be implemented by us.
Matteo
On Sun, Dec 21, 2025, 3:20 PM Tomek CEDRO <to...@cedro.info> wrote:
> Question 3 was about test-sets/suites that you already mentioned thanks!
> :-)
>
> Yes some simple existing ostest would be great as poc so we can compare :-)
>
> I can see cmocka framework already present in the nuttx-apps how would
> unity relate to that is it something different or similar? :-)
>
> https://github.com/apache/nuttx-apps/tree/master/testing/cmocka
>
> Thanks! :-)
> Tomek
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>
> On Sun, Dec 21, 2025 at 9:14 PM Matteo Golin <ma...@gmail.com>
> wrote:
> >
> > 1. From my understanding, very stable. I also believe that we fix the
> > version in NuttX so we'd be in control when/if we upgrade the version
> used.
> >
> > 2. Yes, it has PASS/FAIL for certain. I'm not sure about SKIP, but I can
> > check that. I also seem to remember a feature where you can choose to run
> > single tests, suites or all test cases with names matching a certain
> > pattern, but don't quote me on that. I will refer back to their docs and
> > send an update here for those questions.
> >
> > 3. Can you clarify what you mean?
> >
> > I could definitely make a proof of concept to demonstrate first. Would
> you
> > suggest just a dummy application with fake tests, or creating a new demo
> > app where a few test cases from OS test are adapted to use Unity?
> >
> > Matteo
> >
> > On Sun, Dec 21, 2025, 3:06 PM Tomek CEDRO <to...@cedro.info> wrote:
> >
> > > Would it be possible to create some proof-of-concept to show how it
> > > would look and work? :-)
> > >
> > > --
> > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > >
> > > On Sun, Dec 21, 2025 at 8:58 PM Tomek CEDRO <to...@cedro.info> wrote:
> > > >
> > > > Sounds great Linguini :-)
> > > >
> > > > 1. How stable is the API of that Unity framework? Can this impact
> > > > self-containment or long term self compatibility problems?
> > > >
> > > > 2. Would that allow marking status of each step like FAIL, PASS,
> SKIP,
> > > > ERROR? We miss that now, only errors are reported but
> > > >
> > > > 3. Can this enable test sets?
> > > >
> > > > Thanks! :-)
> > > > Tomek
> > > >
> > > > --
> > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > >
> > > > On Sun, Dec 21, 2025 at 8:38 PM Matteo Golin <matteo.golin@gmail.com
> >
> > > wrote:
> > > > >
> > > > > Hello everyone,
> > > > >
> > > > > I wanted to get some feedback on my proposal for re-factor of OS
> test:
> > > https://github.com/apache/nuttx-apps/issues/3258
> > > > >
> > > > > Since it's the holidays and I have some time on my hands, I figured
> > > that OS test could use some improvements, especially
> > > > > since it is the primary test used to check for regressions/correct
> > > functionality of kernel logic on NuttX.
> > > > >
> > > > > My proposal basically aims to use the Unity test framework to
> organize
> > > all of the existing tests in OS test so that
> > > > > there is a logical code structure, and also logical console output
> > > which makes it much clearer to tell if a test has
> > > > > passed/failed. The goal is that a) this should make reviewing test
> > > results much easier in PRs and b) the improved code
> > > > > readability/extendability should make it easier for contributors to
> > > identify gaps in the testing and also add their own
> > > > > suggested tests.
> > > > >
> > > > > I have used the Unity support on NuttX for embedded testing for
> > > rocketry and really enjoyed it; I think it would be a
> > > > > huge improvement for NuttX.
> > > > >
> > > > > If this refactor is accepted by the community, once it is done I
> plan
> > > on completing more documentation for the OS test
> > > > > on the NuttX website. There is currently a lack of information
> about
> > > all of the test suites that exist for the OS and
> > > > > what exactly is being verified.
> > > > >
> > > > > Please let me know what you think!
> > > > >
> > > > > --
> > > > > Matteo Golin
> > >
>
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
Okay, I have re-written some of the tests with cmocka and my observations
are:
- cmocka has less `assert` macros than Unity to work with
- The output from cmocka is a little messier to look at than Unity, perhaps
that can be tweaked (?)
- The output from cmocka doesn't include plain-english descriptions like
"expected _, was _", but maybe this is also a good thing to cut down on
strings taking up memory
- cmocka test collections are much nicer to use, they allow bundling of
tests into groups with per-test setup and teardown functions
- cmocka test groups can be named and are displayed under a name in the
output, as opposed to Unity's large blob of tests
All in all, I think cmocka is a good solution too, and it honestly might be
easier to manage large amounts of tests with it. I haven't done a size
comparison yet, but now I am starting to think that cmocka might be better
suited for this than Unity, even if it has a smaller user base.
Also, while compiling for cmocka, I noticed that it includes a `cmocka`
binary which looks to be a template for setting up a test-runner
command-line tool. This isn't necessary for creating other cmocka test
applications. So, before I move further with this, I will submit a patch to
allow disabling the compilation of this cmocka binary to remove that extra
bit of dependency.
Matteo
On Mon, Dec 22, 2025 at 11:00 PM Matteo Golin <ma...@gmail.com>
wrote:
> Okay this is great information! And I learned a new testing method today.
> I will play with cmocka then since it seems like it could also work well
> for the smaller NuttX devices.
>
> Thank you Xiang!
>
> On Mon, Dec 22, 2025, 10:57 PM Xiang Xiao <xi...@gmail.com>
> wrote:
>
>> On Tue, Dec 23, 2025 at 10:29 AM Matteo Golin <ma...@gmail.com>
>> wrote:
>>
>> > Thank you Xiang for the information! It makes sense of course that
>> you're
>> > not able to share the Xiaomi internal test suites.
>> >
>> >
>> Actually, raiden00pl is working on upstreaming the part of the test suite
>> which tests NuttX's components to the NuttX community.
>> The part we can't share is that:
>>
>> 1. The test suite specific for openvela framework
>> 2. Internal CI/CD infrastructure which is couple with our workflow and
>> very different from github
>>
>>
>>
>> > Are there any modifications Xiaomi has needed to make to cmocka to
>> improve
>> > it's ability to run on small devices? I know some of your targets are
>> > wearables which I imagine are very resource constrained?
>> >
>> >
>> No, we don't do any size optimization for cmocka since it is already a
>> small unit test framework containing one .h and one .c.
>> We run many test cases(including cmocka's) on the smallest BLE only module
>> which has 256KB SRAM and 2MB flash.
>>
>>
>>
>> > Can you clarify what you mean by monkey test? Is that another test
>> > framework? I'm interested since you mentioned it running on watches and
>> > speakers.
>> >
>> >
>> Here is an explanation from wiki:
>> https://en.wikipedia.org/wiki/Monkey_testing
>> https://en.wikipedia.org/wiki/Mean_time_between_failures
>> Here is the our monkey test requirement:
>>
>> 1. Boot 50 instances for each simulator/qemu/hardware with the product
>> config
>> 2. Generate the random input quickly through the fake drivers(e.g.
>> touch/button/audio...)
>>
>> All devices need to run beyond 24 hours without crash or hang for three
>> consecutive days.
>>
>> Thanks,
>> > Matteo
>> >
>> > On Mon, Dec 22, 2025, 9:08 PM Xiang Xiao <xi...@gmail.com>
>> > wrote:
>> >
>> > > On Tue, Dec 23, 2025 at 1:08 AM Matteo Golin <ma...@gmail.com>
>> > > wrote:
>> > >
>> > > > Hello everyone,
>> > > >
>> > > > Thanks for all the feedback! I'm glad I discussed this here first.
>> I'll
>> > > try
>> > > > to address everything that was brought up.
>> > > >
>> > > > Thanks Guiding for the input! I must just be missing some library
>> on my
>> > > > machine if you're able to compile it just fine. Maybe I'll track
>> down
>> > > that
>> > > > error and see if you can recognize what's wrong.
>> > > >
>> > > > I don't intend to replace OS test anymore. I think it serves its
>> > purpose
>> > > as
>> > > > a minimal sanity check. Instead, I think we need something a little
>> > more
>> > > > involved to test PRs since OS test can't catch many issues. It is
>> > > currently
>> > > > the standard everyone uses for verifying their patches and it would
>> be
>> > > > better to have more complete, alternate options. I do think that we
>> can
>> > > all
>> > > > agree that OS test needs a revision of its console output so that
>> it is
>> > > > much easier to see pass/fail results, but that is just a matter of
>> > print
>> > > > statements and can be done in a different effort.
>> > > >
>> > > > I think the overwhelming concern here is mixing two unit testing
>> > > frameworks
>> > > > and no longer having one resource where we can go for tests. I share
>> > > raiden
>> > > > and Guiding's concern about that, which is why I'm glad I opened for
>> > > > discussion here. On the one hand, Xiaomi has a pretty substantial
>> > > > collection of tests in the `testsuite` application which they are
>> using
>> > > > internally I believe (in their CI). These run well on virtual
>> machines,
>> > > > which I assume is why the only configurations with that test suite
>> > > included
>> > > > is sim and rv-virt. I do not know nearly enough about cmocka to make
>> > any
>> > > > claim that it *wouldn't* work on smaller hardware devices, but it
>> also
>> > > > definitely looks to be less light-weight from the feature set in the
>> > > > documentation. On the other hand, the Xiaomi test suites are also
>> not a
>> > > > complete silver bullet since they seem not to be intended to run on
>> > > > hardware, and there are *a lot *of tests that need hardware
>> > verification.
>> > > >
>> > >
>> > > The test suite also runs on hardware daily, but sorry it's impossible
>> to
>> > > share our hardware and internal CI/CD infrastructure for the
>> community.
>> > > But, we already contract raiden00pl to refactor our test automation
>> > > framework for public use.
>> > >
>> > >
>> > > > In PRs we often request more than just a simulator test for certain
>> > > > changes. For instance, sim isn't able to have `/dev/rand`, so that
>> > needs
>> > > to
>> > > > be tested on a different target.
>> > > >
>> > >
>> > > Maybe someone more familiar with Xiaomi's testing practices can weigh
>> in
>> > on
>> > > > whether cmocka has been run on hardware. Going purely based off of
>> > their
>> > > > PRs, which almost exclusively cite tests on simulators, my guess is
>> > that
>> > > > cmocka was not designed with embedded in mind when it was written.
>> > > >
>> > > >
>> > > CI/CD in Xiaomi is far more complex than your description:
>> > >
>> > > 1. Each patch will test on 10+ devices(include sim/qemu/hardware,
>> > > arm/arm64/riscv/xtensa/x64, smp/up) with 1000+ test case
>> > > 2. Run the full test suite(10000+ test case) every day
>> > > 3. Run the monkey test every day on watch and speaker product:
>> 24hour
>> > X
>> > > 50 instance X (sim + qemu + hardware)
>> > >
>> > > Many stability bugs only happen randomly in monkey test.
>> > >
>> > > As for size: unit testing can get large with Unity or cmocka (as
>> raiden
>> > > > mentioned) as more tests are added. Hence my effort to
>> enable/disable
>> > > > individual test suites from getting compiled. It might not be
>> possible
>> > to
>> > > > ever get good testing ability for AVR just because the devices are
>> so
>> > > > small. Whatever solution is used there is probably going to involve
>> > > > flashing the device multiple times with different tests. The size
>> goal
>> > > > should just be to target the most NuttX devices possible.
>> > > >
>> > > > Then there are the pros and cons of each framework (again, I don't
>> know
>> > > > much about cmocka so take it with a grain of salt):
>> > > >
>> > > > Unity's cons:
>> > > > - Designed for each test-suite to be a standalone binary. This is
>> > clunky
>> > > on
>> > > > NuttX, so `nest` is currently implemented as a mega-suite where test
>> > > cases
>> > > > just follow a naming convention to make it clear what is being
>> tested
>> > > > - setUp/tearDown can't really be leveraged because of the above
>> > > > - I've noticed some funky behaviour using TEST_IGNORE in certain
>> > > scenarios
>> > > > but that's possibly my own configuration issue
>> > > > - Would be in conflict with existing cmocka tests
>> > > >
>> > > > Unity's pros:
>> > > > - Highly configurable, like NuttX
>> > > > - Light-weight and works for embedded devices
>> > > > - Significantly larger user base than cmocka
>> > > >
>> > > > Cmocka's cons:
>> > > > - Possibly (?) not light-weight
>> > > > - No documentation for existing test cases in NuttX
>> > > > - Small user base
>> > > >
>> > > > Cmocka's pros:
>> > > > - Seems to have a larger set of features, including setup/teardown
>> > > > functions per-test case
>> > > > - Existing tests in NuttX
>> > > >
>> > > >
>> > > Xiaomi selected cmocka just because it has the test suite concept
>> which
>> > > could help us manage 10000+ tests.
>> > > But we also use gtest for the c++ code base, and run many test cases
>> from
>> > > 3rd parties(e.g. LTP, libuv, libcxx testsuite, PSE52...) which may
>> use a
>> > > different test framework.
>> > > So, is it hard or meaningless to stick to one unit test framework for
>> an
>> > > open source project.
>> > >
>> > > Irrespective of whether we go with Unity or cmocka, the collection of
>> > tests
>> > > > that we write will be intended to run on the actual hardware devices
>> > > > themselves for the purposes of testing patches and doing regression
>> > > tests.
>> > > > As a result, the tests will be separate from the Xiaomi 'testsuite'
>> > > > application which is intended for CI testing. I think this
>> > realistically
>> > > > nullifies choosing which framework to use on the basis of
>> consistency,
>> > > > since the tests are not coexisting with the cmocka tests Xiaomi has
>> > > > written. Instead, the choice should be based on what best achieves
>> the
>> > > goal
>> > > > from a functionality perspective. Unity seems like a really great
>> > choice
>> > > > for this in my opinion, but I am bothered by the fact that it's
>> > designed
>> > > > with multiple test binaries in mind.
>> > > >
>> > > > Let me know what you think,
>> > > > Matteo
>> > > >
>> > > > On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <ac...@gmail.com>
>> > > wrote:
>> > > >
>> > > > > Yes, I agree! We need data size from each solution for comparison,
>> > this
>> > > > is
>> > > > > the first step.
>> > > > >
>> > > > > Maybe the conversion could be facilitated if we create equivalent
>> > > > functions
>> > > > > like those used in the apps/testing/testsuites:
>> > > > >
>> > > > > cmocka_run_group_tests -> unity_run_group_tests
>> > > > > cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown
>> > > > > etc...
>> > > > >
>> > > > > This way we avoid end up with two unit test framework, we already
>> > have
>> > > > the
>> > > > > Makefile vs CMake precedent! :-D
>> > > > >
>> > > > > On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <raiden00pl@gmail.com
>> >
>> > > > wrote:
>> > > > >
>> > > > > > The discussion about small systems doesn't make much sense
>> without
>> > > > > > data comparing resources of the same test cases for cmock and
>> > unity.
>> > > > > >
>> > > > > > But let's assume that Unity is better. Should we use two
>> frameworks
>> > > > > > to implement unit tests for OS? Or choose one framework? If we
>> > choose
>> > > > > only
>> > > > > > Unity, who
>> > > > > > will migrate the current cmocka tests to Unity? Remember that
>> cmoka
>> > > > tests
>> > > > > > are used in CI, so all CI tools must be also migrated.
>> > > > > > Or should we use two different frameworks to test different
>> things
>> > > and
>> > > > > make
>> > > > > > even
>> > > > > > more mess in apps/testing as it is now?
>> > > > > >
>> > > > > > No one is forcing anyone to do anything. This is an open-source
>> > > > project,
>> > > > > > everyone
>> > > > > > should do what they feel is right. But whether the change
>> should be
>> > > > > merged
>> > > > > > is
>> > > > > > a different question. In this case, it's a very important
>> question,
>> > > > > because
>> > > > > > unit testing
>> > > > > > isn't just about test cases. It's about the entire
>> infrastructure
>> > > that
>> > > > > > runs, captures and
>> > > > > > handles test results later. Writing the test cases is the least
>> of
>> > > the
>> > > > > > problems.
>> > > > > > The infrastructure for cmoka testing is already in place, the
>> > > > > > infrastructure for Unity
>> > > > > > testing isn't.
>> > > > > >
>> > > > > > pon., 22 gru 2025 o 14:16 Alan C. Assis <ac...@gmail.com>
>> > > > napisał(a):
>> > > > > >
>> > > > > > > Mateusz,
>> > > > > > > but some chips like AVR DA have big flash, but not so much
>> RAM.
>> > > > > > >
>> > > > > > > It is better to have some possible way to test on a small
>> system,
>> > > > using
>> > > > > > > cmocka we are removing this option from the table.
>> > > > > > >
>> > > > > > > I liked the test Matteo did with Unity and I think we should
>> let
>> > > him
>> > > > > > > continue. Forcing him to migrate to cmocka just because
>> there's a
>> > > > test
>> > > > > > > suite from Xiaomi doesn't seem prudent to me.
>> > > > > > >
>> > > > > > > This is the advantage of an open mind open-source project:
>> "the
>> > > > > > possibility
>> > > > > > > to test new ideas", if someone had said to Michal Lenc he
>> > couldn't
>> > > > > create
>> > > > > > > NXBoot because where is already MCUBoot integrated on NuttX,
>> we
>> > > > didn't
>> > > > > > have
>> > > > > > > the option to use NXBoot today. ;-)
>> > > > > > >
>> > > > > > > BR,
>> > > > > > >
>> > > > > > > Alan
>> > > > > > >
>> > > > > > > On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <
>> > raiden00pl@gmail.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > Unit testing by itself isn't suitable for small systems.
>> Ostest
>> > > is
>> > > > > > > > something
>> > > > > > > > different (system level testing), so running it on small
>> > systems
>> > > > > makes
>> > > > > > > > sense.
>> > > > > > > > Running unit tests on small systems is pointless because you
>> > > won't
>> > > > > fit
>> > > > > > in
>> > > > > > > > FLASH
>> > > > > > > > either way. You can run the same on large targets
>> (preferably
>> > on
>> > > > > host:
>> > > > > > > sim,
>> > > > > > > > qemu)
>> > > > > > > > and achieve the same result.
>> > > > > > > >
>> > > > > > > > pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com>
>> > > > > > napisał(a):
>> > > > > > > >
>> > > > > > > > > Is it too complicated to rewrite the cmocka tests in
>> Unity?
>> > > > > > > > >
>> > > > > > > > > If cmoka is not feasible for small microcontrollers, like
>> > AVRs
>> > > > and
>> > > > > > > > others,
>> > > > > > > > > our test coverage will be faulty.
>> > > > > > > > >
>> > > > > > > > > BR,
>> > > > > > > > >
>> > > > > > > > > Alan
>> > > > > > > > >
>> > > > > > > > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <
>> > > raiden00pl@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > I'm a fan of Unity and have been using it in my projects
>> > for
>> > > > > years,
>> > > > > > > > but I
>> > > > > > > > > > think
>> > > > > > > > > > we should stick with one testing framework. Currently,
>> > Cmocka
>> > > > is
>> > > > > > used
>> > > > > > > > in
>> > > > > > > > > > nuttx-apps, and we should stick with it. The only
>> question
>> > is
>> > > > how
>> > > > > > > well
>> > > > > > > > > > Cmocka
>> > > > > > > > > > is suitable for running on small systems? Will ostest
>> still
>> > > be
>> > > > > able
>> > > > > > > to
>> > > > > > > > > run
>> > > > > > > > > > on small MCUs?
>> > > > > > > > > > With Unity this would be quite likely, so maybe an
>> > exception
>> > > > for
>> > > > > > > Unity
>> > > > > > > > > and
>> > > > > > > > > > ostest would
>> > > > > > > > > > be acceptable, but using it for other test cases will
>> only
>> > > > cause
>> > > > > > > chaos.
>> > > > > > > > > >
>> > > > > > > > > > pon., 22 gru 2025 o 10:05 Guiding Li <ligdbyy@gmail.com
>> >
>> > > > > > napisał(a):
>> > > > > > > > > >
>> > > > > > > > > > > Hi Matteo,
>> > > > > > > > > > >
>> > > > > > > > > > > Good idea !
>> > > > > > > > > > >
>> > > > > > > > > > > But it is best to stick with a single testing
>> framework.
>> > > > > > Currently,
>> > > > > > > > > many
>> > > > > > > > > > > test cases are using cmocka, so I suggest reusing
>> cmocka
>> > > for
>> > > > > > > > > consistency.
>> > > > > > > > > > >
>> > > > > > > > > > > ostest is a minimal basic test suite, typically used
>> for
>> > CT
>> > > > > > > > (continuous
>> > > > > > > > > > > testing) of each patch.
>> > > > > > > > > > > If we move ostest to cmocka, it would make ostest
>> > dependent
>> > > > on
>> > > > > a
>> > > > > > > > > specific
>> > > > > > > > > > > framework before it can be run.
>> > > > > > > > > > > And It may increase the code size, which is not very
>> > > friendly
>> > > > > for
>> > > > > > > > > > > configurations with small flash memory.
>> > > > > > > > > > >
>> > > > > > > > > > > As Greg mentioned: the answer depends on a tradeoff
>> > between
>> > > > > > effort
>> > > > > > > > and
>> > > > > > > > > > test
>> > > > > > > > > > > coverage.
>> > > > > > > > > > >
>> > > > > > > > > > > We need to clarify whether we define ostest as a
>> > > lightweight,
>> > > > > > > > > > ready-to-run
>> > > > > > > > > > > test set for quick verification, or as part of the
>> full
>> > > test
>> > > > > > suite.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > The follow is my compiling for cmocka:
>> > > > > > > > > > >
>> > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$
>> > ./tools/configure.sh
>> > > > > > > sim:citest
>> > > > > > > > > > -j8*
>> > > > > > > > > > > Copy files
>> > > > > > > > > > > Select CONFIG_HOST_LINUX=y
>> > > > > > > > > > > Refreshing...
>> > > > > > > > > > > CP: arch/dummy/Kconfig to
>> > > > > > > > > > >
>> > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
>> > > > > > > > > > > CP: boards/dummy/Kconfig to
>> > > > > > > > > > >
>> > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
>> > > > > > > > > > > LN: include/arch to arch/sim/include
>> > > > > > > > > > > LN: drivers/platform to
>> > > > > > > > > /home/ligd/platform/mainline/nuttx/drivers/dummy
>> > > > > > > > > > > LN: arch/sim/src/chip to
>> > > > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
>> > > > > > > > > > > LN: arch/sim/src/board to
>> > > > > > > > > > >
>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
>> > > > > > > > > > > LN: include/arch/board to
>> > > > > > > > > > >
>> > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
>> > > > > > > > > > > LN: include/arch/chip to
>> > > > > > > > > > >
>> /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
>> > > > > > > > > > > LN: platform/board to
>> > > > > > > > /home/ligd/platform/mainline/apps/platform/dummy
>> > > > > > > > > > > mkkconfig in
>> /home/ligd/platform/mainline/apps/audioutils
>> > > > > > > > > > > mkkconfig in
>> /home/ligd/platform/mainline/apps/benchmarks
>> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/boot
>> > > > > > > > > > > mkkconfig in
>> /home/ligd/platform/mainline/apps/canutils
>> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto
>> > > > > > > > > > > mkkconfig in
>> /home/ligd/platform/mainline/apps/database
>> > > > > > > > > > > mkkconfig in
>> > > /home/ligd/platform/mainline/apps/examples/rust
>> > > > > > > > > > > mkkconfig in
>> > > > /home/ligd/platform/mainline/apps/examples/sotest
>> > > > > > > > > > > mkkconfig in
>> > > > /home/ligd/platform/mainline/apps/examples/module
>> > > > > > > > > > > mkkconfig in
>> > > > /home/ligd/platform/mainline/apps/examples/mcuboot
>> > > > > > > > > > > mkkconfig in
>> /home/ligd/platform/mainline/apps/examples
>> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
>> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/games
>> > > > > > > > > > > mkkconfig in
>> /home/ligd/platform/mainline/apps/graphics
>> > > > > > > > > > > mkkconfig in
>> /home/ligd/platform/mainline/apps/industry
>> > > > > > > > > > > mkkconfig in
>> /home/ligd/platform/mainline/apps/inertial
>> > > > > > > > > > > mkkconfig in
>> > > > > > > > /home/ligd/platform/mainline/apps/interpreters/luamodules
>> > > > > > > > > > > mkkconfig in
>> > /home/ligd/platform/mainline/apps/interpreters
>> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/logging
>> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/lte
>> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/math
>> > > > > > > > > > > mkkconfig in
>> /home/ligd/platform/mainline/apps/mlearning
>> > > > > > > > > > > mkkconfig in
>> /home/ligd/platform/mainline/apps/netutils
>> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr
>> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/system
>> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/tee
>> > > > > > > > > > > mkkconfig in
>> > > > /home/ligd/platform/mainline/apps/testing/drivers
>> > > > > > > > > > > mkkconfig in
>> > > /home/ligd/platform/mainline/apps/testing/sched
>> > > > > > > > > > > mkkconfig in
>> /home/ligd/platform/mainline/apps/testing/fs
>> > > > > > > > > > > mkkconfig in
>> /home/ligd/platform/mainline/apps/testing/mm
>> > > > > > > > > > > mkkconfig in
>> > /home/ligd/platform/mainline/apps/testing/libc
>> > > > > > > > > > > mkkconfig in
>> > /home/ligd/platform/mainline/apps/testing/arch
>> > > > > > > > > > > mkkconfig in
>> > /home/ligd/platform/mainline/apps/testing/cxx
>> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing
>> > > > > > > > > > > mkkconfig in
>> /home/ligd/platform/mainline/apps/videoutils
>> > > > > > > > > > > mkkconfig in
>> > > > > /home/ligd/platform/mainline/apps/wireless/bluetooth
>> > > > > > > > > > > mkkconfig in
>> > > > > > /home/ligd/platform/mainline/apps/wireless/ieee802154
>> > > > > > > > > > > mkkconfig in
>> /home/ligd/platform/mainline/apps/wireless
>> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps
>> > > > > > > > > > > Loaded configuration '.config'
>> > > > > > > > > > > Configuration saved to '.config'
>> > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
>> > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ make*
>> > > > > > > > > > > Create version.h
>> > > > > > > > > > > Downloading: libcxx-17.0.6.src.tar.xz LN:
>> platform/board
>> > to
>> > > > > > > > > > > /home/ligd/platform/mainline/apps/platform/dummy
>> > > > > > > > > > > Downloading: v1.46.0.zip
>> > > > > > > > > > > ...
>> > > > > > > > > > > patching file
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
>> > > > > > > > > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8
>> lines).
>> > > > > > > > > > >
>> > > > > > > > > > > CP:
>> > > > /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
>> > > > > > > > > > > CP:
>> > > > > /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
>> > > > > > > > > > > LD: nuttx
>> > > > > > > > > > > Pac SIM with dynamic libs..
>> > > > > > > > > > > '/lib/x86_64-linux-gnu/libz.so.1' ->
>> > > 'sim-pac/libs/libz.so.1'
>> > > > > > > > > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
>> > > > > > > > 'sim-pac/libs/libstdc++.so.6'
>> > > > > > > > > > > '/lib/x86_64-linux-gnu/libc.so.6' ->
>> > > 'sim-pac/libs/libc.so.6'
>> > > > > > > > > > > '/lib/x86_64-linux-gnu/libm.so.6' ->
>> > > 'sim-pac/libs/libm.so.6'
>> > > > > > > > > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' ->
>> > > > > > > 'sim-pac/libs/libgcc_s.so.1'
>> > > > > > > > > > > '/lib64/ld-linux-x86-64.so.2' ->
>> > > > 'sim-pac/ld-linux-x86-64.so.2'
>> > > > > > > > > > > SIM elf with dynamic libs archive in nuttx.tgz
>> > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
>> > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
>> > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
>> > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
>> > > > > > > > > > > CHelloWorld: Constructor: mSecret=42
>> > > > > > > > > > > nsh: init: open failed: 2
>> > > > > > > > > > >
>> > > > > > > > > > > NuttShell (NSH) NuttX-12.11.0
>> > > > > > > > > > > MOTD: username=admin password=Administrator
>> > > > > > > > > > > nsh>
>> > > > > > > > > > > nsh> help
>> > > > > > > > > > > help usage: help [-v] [<cmd>]
>> > > > > > > > > > >
>> > > > > > > > > > > . df ifconfig mkfifo
>> > > readlink
>> > > > > > > > umount
>> > > > > > > > > > > [ dmesg ifdown mkrd rm
>> > > > > > > unset
>> > > > > > > > > > > ? echo ifup mount
>> rmdir
>> > > > > > > > uptime
>> > > > > > > > > > > alias env insmod mv
>> rmmod
>> > > > > > > > usleep
>> > > > > > > > > > > unalias exec kill nslookup
>> set
>> > > > > > > watch
>> > > > > > > > > > > basename exit pkill pidof
>> sleep
>> > > > > > xd
>> > > > > > > > > > > break expr losetup pmconfig
>> > source
>> > > > > > > wait
>> > > > > > > > > > > cat false ln poweroff
>> test
>> > > > > > > > > > > cd fdinfo ls quit
>> time
>> > > > > > > > > > > cp free lsmod printf
>> true
>> > > > > > > > > > > cmp help mkdir ps
>> > > truncate
>> > > > > > > > > > > dirname hexdump mkfatfs pwd
>> uname
>> > > > > > > > > > >
>> > > > > > > > > > > Builtin Apps:
>> > > > > > > > > > > cmocka
>> > > > > > > > > > > cmocka_driver_block
>> > > > > > > > > > > cmocka_driver_gpio
>> > > > > > > > > > > cmocka_driver_pm
>> > > > > > > > > > > cmocka_driver_pm_runtime
>> > > > > > > > > > > cmocka_driver_regulator
>> > > > > > > > > > >
>> > > > > > > > > > > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一
>> > > > 10:15写道：
>> > > > > > > > > > >
>> > > > > > > > > > > > I think so. Best to leave OS test alone and slowly
>> > > replace
>> > > > it
>> > > > > > > with
>> > > > > > > > a
>> > > > > > > > > > new
>> > > > > > > > > > > > tool that has improvements. The only reason I can
>> see
>> > to
>> > > > > modify
>> > > > > > > OS
>> > > > > > > > > test
>> > > > > > > > > > > now
>> > > > > > > > > > > > would just be to have clearer output about what
>> > > > > passed/failed.
>> > > > > > > > > > > >
>> > > > > > > > > > > > I'll come up with an improved proof of concept for a
>> > new
>> > > > tool
>> > > > > > to
>> > > > > > > > > share
>> > > > > > > > > > in
>> > > > > > > > > > > > the next few days.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Matteo
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <
>> > > > tomek@cedro.info>
>> > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Small steps with measurable results may be the
>> best
>> > > > > approach
>> > > > > > > > here,
>> > > > > > > > > to
>> > > > > > > > > > > > > create some new solution next to existing ostest
>> but
>> > > not
>> > > > > > > > replacing
>> > > > > > > > > it
>> > > > > > > > > > > > > until ready? :-)
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > --
>> > > > > > > > > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
>> > > > > > > > > matteo.golin@gmail.com
>> > > > > > > > > > >
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I agree, I think it would take a long time to
>> have
>> > > > proper
>> > > > > > > > > coverage
>> > > > > > > > > > of
>> > > > > > > > > > > > > NuttX
>> > > > > > > > > > > > > > as a whole. Perhaps even impossible, since
>> Unity is
>> > > > more
>> > > > > > of a
>> > > > > > > > > unit
>> > > > > > > > > > > > > testing
>> > > > > > > > > > > > > > framework and thus there are definitely test
>> cases
>> > > for
>> > > > > > NuttX
>> > > > > > > > that
>> > > > > > > > > > > would
>> > > > > > > > > > > > > be
>> > > > > > > > > > > > > > outside of its realm of "applicability". There
>> is
>> > > > still a
>> > > > > > > need
>> > > > > > > > > for
>> > > > > > > > > > a
>> > > > > > > > > > > > > > variety of testing applications and
>> bench-marking
>> > > > > > > applications
>> > > > > > > > > for
>> > > > > > > > > > > > those
>> > > > > > > > > > > > > > scenarios.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > That makes sense, prerequisites would need to be
>> > > > > > considered.
>> > > > > > > I
>> > > > > > > > > > think
>> > > > > > > > > > > > > > targeting the low-hanging fruit of unit-testable
>> > code
>> > > > > would
>> > > > > > > be
>> > > > > > > > > easy
>> > > > > > > > > > > in
>> > > > > > > > > > > > > that
>> > > > > > > > > > > > > > case, since prerequisites are mostly encoded in
>> the
>> > > > > Kconfig
>> > > > > > > > > > > definitions
>> > > > > > > > > > > > > > existing or not for features to be tested. Any
>> > > complex
>> > > > > > > > behaviour
>> > > > > > > > > > > > > > (interrupts, multi-threading, etc.) might be
>> out of
>> > > the
>> > > > > > realm
>> > > > > > > > of
>> > > > > > > > > > > what's
>> > > > > > > > > > > > > > testable by Unity in the first place. We'll
>> always
>> > > have
>> > > > > OS
>> > > > > > > test
>> > > > > > > > > in
>> > > > > > > > > > > that
>> > > > > > > > > > > > > > case, but maybe then it's better to do what Alan
>> > > > > suggested
>> > > > > > > and
>> > > > > > > > at
>> > > > > > > > > > > least
>> > > > > > > > > > > > > > make its output more clear in the meantime. I
>> > suppose
>> > > > > RTOS
>> > > > > > > > > testing
>> > > > > > > > > > > is a
>> > > > > > > > > > > > > > whole field in itself.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
>> > > > > > > > > spudaneco@gmail.com>
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > I think that the answer depends on a tradeoff
>> > > between
>> > > > > > > effort
>> > > > > > > > > and
>> > > > > > > > > > > test
>> > > > > > > > > > > > > > > coverage. I could imagine that a proper job
>> > could
>> > > be
>> > > > > > > > man-years
>> > > > > > > > > > of
>> > > > > > > > > > > > > effort.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > My only concern with a re-design is to be
>> certain
>> > > > that
>> > > > > > > tests
>> > > > > > > > > are
>> > > > > > > > > > > > > > > arranged so that the prerequisites of later
>> test
>> > > are
>> > > > > > tested
>> > > > > > > > > > first.
>> > > > > > > > > > > > If
>> > > > > > > > > > > > > > > tests are grouped by, for example, functional
>> > area,
>> > > > > then
>> > > > > > > > > failures
>> > > > > > > > > > > can
>> > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > difficult to understand if there is a failure
>> in
>> > an
>> > > > > > > > unverified
>> > > > > > > > > > test
>> > > > > > > > > > > > > > > prerequisite.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Some of the "tests" are bogus and only exist
>> to
>> > > > provide
>> > > > > > > some
>> > > > > > > > > > > minimum
>> > > > > > > > > > > > > > > confidence that we have the functionality to
>> run
>> > > the
>> > > > > > test.
>> > > > > > > > For
>> > > > > > > > > > > > > > > example: Can I start tasks? Can I pass
>> > > parameters?
>> > > > > Can
>> > > > > > I
>> > > > > > > > > > survive
>> > > > > > > > > > > a
>> > > > > > > > > > > > > > > context switch. Some of the tests were
>> intended
>> > > only
>> > > > > to
>> > > > > > > > > support
>> > > > > > > > > > > > debug.
>> > > > > > > > > > > > > > > Some were created only to track down specific
>> > bugs.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote:
>> > > > > > > > > > > > > > > > LTS being the Long-Term-Stable release of
>> > NuttX?
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > I'm starting to think that it would be best
>> to
>> > > > create
>> > > > > > new
>> > > > > > > > > suite
>> > > > > > > > > > > of
>> > > > > > > > > > > > > tests
>> > > > > > > > > > > > > > > > using Unity, using what's in OS Test as a
>> basis
>> > > and
>> > > > > > then
>> > > > > > > > > > > expanding
>> > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > further. Greg has pointed out that OS test
>> is
>> > not
>> > > > > > > intended
>> > > > > > > > to
>> > > > > > > > > > be
>> > > > > > > > > > > a
>> > > > > > > > > > > > > > > complete
>> > > > > > > > > > > > > > > > test, so maybe a more solid testing
>> framework
>> > > base
>> > > > > in a
>> > > > > > > new
>> > > > > > > > > > form
>> > > > > > > > > > > > > would be
>> > > > > > > > > > > > > > > > in order. At that point it would also be
>> good
>> > to
>> > > > > > document
>> > > > > > > > and
>> > > > > > > > > > > > educate
>> > > > > > > > > > > > > > > users
>> > > > > > > > > > > > > > > > about other test options, since OS test is a
>> > > > current
>> > > > > > > > > favourite
>> > > > > > > > > > > for
>> > > > > > > > > > > > > > > showing
>> > > > > > > > > > > > > > > > PRs haven't broken anything. Maybe that's
>> not
>> > the
>> > > > > best
>> > > > > > > > choice
>> > > > > > > > > > > > anymore
>> > > > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > this information.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > I don't want to re-invent the wheel of
>> what's
>> > > > already
>> > > > > > in
>> > > > > > > > > > > > `testsuite`,
>> > > > > > > > > > > > > > > but I
>> > > > > > > > > > > > > > > > think Unity might end up being
>> lighter-weight
>> > in
>> > > > the
>> > > > > > long
>> > > > > > > > > run?
>> > > > > > > > > > > And
>> > > > > > > > > > > > > it is
>> > > > > > > > > > > > > > > > also not a Xiaomi owned/maintained
>> framework so
>> > > it
>> > > > > may
>> > > > > > > > > receive
>> > > > > > > > > > > > better
>> > > > > > > > > > > > > > > > external support (i.e. more users rely on
>> > it?). I
>> > > > > tried
>> > > > > > > to
>> > > > > > > > > > > compile
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > sim:citest configuration to run the
>> testsuite
>> > > > example
>> > > > > > and
>> > > > > > > > see
>> > > > > > > > > > > what
>> > > > > > > > > > > > > it was
>> > > > > > > > > > > > > > > > like, but I was met with a long list of C++
>> > > > > compilation
>> > > > > > > > > errors
>> > > > > > > > > > > that
>> > > > > > > > > > > > > I'd
>> > > > > > > > > > > > > > > > rather not deal with. There also seems to be
>> > zero
>> > > > > docs
>> > > > > > > > about
>> > > > > > > > > it
>> > > > > > > > > > > or
>> > > > > > > > > > > > > cmocka
>> > > > > > > > > > > > > > > > in our NuttX docs at the moment.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > What do you think? Start a Unity test suite,
>> > add
>> > > > some
>> > > > > > > > > switches
>> > > > > > > > > > to
>> > > > > > > > > > > > > choose
>> > > > > > > > > > > > > > > > which test cases get compiled into the
>> binary,
>> > > open
>> > > > > for
>> > > > > > > > > > > extensions
>> > > > > > > > > > > > by
>> > > > > > > > > > > > > > > users
>> > > > > > > > > > > > > > > > who know more about different subsystems?
>> This
>> > > way
>> > > > we
>> > > > > > can
>> > > > > > > > > keep
>> > > > > > > > > > OS
>> > > > > > > > > > > > > test
>> > > > > > > > > > > > > > > how
>> > > > > > > > > > > > > > > > it is for a quick, low-resource sanity
>> check of
>> > > > > "board
>> > > > > > > > > bringup"
>> > > > > > > > > > > > while
>> > > > > > > > > > > > > > > > having a dedicated testing application for
>> > > patches?
>> > > > > > > Someone
>> > > > > > > > > > > > changing
>> > > > > > > > > > > > > file
>> > > > > > > > > > > > > > > > system stuff can just compile to include
>> file
>> > > > system
>> > > > > > test
>> > > > > > > > > cases
>> > > > > > > > > > > and
>> > > > > > > > > > > > > run
>> > > > > > > > > > > > > > > > that suite only as proof that their PR
>> works.
>> > The
>> > > > > > output
>> > > > > > > is
>> > > > > > > > > > > really
>> > > > > > > > > > > > > easy
>> > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > glance at to determine pass/fail results.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Anyways, here's my draft demonstrating how
>> > Unity
>> > > > > works
>> > > > > > > with
>> > > > > > > > > the
>> > > > > > > > > > > FPU
>> > > > > > > > > > > > > test
>> > > > > > > > > > > > > > > > suite from OS test:
>> > > > > > > > > > > https://github.com/apache/nuttx-apps/pull/3259
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Matteo
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO
>> <
>> > > > > > > > > tomek@cedro.info>
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory
>> Nutt <
>> > > > > > > > > > > spudaneco@gmail.com
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > >>> (..)
>> > > > > > > > > > > > > > > >>> Starting with bits and pieces of something
>> > like
>> > > > the
>> > > > > > LTS
>> > > > > > > > > might
>> > > > > > > > > > > > make
>> > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > >>> sense other than its licensing problems.
>> > Since
>> > > > it
>> > > > > > > would
>> > > > > > > > > > never
>> > > > > > > > > > > be
>> > > > > > > > > > > > > > > >>> included in a distributed binary, the
>> > licensing
>> > > > > > really
>> > > > > > > > > should
>> > > > > > > > > > > not
>> > > > > > > > > > > > > > > matter
>> > > > > > > > > > > > > > > >>> much in practice.
>> > > > > > > > > > > > > > > >> Unless its GPL and some company would like
>> to
>> > > have
>> > > > > > their
>> > > > > > > > own
>> > > > > > > > > > > > > internal
>> > > > > > > > > > > > > > > >> bits and pieces.. luckily its MIT and the
>> > whole
>> > > > > thing
>> > > > > > is
>> > > > > > > > > > > > open-source
>> > > > > > > > > > > > > > > >> anyways :-)
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >> Yes we had many discussions about the LTS
>> and
>> > > this
>> > > > > > could
>> > > > > > > > be
>> > > > > > > > > > > great
>> > > > > > > > > > > > > start!
>> > > > > > > > > > > > > > > >> :-)
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >> --
>> > > > > > > > > > > > > > > >> CeDeROM, SQ7MHZ,
>> http://www.tomek.cedro.info
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
Okay this is great information! And I learned a new testing method today. I
will play with cmocka then since it seems like it could also work well for
the smaller NuttX devices.
Thank you Xiang!
On Mon, Dec 22, 2025, 10:57 PM Xiang Xiao <xi...@gmail.com> wrote:
> On Tue, Dec 23, 2025 at 10:29 AM Matteo Golin <ma...@gmail.com>
> wrote:
>
> > Thank you Xiang for the information! It makes sense of course that you're
> > not able to share the Xiaomi internal test suites.
> >
> >
> Actually, raiden00pl is working on upstreaming the part of the test suite
> which tests NuttX's components to the NuttX community.
> The part we can't share is that:
>
> 1. The test suite specific for openvela framework
> 2. Internal CI/CD infrastructure which is couple with our workflow and
> very different from github
>
>
>
> > Are there any modifications Xiaomi has needed to make to cmocka to
> improve
> > it's ability to run on small devices? I know some of your targets are
> > wearables which I imagine are very resource constrained?
> >
> >
> No, we don't do any size optimization for cmocka since it is already a
> small unit test framework containing one .h and one .c.
> We run many test cases(including cmocka's) on the smallest BLE only module
> which has 256KB SRAM and 2MB flash.
>
>
>
> > Can you clarify what you mean by monkey test? Is that another test
> > framework? I'm interested since you mentioned it running on watches and
> > speakers.
> >
> >
> Here is an explanation from wiki:
> https://en.wikipedia.org/wiki/Monkey_testing
> https://en.wikipedia.org/wiki/Mean_time_between_failures
> Here is the our monkey test requirement:
>
> 1. Boot 50 instances for each simulator/qemu/hardware with the product
> config
> 2. Generate the random input quickly through the fake drivers(e.g.
> touch/button/audio...)
>
> All devices need to run beyond 24 hours without crash or hang for three
> consecutive days.
>
> Thanks,
> > Matteo
> >
> > On Mon, Dec 22, 2025, 9:08 PM Xiang Xiao <xi...@gmail.com>
> > wrote:
> >
> > > On Tue, Dec 23, 2025 at 1:08 AM Matteo Golin <ma...@gmail.com>
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > Thanks for all the feedback! I'm glad I discussed this here first.
> I'll
> > > try
> > > > to address everything that was brought up.
> > > >
> > > > Thanks Guiding for the input! I must just be missing some library on
> my
> > > > machine if you're able to compile it just fine. Maybe I'll track down
> > > that
> > > > error and see if you can recognize what's wrong.
> > > >
> > > > I don't intend to replace OS test anymore. I think it serves its
> > purpose
> > > as
> > > > a minimal sanity check. Instead, I think we need something a little
> > more
> > > > involved to test PRs since OS test can't catch many issues. It is
> > > currently
> > > > the standard everyone uses for verifying their patches and it would
> be
> > > > better to have more complete, alternate options. I do think that we
> can
> > > all
> > > > agree that OS test needs a revision of its console output so that it
> is
> > > > much easier to see pass/fail results, but that is just a matter of
> > print
> > > > statements and can be done in a different effort.
> > > >
> > > > I think the overwhelming concern here is mixing two unit testing
> > > frameworks
> > > > and no longer having one resource where we can go for tests. I share
> > > raiden
> > > > and Guiding's concern about that, which is why I'm glad I opened for
> > > > discussion here. On the one hand, Xiaomi has a pretty substantial
> > > > collection of tests in the `testsuite` application which they are
> using
> > > > internally I believe (in their CI). These run well on virtual
> machines,
> > > > which I assume is why the only configurations with that test suite
> > > included
> > > > is sim and rv-virt. I do not know nearly enough about cmocka to make
> > any
> > > > claim that it *wouldn't* work on smaller hardware devices, but it
> also
> > > > definitely looks to be less light-weight from the feature set in the
> > > > documentation. On the other hand, the Xiaomi test suites are also
> not a
> > > > complete silver bullet since they seem not to be intended to run on
> > > > hardware, and there are *a lot *of tests that need hardware
> > verification.
> > > >
> > >
> > > The test suite also runs on hardware daily, but sorry it's impossible
> to
> > > share our hardware and internal CI/CD infrastructure for the community.
> > > But, we already contract raiden00pl to refactor our test automation
> > > framework for public use.
> > >
> > >
> > > > In PRs we often request more than just a simulator test for certain
> > > > changes. For instance, sim isn't able to have `/dev/rand`, so that
> > needs
> > > to
> > > > be tested on a different target.
> > > >
> > >
> > > Maybe someone more familiar with Xiaomi's testing practices can weigh
> in
> > on
> > > > whether cmocka has been run on hardware. Going purely based off of
> > their
> > > > PRs, which almost exclusively cite tests on simulators, my guess is
> > that
> > > > cmocka was not designed with embedded in mind when it was written.
> > > >
> > > >
> > > CI/CD in Xiaomi is far more complex than your description:
> > >
> > > 1. Each patch will test on 10+ devices(include sim/qemu/hardware,
> > > arm/arm64/riscv/xtensa/x64, smp/up) with 1000+ test case
> > > 2. Run the full test suite(10000+ test case) every day
> > > 3. Run the monkey test every day on watch and speaker product:
> 24hour
> > X
> > > 50 instance X (sim + qemu + hardware)
> > >
> > > Many stability bugs only happen randomly in monkey test.
> > >
> > > As for size: unit testing can get large with Unity or cmocka (as raiden
> > > > mentioned) as more tests are added. Hence my effort to enable/disable
> > > > individual test suites from getting compiled. It might not be
> possible
> > to
> > > > ever get good testing ability for AVR just because the devices are so
> > > > small. Whatever solution is used there is probably going to involve
> > > > flashing the device multiple times with different tests. The size
> goal
> > > > should just be to target the most NuttX devices possible.
> > > >
> > > > Then there are the pros and cons of each framework (again, I don't
> know
> > > > much about cmocka so take it with a grain of salt):
> > > >
> > > > Unity's cons:
> > > > - Designed for each test-suite to be a standalone binary. This is
> > clunky
> > > on
> > > > NuttX, so `nest` is currently implemented as a mega-suite where test
> > > cases
> > > > just follow a naming convention to make it clear what is being tested
> > > > - setUp/tearDown can't really be leveraged because of the above
> > > > - I've noticed some funky behaviour using TEST_IGNORE in certain
> > > scenarios
> > > > but that's possibly my own configuration issue
> > > > - Would be in conflict with existing cmocka tests
> > > >
> > > > Unity's pros:
> > > > - Highly configurable, like NuttX
> > > > - Light-weight and works for embedded devices
> > > > - Significantly larger user base than cmocka
> > > >
> > > > Cmocka's cons:
> > > > - Possibly (?) not light-weight
> > > > - No documentation for existing test cases in NuttX
> > > > - Small user base
> > > >
> > > > Cmocka's pros:
> > > > - Seems to have a larger set of features, including setup/teardown
> > > > functions per-test case
> > > > - Existing tests in NuttX
> > > >
> > > >
> > > Xiaomi selected cmocka just because it has the test suite concept which
> > > could help us manage 10000+ tests.
> > > But we also use gtest for the c++ code base, and run many test cases
> from
> > > 3rd parties(e.g. LTP, libuv, libcxx testsuite, PSE52...) which may use
> a
> > > different test framework.
> > > So, is it hard or meaningless to stick to one unit test framework for
> an
> > > open source project.
> > >
> > > Irrespective of whether we go with Unity or cmocka, the collection of
> > tests
> > > > that we write will be intended to run on the actual hardware devices
> > > > themselves for the purposes of testing patches and doing regression
> > > tests.
> > > > As a result, the tests will be separate from the Xiaomi 'testsuite'
> > > > application which is intended for CI testing. I think this
> > realistically
> > > > nullifies choosing which framework to use on the basis of
> consistency,
> > > > since the tests are not coexisting with the cmocka tests Xiaomi has
> > > > written. Instead, the choice should be based on what best achieves
> the
> > > goal
> > > > from a functionality perspective. Unity seems like a really great
> > choice
> > > > for this in my opinion, but I am bothered by the fact that it's
> > designed
> > > > with multiple test binaries in mind.
> > > >
> > > > Let me know what you think,
> > > > Matteo
> > > >
> > > > On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <ac...@gmail.com>
> > > wrote:
> > > >
> > > > > Yes, I agree! We need data size from each solution for comparison,
> > this
> > > > is
> > > > > the first step.
> > > > >
> > > > > Maybe the conversion could be facilitated if we create equivalent
> > > > functions
> > > > > like those used in the apps/testing/testsuites:
> > > > >
> > > > > cmocka_run_group_tests -> unity_run_group_tests
> > > > > cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown
> > > > > etc...
> > > > >
> > > > > This way we avoid end up with two unit test framework, we already
> > have
> > > > the
> > > > > Makefile vs CMake precedent! :-D
> > > > >
> > > > > On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <ra...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > The discussion about small systems doesn't make much sense
> without
> > > > > > data comparing resources of the same test cases for cmock and
> > unity.
> > > > > >
> > > > > > But let's assume that Unity is better. Should we use two
> frameworks
> > > > > > to implement unit tests for OS? Or choose one framework? If we
> > choose
> > > > > only
> > > > > > Unity, who
> > > > > > will migrate the current cmocka tests to Unity? Remember that
> cmoka
> > > > tests
> > > > > > are used in CI, so all CI tools must be also migrated.
> > > > > > Or should we use two different frameworks to test different
> things
> > > and
> > > > > make
> > > > > > even
> > > > > > more mess in apps/testing as it is now?
> > > > > >
> > > > > > No one is forcing anyone to do anything. This is an open-source
> > > > project,
> > > > > > everyone
> > > > > > should do what they feel is right. But whether the change should
> be
> > > > > merged
> > > > > > is
> > > > > > a different question. In this case, it's a very important
> question,
> > > > > because
> > > > > > unit testing
> > > > > > isn't just about test cases. It's about the entire infrastructure
> > > that
> > > > > > runs, captures and
> > > > > > handles test results later. Writing the test cases is the least
> of
> > > the
> > > > > > problems.
> > > > > > The infrastructure for cmoka testing is already in place, the
> > > > > > infrastructure for Unity
> > > > > > testing isn't.
> > > > > >
> > > > > > pon., 22 gru 2025 o 14:16 Alan C. Assis <ac...@gmail.com>
> > > > napisał(a):
> > > > > >
> > > > > > > Mateusz,
> > > > > > > but some chips like AVR DA have big flash, but not so much RAM.
> > > > > > >
> > > > > > > It is better to have some possible way to test on a small
> system,
> > > > using
> > > > > > > cmocka we are removing this option from the table.
> > > > > > >
> > > > > > > I liked the test Matteo did with Unity and I think we should
> let
> > > him
> > > > > > > continue. Forcing him to migrate to cmocka just because
> there's a
> > > > test
> > > > > > > suite from Xiaomi doesn't seem prudent to me.
> > > > > > >
> > > > > > > This is the advantage of an open mind open-source project: "the
> > > > > > possibility
> > > > > > > to test new ideas", if someone had said to Michal Lenc he
> > couldn't
> > > > > create
> > > > > > > NXBoot because where is already MCUBoot integrated on NuttX, we
> > > > didn't
> > > > > > have
> > > > > > > the option to use NXBoot today. ;-)
> > > > > > >
> > > > > > > BR,
> > > > > > >
> > > > > > > Alan
> > > > > > >
> > > > > > > On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <
> > raiden00pl@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Unit testing by itself isn't suitable for small systems.
> Ostest
> > > is
> > > > > > > > something
> > > > > > > > different (system level testing), so running it on small
> > systems
> > > > > makes
> > > > > > > > sense.
> > > > > > > > Running unit tests on small systems is pointless because you
> > > won't
> > > > > fit
> > > > > > in
> > > > > > > > FLASH
> > > > > > > > either way. You can run the same on large targets (preferably
> > on
> > > > > host:
> > > > > > > sim,
> > > > > > > > qemu)
> > > > > > > > and achieve the same result.
> > > > > > > >
> > > > > > > > pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com>
> > > > > > napisał(a):
> > > > > > > >
> > > > > > > > > Is it too complicated to rewrite the cmocka tests in Unity?
> > > > > > > > >
> > > > > > > > > If cmoka is not feasible for small microcontrollers, like
> > AVRs
> > > > and
> > > > > > > > others,
> > > > > > > > > our test coverage will be faulty.
> > > > > > > > >
> > > > > > > > > BR,
> > > > > > > > >
> > > > > > > > > Alan
> > > > > > > > >
> > > > > > > > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <
> > > raiden00pl@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I'm a fan of Unity and have been using it in my projects
> > for
> > > > > years,
> > > > > > > > but I
> > > > > > > > > > think
> > > > > > > > > > we should stick with one testing framework. Currently,
> > Cmocka
> > > > is
> > > > > > used
> > > > > > > > in
> > > > > > > > > > nuttx-apps, and we should stick with it. The only
> question
> > is
> > > > how
> > > > > > > well
> > > > > > > > > > Cmocka
> > > > > > > > > > is suitable for running on small systems? Will ostest
> still
> > > be
> > > > > able
> > > > > > > to
> > > > > > > > > run
> > > > > > > > > > on small MCUs?
> > > > > > > > > > With Unity this would be quite likely, so maybe an
> > exception
> > > > for
> > > > > > > Unity
> > > > > > > > > and
> > > > > > > > > > ostest would
> > > > > > > > > > be acceptable, but using it for other test cases will
> only
> > > > cause
> > > > > > > chaos.
> > > > > > > > > >
> > > > > > > > > > pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com>
> > > > > > napisał(a):
> > > > > > > > > >
> > > > > > > > > > > Hi Matteo,
> > > > > > > > > > >
> > > > > > > > > > > Good idea !
> > > > > > > > > > >
> > > > > > > > > > > But it is best to stick with a single testing
> framework.
> > > > > > Currently,
> > > > > > > > > many
> > > > > > > > > > > test cases are using cmocka, so I suggest reusing
> cmocka
> > > for
> > > > > > > > > consistency.
> > > > > > > > > > >
> > > > > > > > > > > ostest is a minimal basic test suite, typically used
> for
> > CT
> > > > > > > > (continuous
> > > > > > > > > > > testing) of each patch.
> > > > > > > > > > > If we move ostest to cmocka, it would make ostest
> > dependent
> > > > on
> > > > > a
> > > > > > > > > specific
> > > > > > > > > > > framework before it can be run.
> > > > > > > > > > > And It may increase the code size, which is not very
> > > friendly
> > > > > for
> > > > > > > > > > > configurations with small flash memory.
> > > > > > > > > > >
> > > > > > > > > > > As Greg mentioned: the answer depends on a tradeoff
> > between
> > > > > > effort
> > > > > > > > and
> > > > > > > > > > test
> > > > > > > > > > > coverage.
> > > > > > > > > > >
> > > > > > > > > > > We need to clarify whether we define ostest as a
> > > lightweight,
> > > > > > > > > > ready-to-run
> > > > > > > > > > > test set for quick verification, or as part of the full
> > > test
> > > > > > suite.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The follow is my compiling for cmocka:
> > > > > > > > > > >
> > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$
> > ./tools/configure.sh
> > > > > > > sim:citest
> > > > > > > > > > -j8*
> > > > > > > > > > > Copy files
> > > > > > > > > > > Select CONFIG_HOST_LINUX=y
> > > > > > > > > > > Refreshing...
> > > > > > > > > > > CP: arch/dummy/Kconfig to
> > > > > > > > > > >
> > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> > > > > > > > > > > CP: boards/dummy/Kconfig to
> > > > > > > > > > >
> > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> > > > > > > > > > > LN: include/arch to arch/sim/include
> > > > > > > > > > > LN: drivers/platform to
> > > > > > > > > /home/ligd/platform/mainline/nuttx/drivers/dummy
> > > > > > > > > > > LN: arch/sim/src/chip to
> > > > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> > > > > > > > > > > LN: arch/sim/src/board to
> > > > > > > > > > >
> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> > > > > > > > > > > LN: include/arch/board to
> > > > > > > > > > >
> > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> > > > > > > > > > > LN: include/arch/chip to
> > > > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> > > > > > > > > > > LN: platform/board to
> > > > > > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/audioutils
> > > > > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/benchmarks
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/boot
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/canutils
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/database
> > > > > > > > > > > mkkconfig in
> > > /home/ligd/platform/mainline/apps/examples/rust
> > > > > > > > > > > mkkconfig in
> > > > /home/ligd/platform/mainline/apps/examples/sotest
> > > > > > > > > > > mkkconfig in
> > > > /home/ligd/platform/mainline/apps/examples/module
> > > > > > > > > > > mkkconfig in
> > > > /home/ligd/platform/mainline/apps/examples/mcuboot
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/games
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/graphics
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/industry
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/inertial
> > > > > > > > > > > mkkconfig in
> > > > > > > > /home/ligd/platform/mainline/apps/interpreters/luamodules
> > > > > > > > > > > mkkconfig in
> > /home/ligd/platform/mainline/apps/interpreters
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/logging
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/lte
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/math
> > > > > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/mlearning
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/netutils
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/system
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/tee
> > > > > > > > > > > mkkconfig in
> > > > /home/ligd/platform/mainline/apps/testing/drivers
> > > > > > > > > > > mkkconfig in
> > > /home/ligd/platform/mainline/apps/testing/sched
> > > > > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/testing/fs
> > > > > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/testing/mm
> > > > > > > > > > > mkkconfig in
> > /home/ligd/platform/mainline/apps/testing/libc
> > > > > > > > > > > mkkconfig in
> > /home/ligd/platform/mainline/apps/testing/arch
> > > > > > > > > > > mkkconfig in
> > /home/ligd/platform/mainline/apps/testing/cxx
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing
> > > > > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/videoutils
> > > > > > > > > > > mkkconfig in
> > > > > /home/ligd/platform/mainline/apps/wireless/bluetooth
> > > > > > > > > > > mkkconfig in
> > > > > > /home/ligd/platform/mainline/apps/wireless/ieee802154
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless
> > > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps
> > > > > > > > > > > Loaded configuration '.config'
> > > > > > > > > > > Configuration saved to '.config'
> > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ make*
> > > > > > > > > > > Create version.h
> > > > > > > > > > > Downloading: libcxx-17.0.6.src.tar.xz LN:
> platform/board
> > to
> > > > > > > > > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > > > > > > > Downloading: v1.46.0.zip
> > > > > > > > > > > ...
> > > > > > > > > > > patching file
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> > > > > > > > > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> > > > > > > > > > >
> > > > > > > > > > > CP:
> > > > /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> > > > > > > > > > > CP:
> > > > > /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> > > > > > > > > > > LD: nuttx
> > > > > > > > > > > Pac SIM with dynamic libs..
> > > > > > > > > > > '/lib/x86_64-linux-gnu/libz.so.1' ->
> > > 'sim-pac/libs/libz.so.1'
> > > > > > > > > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
> > > > > > > > 'sim-pac/libs/libstdc++.so.6'
> > > > > > > > > > > '/lib/x86_64-linux-gnu/libc.so.6' ->
> > > 'sim-pac/libs/libc.so.6'
> > > > > > > > > > > '/lib/x86_64-linux-gnu/libm.so.6' ->
> > > 'sim-pac/libs/libm.so.6'
> > > > > > > > > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' ->
> > > > > > > 'sim-pac/libs/libgcc_s.so.1'
> > > > > > > > > > > '/lib64/ld-linux-x86-64.so.2' ->
> > > > 'sim-pac/ld-linux-x86-64.so.2'
> > > > > > > > > > > SIM elf with dynamic libs archive in nuttx.tgz
> > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> > > > > > > > > > > CHelloWorld: Constructor: mSecret=42
> > > > > > > > > > > nsh: init: open failed: 2
> > > > > > > > > > >
> > > > > > > > > > > NuttShell (NSH) NuttX-12.11.0
> > > > > > > > > > > MOTD: username=admin password=Administrator
> > > > > > > > > > > nsh>
> > > > > > > > > > > nsh> help
> > > > > > > > > > > help usage: help [-v] [<cmd>]
> > > > > > > > > > >
> > > > > > > > > > > . df ifconfig mkfifo
> > > readlink
> > > > > > > > umount
> > > > > > > > > > > [ dmesg ifdown mkrd rm
> > > > > > > unset
> > > > > > > > > > > ? echo ifup mount
> rmdir
> > > > > > > > uptime
> > > > > > > > > > > alias env insmod mv
> rmmod
> > > > > > > > usleep
> > > > > > > > > > > unalias exec kill nslookup set
> > > > > > > watch
> > > > > > > > > > > basename exit pkill pidof
> sleep
> > > > > > xd
> > > > > > > > > > > break expr losetup pmconfig
> > source
> > > > > > > wait
> > > > > > > > > > > cat false ln poweroff
> test
> > > > > > > > > > > cd fdinfo ls quit
> time
> > > > > > > > > > > cp free lsmod printf
> true
> > > > > > > > > > > cmp help mkdir ps
> > > truncate
> > > > > > > > > > > dirname hexdump mkfatfs pwd
> uname
> > > > > > > > > > >
> > > > > > > > > > > Builtin Apps:
> > > > > > > > > > > cmocka
> > > > > > > > > > > cmocka_driver_block
> > > > > > > > > > > cmocka_driver_gpio
> > > > > > > > > > > cmocka_driver_pm
> > > > > > > > > > > cmocka_driver_pm_runtime
> > > > > > > > > > > cmocka_driver_regulator
> > > > > > > > > > >
> > > > > > > > > > > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一
> > > > 10:15写道：
> > > > > > > > > > >
> > > > > > > > > > > > I think so. Best to leave OS test alone and slowly
> > > replace
> > > > it
> > > > > > > with
> > > > > > > > a
> > > > > > > > > > new
> > > > > > > > > > > > tool that has improvements. The only reason I can see
> > to
> > > > > modify
> > > > > > > OS
> > > > > > > > > test
> > > > > > > > > > > now
> > > > > > > > > > > > would just be to have clearer output about what
> > > > > passed/failed.
> > > > > > > > > > > >
> > > > > > > > > > > > I'll come up with an improved proof of concept for a
> > new
> > > > tool
> > > > > > to
> > > > > > > > > share
> > > > > > > > > > in
> > > > > > > > > > > > the next few days.
> > > > > > > > > > > >
> > > > > > > > > > > > Matteo
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <
> > > > tomek@cedro.info>
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Small steps with measurable results may be the best
> > > > > approach
> > > > > > > > here,
> > > > > > > > > to
> > > > > > > > > > > > > create some new solution next to existing ostest
> but
> > > not
> > > > > > > > replacing
> > > > > > > > > it
> > > > > > > > > > > > > until ready? :-)
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> > > > > > > > > matteo.golin@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I agree, I think it would take a long time to
> have
> > > > proper
> > > > > > > > > coverage
> > > > > > > > > > of
> > > > > > > > > > > > > NuttX
> > > > > > > > > > > > > > as a whole. Perhaps even impossible, since Unity
> is
> > > > more
> > > > > > of a
> > > > > > > > > unit
> > > > > > > > > > > > > testing
> > > > > > > > > > > > > > framework and thus there are definitely test
> cases
> > > for
> > > > > > NuttX
> > > > > > > > that
> > > > > > > > > > > would
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > outside of its realm of "applicability". There is
> > > > still a
> > > > > > > need
> > > > > > > > > for
> > > > > > > > > > a
> > > > > > > > > > > > > > variety of testing applications and bench-marking
> > > > > > > applications
> > > > > > > > > for
> > > > > > > > > > > > those
> > > > > > > > > > > > > > scenarios.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > That makes sense, prerequisites would need to be
> > > > > > considered.
> > > > > > > I
> > > > > > > > > > think
> > > > > > > > > > > > > > targeting the low-hanging fruit of unit-testable
> > code
> > > > > would
> > > > > > > be
> > > > > > > > > easy
> > > > > > > > > > > in
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > case, since prerequisites are mostly encoded in
> the
> > > > > Kconfig
> > > > > > > > > > > definitions
> > > > > > > > > > > > > > existing or not for features to be tested. Any
> > > complex
> > > > > > > > behaviour
> > > > > > > > > > > > > > (interrupts, multi-threading, etc.) might be out
> of
> > > the
> > > > > > realm
> > > > > > > > of
> > > > > > > > > > > what's
> > > > > > > > > > > > > > testable by Unity in the first place. We'll
> always
> > > have
> > > > > OS
> > > > > > > test
> > > > > > > > > in
> > > > > > > > > > > that
> > > > > > > > > > > > > > case, but maybe then it's better to do what Alan
> > > > > suggested
> > > > > > > and
> > > > > > > > at
> > > > > > > > > > > least
> > > > > > > > > > > > > > make its output more clear in the meantime. I
> > suppose
> > > > > RTOS
> > > > > > > > > testing
> > > > > > > > > > > is a
> > > > > > > > > > > > > > whole field in itself.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> > > > > > > > > spudaneco@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think that the answer depends on a tradeoff
> > > between
> > > > > > > effort
> > > > > > > > > and
> > > > > > > > > > > test
> > > > > > > > > > > > > > > coverage. I could imagine that a proper job
> > could
> > > be
> > > > > > > > man-years
> > > > > > > > > > of
> > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > My only concern with a re-design is to be
> certain
> > > > that
> > > > > > > tests
> > > > > > > > > are
> > > > > > > > > > > > > > > arranged so that the prerequisites of later
> test
> > > are
> > > > > > tested
> > > > > > > > > > first.
> > > > > > > > > > > > If
> > > > > > > > > > > > > > > tests are grouped by, for example, functional
> > area,
> > > > > then
> > > > > > > > > failures
> > > > > > > > > > > can
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > difficult to understand if there is a failure
> in
> > an
> > > > > > > > unverified
> > > > > > > > > > test
> > > > > > > > > > > > > > > prerequisite.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Some of the "tests" are bogus and only exist to
> > > > provide
> > > > > > > some
> > > > > > > > > > > minimum
> > > > > > > > > > > > > > > confidence that we have the functionality to
> run
> > > the
> > > > > > test.
> > > > > > > > For
> > > > > > > > > > > > > > > example: Can I start tasks? Can I pass
> > > parameters?
> > > > > Can
> > > > > > I
> > > > > > > > > > survive
> > > > > > > > > > > a
> > > > > > > > > > > > > > > context switch. Some of the tests were
> intended
> > > only
> > > > > to
> > > > > > > > > support
> > > > > > > > > > > > debug.
> > > > > > > > > > > > > > > Some were created only to track down specific
> > bugs.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > > > > > > > > > > > > LTS being the Long-Term-Stable release of
> > NuttX?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm starting to think that it would be best
> to
> > > > create
> > > > > > new
> > > > > > > > > suite
> > > > > > > > > > > of
> > > > > > > > > > > > > tests
> > > > > > > > > > > > > > > > using Unity, using what's in OS Test as a
> basis
> > > and
> > > > > > then
> > > > > > > > > > > expanding
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > > further. Greg has pointed out that OS test is
> > not
> > > > > > > intended
> > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > a
> > > > > > > > > > > > > > > complete
> > > > > > > > > > > > > > > > test, so maybe a more solid testing framework
> > > base
> > > > > in a
> > > > > > > new
> > > > > > > > > > form
> > > > > > > > > > > > > would be
> > > > > > > > > > > > > > > > in order. At that point it would also be good
> > to
> > > > > > document
> > > > > > > > and
> > > > > > > > > > > > educate
> > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > about other test options, since OS test is a
> > > > current
> > > > > > > > > favourite
> > > > > > > > > > > for
> > > > > > > > > > > > > > > showing
> > > > > > > > > > > > > > > > PRs haven't broken anything. Maybe that's not
> > the
> > > > > best
> > > > > > > > choice
> > > > > > > > > > > > anymore
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > this information.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I don't want to re-invent the wheel of what's
> > > > already
> > > > > > in
> > > > > > > > > > > > `testsuite`,
> > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > think Unity might end up being lighter-weight
> > in
> > > > the
> > > > > > long
> > > > > > > > > run?
> > > > > > > > > > > And
> > > > > > > > > > > > > it is
> > > > > > > > > > > > > > > > also not a Xiaomi owned/maintained framework
> so
> > > it
> > > > > may
> > > > > > > > > receive
> > > > > > > > > > > > better
> > > > > > > > > > > > > > > > external support (i.e. more users rely on
> > it?). I
> > > > > tried
> > > > > > > to
> > > > > > > > > > > compile
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > sim:citest configuration to run the testsuite
> > > > example
> > > > > > and
> > > > > > > > see
> > > > > > > > > > > what
> > > > > > > > > > > > > it was
> > > > > > > > > > > > > > > > like, but I was met with a long list of C++
> > > > > compilation
> > > > > > > > > errors
> > > > > > > > > > > that
> > > > > > > > > > > > > I'd
> > > > > > > > > > > > > > > > rather not deal with. There also seems to be
> > zero
> > > > > docs
> > > > > > > > about
> > > > > > > > > it
> > > > > > > > > > > or
> > > > > > > > > > > > > cmocka
> > > > > > > > > > > > > > > > in our NuttX docs at the moment.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > What do you think? Start a Unity test suite,
> > add
> > > > some
> > > > > > > > > switches
> > > > > > > > > > to
> > > > > > > > > > > > > choose
> > > > > > > > > > > > > > > > which test cases get compiled into the
> binary,
> > > open
> > > > > for
> > > > > > > > > > > extensions
> > > > > > > > > > > > by
> > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > who know more about different subsystems?
> This
> > > way
> > > > we
> > > > > > can
> > > > > > > > > keep
> > > > > > > > > > OS
> > > > > > > > > > > > > test
> > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > it is for a quick, low-resource sanity check
> of
> > > > > "board
> > > > > > > > > bringup"
> > > > > > > > > > > > while
> > > > > > > > > > > > > > > > having a dedicated testing application for
> > > patches?
> > > > > > > Someone
> > > > > > > > > > > > changing
> > > > > > > > > > > > > file
> > > > > > > > > > > > > > > > system stuff can just compile to include file
> > > > system
> > > > > > test
> > > > > > > > > cases
> > > > > > > > > > > and
> > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > that suite only as proof that their PR works.
> > The
> > > > > > output
> > > > > > > is
> > > > > > > > > > > really
> > > > > > > > > > > > > easy
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > glance at to determine pass/fail results.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Anyways, here's my draft demonstrating how
> > Unity
> > > > > works
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > FPU
> > > > > > > > > > > > > test
> > > > > > > > > > > > > > > > suite from OS test:
> > > > > > > > > > > https://github.com/apache/nuttx-apps/pull/3259
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Matteo
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> > > > > > > > > tomek@cedro.info>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory
> Nutt <
> > > > > > > > > > > spudaneco@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >>> (..)
> > > > > > > > > > > > > > > >>> Starting with bits and pieces of something
> > like
> > > > the
> > > > > > LTS
> > > > > > > > > might
> > > > > > > > > > > > make
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > >>> sense other than its licensing problems.
> > Since
> > > > it
> > > > > > > would
> > > > > > > > > > never
> > > > > > > > > > > be
> > > > > > > > > > > > > > > >>> included in a distributed binary, the
> > licensing
> > > > > > really
> > > > > > > > > should
> > > > > > > > > > > not
> > > > > > > > > > > > > > > matter
> > > > > > > > > > > > > > > >>> much in practice.
> > > > > > > > > > > > > > > >> Unless its GPL and some company would like
> to
> > > have
> > > > > > their
> > > > > > > > own
> > > > > > > > > > > > > internal
> > > > > > > > > > > > > > > >> bits and pieces.. luckily its MIT and the
> > whole
> > > > > thing
> > > > > > is
> > > > > > > > > > > > open-source
> > > > > > > > > > > > > > > >> anyways :-)
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Yes we had many discussions about the LTS
> and
> > > this
> > > > > > could
> > > > > > > > be
> > > > > > > > > > > great
> > > > > > > > > > > > > start!
> > > > > > > > > > > > > > > >> :-)
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> --
> > > > > > > > > > > > > > > >> CeDeROM, SQ7MHZ,
> http://www.tomek.cedro.info
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: OS Test re-factor proposal
Posted by Xiang Xiao <xi...@gmail.com>.
On Tue, Dec 23, 2025 at 10:29 AM Matteo Golin <ma...@gmail.com>
wrote:
> Thank you Xiang for the information! It makes sense of course that you're
> not able to share the Xiaomi internal test suites.
>
>
Actually, raiden00pl is working on upstreaming the part of the test suite
which tests NuttX's components to the NuttX community.
The part we can't share is that:
1. The test suite specific for openvela framework
2. Internal CI/CD infrastructure which is couple with our workflow and
very different from github
> Are there any modifications Xiaomi has needed to make to cmocka to improve
> it's ability to run on small devices? I know some of your targets are
> wearables which I imagine are very resource constrained?
>
>
No, we don't do any size optimization for cmocka since it is already a
small unit test framework containing one .h and one .c.
We run many test cases(including cmocka's) on the smallest BLE only module
which has 256KB SRAM and 2MB flash.
> Can you clarify what you mean by monkey test? Is that another test
> framework? I'm interested since you mentioned it running on watches and
> speakers.
>
>
Here is an explanation from wiki:
https://en.wikipedia.org/wiki/Monkey_testing
https://en.wikipedia.org/wiki/Mean_time_between_failures
Here is the our monkey test requirement:
1. Boot 50 instances for each simulator/qemu/hardware with the product
config
2. Generate the random input quickly through the fake drivers(e.g.
touch/button/audio...)
All devices need to run beyond 24 hours without crash or hang for three
consecutive days.
Thanks,
> Matteo
>
> On Mon, Dec 22, 2025, 9:08 PM Xiang Xiao <xi...@gmail.com>
> wrote:
>
> > On Tue, Dec 23, 2025 at 1:08 AM Matteo Golin <ma...@gmail.com>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > Thanks for all the feedback! I'm glad I discussed this here first. I'll
> > try
> > > to address everything that was brought up.
> > >
> > > Thanks Guiding for the input! I must just be missing some library on my
> > > machine if you're able to compile it just fine. Maybe I'll track down
> > that
> > > error and see if you can recognize what's wrong.
> > >
> > > I don't intend to replace OS test anymore. I think it serves its
> purpose
> > as
> > > a minimal sanity check. Instead, I think we need something a little
> more
> > > involved to test PRs since OS test can't catch many issues. It is
> > currently
> > > the standard everyone uses for verifying their patches and it would be
> > > better to have more complete, alternate options. I do think that we can
> > all
> > > agree that OS test needs a revision of its console output so that it is
> > > much easier to see pass/fail results, but that is just a matter of
> print
> > > statements and can be done in a different effort.
> > >
> > > I think the overwhelming concern here is mixing two unit testing
> > frameworks
> > > and no longer having one resource where we can go for tests. I share
> > raiden
> > > and Guiding's concern about that, which is why I'm glad I opened for
> > > discussion here. On the one hand, Xiaomi has a pretty substantial
> > > collection of tests in the `testsuite` application which they are using
> > > internally I believe (in their CI). These run well on virtual machines,
> > > which I assume is why the only configurations with that test suite
> > included
> > > is sim and rv-virt. I do not know nearly enough about cmocka to make
> any
> > > claim that it *wouldn't* work on smaller hardware devices, but it also
> > > definitely looks to be less light-weight from the feature set in the
> > > documentation. On the other hand, the Xiaomi test suites are also not a
> > > complete silver bullet since they seem not to be intended to run on
> > > hardware, and there are *a lot *of tests that need hardware
> verification.
> > >
> >
> > The test suite also runs on hardware daily, but sorry it's impossible to
> > share our hardware and internal CI/CD infrastructure for the community.
> > But, we already contract raiden00pl to refactor our test automation
> > framework for public use.
> >
> >
> > > In PRs we often request more than just a simulator test for certain
> > > changes. For instance, sim isn't able to have `/dev/rand`, so that
> needs
> > to
> > > be tested on a different target.
> > >
> >
> > Maybe someone more familiar with Xiaomi's testing practices can weigh in
> on
> > > whether cmocka has been run on hardware. Going purely based off of
> their
> > > PRs, which almost exclusively cite tests on simulators, my guess is
> that
> > > cmocka was not designed with embedded in mind when it was written.
> > >
> > >
> > CI/CD in Xiaomi is far more complex than your description:
> >
> > 1. Each patch will test on 10+ devices(include sim/qemu/hardware,
> > arm/arm64/riscv/xtensa/x64, smp/up) with 1000+ test case
> > 2. Run the full test suite(10000+ test case) every day
> > 3. Run the monkey test every day on watch and speaker product: 24hour
> X
> > 50 instance X (sim + qemu + hardware)
> >
> > Many stability bugs only happen randomly in monkey test.
> >
> > As for size: unit testing can get large with Unity or cmocka (as raiden
> > > mentioned) as more tests are added. Hence my effort to enable/disable
> > > individual test suites from getting compiled. It might not be possible
> to
> > > ever get good testing ability for AVR just because the devices are so
> > > small. Whatever solution is used there is probably going to involve
> > > flashing the device multiple times with different tests. The size goal
> > > should just be to target the most NuttX devices possible.
> > >
> > > Then there are the pros and cons of each framework (again, I don't know
> > > much about cmocka so take it with a grain of salt):
> > >
> > > Unity's cons:
> > > - Designed for each test-suite to be a standalone binary. This is
> clunky
> > on
> > > NuttX, so `nest` is currently implemented as a mega-suite where test
> > cases
> > > just follow a naming convention to make it clear what is being tested
> > > - setUp/tearDown can't really be leveraged because of the above
> > > - I've noticed some funky behaviour using TEST_IGNORE in certain
> > scenarios
> > > but that's possibly my own configuration issue
> > > - Would be in conflict with existing cmocka tests
> > >
> > > Unity's pros:
> > > - Highly configurable, like NuttX
> > > - Light-weight and works for embedded devices
> > > - Significantly larger user base than cmocka
> > >
> > > Cmocka's cons:
> > > - Possibly (?) not light-weight
> > > - No documentation for existing test cases in NuttX
> > > - Small user base
> > >
> > > Cmocka's pros:
> > > - Seems to have a larger set of features, including setup/teardown
> > > functions per-test case
> > > - Existing tests in NuttX
> > >
> > >
> > Xiaomi selected cmocka just because it has the test suite concept which
> > could help us manage 10000+ tests.
> > But we also use gtest for the c++ code base, and run many test cases from
> > 3rd parties(e.g. LTP, libuv, libcxx testsuite, PSE52...) which may use a
> > different test framework.
> > So, is it hard or meaningless to stick to one unit test framework for an
> > open source project.
> >
> > Irrespective of whether we go with Unity or cmocka, the collection of
> tests
> > > that we write will be intended to run on the actual hardware devices
> > > themselves for the purposes of testing patches and doing regression
> > tests.
> > > As a result, the tests will be separate from the Xiaomi 'testsuite'
> > > application which is intended for CI testing. I think this
> realistically
> > > nullifies choosing which framework to use on the basis of consistency,
> > > since the tests are not coexisting with the cmocka tests Xiaomi has
> > > written. Instead, the choice should be based on what best achieves the
> > goal
> > > from a functionality perspective. Unity seems like a really great
> choice
> > > for this in my opinion, but I am bothered by the fact that it's
> designed
> > > with multiple test binaries in mind.
> > >
> > > Let me know what you think,
> > > Matteo
> > >
> > > On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <ac...@gmail.com>
> > wrote:
> > >
> > > > Yes, I agree! We need data size from each solution for comparison,
> this
> > > is
> > > > the first step.
> > > >
> > > > Maybe the conversion could be facilitated if we create equivalent
> > > functions
> > > > like those used in the apps/testing/testsuites:
> > > >
> > > > cmocka_run_group_tests -> unity_run_group_tests
> > > > cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown
> > > > etc...
> > > >
> > > > This way we avoid end up with two unit test framework, we already
> have
> > > the
> > > > Makefile vs CMake precedent! :-D
> > > >
> > > > On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <ra...@gmail.com>
> > > wrote:
> > > >
> > > > > The discussion about small systems doesn't make much sense without
> > > > > data comparing resources of the same test cases for cmock and
> unity.
> > > > >
> > > > > But let's assume that Unity is better. Should we use two frameworks
> > > > > to implement unit tests for OS? Or choose one framework? If we
> choose
> > > > only
> > > > > Unity, who
> > > > > will migrate the current cmocka tests to Unity? Remember that cmoka
> > > tests
> > > > > are used in CI, so all CI tools must be also migrated.
> > > > > Or should we use two different frameworks to test different things
> > and
> > > > make
> > > > > even
> > > > > more mess in apps/testing as it is now?
> > > > >
> > > > > No one is forcing anyone to do anything. This is an open-source
> > > project,
> > > > > everyone
> > > > > should do what they feel is right. But whether the change should be
> > > > merged
> > > > > is
> > > > > a different question. In this case, it's a very important question,
> > > > because
> > > > > unit testing
> > > > > isn't just about test cases. It's about the entire infrastructure
> > that
> > > > > runs, captures and
> > > > > handles test results later. Writing the test cases is the least of
> > the
> > > > > problems.
> > > > > The infrastructure for cmoka testing is already in place, the
> > > > > infrastructure for Unity
> > > > > testing isn't.
> > > > >
> > > > > pon., 22 gru 2025 o 14:16 Alan C. Assis <ac...@gmail.com>
> > > napisał(a):
> > > > >
> > > > > > Mateusz,
> > > > > > but some chips like AVR DA have big flash, but not so much RAM.
> > > > > >
> > > > > > It is better to have some possible way to test on a small system,
> > > using
> > > > > > cmocka we are removing this option from the table.
> > > > > >
> > > > > > I liked the test Matteo did with Unity and I think we should let
> > him
> > > > > > continue. Forcing him to migrate to cmocka just because there's a
> > > test
> > > > > > suite from Xiaomi doesn't seem prudent to me.
> > > > > >
> > > > > > This is the advantage of an open mind open-source project: "the
> > > > > possibility
> > > > > > to test new ideas", if someone had said to Michal Lenc he
> couldn't
> > > > create
> > > > > > NXBoot because where is already MCUBoot integrated on NuttX, we
> > > didn't
> > > > > have
> > > > > > the option to use NXBoot today. ;-)
> > > > > >
> > > > > > BR,
> > > > > >
> > > > > > Alan
> > > > > >
> > > > > > On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <
> raiden00pl@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Unit testing by itself isn't suitable for small systems. Ostest
> > is
> > > > > > > something
> > > > > > > different (system level testing), so running it on small
> systems
> > > > makes
> > > > > > > sense.
> > > > > > > Running unit tests on small systems is pointless because you
> > won't
> > > > fit
> > > > > in
> > > > > > > FLASH
> > > > > > > either way. You can run the same on large targets (preferably
> on
> > > > host:
> > > > > > sim,
> > > > > > > qemu)
> > > > > > > and achieve the same result.
> > > > > > >
> > > > > > > pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com>
> > > > > napisał(a):
> > > > > > >
> > > > > > > > Is it too complicated to rewrite the cmocka tests in Unity?
> > > > > > > >
> > > > > > > > If cmoka is not feasible for small microcontrollers, like
> AVRs
> > > and
> > > > > > > others,
> > > > > > > > our test coverage will be faulty.
> > > > > > > >
> > > > > > > > BR,
> > > > > > > >
> > > > > > > > Alan
> > > > > > > >
> > > > > > > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <
> > raiden00pl@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > I'm a fan of Unity and have been using it in my projects
> for
> > > > years,
> > > > > > > but I
> > > > > > > > > think
> > > > > > > > > we should stick with one testing framework. Currently,
> Cmocka
> > > is
> > > > > used
> > > > > > > in
> > > > > > > > > nuttx-apps, and we should stick with it. The only question
> is
> > > how
> > > > > > well
> > > > > > > > > Cmocka
> > > > > > > > > is suitable for running on small systems? Will ostest still
> > be
> > > > able
> > > > > > to
> > > > > > > > run
> > > > > > > > > on small MCUs?
> > > > > > > > > With Unity this would be quite likely, so maybe an
> exception
> > > for
> > > > > > Unity
> > > > > > > > and
> > > > > > > > > ostest would
> > > > > > > > > be acceptable, but using it for other test cases will only
> > > cause
> > > > > > chaos.
> > > > > > > > >
> > > > > > > > > pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com>
> > > > > napisał(a):
> > > > > > > > >
> > > > > > > > > > Hi Matteo,
> > > > > > > > > >
> > > > > > > > > > Good idea !
> > > > > > > > > >
> > > > > > > > > > But it is best to stick with a single testing framework.
> > > > > Currently,
> > > > > > > > many
> > > > > > > > > > test cases are using cmocka, so I suggest reusing cmocka
> > for
> > > > > > > > consistency.
> > > > > > > > > >
> > > > > > > > > > ostest is a minimal basic test suite, typically used for
> CT
> > > > > > > (continuous
> > > > > > > > > > testing) of each patch.
> > > > > > > > > > If we move ostest to cmocka, it would make ostest
> dependent
> > > on
> > > > a
> > > > > > > > specific
> > > > > > > > > > framework before it can be run.
> > > > > > > > > > And It may increase the code size, which is not very
> > friendly
> > > > for
> > > > > > > > > > configurations with small flash memory.
> > > > > > > > > >
> > > > > > > > > > As Greg mentioned: the answer depends on a tradeoff
> between
> > > > > effort
> > > > > > > and
> > > > > > > > > test
> > > > > > > > > > coverage.
> > > > > > > > > >
> > > > > > > > > > We need to clarify whether we define ostest as a
> > lightweight,
> > > > > > > > > ready-to-run
> > > > > > > > > > test set for quick verification, or as part of the full
> > test
> > > > > suite.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The follow is my compiling for cmocka:
> > > > > > > > > >
> > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$
> ./tools/configure.sh
> > > > > > sim:citest
> > > > > > > > > -j8*
> > > > > > > > > > Copy files
> > > > > > > > > > Select CONFIG_HOST_LINUX=y
> > > > > > > > > > Refreshing...
> > > > > > > > > > CP: arch/dummy/Kconfig to
> > > > > > > > > >
> /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> > > > > > > > > > CP: boards/dummy/Kconfig to
> > > > > > > > > >
> > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> > > > > > > > > > LN: include/arch to arch/sim/include
> > > > > > > > > > LN: drivers/platform to
> > > > > > > > /home/ligd/platform/mainline/nuttx/drivers/dummy
> > > > > > > > > > LN: arch/sim/src/chip to
> > > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> > > > > > > > > > LN: arch/sim/src/board to
> > > > > > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> > > > > > > > > > LN: include/arch/board to
> > > > > > > > > >
> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> > > > > > > > > > LN: include/arch/chip to
> > > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> > > > > > > > > > LN: platform/board to
> > > > > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/boot
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/canutils
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/database
> > > > > > > > > > mkkconfig in
> > /home/ligd/platform/mainline/apps/examples/rust
> > > > > > > > > > mkkconfig in
> > > /home/ligd/platform/mainline/apps/examples/sotest
> > > > > > > > > > mkkconfig in
> > > /home/ligd/platform/mainline/apps/examples/module
> > > > > > > > > > mkkconfig in
> > > /home/ligd/platform/mainline/apps/examples/mcuboot
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/games
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/graphics
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/industry
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/inertial
> > > > > > > > > > mkkconfig in
> > > > > > > /home/ligd/platform/mainline/apps/interpreters/luamodules
> > > > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/interpreters
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/logging
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/lte
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/math
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/netutils
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/system
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/tee
> > > > > > > > > > mkkconfig in
> > > /home/ligd/platform/mainline/apps/testing/drivers
> > > > > > > > > > mkkconfig in
> > /home/ligd/platform/mainline/apps/testing/sched
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> > > > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/testing/libc
> > > > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/testing/arch
> > > > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/testing/cxx
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> > > > > > > > > > mkkconfig in
> > > > /home/ligd/platform/mainline/apps/wireless/bluetooth
> > > > > > > > > > mkkconfig in
> > > > > /home/ligd/platform/mainline/apps/wireless/ieee802154
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless
> > > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps
> > > > > > > > > > Loaded configuration '.config'
> > > > > > > > > > Configuration saved to '.config'
> > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ make*
> > > > > > > > > > Create version.h
> > > > > > > > > > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board
> to
> > > > > > > > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > > > > > > Downloading: v1.46.0.zip
> > > > > > > > > > ...
> > > > > > > > > > patching file
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> > > > > > > > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> > > > > > > > > >
> > > > > > > > > > CP:
> > > /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> > > > > > > > > > CP:
> > > > /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> > > > > > > > > > LD: nuttx
> > > > > > > > > > Pac SIM with dynamic libs..
> > > > > > > > > > '/lib/x86_64-linux-gnu/libz.so.1' ->
> > 'sim-pac/libs/libz.so.1'
> > > > > > > > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
> > > > > > > 'sim-pac/libs/libstdc++.so.6'
> > > > > > > > > > '/lib/x86_64-linux-gnu/libc.so.6' ->
> > 'sim-pac/libs/libc.so.6'
> > > > > > > > > > '/lib/x86_64-linux-gnu/libm.so.6' ->
> > 'sim-pac/libs/libm.so.6'
> > > > > > > > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' ->
> > > > > > 'sim-pac/libs/libgcc_s.so.1'
> > > > > > > > > > '/lib64/ld-linux-x86-64.so.2' ->
> > > 'sim-pac/ld-linux-x86-64.so.2'
> > > > > > > > > > SIM elf with dynamic libs archive in nuttx.tgz
> > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> > > > > > > > > > CHelloWorld: Constructor: mSecret=42
> > > > > > > > > > nsh: init: open failed: 2
> > > > > > > > > >
> > > > > > > > > > NuttShell (NSH) NuttX-12.11.0
> > > > > > > > > > MOTD: username=admin password=Administrator
> > > > > > > > > > nsh>
> > > > > > > > > > nsh> help
> > > > > > > > > > help usage: help [-v] [<cmd>]
> > > > > > > > > >
> > > > > > > > > > . df ifconfig mkfifo
> > readlink
> > > > > > > umount
> > > > > > > > > > [ dmesg ifdown mkrd rm
> > > > > > unset
> > > > > > > > > > ? echo ifup mount rmdir
> > > > > > > uptime
> > > > > > > > > > alias env insmod mv rmmod
> > > > > > > usleep
> > > > > > > > > > unalias exec kill nslookup set
> > > > > > watch
> > > > > > > > > > basename exit pkill pidof sleep
> > > > > xd
> > > > > > > > > > break expr losetup pmconfig
> source
> > > > > > wait
> > > > > > > > > > cat false ln poweroff test
> > > > > > > > > > cd fdinfo ls quit time
> > > > > > > > > > cp free lsmod printf true
> > > > > > > > > > cmp help mkdir ps
> > truncate
> > > > > > > > > > dirname hexdump mkfatfs pwd uname
> > > > > > > > > >
> > > > > > > > > > Builtin Apps:
> > > > > > > > > > cmocka
> > > > > > > > > > cmocka_driver_block
> > > > > > > > > > cmocka_driver_gpio
> > > > > > > > > > cmocka_driver_pm
> > > > > > > > > > cmocka_driver_pm_runtime
> > > > > > > > > > cmocka_driver_regulator
> > > > > > > > > >
> > > > > > > > > > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一
> > > 10:15写道：
> > > > > > > > > >
> > > > > > > > > > > I think so. Best to leave OS test alone and slowly
> > replace
> > > it
> > > > > > with
> > > > > > > a
> > > > > > > > > new
> > > > > > > > > > > tool that has improvements. The only reason I can see
> to
> > > > modify
> > > > > > OS
> > > > > > > > test
> > > > > > > > > > now
> > > > > > > > > > > would just be to have clearer output about what
> > > > passed/failed.
> > > > > > > > > > >
> > > > > > > > > > > I'll come up with an improved proof of concept for a
> new
> > > tool
> > > > > to
> > > > > > > > share
> > > > > > > > > in
> > > > > > > > > > > the next few days.
> > > > > > > > > > >
> > > > > > > > > > > Matteo
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <
> > > tomek@cedro.info>
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Small steps with measurable results may be the best
> > > > approach
> > > > > > > here,
> > > > > > > > to
> > > > > > > > > > > > create some new solution next to existing ostest but
> > not
> > > > > > > replacing
> > > > > > > > it
> > > > > > > > > > > > until ready? :-)
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> > > > > > > > matteo.golin@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > I agree, I think it would take a long time to have
> > > proper
> > > > > > > > coverage
> > > > > > > > > of
> > > > > > > > > > > > NuttX
> > > > > > > > > > > > > as a whole. Perhaps even impossible, since Unity is
> > > more
> > > > > of a
> > > > > > > > unit
> > > > > > > > > > > > testing
> > > > > > > > > > > > > framework and thus there are definitely test cases
> > for
> > > > > NuttX
> > > > > > > that
> > > > > > > > > > would
> > > > > > > > > > > > be
> > > > > > > > > > > > > outside of its realm of "applicability". There is
> > > still a
> > > > > > need
> > > > > > > > for
> > > > > > > > > a
> > > > > > > > > > > > > variety of testing applications and bench-marking
> > > > > > applications
> > > > > > > > for
> > > > > > > > > > > those
> > > > > > > > > > > > > scenarios.
> > > > > > > > > > > > >
> > > > > > > > > > > > > That makes sense, prerequisites would need to be
> > > > > considered.
> > > > > > I
> > > > > > > > > think
> > > > > > > > > > > > > targeting the low-hanging fruit of unit-testable
> code
> > > > would
> > > > > > be
> > > > > > > > easy
> > > > > > > > > > in
> > > > > > > > > > > > that
> > > > > > > > > > > > > case, since prerequisites are mostly encoded in the
> > > > Kconfig
> > > > > > > > > > definitions
> > > > > > > > > > > > > existing or not for features to be tested. Any
> > complex
> > > > > > > behaviour
> > > > > > > > > > > > > (interrupts, multi-threading, etc.) might be out of
> > the
> > > > > realm
> > > > > > > of
> > > > > > > > > > what's
> > > > > > > > > > > > > testable by Unity in the first place. We'll always
> > have
> > > > OS
> > > > > > test
> > > > > > > > in
> > > > > > > > > > that
> > > > > > > > > > > > > case, but maybe then it's better to do what Alan
> > > > suggested
> > > > > > and
> > > > > > > at
> > > > > > > > > > least
> > > > > > > > > > > > > make its output more clear in the meantime. I
> suppose
> > > > RTOS
> > > > > > > > testing
> > > > > > > > > > is a
> > > > > > > > > > > > > whole field in itself.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> > > > > > > > spudaneco@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I think that the answer depends on a tradeoff
> > between
> > > > > > effort
> > > > > > > > and
> > > > > > > > > > test
> > > > > > > > > > > > > > coverage. I could imagine that a proper job
> could
> > be
> > > > > > > man-years
> > > > > > > > > of
> > > > > > > > > > > > effort.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > My only concern with a re-design is to be certain
> > > that
> > > > > > tests
> > > > > > > > are
> > > > > > > > > > > > > > arranged so that the prerequisites of later test
> > are
> > > > > tested
> > > > > > > > > first.
> > > > > > > > > > > If
> > > > > > > > > > > > > > tests are grouped by, for example, functional
> area,
> > > > then
> > > > > > > > failures
> > > > > > > > > > can
> > > > > > > > > > > > be
> > > > > > > > > > > > > > difficult to understand if there is a failure in
> an
> > > > > > > unverified
> > > > > > > > > test
> > > > > > > > > > > > > > prerequisite.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Some of the "tests" are bogus and only exist to
> > > provide
> > > > > > some
> > > > > > > > > > minimum
> > > > > > > > > > > > > > confidence that we have the functionality to run
> > the
> > > > > test.
> > > > > > > For
> > > > > > > > > > > > > > example: Can I start tasks? Can I pass
> > parameters?
> > > > Can
> > > > > I
> > > > > > > > > survive
> > > > > > > > > > a
> > > > > > > > > > > > > > context switch. Some of the tests were intended
> > only
> > > > to
> > > > > > > > support
> > > > > > > > > > > debug.
> > > > > > > > > > > > > > Some were created only to track down specific
> bugs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > > > > > > > > > > > LTS being the Long-Term-Stable release of
> NuttX?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm starting to think that it would be best to
> > > create
> > > > > new
> > > > > > > > suite
> > > > > > > > > > of
> > > > > > > > > > > > tests
> > > > > > > > > > > > > > > using Unity, using what's in OS Test as a basis
> > and
> > > > > then
> > > > > > > > > > expanding
> > > > > > > > > > > it
> > > > > > > > > > > > > > > further. Greg has pointed out that OS test is
> not
> > > > > > intended
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > a
> > > > > > > > > > > > > > complete
> > > > > > > > > > > > > > > test, so maybe a more solid testing framework
> > base
> > > > in a
> > > > > > new
> > > > > > > > > form
> > > > > > > > > > > > would be
> > > > > > > > > > > > > > > in order. At that point it would also be good
> to
> > > > > document
> > > > > > > and
> > > > > > > > > > > educate
> > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > about other test options, since OS test is a
> > > current
> > > > > > > > favourite
> > > > > > > > > > for
> > > > > > > > > > > > > > showing
> > > > > > > > > > > > > > > PRs haven't broken anything. Maybe that's not
> the
> > > > best
> > > > > > > choice
> > > > > > > > > > > anymore
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > this information.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I don't want to re-invent the wheel of what's
> > > already
> > > > > in
> > > > > > > > > > > `testsuite`,
> > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > think Unity might end up being lighter-weight
> in
> > > the
> > > > > long
> > > > > > > > run?
> > > > > > > > > > And
> > > > > > > > > > > > it is
> > > > > > > > > > > > > > > also not a Xiaomi owned/maintained framework so
> > it
> > > > may
> > > > > > > > receive
> > > > > > > > > > > better
> > > > > > > > > > > > > > > external support (i.e. more users rely on
> it?). I
> > > > tried
> > > > > > to
> > > > > > > > > > compile
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > sim:citest configuration to run the testsuite
> > > example
> > > > > and
> > > > > > > see
> > > > > > > > > > what
> > > > > > > > > > > > it was
> > > > > > > > > > > > > > > like, but I was met with a long list of C++
> > > > compilation
> > > > > > > > errors
> > > > > > > > > > that
> > > > > > > > > > > > I'd
> > > > > > > > > > > > > > > rather not deal with. There also seems to be
> zero
> > > > docs
> > > > > > > about
> > > > > > > > it
> > > > > > > > > > or
> > > > > > > > > > > > cmocka
> > > > > > > > > > > > > > > in our NuttX docs at the moment.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > What do you think? Start a Unity test suite,
> add
> > > some
> > > > > > > > switches
> > > > > > > > > to
> > > > > > > > > > > > choose
> > > > > > > > > > > > > > > which test cases get compiled into the binary,
> > open
> > > > for
> > > > > > > > > > extensions
> > > > > > > > > > > by
> > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > who know more about different subsystems? This
> > way
> > > we
> > > > > can
> > > > > > > > keep
> > > > > > > > > OS
> > > > > > > > > > > > test
> > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > it is for a quick, low-resource sanity check of
> > > > "board
> > > > > > > > bringup"
> > > > > > > > > > > while
> > > > > > > > > > > > > > > having a dedicated testing application for
> > patches?
> > > > > > Someone
> > > > > > > > > > > changing
> > > > > > > > > > > > file
> > > > > > > > > > > > > > > system stuff can just compile to include file
> > > system
> > > > > test
> > > > > > > > cases
> > > > > > > > > > and
> > > > > > > > > > > > run
> > > > > > > > > > > > > > > that suite only as proof that their PR works.
> The
> > > > > output
> > > > > > is
> > > > > > > > > > really
> > > > > > > > > > > > easy
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > glance at to determine pass/fail results.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Anyways, here's my draft demonstrating how
> Unity
> > > > works
> > > > > > with
> > > > > > > > the
> > > > > > > > > > FPU
> > > > > > > > > > > > test
> > > > > > > > > > > > > > > suite from OS test:
> > > > > > > > > > https://github.com/apache/nuttx-apps/pull/3259
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Matteo
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> > > > > > > > tomek@cedro.info>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> > > > > > > > > > spudaneco@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >>> (..)
> > > > > > > > > > > > > > >>> Starting with bits and pieces of something
> like
> > > the
> > > > > LTS
> > > > > > > > might
> > > > > > > > > > > make
> > > > > > > > > > > > more
> > > > > > > > > > > > > > >>> sense other than its licensing problems.
> Since
> > > it
> > > > > > would
> > > > > > > > > never
> > > > > > > > > > be
> > > > > > > > > > > > > > >>> included in a distributed binary, the
> licensing
> > > > > really
> > > > > > > > should
> > > > > > > > > > not
> > > > > > > > > > > > > > matter
> > > > > > > > > > > > > > >>> much in practice.
> > > > > > > > > > > > > > >> Unless its GPL and some company would like to
> > have
> > > > > their
> > > > > > > own
> > > > > > > > > > > > internal
> > > > > > > > > > > > > > >> bits and pieces.. luckily its MIT and the
> whole
> > > > thing
> > > > > is
> > > > > > > > > > > open-source
> > > > > > > > > > > > > > >> anyways :-)
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Yes we had many discussions about the LTS and
> > this
> > > > > could
> > > > > > > be
> > > > > > > > > > great
> > > > > > > > > > > > start!
> > > > > > > > > > > > > > >> :-)
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> --
> > > > > > > > > > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
Thank you Xiang for the information! It makes sense of course that you're
not able to share the Xiaomi internal test suites.
Are there any modifications Xiaomi has needed to make to cmocka to improve
it's ability to run on small devices? I know some of your targets are
wearables which I imagine are very resource constrained?
Can you clarify what you mean by monkey test? Is that another test
framework? I'm interested since you mentioned it running on watches and
speakers.
Thanks,
Matteo
On Mon, Dec 22, 2025, 9:08 PM Xiang Xiao <xi...@gmail.com> wrote:
> On Tue, Dec 23, 2025 at 1:08 AM Matteo Golin <ma...@gmail.com>
> wrote:
>
> > Hello everyone,
> >
> > Thanks for all the feedback! I'm glad I discussed this here first. I'll
> try
> > to address everything that was brought up.
> >
> > Thanks Guiding for the input! I must just be missing some library on my
> > machine if you're able to compile it just fine. Maybe I'll track down
> that
> > error and see if you can recognize what's wrong.
> >
> > I don't intend to replace OS test anymore. I think it serves its purpose
> as
> > a minimal sanity check. Instead, I think we need something a little more
> > involved to test PRs since OS test can't catch many issues. It is
> currently
> > the standard everyone uses for verifying their patches and it would be
> > better to have more complete, alternate options. I do think that we can
> all
> > agree that OS test needs a revision of its console output so that it is
> > much easier to see pass/fail results, but that is just a matter of print
> > statements and can be done in a different effort.
> >
> > I think the overwhelming concern here is mixing two unit testing
> frameworks
> > and no longer having one resource where we can go for tests. I share
> raiden
> > and Guiding's concern about that, which is why I'm glad I opened for
> > discussion here. On the one hand, Xiaomi has a pretty substantial
> > collection of tests in the `testsuite` application which they are using
> > internally I believe (in their CI). These run well on virtual machines,
> > which I assume is why the only configurations with that test suite
> included
> > is sim and rv-virt. I do not know nearly enough about cmocka to make any
> > claim that it *wouldn't* work on smaller hardware devices, but it also
> > definitely looks to be less light-weight from the feature set in the
> > documentation. On the other hand, the Xiaomi test suites are also not a
> > complete silver bullet since they seem not to be intended to run on
> > hardware, and there are *a lot *of tests that need hardware verification.
> >
>
> The test suite also runs on hardware daily, but sorry it's impossible to
> share our hardware and internal CI/CD infrastructure for the community.
> But, we already contract raiden00pl to refactor our test automation
> framework for public use.
>
>
> > In PRs we often request more than just a simulator test for certain
> > changes. For instance, sim isn't able to have `/dev/rand`, so that needs
> to
> > be tested on a different target.
> >
>
> Maybe someone more familiar with Xiaomi's testing practices can weigh in on
> > whether cmocka has been run on hardware. Going purely based off of their
> > PRs, which almost exclusively cite tests on simulators, my guess is that
> > cmocka was not designed with embedded in mind when it was written.
> >
> >
> CI/CD in Xiaomi is far more complex than your description:
>
> 1. Each patch will test on 10+ devices(include sim/qemu/hardware,
> arm/arm64/riscv/xtensa/x64, smp/up) with 1000+ test case
> 2. Run the full test suite(10000+ test case) every day
> 3. Run the monkey test every day on watch and speaker product: 24hour X
> 50 instance X (sim + qemu + hardware)
>
> Many stability bugs only happen randomly in monkey test.
>
> As for size: unit testing can get large with Unity or cmocka (as raiden
> > mentioned) as more tests are added. Hence my effort to enable/disable
> > individual test suites from getting compiled. It might not be possible to
> > ever get good testing ability for AVR just because the devices are so
> > small. Whatever solution is used there is probably going to involve
> > flashing the device multiple times with different tests. The size goal
> > should just be to target the most NuttX devices possible.
> >
> > Then there are the pros and cons of each framework (again, I don't know
> > much about cmocka so take it with a grain of salt):
> >
> > Unity's cons:
> > - Designed for each test-suite to be a standalone binary. This is clunky
> on
> > NuttX, so `nest` is currently implemented as a mega-suite where test
> cases
> > just follow a naming convention to make it clear what is being tested
> > - setUp/tearDown can't really be leveraged because of the above
> > - I've noticed some funky behaviour using TEST_IGNORE in certain
> scenarios
> > but that's possibly my own configuration issue
> > - Would be in conflict with existing cmocka tests
> >
> > Unity's pros:
> > - Highly configurable, like NuttX
> > - Light-weight and works for embedded devices
> > - Significantly larger user base than cmocka
> >
> > Cmocka's cons:
> > - Possibly (?) not light-weight
> > - No documentation for existing test cases in NuttX
> > - Small user base
> >
> > Cmocka's pros:
> > - Seems to have a larger set of features, including setup/teardown
> > functions per-test case
> > - Existing tests in NuttX
> >
> >
> Xiaomi selected cmocka just because it has the test suite concept which
> could help us manage 10000+ tests.
> But we also use gtest for the c++ code base, and run many test cases from
> 3rd parties(e.g. LTP, libuv, libcxx testsuite, PSE52...) which may use a
> different test framework.
> So, is it hard or meaningless to stick to one unit test framework for an
> open source project.
>
> Irrespective of whether we go with Unity or cmocka, the collection of tests
> > that we write will be intended to run on the actual hardware devices
> > themselves for the purposes of testing patches and doing regression
> tests.
> > As a result, the tests will be separate from the Xiaomi 'testsuite'
> > application which is intended for CI testing. I think this realistically
> > nullifies choosing which framework to use on the basis of consistency,
> > since the tests are not coexisting with the cmocka tests Xiaomi has
> > written. Instead, the choice should be based on what best achieves the
> goal
> > from a functionality perspective. Unity seems like a really great choice
> > for this in my opinion, but I am bothered by the fact that it's designed
> > with multiple test binaries in mind.
> >
> > Let me know what you think,
> > Matteo
> >
> > On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <ac...@gmail.com>
> wrote:
> >
> > > Yes, I agree! We need data size from each solution for comparison, this
> > is
> > > the first step.
> > >
> > > Maybe the conversion could be facilitated if we create equivalent
> > functions
> > > like those used in the apps/testing/testsuites:
> > >
> > > cmocka_run_group_tests -> unity_run_group_tests
> > > cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown
> > > etc...
> > >
> > > This way we avoid end up with two unit test framework, we already have
> > the
> > > Makefile vs CMake precedent! :-D
> > >
> > > On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <ra...@gmail.com>
> > wrote:
> > >
> > > > The discussion about small systems doesn't make much sense without
> > > > data comparing resources of the same test cases for cmock and unity.
> > > >
> > > > But let's assume that Unity is better. Should we use two frameworks
> > > > to implement unit tests for OS? Or choose one framework? If we choose
> > > only
> > > > Unity, who
> > > > will migrate the current cmocka tests to Unity? Remember that cmoka
> > tests
> > > > are used in CI, so all CI tools must be also migrated.
> > > > Or should we use two different frameworks to test different things
> and
> > > make
> > > > even
> > > > more mess in apps/testing as it is now?
> > > >
> > > > No one is forcing anyone to do anything. This is an open-source
> > project,
> > > > everyone
> > > > should do what they feel is right. But whether the change should be
> > > merged
> > > > is
> > > > a different question. In this case, it's a very important question,
> > > because
> > > > unit testing
> > > > isn't just about test cases. It's about the entire infrastructure
> that
> > > > runs, captures and
> > > > handles test results later. Writing the test cases is the least of
> the
> > > > problems.
> > > > The infrastructure for cmoka testing is already in place, the
> > > > infrastructure for Unity
> > > > testing isn't.
> > > >
> > > > pon., 22 gru 2025 o 14:16 Alan C. Assis <ac...@gmail.com>
> > napisał(a):
> > > >
> > > > > Mateusz,
> > > > > but some chips like AVR DA have big flash, but not so much RAM.
> > > > >
> > > > > It is better to have some possible way to test on a small system,
> > using
> > > > > cmocka we are removing this option from the table.
> > > > >
> > > > > I liked the test Matteo did with Unity and I think we should let
> him
> > > > > continue. Forcing him to migrate to cmocka just because there's a
> > test
> > > > > suite from Xiaomi doesn't seem prudent to me.
> > > > >
> > > > > This is the advantage of an open mind open-source project: "the
> > > > possibility
> > > > > to test new ideas", if someone had said to Michal Lenc he couldn't
> > > create
> > > > > NXBoot because where is already MCUBoot integrated on NuttX, we
> > didn't
> > > > have
> > > > > the option to use NXBoot today. ;-)
> > > > >
> > > > > BR,
> > > > >
> > > > > Alan
> > > > >
> > > > > On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <ra...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Unit testing by itself isn't suitable for small systems. Ostest
> is
> > > > > > something
> > > > > > different (system level testing), so running it on small systems
> > > makes
> > > > > > sense.
> > > > > > Running unit tests on small systems is pointless because you
> won't
> > > fit
> > > > in
> > > > > > FLASH
> > > > > > either way. You can run the same on large targets (preferably on
> > > host:
> > > > > sim,
> > > > > > qemu)
> > > > > > and achieve the same result.
> > > > > >
> > > > > > pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com>
> > > > napisał(a):
> > > > > >
> > > > > > > Is it too complicated to rewrite the cmocka tests in Unity?
> > > > > > >
> > > > > > > If cmoka is not feasible for small microcontrollers, like AVRs
> > and
> > > > > > others,
> > > > > > > our test coverage will be faulty.
> > > > > > >
> > > > > > > BR,
> > > > > > >
> > > > > > > Alan
> > > > > > >
> > > > > > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <
> raiden00pl@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > I'm a fan of Unity and have been using it in my projects for
> > > years,
> > > > > > but I
> > > > > > > > think
> > > > > > > > we should stick with one testing framework. Currently, Cmocka
> > is
> > > > used
> > > > > > in
> > > > > > > > nuttx-apps, and we should stick with it. The only question is
> > how
> > > > > well
> > > > > > > > Cmocka
> > > > > > > > is suitable for running on small systems? Will ostest still
> be
> > > able
> > > > > to
> > > > > > > run
> > > > > > > > on small MCUs?
> > > > > > > > With Unity this would be quite likely, so maybe an exception
> > for
> > > > > Unity
> > > > > > > and
> > > > > > > > ostest would
> > > > > > > > be acceptable, but using it for other test cases will only
> > cause
> > > > > chaos.
> > > > > > > >
> > > > > > > > pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com>
> > > > napisał(a):
> > > > > > > >
> > > > > > > > > Hi Matteo,
> > > > > > > > >
> > > > > > > > > Good idea !
> > > > > > > > >
> > > > > > > > > But it is best to stick with a single testing framework.
> > > > Currently,
> > > > > > > many
> > > > > > > > > test cases are using cmocka, so I suggest reusing cmocka
> for
> > > > > > > consistency.
> > > > > > > > >
> > > > > > > > > ostest is a minimal basic test suite, typically used for CT
> > > > > > (continuous
> > > > > > > > > testing) of each patch.
> > > > > > > > > If we move ostest to cmocka, it would make ostest dependent
> > on
> > > a
> > > > > > > specific
> > > > > > > > > framework before it can be run.
> > > > > > > > > And It may increase the code size, which is not very
> friendly
> > > for
> > > > > > > > > configurations with small flash memory.
> > > > > > > > >
> > > > > > > > > As Greg mentioned: the answer depends on a tradeoff between
> > > > effort
> > > > > > and
> > > > > > > > test
> > > > > > > > > coverage.
> > > > > > > > >
> > > > > > > > > We need to clarify whether we define ostest as a
> lightweight,
> > > > > > > > ready-to-run
> > > > > > > > > test set for quick verification, or as part of the full
> test
> > > > suite.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The follow is my compiling for cmocka:
> > > > > > > > >
> > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh
> > > > > sim:citest
> > > > > > > > -j8*
> > > > > > > > > Copy files
> > > > > > > > > Select CONFIG_HOST_LINUX=y
> > > > > > > > > Refreshing...
> > > > > > > > > CP: arch/dummy/Kconfig to
> > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> > > > > > > > > CP: boards/dummy/Kconfig to
> > > > > > > > >
> /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> > > > > > > > > LN: include/arch to arch/sim/include
> > > > > > > > > LN: drivers/platform to
> > > > > > > /home/ligd/platform/mainline/nuttx/drivers/dummy
> > > > > > > > > LN: arch/sim/src/chip to
> > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> > > > > > > > > LN: arch/sim/src/board to
> > > > > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> > > > > > > > > LN: include/arch/board to
> > > > > > > > >
> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> > > > > > > > > LN: include/arch/chip to
> > > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> > > > > > > > > LN: platform/board to
> > > > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/boot
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/canutils
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/database
> > > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/examples/rust
> > > > > > > > > mkkconfig in
> > /home/ligd/platform/mainline/apps/examples/sotest
> > > > > > > > > mkkconfig in
> > /home/ligd/platform/mainline/apps/examples/module
> > > > > > > > > mkkconfig in
> > /home/ligd/platform/mainline/apps/examples/mcuboot
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/games
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/graphics
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/industry
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/inertial
> > > > > > > > > mkkconfig in
> > > > > > /home/ligd/platform/mainline/apps/interpreters/luamodules
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/logging
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/lte
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/math
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/netutils
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/system
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/tee
> > > > > > > > > mkkconfig in
> > /home/ligd/platform/mainline/apps/testing/drivers
> > > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/testing/sched
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> > > > > > > > > mkkconfig in
> > > /home/ligd/platform/mainline/apps/wireless/bluetooth
> > > > > > > > > mkkconfig in
> > > > /home/ligd/platform/mainline/apps/wireless/ieee802154
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless
> > > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps
> > > > > > > > > Loaded configuration '.config'
> > > > > > > > > Configuration saved to '.config'
> > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ make*
> > > > > > > > > Create version.h
> > > > > > > > > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> > > > > > > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > > > > > Downloading: v1.46.0.zip
> > > > > > > > > ...
> > > > > > > > > patching file
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> > > > > > > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> > > > > > > > >
> > > > > > > > > CP:
> > /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> > > > > > > > > CP:
> > > /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> > > > > > > > > LD: nuttx
> > > > > > > > > Pac SIM with dynamic libs..
> > > > > > > > > '/lib/x86_64-linux-gnu/libz.so.1' ->
> 'sim-pac/libs/libz.so.1'
> > > > > > > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
> > > > > > 'sim-pac/libs/libstdc++.so.6'
> > > > > > > > > '/lib/x86_64-linux-gnu/libc.so.6' ->
> 'sim-pac/libs/libc.so.6'
> > > > > > > > > '/lib/x86_64-linux-gnu/libm.so.6' ->
> 'sim-pac/libs/libm.so.6'
> > > > > > > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' ->
> > > > > 'sim-pac/libs/libgcc_s.so.1'
> > > > > > > > > '/lib64/ld-linux-x86-64.so.2' ->
> > 'sim-pac/ld-linux-x86-64.so.2'
> > > > > > > > > SIM elf with dynamic libs archive in nuttx.tgz
> > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> > > > > > > > > CHelloWorld: Constructor: mSecret=42
> > > > > > > > > nsh: init: open failed: 2
> > > > > > > > >
> > > > > > > > > NuttShell (NSH) NuttX-12.11.0
> > > > > > > > > MOTD: username=admin password=Administrator
> > > > > > > > > nsh>
> > > > > > > > > nsh> help
> > > > > > > > > help usage: help [-v] [<cmd>]
> > > > > > > > >
> > > > > > > > > . df ifconfig mkfifo
> readlink
> > > > > > umount
> > > > > > > > > [ dmesg ifdown mkrd rm
> > > > > unset
> > > > > > > > > ? echo ifup mount rmdir
> > > > > > uptime
> > > > > > > > > alias env insmod mv rmmod
> > > > > > usleep
> > > > > > > > > unalias exec kill nslookup set
> > > > > watch
> > > > > > > > > basename exit pkill pidof sleep
> > > > xd
> > > > > > > > > break expr losetup pmconfig source
> > > > > wait
> > > > > > > > > cat false ln poweroff test
> > > > > > > > > cd fdinfo ls quit time
> > > > > > > > > cp free lsmod printf true
> > > > > > > > > cmp help mkdir ps
> truncate
> > > > > > > > > dirname hexdump mkfatfs pwd uname
> > > > > > > > >
> > > > > > > > > Builtin Apps:
> > > > > > > > > cmocka
> > > > > > > > > cmocka_driver_block
> > > > > > > > > cmocka_driver_gpio
> > > > > > > > > cmocka_driver_pm
> > > > > > > > > cmocka_driver_pm_runtime
> > > > > > > > > cmocka_driver_regulator
> > > > > > > > >
> > > > > > > > > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一
> > 10:15写道：
> > > > > > > > >
> > > > > > > > > > I think so. Best to leave OS test alone and slowly
> replace
> > it
> > > > > with
> > > > > > a
> > > > > > > > new
> > > > > > > > > > tool that has improvements. The only reason I can see to
> > > modify
> > > > > OS
> > > > > > > test
> > > > > > > > > now
> > > > > > > > > > would just be to have clearer output about what
> > > passed/failed.
> > > > > > > > > >
> > > > > > > > > > I'll come up with an improved proof of concept for a new
> > tool
> > > > to
> > > > > > > share
> > > > > > > > in
> > > > > > > > > > the next few days.
> > > > > > > > > >
> > > > > > > > > > Matteo
> > > > > > > > > >
> > > > > > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <
> > tomek@cedro.info>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Small steps with measurable results may be the best
> > > approach
> > > > > > here,
> > > > > > > to
> > > > > > > > > > > create some new solution next to existing ostest but
> not
> > > > > > replacing
> > > > > > > it
> > > > > > > > > > > until ready? :-)
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> > > > > > > matteo.golin@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > I agree, I think it would take a long time to have
> > proper
> > > > > > > coverage
> > > > > > > > of
> > > > > > > > > > > NuttX
> > > > > > > > > > > > as a whole. Perhaps even impossible, since Unity is
> > more
> > > > of a
> > > > > > > unit
> > > > > > > > > > > testing
> > > > > > > > > > > > framework and thus there are definitely test cases
> for
> > > > NuttX
> > > > > > that
> > > > > > > > > would
> > > > > > > > > > > be
> > > > > > > > > > > > outside of its realm of "applicability". There is
> > still a
> > > > > need
> > > > > > > for
> > > > > > > > a
> > > > > > > > > > > > variety of testing applications and bench-marking
> > > > > applications
> > > > > > > for
> > > > > > > > > > those
> > > > > > > > > > > > scenarios.
> > > > > > > > > > > >
> > > > > > > > > > > > That makes sense, prerequisites would need to be
> > > > considered.
> > > > > I
> > > > > > > > think
> > > > > > > > > > > > targeting the low-hanging fruit of unit-testable code
> > > would
> > > > > be
> > > > > > > easy
> > > > > > > > > in
> > > > > > > > > > > that
> > > > > > > > > > > > case, since prerequisites are mostly encoded in the
> > > Kconfig
> > > > > > > > > definitions
> > > > > > > > > > > > existing or not for features to be tested. Any
> complex
> > > > > > behaviour
> > > > > > > > > > > > (interrupts, multi-threading, etc.) might be out of
> the
> > > > realm
> > > > > > of
> > > > > > > > > what's
> > > > > > > > > > > > testable by Unity in the first place. We'll always
> have
> > > OS
> > > > > test
> > > > > > > in
> > > > > > > > > that
> > > > > > > > > > > > case, but maybe then it's better to do what Alan
> > > suggested
> > > > > and
> > > > > > at
> > > > > > > > > least
> > > > > > > > > > > > make its output more clear in the meantime. I suppose
> > > RTOS
> > > > > > > testing
> > > > > > > > > is a
> > > > > > > > > > > > whole field in itself.
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> > > > > > > spudaneco@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I think that the answer depends on a tradeoff
> between
> > > > > effort
> > > > > > > and
> > > > > > > > > test
> > > > > > > > > > > > > coverage. I could imagine that a proper job could
> be
> > > > > > man-years
> > > > > > > > of
> > > > > > > > > > > effort.
> > > > > > > > > > > > >
> > > > > > > > > > > > > My only concern with a re-design is to be certain
> > that
> > > > > tests
> > > > > > > are
> > > > > > > > > > > > > arranged so that the prerequisites of later test
> are
> > > > tested
> > > > > > > > first.
> > > > > > > > > > If
> > > > > > > > > > > > > tests are grouped by, for example, functional area,
> > > then
> > > > > > > failures
> > > > > > > > > can
> > > > > > > > > > > be
> > > > > > > > > > > > > difficult to understand if there is a failure in an
> > > > > > unverified
> > > > > > > > test
> > > > > > > > > > > > > prerequisite.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Some of the "tests" are bogus and only exist to
> > provide
> > > > > some
> > > > > > > > > minimum
> > > > > > > > > > > > > confidence that we have the functionality to run
> the
> > > > test.
> > > > > > For
> > > > > > > > > > > > > example: Can I start tasks? Can I pass
> parameters?
> > > Can
> > > > I
> > > > > > > > survive
> > > > > > > > > a
> > > > > > > > > > > > > context switch. Some of the tests were intended
> only
> > > to
> > > > > > > support
> > > > > > > > > > debug.
> > > > > > > > > > > > > Some were created only to track down specific bugs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > > > > > > > > > > LTS being the Long-Term-Stable release of NuttX?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'm starting to think that it would be best to
> > create
> > > > new
> > > > > > > suite
> > > > > > > > > of
> > > > > > > > > > > tests
> > > > > > > > > > > > > > using Unity, using what's in OS Test as a basis
> and
> > > > then
> > > > > > > > > expanding
> > > > > > > > > > it
> > > > > > > > > > > > > > further. Greg has pointed out that OS test is not
> > > > > intended
> > > > > > to
> > > > > > > > be
> > > > > > > > > a
> > > > > > > > > > > > > complete
> > > > > > > > > > > > > > test, so maybe a more solid testing framework
> base
> > > in a
> > > > > new
> > > > > > > > form
> > > > > > > > > > > would be
> > > > > > > > > > > > > > in order. At that point it would also be good to
> > > > document
> > > > > > and
> > > > > > > > > > educate
> > > > > > > > > > > > > users
> > > > > > > > > > > > > > about other test options, since OS test is a
> > current
> > > > > > > favourite
> > > > > > > > > for
> > > > > > > > > > > > > showing
> > > > > > > > > > > > > > PRs haven't broken anything. Maybe that's not the
> > > best
> > > > > > choice
> > > > > > > > > > anymore
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > this information.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I don't want to re-invent the wheel of what's
> > already
> > > > in
> > > > > > > > > > `testsuite`,
> > > > > > > > > > > > > but I
> > > > > > > > > > > > > > think Unity might end up being lighter-weight in
> > the
> > > > long
> > > > > > > run?
> > > > > > > > > And
> > > > > > > > > > > it is
> > > > > > > > > > > > > > also not a Xiaomi owned/maintained framework so
> it
> > > may
> > > > > > > receive
> > > > > > > > > > better
> > > > > > > > > > > > > > external support (i.e. more users rely on it?). I
> > > tried
> > > > > to
> > > > > > > > > compile
> > > > > > > > > > > the
> > > > > > > > > > > > > > sim:citest configuration to run the testsuite
> > example
> > > > and
> > > > > > see
> > > > > > > > > what
> > > > > > > > > > > it was
> > > > > > > > > > > > > > like, but I was met with a long list of C++
> > > compilation
> > > > > > > errors
> > > > > > > > > that
> > > > > > > > > > > I'd
> > > > > > > > > > > > > > rather not deal with. There also seems to be zero
> > > docs
> > > > > > about
> > > > > > > it
> > > > > > > > > or
> > > > > > > > > > > cmocka
> > > > > > > > > > > > > > in our NuttX docs at the moment.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > What do you think? Start a Unity test suite, add
> > some
> > > > > > > switches
> > > > > > > > to
> > > > > > > > > > > choose
> > > > > > > > > > > > > > which test cases get compiled into the binary,
> open
> > > for
> > > > > > > > > extensions
> > > > > > > > > > by
> > > > > > > > > > > > > users
> > > > > > > > > > > > > > who know more about different subsystems? This
> way
> > we
> > > > can
> > > > > > > keep
> > > > > > > > OS
> > > > > > > > > > > test
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > it is for a quick, low-resource sanity check of
> > > "board
> > > > > > > bringup"
> > > > > > > > > > while
> > > > > > > > > > > > > > having a dedicated testing application for
> patches?
> > > > > Someone
> > > > > > > > > > changing
> > > > > > > > > > > file
> > > > > > > > > > > > > > system stuff can just compile to include file
> > system
> > > > test
> > > > > > > cases
> > > > > > > > > and
> > > > > > > > > > > run
> > > > > > > > > > > > > > that suite only as proof that their PR works. The
> > > > output
> > > > > is
> > > > > > > > > really
> > > > > > > > > > > easy
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > glance at to determine pass/fail results.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Anyways, here's my draft demonstrating how Unity
> > > works
> > > > > with
> > > > > > > the
> > > > > > > > > FPU
> > > > > > > > > > > test
> > > > > > > > > > > > > > suite from OS test:
> > > > > > > > > https://github.com/apache/nuttx-apps/pull/3259
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Matteo
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> > > > > > > tomek@cedro.info>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> > > > > > > > > spudaneco@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >>> (..)
> > > > > > > > > > > > > >>> Starting with bits and pieces of something like
> > the
> > > > LTS
> > > > > > > might
> > > > > > > > > > make
> > > > > > > > > > > more
> > > > > > > > > > > > > >>> sense other than its licensing problems. Since
> > it
> > > > > would
> > > > > > > > never
> > > > > > > > > be
> > > > > > > > > > > > > >>> included in a distributed binary, the licensing
> > > > really
> > > > > > > should
> > > > > > > > > not
> > > > > > > > > > > > > matter
> > > > > > > > > > > > > >>> much in practice.
> > > > > > > > > > > > > >> Unless its GPL and some company would like to
> have
> > > > their
> > > > > > own
> > > > > > > > > > > internal
> > > > > > > > > > > > > >> bits and pieces.. luckily its MIT and the whole
> > > thing
> > > > is
> > > > > > > > > > open-source
> > > > > > > > > > > > > >> anyways :-)
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Yes we had many discussions about the LTS and
> this
> > > > could
> > > > > > be
> > > > > > > > > great
> > > > > > > > > > > start!
> > > > > > > > > > > > > >> :-)
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> --
> > > > > > > > > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: OS Test re-factor proposal
Posted by Xiang Xiao <xi...@gmail.com>.
On Tue, Dec 23, 2025 at 1:08 AM Matteo Golin <ma...@gmail.com> wrote:
> Hello everyone,
>
> Thanks for all the feedback! I'm glad I discussed this here first. I'll try
> to address everything that was brought up.
>
> Thanks Guiding for the input! I must just be missing some library on my
> machine if you're able to compile it just fine. Maybe I'll track down that
> error and see if you can recognize what's wrong.
>
> I don't intend to replace OS test anymore. I think it serves its purpose as
> a minimal sanity check. Instead, I think we need something a little more
> involved to test PRs since OS test can't catch many issues. It is currently
> the standard everyone uses for verifying their patches and it would be
> better to have more complete, alternate options. I do think that we can all
> agree that OS test needs a revision of its console output so that it is
> much easier to see pass/fail results, but that is just a matter of print
> statements and can be done in a different effort.
>
> I think the overwhelming concern here is mixing two unit testing frameworks
> and no longer having one resource where we can go for tests. I share raiden
> and Guiding's concern about that, which is why I'm glad I opened for
> discussion here. On the one hand, Xiaomi has a pretty substantial
> collection of tests in the `testsuite` application which they are using
> internally I believe (in their CI). These run well on virtual machines,
> which I assume is why the only configurations with that test suite included
> is sim and rv-virt. I do not know nearly enough about cmocka to make any
> claim that it *wouldn't* work on smaller hardware devices, but it also
> definitely looks to be less light-weight from the feature set in the
> documentation. On the other hand, the Xiaomi test suites are also not a
> complete silver bullet since they seem not to be intended to run on
> hardware, and there are *a lot *of tests that need hardware verification.
>
The test suite also runs on hardware daily, but sorry it's impossible to
share our hardware and internal CI/CD infrastructure for the community.
But, we already contract raiden00pl to refactor our test automation
framework for public use.
> In PRs we often request more than just a simulator test for certain
> changes. For instance, sim isn't able to have `/dev/rand`, so that needs to
> be tested on a different target.
>
Maybe someone more familiar with Xiaomi's testing practices can weigh in on
> whether cmocka has been run on hardware. Going purely based off of their
> PRs, which almost exclusively cite tests on simulators, my guess is that
> cmocka was not designed with embedded in mind when it was written.
>
>
CI/CD in Xiaomi is far more complex than your description:
1. Each patch will test on 10+ devices(include sim/qemu/hardware,
arm/arm64/riscv/xtensa/x64, smp/up) with 1000+ test case
2. Run the full test suite(10000+ test case) every day
3. Run the monkey test every day on watch and speaker product: 24hour X
50 instance X (sim + qemu + hardware)
Many stability bugs only happen randomly in monkey test.
As for size: unit testing can get large with Unity or cmocka (as raiden
> mentioned) as more tests are added. Hence my effort to enable/disable
> individual test suites from getting compiled. It might not be possible to
> ever get good testing ability for AVR just because the devices are so
> small. Whatever solution is used there is probably going to involve
> flashing the device multiple times with different tests. The size goal
> should just be to target the most NuttX devices possible.
>
> Then there are the pros and cons of each framework (again, I don't know
> much about cmocka so take it with a grain of salt):
>
> Unity's cons:
> - Designed for each test-suite to be a standalone binary. This is clunky on
> NuttX, so `nest` is currently implemented as a mega-suite where test cases
> just follow a naming convention to make it clear what is being tested
> - setUp/tearDown can't really be leveraged because of the above
> - I've noticed some funky behaviour using TEST_IGNORE in certain scenarios
> but that's possibly my own configuration issue
> - Would be in conflict with existing cmocka tests
>
> Unity's pros:
> - Highly configurable, like NuttX
> - Light-weight and works for embedded devices
> - Significantly larger user base than cmocka
>
> Cmocka's cons:
> - Possibly (?) not light-weight
> - No documentation for existing test cases in NuttX
> - Small user base
>
> Cmocka's pros:
> - Seems to have a larger set of features, including setup/teardown
> functions per-test case
> - Existing tests in NuttX
>
>
Xiaomi selected cmocka just because it has the test suite concept which
could help us manage 10000+ tests.
But we also use gtest for the c++ code base, and run many test cases from
3rd parties(e.g. LTP, libuv, libcxx testsuite, PSE52...) which may use a
different test framework.
So, is it hard or meaningless to stick to one unit test framework for an
open source project.
Irrespective of whether we go with Unity or cmocka, the collection of tests
> that we write will be intended to run on the actual hardware devices
> themselves for the purposes of testing patches and doing regression tests.
> As a result, the tests will be separate from the Xiaomi 'testsuite'
> application which is intended for CI testing. I think this realistically
> nullifies choosing which framework to use on the basis of consistency,
> since the tests are not coexisting with the cmocka tests Xiaomi has
> written. Instead, the choice should be based on what best achieves the goal
> from a functionality perspective. Unity seems like a really great choice
> for this in my opinion, but I am bothered by the fact that it's designed
> with multiple test binaries in mind.
>
> Let me know what you think,
> Matteo
>
> On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <ac...@gmail.com> wrote:
>
> > Yes, I agree! We need data size from each solution for comparison, this
> is
> > the first step.
> >
> > Maybe the conversion could be facilitated if we create equivalent
> functions
> > like those used in the apps/testing/testsuites:
> >
> > cmocka_run_group_tests -> unity_run_group_tests
> > cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown
> > etc...
> >
> > This way we avoid end up with two unit test framework, we already have
> the
> > Makefile vs CMake precedent! :-D
> >
> > On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <ra...@gmail.com>
> wrote:
> >
> > > The discussion about small systems doesn't make much sense without
> > > data comparing resources of the same test cases for cmock and unity.
> > >
> > > But let's assume that Unity is better. Should we use two frameworks
> > > to implement unit tests for OS? Or choose one framework? If we choose
> > only
> > > Unity, who
> > > will migrate the current cmocka tests to Unity? Remember that cmoka
> tests
> > > are used in CI, so all CI tools must be also migrated.
> > > Or should we use two different frameworks to test different things and
> > make
> > > even
> > > more mess in apps/testing as it is now?
> > >
> > > No one is forcing anyone to do anything. This is an open-source
> project,
> > > everyone
> > > should do what they feel is right. But whether the change should be
> > merged
> > > is
> > > a different question. In this case, it's a very important question,
> > because
> > > unit testing
> > > isn't just about test cases. It's about the entire infrastructure that
> > > runs, captures and
> > > handles test results later. Writing the test cases is the least of the
> > > problems.
> > > The infrastructure for cmoka testing is already in place, the
> > > infrastructure for Unity
> > > testing isn't.
> > >
> > > pon., 22 gru 2025 o 14:16 Alan C. Assis <ac...@gmail.com>
> napisał(a):
> > >
> > > > Mateusz,
> > > > but some chips like AVR DA have big flash, but not so much RAM.
> > > >
> > > > It is better to have some possible way to test on a small system,
> using
> > > > cmocka we are removing this option from the table.
> > > >
> > > > I liked the test Matteo did with Unity and I think we should let him
> > > > continue. Forcing him to migrate to cmocka just because there's a
> test
> > > > suite from Xiaomi doesn't seem prudent to me.
> > > >
> > > > This is the advantage of an open mind open-source project: "the
> > > possibility
> > > > to test new ideas", if someone had said to Michal Lenc he couldn't
> > create
> > > > NXBoot because where is already MCUBoot integrated on NuttX, we
> didn't
> > > have
> > > > the option to use NXBoot today. ;-)
> > > >
> > > > BR,
> > > >
> > > > Alan
> > > >
> > > > On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <ra...@gmail.com>
> > > wrote:
> > > >
> > > > > Unit testing by itself isn't suitable for small systems. Ostest is
> > > > > something
> > > > > different (system level testing), so running it on small systems
> > makes
> > > > > sense.
> > > > > Running unit tests on small systems is pointless because you won't
> > fit
> > > in
> > > > > FLASH
> > > > > either way. You can run the same on large targets (preferably on
> > host:
> > > > sim,
> > > > > qemu)
> > > > > and achieve the same result.
> > > > >
> > > > > pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com>
> > > napisał(a):
> > > > >
> > > > > > Is it too complicated to rewrite the cmocka tests in Unity?
> > > > > >
> > > > > > If cmoka is not feasible for small microcontrollers, like AVRs
> and
> > > > > others,
> > > > > > our test coverage will be faulty.
> > > > > >
> > > > > > BR,
> > > > > >
> > > > > > Alan
> > > > > >
> > > > > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <raiden00pl@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > I'm a fan of Unity and have been using it in my projects for
> > years,
> > > > > but I
> > > > > > > think
> > > > > > > we should stick with one testing framework. Currently, Cmocka
> is
> > > used
> > > > > in
> > > > > > > nuttx-apps, and we should stick with it. The only question is
> how
> > > > well
> > > > > > > Cmocka
> > > > > > > is suitable for running on small systems? Will ostest still be
> > able
> > > > to
> > > > > > run
> > > > > > > on small MCUs?
> > > > > > > With Unity this would be quite likely, so maybe an exception
> for
> > > > Unity
> > > > > > and
> > > > > > > ostest would
> > > > > > > be acceptable, but using it for other test cases will only
> cause
> > > > chaos.
> > > > > > >
> > > > > > > pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com>
> > > napisał(a):
> > > > > > >
> > > > > > > > Hi Matteo,
> > > > > > > >
> > > > > > > > Good idea !
> > > > > > > >
> > > > > > > > But it is best to stick with a single testing framework.
> > > Currently,
> > > > > > many
> > > > > > > > test cases are using cmocka, so I suggest reusing cmocka for
> > > > > > consistency.
> > > > > > > >
> > > > > > > > ostest is a minimal basic test suite, typically used for CT
> > > > > (continuous
> > > > > > > > testing) of each patch.
> > > > > > > > If we move ostest to cmocka, it would make ostest dependent
> on
> > a
> > > > > > specific
> > > > > > > > framework before it can be run.
> > > > > > > > And It may increase the code size, which is not very friendly
> > for
> > > > > > > > configurations with small flash memory.
> > > > > > > >
> > > > > > > > As Greg mentioned: the answer depends on a tradeoff between
> > > effort
> > > > > and
> > > > > > > test
> > > > > > > > coverage.
> > > > > > > >
> > > > > > > > We need to clarify whether we define ostest as a lightweight,
> > > > > > > ready-to-run
> > > > > > > > test set for quick verification, or as part of the full test
> > > suite.
> > > > > > > >
> > > > > > > >
> > > > > > > > The follow is my compiling for cmocka:
> > > > > > > >
> > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh
> > > > sim:citest
> > > > > > > -j8*
> > > > > > > > Copy files
> > > > > > > > Select CONFIG_HOST_LINUX=y
> > > > > > > > Refreshing...
> > > > > > > > CP: arch/dummy/Kconfig to
> > > > > > > > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> > > > > > > > CP: boards/dummy/Kconfig to
> > > > > > > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> > > > > > > > LN: include/arch to arch/sim/include
> > > > > > > > LN: drivers/platform to
> > > > > > /home/ligd/platform/mainline/nuttx/drivers/dummy
> > > > > > > > LN: arch/sim/src/chip to
> > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> > > > > > > > LN: arch/sim/src/board to
> > > > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> > > > > > > > LN: include/arch/board to
> > > > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> > > > > > > > LN: include/arch/chip to
> > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> > > > > > > > LN: platform/board to
> > > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/boot
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/canutils
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/database
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/examples/sotest
> > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/examples/module
> > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/examples/mcuboot
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/games
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/graphics
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/industry
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/inertial
> > > > > > > > mkkconfig in
> > > > > /home/ligd/platform/mainline/apps/interpreters/luamodules
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/logging
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/lte
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/math
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/netutils
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/system
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/tee
> > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/testing/drivers
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> > > > > > > > mkkconfig in
> > /home/ligd/platform/mainline/apps/wireless/bluetooth
> > > > > > > > mkkconfig in
> > > /home/ligd/platform/mainline/apps/wireless/ieee802154
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps
> > > > > > > > Loaded configuration '.config'
> > > > > > > > Configuration saved to '.config'
> > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ make*
> > > > > > > > Create version.h
> > > > > > > > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> > > > > > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > > > > Downloading: v1.46.0.zip
> > > > > > > > ...
> > > > > > > > patching file
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> > > > > > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> > > > > > > >
> > > > > > > > CP:
> /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> > > > > > > > CP:
> > /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> > > > > > > > LD: nuttx
> > > > > > > > Pac SIM with dynamic libs..
> > > > > > > > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> > > > > > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
> > > > > 'sim-pac/libs/libstdc++.so.6'
> > > > > > > > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> > > > > > > > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> > > > > > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' ->
> > > > 'sim-pac/libs/libgcc_s.so.1'
> > > > > > > > '/lib64/ld-linux-x86-64.so.2' ->
> 'sim-pac/ld-linux-x86-64.so.2'
> > > > > > > > SIM elf with dynamic libs archive in nuttx.tgz
> > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> > > > > > > > CHelloWorld: Constructor: mSecret=42
> > > > > > > > nsh: init: open failed: 2
> > > > > > > >
> > > > > > > > NuttShell (NSH) NuttX-12.11.0
> > > > > > > > MOTD: username=admin password=Administrator
> > > > > > > > nsh>
> > > > > > > > nsh> help
> > > > > > > > help usage: help [-v] [<cmd>]
> > > > > > > >
> > > > > > > > . df ifconfig mkfifo readlink
> > > > > umount
> > > > > > > > [ dmesg ifdown mkrd rm
> > > > unset
> > > > > > > > ? echo ifup mount rmdir
> > > > > uptime
> > > > > > > > alias env insmod mv rmmod
> > > > > usleep
> > > > > > > > unalias exec kill nslookup set
> > > > watch
> > > > > > > > basename exit pkill pidof sleep
> > > xd
> > > > > > > > break expr losetup pmconfig source
> > > > wait
> > > > > > > > cat false ln poweroff test
> > > > > > > > cd fdinfo ls quit time
> > > > > > > > cp free lsmod printf true
> > > > > > > > cmp help mkdir ps truncate
> > > > > > > > dirname hexdump mkfatfs pwd uname
> > > > > > > >
> > > > > > > > Builtin Apps:
> > > > > > > > cmocka
> > > > > > > > cmocka_driver_block
> > > > > > > > cmocka_driver_gpio
> > > > > > > > cmocka_driver_pm
> > > > > > > > cmocka_driver_pm_runtime
> > > > > > > > cmocka_driver_regulator
> > > > > > > >
> > > > > > > > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一
> 10:15写道：
> > > > > > > >
> > > > > > > > > I think so. Best to leave OS test alone and slowly replace
> it
> > > > with
> > > > > a
> > > > > > > new
> > > > > > > > > tool that has improvements. The only reason I can see to
> > modify
> > > > OS
> > > > > > test
> > > > > > > > now
> > > > > > > > > would just be to have clearer output about what
> > passed/failed.
> > > > > > > > >
> > > > > > > > > I'll come up with an improved proof of concept for a new
> tool
> > > to
> > > > > > share
> > > > > > > in
> > > > > > > > > the next few days.
> > > > > > > > >
> > > > > > > > > Matteo
> > > > > > > > >
> > > > > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <
> tomek@cedro.info>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Small steps with measurable results may be the best
> > approach
> > > > > here,
> > > > > > to
> > > > > > > > > > create some new solution next to existing ostest but not
> > > > > replacing
> > > > > > it
> > > > > > > > > > until ready? :-)
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > > > >
> > > > > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> > > > > > matteo.golin@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > I agree, I think it would take a long time to have
> proper
> > > > > > coverage
> > > > > > > of
> > > > > > > > > > NuttX
> > > > > > > > > > > as a whole. Perhaps even impossible, since Unity is
> more
> > > of a
> > > > > > unit
> > > > > > > > > > testing
> > > > > > > > > > > framework and thus there are definitely test cases for
> > > NuttX
> > > > > that
> > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > > outside of its realm of "applicability". There is
> still a
> > > > need
> > > > > > for
> > > > > > > a
> > > > > > > > > > > variety of testing applications and bench-marking
> > > > applications
> > > > > > for
> > > > > > > > > those
> > > > > > > > > > > scenarios.
> > > > > > > > > > >
> > > > > > > > > > > That makes sense, prerequisites would need to be
> > > considered.
> > > > I
> > > > > > > think
> > > > > > > > > > > targeting the low-hanging fruit of unit-testable code
> > would
> > > > be
> > > > > > easy
> > > > > > > > in
> > > > > > > > > > that
> > > > > > > > > > > case, since prerequisites are mostly encoded in the
> > Kconfig
> > > > > > > > definitions
> > > > > > > > > > > existing or not for features to be tested. Any complex
> > > > > behaviour
> > > > > > > > > > > (interrupts, multi-threading, etc.) might be out of the
> > > realm
> > > > > of
> > > > > > > > what's
> > > > > > > > > > > testable by Unity in the first place. We'll always have
> > OS
> > > > test
> > > > > > in
> > > > > > > > that
> > > > > > > > > > > case, but maybe then it's better to do what Alan
> > suggested
> > > > and
> > > > > at
> > > > > > > > least
> > > > > > > > > > > make its output more clear in the meantime. I suppose
> > RTOS
> > > > > > testing
> > > > > > > > is a
> > > > > > > > > > > whole field in itself.
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> > > > > > spudaneco@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I think that the answer depends on a tradeoff between
> > > > effort
> > > > > > and
> > > > > > > > test
> > > > > > > > > > > > coverage. I could imagine that a proper job could be
> > > > > man-years
> > > > > > > of
> > > > > > > > > > effort.
> > > > > > > > > > > >
> > > > > > > > > > > > My only concern with a re-design is to be certain
> that
> > > > tests
> > > > > > are
> > > > > > > > > > > > arranged so that the prerequisites of later test are
> > > tested
> > > > > > > first.
> > > > > > > > > If
> > > > > > > > > > > > tests are grouped by, for example, functional area,
> > then
> > > > > > failures
> > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > > > difficult to understand if there is a failure in an
> > > > > unverified
> > > > > > > test
> > > > > > > > > > > > prerequisite.
> > > > > > > > > > > >
> > > > > > > > > > > > Some of the "tests" are bogus and only exist to
> provide
> > > > some
> > > > > > > > minimum
> > > > > > > > > > > > confidence that we have the functionality to run the
> > > test.
> > > > > For
> > > > > > > > > > > > example: Can I start tasks? Can I pass parameters?
> > Can
> > > I
> > > > > > > survive
> > > > > > > > a
> > > > > > > > > > > > context switch. Some of the tests were intended only
> > to
> > > > > > support
> > > > > > > > > debug.
> > > > > > > > > > > > Some were created only to track down specific bugs.
> > > > > > > > > > > >
> > > > > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > > > > > > > > > LTS being the Long-Term-Stable release of NuttX?
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm starting to think that it would be best to
> create
> > > new
> > > > > > suite
> > > > > > > > of
> > > > > > > > > > tests
> > > > > > > > > > > > > using Unity, using what's in OS Test as a basis and
> > > then
> > > > > > > > expanding
> > > > > > > > > it
> > > > > > > > > > > > > further. Greg has pointed out that OS test is not
> > > > intended
> > > > > to
> > > > > > > be
> > > > > > > > a
> > > > > > > > > > > > complete
> > > > > > > > > > > > > test, so maybe a more solid testing framework base
> > in a
> > > > new
> > > > > > > form
> > > > > > > > > > would be
> > > > > > > > > > > > > in order. At that point it would also be good to
> > > document
> > > > > and
> > > > > > > > > educate
> > > > > > > > > > > > users
> > > > > > > > > > > > > about other test options, since OS test is a
> current
> > > > > > favourite
> > > > > > > > for
> > > > > > > > > > > > showing
> > > > > > > > > > > > > PRs haven't broken anything. Maybe that's not the
> > best
> > > > > choice
> > > > > > > > > anymore
> > > > > > > > > > > > with
> > > > > > > > > > > > > this information.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I don't want to re-invent the wheel of what's
> already
> > > in
> > > > > > > > > `testsuite`,
> > > > > > > > > > > > but I
> > > > > > > > > > > > > think Unity might end up being lighter-weight in
> the
> > > long
> > > > > > run?
> > > > > > > > And
> > > > > > > > > > it is
> > > > > > > > > > > > > also not a Xiaomi owned/maintained framework so it
> > may
> > > > > > receive
> > > > > > > > > better
> > > > > > > > > > > > > external support (i.e. more users rely on it?). I
> > tried
> > > > to
> > > > > > > > compile
> > > > > > > > > > the
> > > > > > > > > > > > > sim:citest configuration to run the testsuite
> example
> > > and
> > > > > see
> > > > > > > > what
> > > > > > > > > > it was
> > > > > > > > > > > > > like, but I was met with a long list of C++
> > compilation
> > > > > > errors
> > > > > > > > that
> > > > > > > > > > I'd
> > > > > > > > > > > > > rather not deal with. There also seems to be zero
> > docs
> > > > > about
> > > > > > it
> > > > > > > > or
> > > > > > > > > > cmocka
> > > > > > > > > > > > > in our NuttX docs at the moment.
> > > > > > > > > > > > >
> > > > > > > > > > > > > What do you think? Start a Unity test suite, add
> some
> > > > > > switches
> > > > > > > to
> > > > > > > > > > choose
> > > > > > > > > > > > > which test cases get compiled into the binary, open
> > for
> > > > > > > > extensions
> > > > > > > > > by
> > > > > > > > > > > > users
> > > > > > > > > > > > > who know more about different subsystems? This way
> we
> > > can
> > > > > > keep
> > > > > > > OS
> > > > > > > > > > test
> > > > > > > > > > > > how
> > > > > > > > > > > > > it is for a quick, low-resource sanity check of
> > "board
> > > > > > bringup"
> > > > > > > > > while
> > > > > > > > > > > > > having a dedicated testing application for patches?
> > > > Someone
> > > > > > > > > changing
> > > > > > > > > > file
> > > > > > > > > > > > > system stuff can just compile to include file
> system
> > > test
> > > > > > cases
> > > > > > > > and
> > > > > > > > > > run
> > > > > > > > > > > > > that suite only as proof that their PR works. The
> > > output
> > > > is
> > > > > > > > really
> > > > > > > > > > easy
> > > > > > > > > > > > to
> > > > > > > > > > > > > glance at to determine pass/fail results.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Anyways, here's my draft demonstrating how Unity
> > works
> > > > with
> > > > > > the
> > > > > > > > FPU
> > > > > > > > > > test
> > > > > > > > > > > > > suite from OS test:
> > > > > > > > https://github.com/apache/nuttx-apps/pull/3259
> > > > > > > > > > > > >
> > > > > > > > > > > > > Matteo
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> > > > > > tomek@cedro.info>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> > > > > > > > spudaneco@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >>> (..)
> > > > > > > > > > > > >>> Starting with bits and pieces of something like
> the
> > > LTS
> > > > > > might
> > > > > > > > > make
> > > > > > > > > > more
> > > > > > > > > > > > >>> sense other than its licensing problems. Since
> it
> > > > would
> > > > > > > never
> > > > > > > > be
> > > > > > > > > > > > >>> included in a distributed binary, the licensing
> > > really
> > > > > > should
> > > > > > > > not
> > > > > > > > > > > > matter
> > > > > > > > > > > > >>> much in practice.
> > > > > > > > > > > > >> Unless its GPL and some company would like to have
> > > their
> > > > > own
> > > > > > > > > > internal
> > > > > > > > > > > > >> bits and pieces.. luckily its MIT and the whole
> > thing
> > > is
> > > > > > > > > open-source
> > > > > > > > > > > > >> anyways :-)
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Yes we had many discussions about the LTS and this
> > > could
> > > > > be
> > > > > > > > great
> > > > > > > > > > start!
> > > > > > > > > > > > >> :-)
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> --
> > > > > > > > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: OS Test re-factor proposal
Posted by "Alan C. Assis" <ac...@gmail.com>.
Yes, that is a good idea!
On Mon, Dec 22, 2025 at 5:56 PM Matteo Golin <ma...@gmail.com> wrote:
> I think I will try to make a version of the sample I wrote which uses
> cmocka to compare a bit better and report back.
>
> On Mon, Dec 22, 2025 at 1:28 PM Alan C. Assis <ac...@gmail.com> wrote:
>
> > Hi Pierre-Noel,
> >
> > The instructions are here: https://nuttx.apache.org/community/
> >
> > We are glad for all the things you did for NuttX (EFM32 port, ESP8266,
> > etc.)
> >
> > Thank you very much!
> >
> > BR,
> >
> > Alan
> >
> > On Mon, Dec 22, 2025 at 3:16 PM Pierre-Noel Bouteville <
> pnb666@hotmail.com
> > >
> > wrote:
> >
> > > How to unsbcribe?
> > > Pierre-Noël Bouteville
> > > Envoyé de mon iPhone
> > >
> > > > Le 22 déc. 2025 à 19:13, raiden00pl <ra...@gmail.com> a écrit :
> > > >
> > > > Cmocka tests can run on any target. It doesn't matter if it's sim,
> > qemu
> > > or
> > > > real hardware.
> > > > You just need to prepare a defconfig and have the right amount of
> > FLASH.
> > > >
> > > > For small systems we can always try to reduce the size of the cmock,
> > e.g.
> > > > by disabling
> > > > unnecessary prints. I doubt that the logic of running tests in cmoka
> is
> > > > heavy, running
> > > > test cases are not a complicated operation, I think most of the
> > required
> > > > FLASH is wasted
> > > > on strings.
> > > >
> > > > NTFC supports serial port communication and sample test scenario
> based
> > on
> > > > nucleo-h743z
> > > > with cmocka tests enabled. The only thing missing for full test
> > > automation
> > > > (compile, flash,
> > > > test, result) is cmake support for testsuites.
> > > >
> > > > pon., 22 gru 2025 o 18:08 Matteo Golin <ma...@gmail.com>
> > > napisał(a):
> > > >
> > > >> Hello everyone,
> > > >>
> > > >> Thanks for all the feedback! I'm glad I discussed this here first.
> > I'll
> > > try
> > > >> to address everything that was brought up.
> > > >>
> > > >> Thanks Guiding for the input! I must just be missing some library on
> > my
> > > >> machine if you're able to compile it just fine. Maybe I'll track
> down
> > > that
> > > >> error and see if you can recognize what's wrong.
> > > >>
> > > >> I don't intend to replace OS test anymore. I think it serves its
> > > purpose as
> > > >> a minimal sanity check. Instead, I think we need something a little
> > more
> > > >> involved to test PRs since OS test can't catch many issues. It is
> > > currently
> > > >> the standard everyone uses for verifying their patches and it would
> be
> > > >> better to have more complete, alternate options. I do think that we
> > can
> > > all
> > > >> agree that OS test needs a revision of its console output so that it
> > is
> > > >> much easier to see pass/fail results, but that is just a matter of
> > print
> > > >> statements and can be done in a different effort.
> > > >>
> > > >> I think the overwhelming concern here is mixing two unit testing
> > > frameworks
> > > >> and no longer having one resource where we can go for tests. I share
> > > raiden
> > > >> and Guiding's concern about that, which is why I'm glad I opened for
> > > >> discussion here. On the one hand, Xiaomi has a pretty substantial
> > > >> collection of tests in the `testsuite` application which they are
> > using
> > > >> internally I believe (in their CI). These run well on virtual
> > machines,
> > > >> which I assume is why the only configurations with that test suite
> > > included
> > > >> is sim and rv-virt. I do not know nearly enough about cmocka to make
> > any
> > > >> claim that it *wouldn't* work on smaller hardware devices, but it
> also
> > > >> definitely looks to be less light-weight from the feature set in the
> > > >> documentation. On the other hand, the Xiaomi test suites are also
> not
> > a
> > > >> complete silver bullet since they seem not to be intended to run on
> > > >> hardware, and there are *a lot *of tests that need hardware
> > > verification.
> > > >> In PRs we often request more than just a simulator test for certain
> > > >> changes. For instance, sim isn't able to have `/dev/rand`, so that
> > > needs to
> > > >> be tested on a different target.
> > > >>
> > > >> Maybe someone more familiar with Xiaomi's testing practices can
> weigh
> > > in on
> > > >> whether cmocka has been run on hardware. Going purely based off of
> > their
> > > >> PRs, which almost exclusively cite tests on simulators, my guess is
> > that
> > > >> cmocka was not designed with embedded in mind when it was written.
> > > >>
> > > >> As for size: unit testing can get large with Unity or cmocka (as
> > raiden
> > > >> mentioned) as more tests are added. Hence my effort to
> enable/disable
> > > >> individual test suites from getting compiled. It might not be
> possible
> > > to
> > > >> ever get good testing ability for AVR just because the devices are
> so
> > > >> small. Whatever solution is used there is probably going to involve
> > > >> flashing the device multiple times with different tests. The size
> goal
> > > >> should just be to target the most NuttX devices possible.
> > > >>
> > > >> Then there are the pros and cons of each framework (again, I don't
> > know
> > > >> much about cmocka so take it with a grain of salt):
> > > >>
> > > >> Unity's cons:
> > > >> - Designed for each test-suite to be a standalone binary. This is
> > > clunky on
> > > >> NuttX, so `nest` is currently implemented as a mega-suite where test
> > > cases
> > > >> just follow a naming convention to make it clear what is being
> tested
> > > >> - setUp/tearDown can't really be leveraged because of the above
> > > >> - I've noticed some funky behaviour using TEST_IGNORE in certain
> > > scenarios
> > > >> but that's possibly my own configuration issue
> > > >> - Would be in conflict with existing cmocka tests
> > > >>
> > > >> Unity's pros:
> > > >> - Highly configurable, like NuttX
> > > >> - Light-weight and works for embedded devices
> > > >> - Significantly larger user base than cmocka
> > > >>
> > > >> Cmocka's cons:
> > > >> - Possibly (?) not light-weight
> > > >> - No documentation for existing test cases in NuttX
> > > >> - Small user base
> > > >>
> > > >> Cmocka's pros:
> > > >> - Seems to have a larger set of features, including setup/teardown
> > > >> functions per-test case
> > > >> - Existing tests in NuttX
> > > >>
> > > >> Irrespective of whether we go with Unity or cmocka, the collection
> of
> > > tests
> > > >> that we write will be intended to run on the actual hardware devices
> > > >> themselves for the purposes of testing patches and doing regression
> > > tests.
> > > >> As a result, the tests will be separate from the Xiaomi 'testsuite'
> > > >> application which is intended for CI testing. I think this
> > realistically
> > > >> nullifies choosing which framework to use on the basis of
> consistency,
> > > >> since the tests are not coexisting with the cmocka tests Xiaomi has
> > > >> written. Instead, the choice should be based on what best achieves
> the
> > > goal
> > > >> from a functionality perspective. Unity seems like a really great
> > choice
> > > >> for this in my opinion, but I am bothered by the fact that it's
> > designed
> > > >> with multiple test binaries in mind.
> > > >>
> > > >> Let me know what you think,
> > > >> Matteo
> > > >>
> > > >>> On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <ac...@gmail.com>
> > > wrote:
> > > >>>
> > > >>> Yes, I agree! We need data size from each solution for comparison,
> > this
> > > >> is
> > > >>> the first step.
> > > >>>
> > > >>> Maybe the conversion could be facilitated if we create equivalent
> > > >> functions
> > > >>> like those used in the apps/testing/testsuites:
> > > >>>
> > > >>> cmocka_run_group_tests -> unity_run_group_tests
> > > >>> cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown
> > > >>> etc...
> > > >>>
> > > >>> This way we avoid end up with two unit test framework, we already
> > have
> > > >> the
> > > >>> Makefile vs CMake precedent! :-D
> > > >>>
> > > >>> On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <ra...@gmail.com>
> > > >> wrote:
> > > >>>
> > > >>>> The discussion about small systems doesn't make much sense without
> > > >>>> data comparing resources of the same test cases for cmock and
> unity.
> > > >>>>
> > > >>>> But let's assume that Unity is better. Should we use two
> frameworks
> > > >>>> to implement unit tests for OS? Or choose one framework? If we
> > choose
> > > >>> only
> > > >>>> Unity, who
> > > >>>> will migrate the current cmocka tests to Unity? Remember that
> cmoka
> > > >> tests
> > > >>>> are used in CI, so all CI tools must be also migrated.
> > > >>>> Or should we use two different frameworks to test different things
> > and
> > > >>> make
> > > >>>> even
> > > >>>> more mess in apps/testing as it is now?
> > > >>>>
> > > >>>> No one is forcing anyone to do anything. This is an open-source
> > > >> project,
> > > >>>> everyone
> > > >>>> should do what they feel is right. But whether the change should
> be
> > > >>> merged
> > > >>>> is
> > > >>>> a different question. In this case, it's a very important
> question,
> > > >>> because
> > > >>>> unit testing
> > > >>>> isn't just about test cases. It's about the entire infrastructure
> > that
> > > >>>> runs, captures and
> > > >>>> handles test results later. Writing the test cases is the least of
> > the
> > > >>>> problems.
> > > >>>> The infrastructure for cmoka testing is already in place, the
> > > >>>> infrastructure for Unity
> > > >>>> testing isn't.
> > > >>>>
> > > >>>> pon., 22 gru 2025 o 14:16 Alan C. Assis <ac...@gmail.com>
> > > >> napisał(a):
> > > >>>>
> > > >>>>> Mateusz,
> > > >>>>> but some chips like AVR DA have big flash, but not so much RAM.
> > > >>>>>
> > > >>>>> It is better to have some possible way to test on a small system,
> > > >> using
> > > >>>>> cmocka we are removing this option from the table.
> > > >>>>>
> > > >>>>> I liked the test Matteo did with Unity and I think we should let
> > him
> > > >>>>> continue. Forcing him to migrate to cmocka just because there's a
> > > >> test
> > > >>>>> suite from Xiaomi doesn't seem prudent to me.
> > > >>>>>
> > > >>>>> This is the advantage of an open mind open-source project: "the
> > > >>>> possibility
> > > >>>>> to test new ideas", if someone had said to Michal Lenc he
> couldn't
> > > >>> create
> > > >>>>> NXBoot because where is already MCUBoot integrated on NuttX, we
> > > >> didn't
> > > >>>> have
> > > >>>>> the option to use NXBoot today. ;-)
> > > >>>>>
> > > >>>>> BR,
> > > >>>>>
> > > >>>>> Alan
> > > >>>>>
> > > >>>>> On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <
> raiden00pl@gmail.com>
> > > >>>> wrote:
> > > >>>>>
> > > >>>>>> Unit testing by itself isn't suitable for small systems. Ostest
> is
> > > >>>>>> something
> > > >>>>>> different (system level testing), so running it on small systems
> > > >>> makes
> > > >>>>>> sense.
> > > >>>>>> Running unit tests on small systems is pointless because you
> won't
> > > >>> fit
> > > >>>> in
> > > >>>>>> FLASH
> > > >>>>>> either way. You can run the same on large targets (preferably on
> > > >>> host:
> > > >>>>> sim,
> > > >>>>>> qemu)
> > > >>>>>> and achieve the same result.
> > > >>>>>>
> > > >>>>>> pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com>
> > > >>>> napisał(a):
> > > >>>>>>
> > > >>>>>>> Is it too complicated to rewrite the cmocka tests in Unity?
> > > >>>>>>>
> > > >>>>>>> If cmoka is not feasible for small microcontrollers, like AVRs
> > > >> and
> > > >>>>>> others,
> > > >>>>>>> our test coverage will be faulty.
> > > >>>>>>>
> > > >>>>>>> BR,
> > > >>>>>>>
> > > >>>>>>> Alan
> > > >>>>>>>
> > > >>>>>>> On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <
> raiden00pl@gmail.com
> > > >>>
> > > >>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> I'm a fan of Unity and have been using it in my projects for
> > > >>> years,
> > > >>>>>> but I
> > > >>>>>>>> think
> > > >>>>>>>> we should stick with one testing framework. Currently, Cmocka
> > > >> is
> > > >>>> used
> > > >>>>>> in
> > > >>>>>>>> nuttx-apps, and we should stick with it. The only question is
> > > >> how
> > > >>>>> well
> > > >>>>>>>> Cmocka
> > > >>>>>>>> is suitable for running on small systems? Will ostest still be
> > > >>> able
> > > >>>>> to
> > > >>>>>>> run
> > > >>>>>>>> on small MCUs?
> > > >>>>>>>> With Unity this would be quite likely, so maybe an exception
> > > >> for
> > > >>>>> Unity
> > > >>>>>>> and
> > > >>>>>>>> ostest would
> > > >>>>>>>> be acceptable, but using it for other test cases will only
> > > >> cause
> > > >>>>> chaos.
> > > >>>>>>>>
> > > >>>>>>>> pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com>
> > > >>>> napisał(a):
> > > >>>>>>>>
> > > >>>>>>>>> Hi Matteo,
> > > >>>>>>>>>
> > > >>>>>>>>> Good idea !
> > > >>>>>>>>>
> > > >>>>>>>>> But it is best to stick with a single testing framework.
> > > >>>> Currently,
> > > >>>>>>> many
> > > >>>>>>>>> test cases are using cmocka, so I suggest reusing cmocka for
> > > >>>>>>> consistency.
> > > >>>>>>>>>
> > > >>>>>>>>> ostest is a minimal basic test suite, typically used for CT
> > > >>>>>> (continuous
> > > >>>>>>>>> testing) of each patch.
> > > >>>>>>>>> If we move ostest to cmocka, it would make ostest dependent
> > > >> on
> > > >>> a
> > > >>>>>>> specific
> > > >>>>>>>>> framework before it can be run.
> > > >>>>>>>>> And It may increase the code size, which is not very friendly
> > > >>> for
> > > >>>>>>>>> configurations with small flash memory.
> > > >>>>>>>>>
> > > >>>>>>>>> As Greg mentioned: the answer depends on a tradeoff between
> > > >>>> effort
> > > >>>>>> and
> > > >>>>>>>> test
> > > >>>>>>>>> coverage.
> > > >>>>>>>>>
> > > >>>>>>>>> We need to clarify whether we define ostest as a lightweight,
> > > >>>>>>>> ready-to-run
> > > >>>>>>>>> test set for quick verification, or as part of the full test
> > > >>>> suite.
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> The follow is my compiling for cmocka:
> > > >>>>>>>>>
> > > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh
> > > >>>>> sim:citest
> > > >>>>>>>> -j8*
> > > >>>>>>>>> Copy files
> > > >>>>>>>>> Select CONFIG_HOST_LINUX=y
> > > >>>>>>>>> Refreshing...
> > > >>>>>>>>> CP: arch/dummy/Kconfig to
> > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> > > >>>>>>>>> CP: boards/dummy/Kconfig to
> > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> > > >>>>>>>>> LN: include/arch to arch/sim/include
> > > >>>>>>>>> LN: drivers/platform to
> > > >>>>>>> /home/ligd/platform/mainline/nuttx/drivers/dummy
> > > >>>>>>>>> LN: arch/sim/src/chip to
> > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> > > >>>>>>>>> LN: arch/sim/src/board to
> > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> > > >>>>>>>>> LN: include/arch/board to
> > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> > > >>>>>>>>> LN: include/arch/chip to
> > > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> > > >>>>>>>>> LN: platform/board to
> > > >>>>>> /home/ligd/platform/mainline/apps/platform/dummy
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/boot
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/canutils
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/crypto
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/database
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> > > >>>>>>>>> mkkconfig in
> > > >> /home/ligd/platform/mainline/apps/examples/sotest
> > > >>>>>>>>> mkkconfig in
> > > >> /home/ligd/platform/mainline/apps/examples/module
> > > >>>>>>>>> mkkconfig in
> > > >> /home/ligd/platform/mainline/apps/examples/mcuboot
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/examples
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/games
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/graphics
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/industry
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/inertial
> > > >>>>>>>>> mkkconfig in
> > > >>>>>> /home/ligd/platform/mainline/apps/interpreters/luamodules
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/logging
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/lte
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/math
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/netutils
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/sdr
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/system
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/tee
> > > >>>>>>>>> mkkconfig in
> > > >> /home/ligd/platform/mainline/apps/testing/drivers
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> > > >>>>>>>>> mkkconfig in
> > > >>> /home/ligd/platform/mainline/apps/wireless/bluetooth
> > > >>>>>>>>> mkkconfig in
> > > >>>> /home/ligd/platform/mainline/apps/wireless/ieee802154
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/wireless
> > > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps
> > > >>>>>>>>> Loaded configuration '.config'
> > > >>>>>>>>> Configuration saved to '.config'
> > > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
> > > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ make*
> > > >>>>>>>>> Create version.h
> > > >>>>>>>>> Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> > > >>>>>>>>> /home/ligd/platform/mainline/apps/platform/dummy
> > > >>>>>>>>> Downloading: v1.46.0.zip
> > > >>>>>>>>> ...
> > > >>>>>>>>> patching file
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> > > >>>>>>>>> Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> > > >>>>>>>>>
> > > >>>>>>>>> CP:
> > > >> /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> > > >>>>>>>>> CP:
> > > >>> /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> > > >>>>>>>>> LD: nuttx
> > > >>>>>>>>> Pac SIM with dynamic libs..
> > > >>>>>>>>> '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> > > >>>>>>>>> '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
> > > >>>>>> 'sim-pac/libs/libstdc++.so.6'
> > > >>>>>>>>> '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> > > >>>>>>>>> '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> > > >>>>>>>>> '/lib/x86_64-linux-gnu/libgcc_s.so.1' ->
> > > >>>>> 'sim-pac/libs/libgcc_s.so.1'
> > > >>>>>>>>> '/lib64/ld-linux-x86-64.so.2' ->
> > > >> 'sim-pac/ld-linux-x86-64.so.2'
> > > >>>>>>>>> SIM elf with dynamic libs archive in nuttx.tgz
> > > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
> > > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
> > > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
> > > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> > > >>>>>>>>> CHelloWorld: Constructor: mSecret=42
> > > >>>>>>>>> nsh: init: open failed: 2
> > > >>>>>>>>>
> > > >>>>>>>>> NuttShell (NSH) NuttX-12.11.0
> > > >>>>>>>>> MOTD: username=admin password=Administrator
> > > >>>>>>>>> nsh>
> > > >>>>>>>>> nsh> help
> > > >>>>>>>>> help usage: help [-v] [<cmd>]
> > > >>>>>>>>>
> > > >>>>>>>>> . df ifconfig mkfifo readlink
> > > >>>>>> umount
> > > >>>>>>>>> [ dmesg ifdown mkrd rm
> > > >>>>> unset
> > > >>>>>>>>> ? echo ifup mount rmdir
> > > >>>>>> uptime
> > > >>>>>>>>> alias env insmod mv rmmod
> > > >>>>>> usleep
> > > >>>>>>>>> unalias exec kill nslookup set
> > > >>>>> watch
> > > >>>>>>>>> basename exit pkill pidof sleep
> > > >>>> xd
> > > >>>>>>>>> break expr losetup pmconfig source
> > > >>>>> wait
> > > >>>>>>>>> cat false ln poweroff test
> > > >>>>>>>>> cd fdinfo ls quit time
> > > >>>>>>>>> cp free lsmod printf true
> > > >>>>>>>>> cmp help mkdir ps truncate
> > > >>>>>>>>> dirname hexdump mkfatfs pwd uname
> > > >>>>>>>>>
> > > >>>>>>>>> Builtin Apps:
> > > >>>>>>>>> cmocka
> > > >>>>>>>>> cmocka_driver_block
> > > >>>>>>>>> cmocka_driver_gpio
> > > >>>>>>>>> cmocka_driver_pm
> > > >>>>>>>>> cmocka_driver_pm_runtime
> > > >>>>>>>>> cmocka_driver_regulator
> > > >>>>>>>>>
> > > >>>>>>>>> Matteo Golin <ma...@gmail.com> 于2025年12月22日周一
> > > >> 10:15写道：
> > > >>>>>>>>>
> > > >>>>>>>>>> I think so. Best to leave OS test alone and slowly replace
> > > >> it
> > > >>>>> with
> > > >>>>>> a
> > > >>>>>>>> new
> > > >>>>>>>>>> tool that has improvements. The only reason I can see to
> > > >>> modify
> > > >>>>> OS
> > > >>>>>>> test
> > > >>>>>>>>> now
> > > >>>>>>>>>> would just be to have clearer output about what
> > > >>> passed/failed.
> > > >>>>>>>>>>
> > > >>>>>>>>>> I'll come up with an improved proof of concept for a new
> > > >> tool
> > > >>>> to
> > > >>>>>>> share
> > > >>>>>>>> in
> > > >>>>>>>>>> the next few days.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Matteo
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <
> > > >> tomek@cedro.info>
> > > >>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Small steps with measurable results may be the best
> > > >>> approach
> > > >>>>>> here,
> > > >>>>>>> to
> > > >>>>>>>>>>> create some new solution next to existing ostest but not
> > > >>>>>> replacing
> > > >>>>>>> it
> > > >>>>>>>>>>> until ready? :-)
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> --
> > > >>>>>>>>>>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> > > >>>>>>> matteo.golin@gmail.com
> > > >>>>>>>>>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I agree, I think it would take a long time to have
> > > >> proper
> > > >>>>>>> coverage
> > > >>>>>>>> of
> > > >>>>>>>>>>> NuttX
> > > >>>>>>>>>>>> as a whole. Perhaps even impossible, since Unity is
> > > >> more
> > > >>>> of a
> > > >>>>>>> unit
> > > >>>>>>>>>>> testing
> > > >>>>>>>>>>>> framework and thus there are definitely test cases for
> > > >>>> NuttX
> > > >>>>>> that
> > > >>>>>>>>> would
> > > >>>>>>>>>>> be
> > > >>>>>>>>>>>> outside of its realm of "applicability". There is
> > > >> still a
> > > >>>>> need
> > > >>>>>>> for
> > > >>>>>>>> a
> > > >>>>>>>>>>>> variety of testing applications and bench-marking
> > > >>>>> applications
> > > >>>>>>> for
> > > >>>>>>>>>> those
> > > >>>>>>>>>>>> scenarios.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> That makes sense, prerequisites would need to be
> > > >>>> considered.
> > > >>>>> I
> > > >>>>>>>> think
> > > >>>>>>>>>>>> targeting the low-hanging fruit of unit-testable code
> > > >>> would
> > > >>>>> be
> > > >>>>>>> easy
> > > >>>>>>>>> in
> > > >>>>>>>>>>> that
> > > >>>>>>>>>>>> case, since prerequisites are mostly encoded in the
> > > >>> Kconfig
> > > >>>>>>>>> definitions
> > > >>>>>>>>>>>> existing or not for features to be tested. Any complex
> > > >>>>>> behaviour
> > > >>>>>>>>>>>> (interrupts, multi-threading, etc.) might be out of the
> > > >>>> realm
> > > >>>>>> of
> > > >>>>>>>>> what's
> > > >>>>>>>>>>>> testable by Unity in the first place. We'll always have
> > > >>> OS
> > > >>>>> test
> > > >>>>>>> in
> > > >>>>>>>>> that
> > > >>>>>>>>>>>> case, but maybe then it's better to do what Alan
> > > >>> suggested
> > > >>>>> and
> > > >>>>>> at
> > > >>>>>>>>> least
> > > >>>>>>>>>>>> make its output more clear in the meantime. I suppose
> > > >>> RTOS
> > > >>>>>>> testing
> > > >>>>>>>>> is a
> > > >>>>>>>>>>>> whole field in itself.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> > > >>>>>>> spudaneco@gmail.com>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> I think that the answer depends on a tradeoff between
> > > >>>>> effort
> > > >>>>>>> and
> > > >>>>>>>>> test
> > > >>>>>>>>>>>>> coverage. I could imagine that a proper job could be
> > > >>>>>> man-years
> > > >>>>>>>> of
> > > >>>>>>>>>>> effort.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> My only concern with a re-design is to be certain
> > > >> that
> > > >>>>> tests
> > > >>>>>>> are
> > > >>>>>>>>>>>>> arranged so that the prerequisites of later test are
> > > >>>> tested
> > > >>>>>>>> first.
> > > >>>>>>>>>> If
> > > >>>>>>>>>>>>> tests are grouped by, for example, functional area,
> > > >>> then
> > > >>>>>>> failures
> > > >>>>>>>>> can
> > > >>>>>>>>>>> be
> > > >>>>>>>>>>>>> difficult to understand if there is a failure in an
> > > >>>>>> unverified
> > > >>>>>>>> test
> > > >>>>>>>>>>>>> prerequisite.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Some of the "tests" are bogus and only exist to
> > > >> provide
> > > >>>>> some
> > > >>>>>>>>> minimum
> > > >>>>>>>>>>>>> confidence that we have the functionality to run the
> > > >>>> test.
> > > >>>>>> For
> > > >>>>>>>>>>>>> example: Can I start tasks? Can I pass parameters?
> > > >>> Can
> > > >>>> I
> > > >>>>>>>> survive
> > > >>>>>>>>> a
> > > >>>>>>>>>>>>> context switch. Some of the tests were intended only
> > > >>> to
> > > >>>>>>> support
> > > >>>>>>>>>> debug.
> > > >>>>>>>>>>>>> Some were created only to track down specific bugs.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On 12/21/25 18:39, Matteo Golin wrote:
> > > >>>>>>>>>>>>>> LTS being the Long-Term-Stable release of NuttX?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I'm starting to think that it would be best to
> > > >> create
> > > >>>> new
> > > >>>>>>> suite
> > > >>>>>>>>> of
> > > >>>>>>>>>>> tests
> > > >>>>>>>>>>>>>> using Unity, using what's in OS Test as a basis and
> > > >>>> then
> > > >>>>>>>>> expanding
> > > >>>>>>>>>> it
> > > >>>>>>>>>>>>>> further. Greg has pointed out that OS test is not
> > > >>>>> intended
> > > >>>>>> to
> > > >>>>>>>> be
> > > >>>>>>>>> a
> > > >>>>>>>>>>>>> complete
> > > >>>>>>>>>>>>>> test, so maybe a more solid testing framework base
> > > >>> in a
> > > >>>>> new
> > > >>>>>>>> form
> > > >>>>>>>>>>> would be
> > > >>>>>>>>>>>>>> in order. At that point it would also be good to
> > > >>>> document
> > > >>>>>> and
> > > >>>>>>>>>> educate
> > > >>>>>>>>>>>>> users
> > > >>>>>>>>>>>>>> about other test options, since OS test is a
> > > >> current
> > > >>>>>>> favourite
> > > >>>>>>>>> for
> > > >>>>>>>>>>>>> showing
> > > >>>>>>>>>>>>>> PRs haven't broken anything. Maybe that's not the
> > > >>> best
> > > >>>>>> choice
> > > >>>>>>>>>> anymore
> > > >>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>> this information.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I don't want to re-invent the wheel of what's
> > > >> already
> > > >>>> in
> > > >>>>>>>>>> `testsuite`,
> > > >>>>>>>>>>>>> but I
> > > >>>>>>>>>>>>>> think Unity might end up being lighter-weight in
> > > >> the
> > > >>>> long
> > > >>>>>>> run?
> > > >>>>>>>>> And
> > > >>>>>>>>>>> it is
> > > >>>>>>>>>>>>>> also not a Xiaomi owned/maintained framework so it
> > > >>> may
> > > >>>>>>> receive
> > > >>>>>>>>>> better
> > > >>>>>>>>>>>>>> external support (i.e. more users rely on it?). I
> > > >>> tried
> > > >>>>> to
> > > >>>>>>>>> compile
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>> sim:citest configuration to run the testsuite
> > > >> example
> > > >>>> and
> > > >>>>>> see
> > > >>>>>>>>> what
> > > >>>>>>>>>>> it was
> > > >>>>>>>>>>>>>> like, but I was met with a long list of C++
> > > >>> compilation
> > > >>>>>>> errors
> > > >>>>>>>>> that
> > > >>>>>>>>>>> I'd
> > > >>>>>>>>>>>>>> rather not deal with. There also seems to be zero
> > > >>> docs
> > > >>>>>> about
> > > >>>>>>> it
> > > >>>>>>>>> or
> > > >>>>>>>>>>> cmocka
> > > >>>>>>>>>>>>>> in our NuttX docs at the moment.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> What do you think? Start a Unity test suite, add
> > > >> some
> > > >>>>>>> switches
> > > >>>>>>>> to
> > > >>>>>>>>>>> choose
> > > >>>>>>>>>>>>>> which test cases get compiled into the binary, open
> > > >>> for
> > > >>>>>>>>> extensions
> > > >>>>>>>>>> by
> > > >>>>>>>>>>>>> users
> > > >>>>>>>>>>>>>> who know more about different subsystems? This way
> > > >> we
> > > >>>> can
> > > >>>>>>> keep
> > > >>>>>>>> OS
> > > >>>>>>>>>>> test
> > > >>>>>>>>>>>>> how
> > > >>>>>>>>>>>>>> it is for a quick, low-resource sanity check of
> > > >>> "board
> > > >>>>>>> bringup"
> > > >>>>>>>>>> while
> > > >>>>>>>>>>>>>> having a dedicated testing application for patches?
> > > >>>>> Someone
> > > >>>>>>>>>> changing
> > > >>>>>>>>>>> file
> > > >>>>>>>>>>>>>> system stuff can just compile to include file
> > > >> system
> > > >>>> test
> > > >>>>>>> cases
> > > >>>>>>>>> and
> > > >>>>>>>>>>> run
> > > >>>>>>>>>>>>>> that suite only as proof that their PR works. The
> > > >>>> output
> > > >>>>> is
> > > >>>>>>>>> really
> > > >>>>>>>>>>> easy
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>> glance at to determine pass/fail results.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Anyways, here's my draft demonstrating how Unity
> > > >>> works
> > > >>>>> with
> > > >>>>>>> the
> > > >>>>>>>>> FPU
> > > >>>>>>>>>>> test
> > > >>>>>>>>>>>>>> suite from OS test:
> > > >>>>>>>>> https://github.com/apache/nuttx-apps/pull/3259
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Matteo
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> > > >>>>>>> tomek@cedro.info>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> > > >>>>>>>>> spudaneco@gmail.com
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>> (..)
> > > >>>>>>>>>>>>>>>> Starting with bits and pieces of something like
> > > >> the
> > > >>>> LTS
> > > >>>>>>> might
> > > >>>>>>>>>> make
> > > >>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>> sense other than its licensing problems. Since
> > > >> it
> > > >>>>> would
> > > >>>>>>>> never
> > > >>>>>>>>> be
> > > >>>>>>>>>>>>>>>> included in a distributed binary, the licensing
> > > >>>> really
> > > >>>>>>> should
> > > >>>>>>>>> not
> > > >>>>>>>>>>>>> matter
> > > >>>>>>>>>>>>>>>> much in practice.
> > > >>>>>>>>>>>>>>> Unless its GPL and some company would like to have
> > > >>>> their
> > > >>>>>> own
> > > >>>>>>>>>>> internal
> > > >>>>>>>>>>>>>>> bits and pieces.. luckily its MIT and the whole
> > > >>> thing
> > > >>>> is
> > > >>>>>>>>>> open-source
> > > >>>>>>>>>>>>>>> anyways :-)
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Yes we had many discussions about the LTS and this
> > > >>>> could
> > > >>>>>> be
> > > >>>>>>>>> great
> > > >>>>>>>>>>> start!
> > > >>>>>>>>>>>>>>> :-)
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> --
> > > >>>>>>>>>>>>>>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
>
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
I think I will try to make a version of the sample I wrote which uses
cmocka to compare a bit better and report back.
On Mon, Dec 22, 2025 at 1:28 PM Alan C. Assis <ac...@gmail.com> wrote:
> Hi Pierre-Noel,
>
> The instructions are here: https://nuttx.apache.org/community/
>
> We are glad for all the things you did for NuttX (EFM32 port, ESP8266,
> etc.)
>
> Thank you very much!
>
> BR,
>
> Alan
>
> On Mon, Dec 22, 2025 at 3:16 PM Pierre-Noel Bouteville <pnb666@hotmail.com
> >
> wrote:
>
> > How to unsbcribe?
> > Pierre-Noël Bouteville
> > Envoyé de mon iPhone
> >
> > > Le 22 déc. 2025 à 19:13, raiden00pl <ra...@gmail.com> a écrit :
> > >
> > > Cmocka tests can run on any target. It doesn't matter if it's sim,
> qemu
> > or
> > > real hardware.
> > > You just need to prepare a defconfig and have the right amount of
> FLASH.
> > >
> > > For small systems we can always try to reduce the size of the cmock,
> e.g.
> > > by disabling
> > > unnecessary prints. I doubt that the logic of running tests in cmoka is
> > > heavy, running
> > > test cases are not a complicated operation, I think most of the
> required
> > > FLASH is wasted
> > > on strings.
> > >
> > > NTFC supports serial port communication and sample test scenario based
> on
> > > nucleo-h743z
> > > with cmocka tests enabled. The only thing missing for full test
> > automation
> > > (compile, flash,
> > > test, result) is cmake support for testsuites.
> > >
> > > pon., 22 gru 2025 o 18:08 Matteo Golin <ma...@gmail.com>
> > napisał(a):
> > >
> > >> Hello everyone,
> > >>
> > >> Thanks for all the feedback! I'm glad I discussed this here first.
> I'll
> > try
> > >> to address everything that was brought up.
> > >>
> > >> Thanks Guiding for the input! I must just be missing some library on
> my
> > >> machine if you're able to compile it just fine. Maybe I'll track down
> > that
> > >> error and see if you can recognize what's wrong.
> > >>
> > >> I don't intend to replace OS test anymore. I think it serves its
> > purpose as
> > >> a minimal sanity check. Instead, I think we need something a little
> more
> > >> involved to test PRs since OS test can't catch many issues. It is
> > currently
> > >> the standard everyone uses for verifying their patches and it would be
> > >> better to have more complete, alternate options. I do think that we
> can
> > all
> > >> agree that OS test needs a revision of its console output so that it
> is
> > >> much easier to see pass/fail results, but that is just a matter of
> print
> > >> statements and can be done in a different effort.
> > >>
> > >> I think the overwhelming concern here is mixing two unit testing
> > frameworks
> > >> and no longer having one resource where we can go for tests. I share
> > raiden
> > >> and Guiding's concern about that, which is why I'm glad I opened for
> > >> discussion here. On the one hand, Xiaomi has a pretty substantial
> > >> collection of tests in the `testsuite` application which they are
> using
> > >> internally I believe (in their CI). These run well on virtual
> machines,
> > >> which I assume is why the only configurations with that test suite
> > included
> > >> is sim and rv-virt. I do not know nearly enough about cmocka to make
> any
> > >> claim that it *wouldn't* work on smaller hardware devices, but it also
> > >> definitely looks to be less light-weight from the feature set in the
> > >> documentation. On the other hand, the Xiaomi test suites are also not
> a
> > >> complete silver bullet since they seem not to be intended to run on
> > >> hardware, and there are *a lot *of tests that need hardware
> > verification.
> > >> In PRs we often request more than just a simulator test for certain
> > >> changes. For instance, sim isn't able to have `/dev/rand`, so that
> > needs to
> > >> be tested on a different target.
> > >>
> > >> Maybe someone more familiar with Xiaomi's testing practices can weigh
> > in on
> > >> whether cmocka has been run on hardware. Going purely based off of
> their
> > >> PRs, which almost exclusively cite tests on simulators, my guess is
> that
> > >> cmocka was not designed with embedded in mind when it was written.
> > >>
> > >> As for size: unit testing can get large with Unity or cmocka (as
> raiden
> > >> mentioned) as more tests are added. Hence my effort to enable/disable
> > >> individual test suites from getting compiled. It might not be possible
> > to
> > >> ever get good testing ability for AVR just because the devices are so
> > >> small. Whatever solution is used there is probably going to involve
> > >> flashing the device multiple times with different tests. The size goal
> > >> should just be to target the most NuttX devices possible.
> > >>
> > >> Then there are the pros and cons of each framework (again, I don't
> know
> > >> much about cmocka so take it with a grain of salt):
> > >>
> > >> Unity's cons:
> > >> - Designed for each test-suite to be a standalone binary. This is
> > clunky on
> > >> NuttX, so `nest` is currently implemented as a mega-suite where test
> > cases
> > >> just follow a naming convention to make it clear what is being tested
> > >> - setUp/tearDown can't really be leveraged because of the above
> > >> - I've noticed some funky behaviour using TEST_IGNORE in certain
> > scenarios
> > >> but that's possibly my own configuration issue
> > >> - Would be in conflict with existing cmocka tests
> > >>
> > >> Unity's pros:
> > >> - Highly configurable, like NuttX
> > >> - Light-weight and works for embedded devices
> > >> - Significantly larger user base than cmocka
> > >>
> > >> Cmocka's cons:
> > >> - Possibly (?) not light-weight
> > >> - No documentation for existing test cases in NuttX
> > >> - Small user base
> > >>
> > >> Cmocka's pros:
> > >> - Seems to have a larger set of features, including setup/teardown
> > >> functions per-test case
> > >> - Existing tests in NuttX
> > >>
> > >> Irrespective of whether we go with Unity or cmocka, the collection of
> > tests
> > >> that we write will be intended to run on the actual hardware devices
> > >> themselves for the purposes of testing patches and doing regression
> > tests.
> > >> As a result, the tests will be separate from the Xiaomi 'testsuite'
> > >> application which is intended for CI testing. I think this
> realistically
> > >> nullifies choosing which framework to use on the basis of consistency,
> > >> since the tests are not coexisting with the cmocka tests Xiaomi has
> > >> written. Instead, the choice should be based on what best achieves the
> > goal
> > >> from a functionality perspective. Unity seems like a really great
> choice
> > >> for this in my opinion, but I am bothered by the fact that it's
> designed
> > >> with multiple test binaries in mind.
> > >>
> > >> Let me know what you think,
> > >> Matteo
> > >>
> > >>> On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <ac...@gmail.com>
> > wrote:
> > >>>
> > >>> Yes, I agree! We need data size from each solution for comparison,
> this
> > >> is
> > >>> the first step.
> > >>>
> > >>> Maybe the conversion could be facilitated if we create equivalent
> > >> functions
> > >>> like those used in the apps/testing/testsuites:
> > >>>
> > >>> cmocka_run_group_tests -> unity_run_group_tests
> > >>> cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown
> > >>> etc...
> > >>>
> > >>> This way we avoid end up with two unit test framework, we already
> have
> > >> the
> > >>> Makefile vs CMake precedent! :-D
> > >>>
> > >>> On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <ra...@gmail.com>
> > >> wrote:
> > >>>
> > >>>> The discussion about small systems doesn't make much sense without
> > >>>> data comparing resources of the same test cases for cmock and unity.
> > >>>>
> > >>>> But let's assume that Unity is better. Should we use two frameworks
> > >>>> to implement unit tests for OS? Or choose one framework? If we
> choose
> > >>> only
> > >>>> Unity, who
> > >>>> will migrate the current cmocka tests to Unity? Remember that cmoka
> > >> tests
> > >>>> are used in CI, so all CI tools must be also migrated.
> > >>>> Or should we use two different frameworks to test different things
> and
> > >>> make
> > >>>> even
> > >>>> more mess in apps/testing as it is now?
> > >>>>
> > >>>> No one is forcing anyone to do anything. This is an open-source
> > >> project,
> > >>>> everyone
> > >>>> should do what they feel is right. But whether the change should be
> > >>> merged
> > >>>> is
> > >>>> a different question. In this case, it's a very important question,
> > >>> because
> > >>>> unit testing
> > >>>> isn't just about test cases. It's about the entire infrastructure
> that
> > >>>> runs, captures and
> > >>>> handles test results later. Writing the test cases is the least of
> the
> > >>>> problems.
> > >>>> The infrastructure for cmoka testing is already in place, the
> > >>>> infrastructure for Unity
> > >>>> testing isn't.
> > >>>>
> > >>>> pon., 22 gru 2025 o 14:16 Alan C. Assis <ac...@gmail.com>
> > >> napisał(a):
> > >>>>
> > >>>>> Mateusz,
> > >>>>> but some chips like AVR DA have big flash, but not so much RAM.
> > >>>>>
> > >>>>> It is better to have some possible way to test on a small system,
> > >> using
> > >>>>> cmocka we are removing this option from the table.
> > >>>>>
> > >>>>> I liked the test Matteo did with Unity and I think we should let
> him
> > >>>>> continue. Forcing him to migrate to cmocka just because there's a
> > >> test
> > >>>>> suite from Xiaomi doesn't seem prudent to me.
> > >>>>>
> > >>>>> This is the advantage of an open mind open-source project: "the
> > >>>> possibility
> > >>>>> to test new ideas", if someone had said to Michal Lenc he couldn't
> > >>> create
> > >>>>> NXBoot because where is already MCUBoot integrated on NuttX, we
> > >> didn't
> > >>>> have
> > >>>>> the option to use NXBoot today. ;-)
> > >>>>>
> > >>>>> BR,
> > >>>>>
> > >>>>> Alan
> > >>>>>
> > >>>>> On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <ra...@gmail.com>
> > >>>> wrote:
> > >>>>>
> > >>>>>> Unit testing by itself isn't suitable for small systems. Ostest is
> > >>>>>> something
> > >>>>>> different (system level testing), so running it on small systems
> > >>> makes
> > >>>>>> sense.
> > >>>>>> Running unit tests on small systems is pointless because you won't
> > >>> fit
> > >>>> in
> > >>>>>> FLASH
> > >>>>>> either way. You can run the same on large targets (preferably on
> > >>> host:
> > >>>>> sim,
> > >>>>>> qemu)
> > >>>>>> and achieve the same result.
> > >>>>>>
> > >>>>>> pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com>
> > >>>> napisał(a):
> > >>>>>>
> > >>>>>>> Is it too complicated to rewrite the cmocka tests in Unity?
> > >>>>>>>
> > >>>>>>> If cmoka is not feasible for small microcontrollers, like AVRs
> > >> and
> > >>>>>> others,
> > >>>>>>> our test coverage will be faulty.
> > >>>>>>>
> > >>>>>>> BR,
> > >>>>>>>
> > >>>>>>> Alan
> > >>>>>>>
> > >>>>>>> On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <raiden00pl@gmail.com
> > >>>
> > >>>>> wrote:
> > >>>>>>>
> > >>>>>>>> I'm a fan of Unity and have been using it in my projects for
> > >>> years,
> > >>>>>> but I
> > >>>>>>>> think
> > >>>>>>>> we should stick with one testing framework. Currently, Cmocka
> > >> is
> > >>>> used
> > >>>>>> in
> > >>>>>>>> nuttx-apps, and we should stick with it. The only question is
> > >> how
> > >>>>> well
> > >>>>>>>> Cmocka
> > >>>>>>>> is suitable for running on small systems? Will ostest still be
> > >>> able
> > >>>>> to
> > >>>>>>> run
> > >>>>>>>> on small MCUs?
> > >>>>>>>> With Unity this would be quite likely, so maybe an exception
> > >> for
> > >>>>> Unity
> > >>>>>>> and
> > >>>>>>>> ostest would
> > >>>>>>>> be acceptable, but using it for other test cases will only
> > >> cause
> > >>>>> chaos.
> > >>>>>>>>
> > >>>>>>>> pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com>
> > >>>> napisał(a):
> > >>>>>>>>
> > >>>>>>>>> Hi Matteo,
> > >>>>>>>>>
> > >>>>>>>>> Good idea !
> > >>>>>>>>>
> > >>>>>>>>> But it is best to stick with a single testing framework.
> > >>>> Currently,
> > >>>>>>> many
> > >>>>>>>>> test cases are using cmocka, so I suggest reusing cmocka for
> > >>>>>>> consistency.
> > >>>>>>>>>
> > >>>>>>>>> ostest is a minimal basic test suite, typically used for CT
> > >>>>>> (continuous
> > >>>>>>>>> testing) of each patch.
> > >>>>>>>>> If we move ostest to cmocka, it would make ostest dependent
> > >> on
> > >>> a
> > >>>>>>> specific
> > >>>>>>>>> framework before it can be run.
> > >>>>>>>>> And It may increase the code size, which is not very friendly
> > >>> for
> > >>>>>>>>> configurations with small flash memory.
> > >>>>>>>>>
> > >>>>>>>>> As Greg mentioned: the answer depends on a tradeoff between
> > >>>> effort
> > >>>>>> and
> > >>>>>>>> test
> > >>>>>>>>> coverage.
> > >>>>>>>>>
> > >>>>>>>>> We need to clarify whether we define ostest as a lightweight,
> > >>>>>>>> ready-to-run
> > >>>>>>>>> test set for quick verification, or as part of the full test
> > >>>> suite.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> The follow is my compiling for cmocka:
> > >>>>>>>>>
> > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh
> > >>>>> sim:citest
> > >>>>>>>> -j8*
> > >>>>>>>>> Copy files
> > >>>>>>>>> Select CONFIG_HOST_LINUX=y
> > >>>>>>>>> Refreshing...
> > >>>>>>>>> CP: arch/dummy/Kconfig to
> > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> > >>>>>>>>> CP: boards/dummy/Kconfig to
> > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> > >>>>>>>>> LN: include/arch to arch/sim/include
> > >>>>>>>>> LN: drivers/platform to
> > >>>>>>> /home/ligd/platform/mainline/nuttx/drivers/dummy
> > >>>>>>>>> LN: arch/sim/src/chip to
> > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> > >>>>>>>>> LN: arch/sim/src/board to
> > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> > >>>>>>>>> LN: include/arch/board to
> > >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> > >>>>>>>>> LN: include/arch/chip to
> > >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> > >>>>>>>>> LN: platform/board to
> > >>>>>> /home/ligd/platform/mainline/apps/platform/dummy
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/boot
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/canutils
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/crypto
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/database
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> > >>>>>>>>> mkkconfig in
> > >> /home/ligd/platform/mainline/apps/examples/sotest
> > >>>>>>>>> mkkconfig in
> > >> /home/ligd/platform/mainline/apps/examples/module
> > >>>>>>>>> mkkconfig in
> > >> /home/ligd/platform/mainline/apps/examples/mcuboot
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/examples
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/games
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/graphics
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/industry
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/inertial
> > >>>>>>>>> mkkconfig in
> > >>>>>> /home/ligd/platform/mainline/apps/interpreters/luamodules
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/logging
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/lte
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/math
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/netutils
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/sdr
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/system
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/tee
> > >>>>>>>>> mkkconfig in
> > >> /home/ligd/platform/mainline/apps/testing/drivers
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> > >>>>>>>>> mkkconfig in
> > >>> /home/ligd/platform/mainline/apps/wireless/bluetooth
> > >>>>>>>>> mkkconfig in
> > >>>> /home/ligd/platform/mainline/apps/wireless/ieee802154
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/wireless
> > >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps
> > >>>>>>>>> Loaded configuration '.config'
> > >>>>>>>>> Configuration saved to '.config'
> > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
> > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ make*
> > >>>>>>>>> Create version.h
> > >>>>>>>>> Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> > >>>>>>>>> /home/ligd/platform/mainline/apps/platform/dummy
> > >>>>>>>>> Downloading: v1.46.0.zip
> > >>>>>>>>> ...
> > >>>>>>>>> patching file
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> > >>>>>>>>> Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> > >>>>>>>>>
> > >>>>>>>>> CP:
> > >> /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> > >>>>>>>>> CP:
> > >>> /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> > >>>>>>>>> LD: nuttx
> > >>>>>>>>> Pac SIM with dynamic libs..
> > >>>>>>>>> '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> > >>>>>>>>> '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
> > >>>>>> 'sim-pac/libs/libstdc++.so.6'
> > >>>>>>>>> '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> > >>>>>>>>> '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> > >>>>>>>>> '/lib/x86_64-linux-gnu/libgcc_s.so.1' ->
> > >>>>> 'sim-pac/libs/libgcc_s.so.1'
> > >>>>>>>>> '/lib64/ld-linux-x86-64.so.2' ->
> > >> 'sim-pac/ld-linux-x86-64.so.2'
> > >>>>>>>>> SIM elf with dynamic libs archive in nuttx.tgz
> > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
> > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
> > >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
> > >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> > >>>>>>>>> CHelloWorld: Constructor: mSecret=42
> > >>>>>>>>> nsh: init: open failed: 2
> > >>>>>>>>>
> > >>>>>>>>> NuttShell (NSH) NuttX-12.11.0
> > >>>>>>>>> MOTD: username=admin password=Administrator
> > >>>>>>>>> nsh>
> > >>>>>>>>> nsh> help
> > >>>>>>>>> help usage: help [-v] [<cmd>]
> > >>>>>>>>>
> > >>>>>>>>> . df ifconfig mkfifo readlink
> > >>>>>> umount
> > >>>>>>>>> [ dmesg ifdown mkrd rm
> > >>>>> unset
> > >>>>>>>>> ? echo ifup mount rmdir
> > >>>>>> uptime
> > >>>>>>>>> alias env insmod mv rmmod
> > >>>>>> usleep
> > >>>>>>>>> unalias exec kill nslookup set
> > >>>>> watch
> > >>>>>>>>> basename exit pkill pidof sleep
> > >>>> xd
> > >>>>>>>>> break expr losetup pmconfig source
> > >>>>> wait
> > >>>>>>>>> cat false ln poweroff test
> > >>>>>>>>> cd fdinfo ls quit time
> > >>>>>>>>> cp free lsmod printf true
> > >>>>>>>>> cmp help mkdir ps truncate
> > >>>>>>>>> dirname hexdump mkfatfs pwd uname
> > >>>>>>>>>
> > >>>>>>>>> Builtin Apps:
> > >>>>>>>>> cmocka
> > >>>>>>>>> cmocka_driver_block
> > >>>>>>>>> cmocka_driver_gpio
> > >>>>>>>>> cmocka_driver_pm
> > >>>>>>>>> cmocka_driver_pm_runtime
> > >>>>>>>>> cmocka_driver_regulator
> > >>>>>>>>>
> > >>>>>>>>> Matteo Golin <ma...@gmail.com> 于2025年12月22日周一
> > >> 10:15写道：
> > >>>>>>>>>
> > >>>>>>>>>> I think so. Best to leave OS test alone and slowly replace
> > >> it
> > >>>>> with
> > >>>>>> a
> > >>>>>>>> new
> > >>>>>>>>>> tool that has improvements. The only reason I can see to
> > >>> modify
> > >>>>> OS
> > >>>>>>> test
> > >>>>>>>>> now
> > >>>>>>>>>> would just be to have clearer output about what
> > >>> passed/failed.
> > >>>>>>>>>>
> > >>>>>>>>>> I'll come up with an improved proof of concept for a new
> > >> tool
> > >>>> to
> > >>>>>>> share
> > >>>>>>>> in
> > >>>>>>>>>> the next few days.
> > >>>>>>>>>>
> > >>>>>>>>>> Matteo
> > >>>>>>>>>>
> > >>>>>>>>>> On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <
> > >> tomek@cedro.info>
> > >>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Small steps with measurable results may be the best
> > >>> approach
> > >>>>>> here,
> > >>>>>>> to
> > >>>>>>>>>>> create some new solution next to existing ostest but not
> > >>>>>> replacing
> > >>>>>>> it
> > >>>>>>>>>>> until ready? :-)
> > >>>>>>>>>>>
> > >>>>>>>>>>> --
> > >>>>>>>>>>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> > >>>>>>> matteo.golin@gmail.com
> > >>>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I agree, I think it would take a long time to have
> > >> proper
> > >>>>>>> coverage
> > >>>>>>>> of
> > >>>>>>>>>>> NuttX
> > >>>>>>>>>>>> as a whole. Perhaps even impossible, since Unity is
> > >> more
> > >>>> of a
> > >>>>>>> unit
> > >>>>>>>>>>> testing
> > >>>>>>>>>>>> framework and thus there are definitely test cases for
> > >>>> NuttX
> > >>>>>> that
> > >>>>>>>>> would
> > >>>>>>>>>>> be
> > >>>>>>>>>>>> outside of its realm of "applicability". There is
> > >> still a
> > >>>>> need
> > >>>>>>> for
> > >>>>>>>> a
> > >>>>>>>>>>>> variety of testing applications and bench-marking
> > >>>>> applications
> > >>>>>>> for
> > >>>>>>>>>> those
> > >>>>>>>>>>>> scenarios.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> That makes sense, prerequisites would need to be
> > >>>> considered.
> > >>>>> I
> > >>>>>>>> think
> > >>>>>>>>>>>> targeting the low-hanging fruit of unit-testable code
> > >>> would
> > >>>>> be
> > >>>>>>> easy
> > >>>>>>>>> in
> > >>>>>>>>>>> that
> > >>>>>>>>>>>> case, since prerequisites are mostly encoded in the
> > >>> Kconfig
> > >>>>>>>>> definitions
> > >>>>>>>>>>>> existing or not for features to be tested. Any complex
> > >>>>>> behaviour
> > >>>>>>>>>>>> (interrupts, multi-threading, etc.) might be out of the
> > >>>> realm
> > >>>>>> of
> > >>>>>>>>> what's
> > >>>>>>>>>>>> testable by Unity in the first place. We'll always have
> > >>> OS
> > >>>>> test
> > >>>>>>> in
> > >>>>>>>>> that
> > >>>>>>>>>>>> case, but maybe then it's better to do what Alan
> > >>> suggested
> > >>>>> and
> > >>>>>> at
> > >>>>>>>>> least
> > >>>>>>>>>>>> make its output more clear in the meantime. I suppose
> > >>> RTOS
> > >>>>>>> testing
> > >>>>>>>>> is a
> > >>>>>>>>>>>> whole field in itself.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> > >>>>>>> spudaneco@gmail.com>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> I think that the answer depends on a tradeoff between
> > >>>>> effort
> > >>>>>>> and
> > >>>>>>>>> test
> > >>>>>>>>>>>>> coverage. I could imagine that a proper job could be
> > >>>>>> man-years
> > >>>>>>>> of
> > >>>>>>>>>>> effort.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> My only concern with a re-design is to be certain
> > >> that
> > >>>>> tests
> > >>>>>>> are
> > >>>>>>>>>>>>> arranged so that the prerequisites of later test are
> > >>>> tested
> > >>>>>>>> first.
> > >>>>>>>>>> If
> > >>>>>>>>>>>>> tests are grouped by, for example, functional area,
> > >>> then
> > >>>>>>> failures
> > >>>>>>>>> can
> > >>>>>>>>>>> be
> > >>>>>>>>>>>>> difficult to understand if there is a failure in an
> > >>>>>> unverified
> > >>>>>>>> test
> > >>>>>>>>>>>>> prerequisite.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Some of the "tests" are bogus and only exist to
> > >> provide
> > >>>>> some
> > >>>>>>>>> minimum
> > >>>>>>>>>>>>> confidence that we have the functionality to run the
> > >>>> test.
> > >>>>>> For
> > >>>>>>>>>>>>> example: Can I start tasks? Can I pass parameters?
> > >>> Can
> > >>>> I
> > >>>>>>>> survive
> > >>>>>>>>> a
> > >>>>>>>>>>>>> context switch. Some of the tests were intended only
> > >>> to
> > >>>>>>> support
> > >>>>>>>>>> debug.
> > >>>>>>>>>>>>> Some were created only to track down specific bugs.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On 12/21/25 18:39, Matteo Golin wrote:
> > >>>>>>>>>>>>>> LTS being the Long-Term-Stable release of NuttX?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I'm starting to think that it would be best to
> > >> create
> > >>>> new
> > >>>>>>> suite
> > >>>>>>>>> of
> > >>>>>>>>>>> tests
> > >>>>>>>>>>>>>> using Unity, using what's in OS Test as a basis and
> > >>>> then
> > >>>>>>>>> expanding
> > >>>>>>>>>> it
> > >>>>>>>>>>>>>> further. Greg has pointed out that OS test is not
> > >>>>> intended
> > >>>>>> to
> > >>>>>>>> be
> > >>>>>>>>> a
> > >>>>>>>>>>>>> complete
> > >>>>>>>>>>>>>> test, so maybe a more solid testing framework base
> > >>> in a
> > >>>>> new
> > >>>>>>>> form
> > >>>>>>>>>>> would be
> > >>>>>>>>>>>>>> in order. At that point it would also be good to
> > >>>> document
> > >>>>>> and
> > >>>>>>>>>> educate
> > >>>>>>>>>>>>> users
> > >>>>>>>>>>>>>> about other test options, since OS test is a
> > >> current
> > >>>>>>> favourite
> > >>>>>>>>> for
> > >>>>>>>>>>>>> showing
> > >>>>>>>>>>>>>> PRs haven't broken anything. Maybe that's not the
> > >>> best
> > >>>>>> choice
> > >>>>>>>>>> anymore
> > >>>>>>>>>>>>> with
> > >>>>>>>>>>>>>> this information.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I don't want to re-invent the wheel of what's
> > >> already
> > >>>> in
> > >>>>>>>>>> `testsuite`,
> > >>>>>>>>>>>>> but I
> > >>>>>>>>>>>>>> think Unity might end up being lighter-weight in
> > >> the
> > >>>> long
> > >>>>>>> run?
> > >>>>>>>>> And
> > >>>>>>>>>>> it is
> > >>>>>>>>>>>>>> also not a Xiaomi owned/maintained framework so it
> > >>> may
> > >>>>>>> receive
> > >>>>>>>>>> better
> > >>>>>>>>>>>>>> external support (i.e. more users rely on it?). I
> > >>> tried
> > >>>>> to
> > >>>>>>>>> compile
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>> sim:citest configuration to run the testsuite
> > >> example
> > >>>> and
> > >>>>>> see
> > >>>>>>>>> what
> > >>>>>>>>>>> it was
> > >>>>>>>>>>>>>> like, but I was met with a long list of C++
> > >>> compilation
> > >>>>>>> errors
> > >>>>>>>>> that
> > >>>>>>>>>>> I'd
> > >>>>>>>>>>>>>> rather not deal with. There also seems to be zero
> > >>> docs
> > >>>>>> about
> > >>>>>>> it
> > >>>>>>>>> or
> > >>>>>>>>>>> cmocka
> > >>>>>>>>>>>>>> in our NuttX docs at the moment.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> What do you think? Start a Unity test suite, add
> > >> some
> > >>>>>>> switches
> > >>>>>>>> to
> > >>>>>>>>>>> choose
> > >>>>>>>>>>>>>> which test cases get compiled into the binary, open
> > >>> for
> > >>>>>>>>> extensions
> > >>>>>>>>>> by
> > >>>>>>>>>>>>> users
> > >>>>>>>>>>>>>> who know more about different subsystems? This way
> > >> we
> > >>>> can
> > >>>>>>> keep
> > >>>>>>>> OS
> > >>>>>>>>>>> test
> > >>>>>>>>>>>>> how
> > >>>>>>>>>>>>>> it is for a quick, low-resource sanity check of
> > >>> "board
> > >>>>>>> bringup"
> > >>>>>>>>>> while
> > >>>>>>>>>>>>>> having a dedicated testing application for patches?
> > >>>>> Someone
> > >>>>>>>>>> changing
> > >>>>>>>>>>> file
> > >>>>>>>>>>>>>> system stuff can just compile to include file
> > >> system
> > >>>> test
> > >>>>>>> cases
> > >>>>>>>>> and
> > >>>>>>>>>>> run
> > >>>>>>>>>>>>>> that suite only as proof that their PR works. The
> > >>>> output
> > >>>>> is
> > >>>>>>>>> really
> > >>>>>>>>>>> easy
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>> glance at to determine pass/fail results.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Anyways, here's my draft demonstrating how Unity
> > >>> works
> > >>>>> with
> > >>>>>>> the
> > >>>>>>>>> FPU
> > >>>>>>>>>>> test
> > >>>>>>>>>>>>>> suite from OS test:
> > >>>>>>>>> https://github.com/apache/nuttx-apps/pull/3259
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Matteo
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> > >>>>>>> tomek@cedro.info>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> > >>>>>>>>> spudaneco@gmail.com
> > >>>>>>>>>>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>> (..)
> > >>>>>>>>>>>>>>>> Starting with bits and pieces of something like
> > >> the
> > >>>> LTS
> > >>>>>>> might
> > >>>>>>>>>> make
> > >>>>>>>>>>> more
> > >>>>>>>>>>>>>>>> sense other than its licensing problems. Since
> > >> it
> > >>>>> would
> > >>>>>>>> never
> > >>>>>>>>> be
> > >>>>>>>>>>>>>>>> included in a distributed binary, the licensing
> > >>>> really
> > >>>>>>> should
> > >>>>>>>>> not
> > >>>>>>>>>>>>> matter
> > >>>>>>>>>>>>>>>> much in practice.
> > >>>>>>>>>>>>>>> Unless its GPL and some company would like to have
> > >>>> their
> > >>>>>> own
> > >>>>>>>>>>> internal
> > >>>>>>>>>>>>>>> bits and pieces.. luckily its MIT and the whole
> > >>> thing
> > >>>> is
> > >>>>>>>>>> open-source
> > >>>>>>>>>>>>>>> anyways :-)
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Yes we had many discussions about the LTS and this
> > >>>> could
> > >>>>>> be
> > >>>>>>>>> great
> > >>>>>>>>>>> start!
> > >>>>>>>>>>>>>>> :-)
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
>
Re: OS Test re-factor proposal
Posted by "Alan C. Assis" <ac...@gmail.com>.
Hi Pierre-Noel,
The instructions are here: https://nuttx.apache.org/community/
We are glad for all the things you did for NuttX (EFM32 port, ESP8266, etc.)
Thank you very much!
BR,
Alan
On Mon, Dec 22, 2025 at 3:16 PM Pierre-Noel Bouteville <pn...@hotmail.com>
wrote:
> How to unsbcribe?
> Pierre-Noël Bouteville
> Envoyé de mon iPhone
>
> > Le 22 déc. 2025 à 19:13, raiden00pl <ra...@gmail.com> a écrit :
> >
> > Cmocka tests can run on any target. It doesn't matter if it's sim, qemu
> or
> > real hardware.
> > You just need to prepare a defconfig and have the right amount of FLASH.
> >
> > For small systems we can always try to reduce the size of the cmock, e.g.
> > by disabling
> > unnecessary prints. I doubt that the logic of running tests in cmoka is
> > heavy, running
> > test cases are not a complicated operation, I think most of the required
> > FLASH is wasted
> > on strings.
> >
> > NTFC supports serial port communication and sample test scenario based on
> > nucleo-h743z
> > with cmocka tests enabled. The only thing missing for full test
> automation
> > (compile, flash,
> > test, result) is cmake support for testsuites.
> >
> > pon., 22 gru 2025 o 18:08 Matteo Golin <ma...@gmail.com>
> napisał(a):
> >
> >> Hello everyone,
> >>
> >> Thanks for all the feedback! I'm glad I discussed this here first. I'll
> try
> >> to address everything that was brought up.
> >>
> >> Thanks Guiding for the input! I must just be missing some library on my
> >> machine if you're able to compile it just fine. Maybe I'll track down
> that
> >> error and see if you can recognize what's wrong.
> >>
> >> I don't intend to replace OS test anymore. I think it serves its
> purpose as
> >> a minimal sanity check. Instead, I think we need something a little more
> >> involved to test PRs since OS test can't catch many issues. It is
> currently
> >> the standard everyone uses for verifying their patches and it would be
> >> better to have more complete, alternate options. I do think that we can
> all
> >> agree that OS test needs a revision of its console output so that it is
> >> much easier to see pass/fail results, but that is just a matter of print
> >> statements and can be done in a different effort.
> >>
> >> I think the overwhelming concern here is mixing two unit testing
> frameworks
> >> and no longer having one resource where we can go for tests. I share
> raiden
> >> and Guiding's concern about that, which is why I'm glad I opened for
> >> discussion here. On the one hand, Xiaomi has a pretty substantial
> >> collection of tests in the `testsuite` application which they are using
> >> internally I believe (in their CI). These run well on virtual machines,
> >> which I assume is why the only configurations with that test suite
> included
> >> is sim and rv-virt. I do not know nearly enough about cmocka to make any
> >> claim that it *wouldn't* work on smaller hardware devices, but it also
> >> definitely looks to be less light-weight from the feature set in the
> >> documentation. On the other hand, the Xiaomi test suites are also not a
> >> complete silver bullet since they seem not to be intended to run on
> >> hardware, and there are *a lot *of tests that need hardware
> verification.
> >> In PRs we often request more than just a simulator test for certain
> >> changes. For instance, sim isn't able to have `/dev/rand`, so that
> needs to
> >> be tested on a different target.
> >>
> >> Maybe someone more familiar with Xiaomi's testing practices can weigh
> in on
> >> whether cmocka has been run on hardware. Going purely based off of their
> >> PRs, which almost exclusively cite tests on simulators, my guess is that
> >> cmocka was not designed with embedded in mind when it was written.
> >>
> >> As for size: unit testing can get large with Unity or cmocka (as raiden
> >> mentioned) as more tests are added. Hence my effort to enable/disable
> >> individual test suites from getting compiled. It might not be possible
> to
> >> ever get good testing ability for AVR just because the devices are so
> >> small. Whatever solution is used there is probably going to involve
> >> flashing the device multiple times with different tests. The size goal
> >> should just be to target the most NuttX devices possible.
> >>
> >> Then there are the pros and cons of each framework (again, I don't know
> >> much about cmocka so take it with a grain of salt):
> >>
> >> Unity's cons:
> >> - Designed for each test-suite to be a standalone binary. This is
> clunky on
> >> NuttX, so `nest` is currently implemented as a mega-suite where test
> cases
> >> just follow a naming convention to make it clear what is being tested
> >> - setUp/tearDown can't really be leveraged because of the above
> >> - I've noticed some funky behaviour using TEST_IGNORE in certain
> scenarios
> >> but that's possibly my own configuration issue
> >> - Would be in conflict with existing cmocka tests
> >>
> >> Unity's pros:
> >> - Highly configurable, like NuttX
> >> - Light-weight and works for embedded devices
> >> - Significantly larger user base than cmocka
> >>
> >> Cmocka's cons:
> >> - Possibly (?) not light-weight
> >> - No documentation for existing test cases in NuttX
> >> - Small user base
> >>
> >> Cmocka's pros:
> >> - Seems to have a larger set of features, including setup/teardown
> >> functions per-test case
> >> - Existing tests in NuttX
> >>
> >> Irrespective of whether we go with Unity or cmocka, the collection of
> tests
> >> that we write will be intended to run on the actual hardware devices
> >> themselves for the purposes of testing patches and doing regression
> tests.
> >> As a result, the tests will be separate from the Xiaomi 'testsuite'
> >> application which is intended for CI testing. I think this realistically
> >> nullifies choosing which framework to use on the basis of consistency,
> >> since the tests are not coexisting with the cmocka tests Xiaomi has
> >> written. Instead, the choice should be based on what best achieves the
> goal
> >> from a functionality perspective. Unity seems like a really great choice
> >> for this in my opinion, but I am bothered by the fact that it's designed
> >> with multiple test binaries in mind.
> >>
> >> Let me know what you think,
> >> Matteo
> >>
> >>> On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <ac...@gmail.com>
> wrote:
> >>>
> >>> Yes, I agree! We need data size from each solution for comparison, this
> >> is
> >>> the first step.
> >>>
> >>> Maybe the conversion could be facilitated if we create equivalent
> >> functions
> >>> like those used in the apps/testing/testsuites:
> >>>
> >>> cmocka_run_group_tests -> unity_run_group_tests
> >>> cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown
> >>> etc...
> >>>
> >>> This way we avoid end up with two unit test framework, we already have
> >> the
> >>> Makefile vs CMake precedent! :-D
> >>>
> >>> On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <ra...@gmail.com>
> >> wrote:
> >>>
> >>>> The discussion about small systems doesn't make much sense without
> >>>> data comparing resources of the same test cases for cmock and unity.
> >>>>
> >>>> But let's assume that Unity is better. Should we use two frameworks
> >>>> to implement unit tests for OS? Or choose one framework? If we choose
> >>> only
> >>>> Unity, who
> >>>> will migrate the current cmocka tests to Unity? Remember that cmoka
> >> tests
> >>>> are used in CI, so all CI tools must be also migrated.
> >>>> Or should we use two different frameworks to test different things and
> >>> make
> >>>> even
> >>>> more mess in apps/testing as it is now?
> >>>>
> >>>> No one is forcing anyone to do anything. This is an open-source
> >> project,
> >>>> everyone
> >>>> should do what they feel is right. But whether the change should be
> >>> merged
> >>>> is
> >>>> a different question. In this case, it's a very important question,
> >>> because
> >>>> unit testing
> >>>> isn't just about test cases. It's about the entire infrastructure that
> >>>> runs, captures and
> >>>> handles test results later. Writing the test cases is the least of the
> >>>> problems.
> >>>> The infrastructure for cmoka testing is already in place, the
> >>>> infrastructure for Unity
> >>>> testing isn't.
> >>>>
> >>>> pon., 22 gru 2025 o 14:16 Alan C. Assis <ac...@gmail.com>
> >> napisał(a):
> >>>>
> >>>>> Mateusz,
> >>>>> but some chips like AVR DA have big flash, but not so much RAM.
> >>>>>
> >>>>> It is better to have some possible way to test on a small system,
> >> using
> >>>>> cmocka we are removing this option from the table.
> >>>>>
> >>>>> I liked the test Matteo did with Unity and I think we should let him
> >>>>> continue. Forcing him to migrate to cmocka just because there's a
> >> test
> >>>>> suite from Xiaomi doesn't seem prudent to me.
> >>>>>
> >>>>> This is the advantage of an open mind open-source project: "the
> >>>> possibility
> >>>>> to test new ideas", if someone had said to Michal Lenc he couldn't
> >>> create
> >>>>> NXBoot because where is already MCUBoot integrated on NuttX, we
> >> didn't
> >>>> have
> >>>>> the option to use NXBoot today. ;-)
> >>>>>
> >>>>> BR,
> >>>>>
> >>>>> Alan
> >>>>>
> >>>>> On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <ra...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>>> Unit testing by itself isn't suitable for small systems. Ostest is
> >>>>>> something
> >>>>>> different (system level testing), so running it on small systems
> >>> makes
> >>>>>> sense.
> >>>>>> Running unit tests on small systems is pointless because you won't
> >>> fit
> >>>> in
> >>>>>> FLASH
> >>>>>> either way. You can run the same on large targets (preferably on
> >>> host:
> >>>>> sim,
> >>>>>> qemu)
> >>>>>> and achieve the same result.
> >>>>>>
> >>>>>> pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com>
> >>>> napisał(a):
> >>>>>>
> >>>>>>> Is it too complicated to rewrite the cmocka tests in Unity?
> >>>>>>>
> >>>>>>> If cmoka is not feasible for small microcontrollers, like AVRs
> >> and
> >>>>>> others,
> >>>>>>> our test coverage will be faulty.
> >>>>>>>
> >>>>>>> BR,
> >>>>>>>
> >>>>>>> Alan
> >>>>>>>
> >>>>>>> On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <raiden00pl@gmail.com
> >>>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> I'm a fan of Unity and have been using it in my projects for
> >>> years,
> >>>>>> but I
> >>>>>>>> think
> >>>>>>>> we should stick with one testing framework. Currently, Cmocka
> >> is
> >>>> used
> >>>>>> in
> >>>>>>>> nuttx-apps, and we should stick with it. The only question is
> >> how
> >>>>> well
> >>>>>>>> Cmocka
> >>>>>>>> is suitable for running on small systems? Will ostest still be
> >>> able
> >>>>> to
> >>>>>>> run
> >>>>>>>> on small MCUs?
> >>>>>>>> With Unity this would be quite likely, so maybe an exception
> >> for
> >>>>> Unity
> >>>>>>> and
> >>>>>>>> ostest would
> >>>>>>>> be acceptable, but using it for other test cases will only
> >> cause
> >>>>> chaos.
> >>>>>>>>
> >>>>>>>> pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com>
> >>>> napisał(a):
> >>>>>>>>
> >>>>>>>>> Hi Matteo,
> >>>>>>>>>
> >>>>>>>>> Good idea !
> >>>>>>>>>
> >>>>>>>>> But it is best to stick with a single testing framework.
> >>>> Currently,
> >>>>>>> many
> >>>>>>>>> test cases are using cmocka, so I suggest reusing cmocka for
> >>>>>>> consistency.
> >>>>>>>>>
> >>>>>>>>> ostest is a minimal basic test suite, typically used for CT
> >>>>>> (continuous
> >>>>>>>>> testing) of each patch.
> >>>>>>>>> If we move ostest to cmocka, it would make ostest dependent
> >> on
> >>> a
> >>>>>>> specific
> >>>>>>>>> framework before it can be run.
> >>>>>>>>> And It may increase the code size, which is not very friendly
> >>> for
> >>>>>>>>> configurations with small flash memory.
> >>>>>>>>>
> >>>>>>>>> As Greg mentioned: the answer depends on a tradeoff between
> >>>> effort
> >>>>>> and
> >>>>>>>> test
> >>>>>>>>> coverage.
> >>>>>>>>>
> >>>>>>>>> We need to clarify whether we define ostest as a lightweight,
> >>>>>>>> ready-to-run
> >>>>>>>>> test set for quick verification, or as part of the full test
> >>>> suite.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> The follow is my compiling for cmocka:
> >>>>>>>>>
> >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh
> >>>>> sim:citest
> >>>>>>>> -j8*
> >>>>>>>>> Copy files
> >>>>>>>>> Select CONFIG_HOST_LINUX=y
> >>>>>>>>> Refreshing...
> >>>>>>>>> CP: arch/dummy/Kconfig to
> >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> >>>>>>>>> CP: boards/dummy/Kconfig to
> >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> >>>>>>>>> LN: include/arch to arch/sim/include
> >>>>>>>>> LN: drivers/platform to
> >>>>>>> /home/ligd/platform/mainline/nuttx/drivers/dummy
> >>>>>>>>> LN: arch/sim/src/chip to
> >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> >>>>>>>>> LN: arch/sim/src/board to
> >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> >>>>>>>>> LN: include/arch/board to
> >>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> >>>>>>>>> LN: include/arch/chip to
> >>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> >>>>>>>>> LN: platform/board to
> >>>>>> /home/ligd/platform/mainline/apps/platform/dummy
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/boot
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/canutils
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/crypto
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/database
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> >>>>>>>>> mkkconfig in
> >> /home/ligd/platform/mainline/apps/examples/sotest
> >>>>>>>>> mkkconfig in
> >> /home/ligd/platform/mainline/apps/examples/module
> >>>>>>>>> mkkconfig in
> >> /home/ligd/platform/mainline/apps/examples/mcuboot
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/examples
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/games
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/graphics
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/industry
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/inertial
> >>>>>>>>> mkkconfig in
> >>>>>> /home/ligd/platform/mainline/apps/interpreters/luamodules
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/logging
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/lte
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/math
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/netutils
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/sdr
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/system
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/tee
> >>>>>>>>> mkkconfig in
> >> /home/ligd/platform/mainline/apps/testing/drivers
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> >>>>>>>>> mkkconfig in
> >>> /home/ligd/platform/mainline/apps/wireless/bluetooth
> >>>>>>>>> mkkconfig in
> >>>> /home/ligd/platform/mainline/apps/wireless/ieee802154
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/wireless
> >>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps
> >>>>>>>>> Loaded configuration '.config'
> >>>>>>>>> Configuration saved to '.config'
> >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
> >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ make*
> >>>>>>>>> Create version.h
> >>>>>>>>> Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> >>>>>>>>> /home/ligd/platform/mainline/apps/platform/dummy
> >>>>>>>>> Downloading: v1.46.0.zip
> >>>>>>>>> ...
> >>>>>>>>> patching file
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> >>>>>>>>> Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> >>>>>>>>>
> >>>>>>>>> CP:
> >> /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> >>>>>>>>> CP:
> >>> /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> >>>>>>>>> LD: nuttx
> >>>>>>>>> Pac SIM with dynamic libs..
> >>>>>>>>> '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> >>>>>>>>> '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
> >>>>>> 'sim-pac/libs/libstdc++.so.6'
> >>>>>>>>> '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> >>>>>>>>> '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> >>>>>>>>> '/lib/x86_64-linux-gnu/libgcc_s.so.1' ->
> >>>>> 'sim-pac/libs/libgcc_s.so.1'
> >>>>>>>>> '/lib64/ld-linux-x86-64.so.2' ->
> >> 'sim-pac/ld-linux-x86-64.so.2'
> >>>>>>>>> SIM elf with dynamic libs archive in nuttx.tgz
> >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
> >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
> >>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
> >>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> >>>>>>>>> CHelloWorld: Constructor: mSecret=42
> >>>>>>>>> nsh: init: open failed: 2
> >>>>>>>>>
> >>>>>>>>> NuttShell (NSH) NuttX-12.11.0
> >>>>>>>>> MOTD: username=admin password=Administrator
> >>>>>>>>> nsh>
> >>>>>>>>> nsh> help
> >>>>>>>>> help usage: help [-v] [<cmd>]
> >>>>>>>>>
> >>>>>>>>> . df ifconfig mkfifo readlink
> >>>>>> umount
> >>>>>>>>> [ dmesg ifdown mkrd rm
> >>>>> unset
> >>>>>>>>> ? echo ifup mount rmdir
> >>>>>> uptime
> >>>>>>>>> alias env insmod mv rmmod
> >>>>>> usleep
> >>>>>>>>> unalias exec kill nslookup set
> >>>>> watch
> >>>>>>>>> basename exit pkill pidof sleep
> >>>> xd
> >>>>>>>>> break expr losetup pmconfig source
> >>>>> wait
> >>>>>>>>> cat false ln poweroff test
> >>>>>>>>> cd fdinfo ls quit time
> >>>>>>>>> cp free lsmod printf true
> >>>>>>>>> cmp help mkdir ps truncate
> >>>>>>>>> dirname hexdump mkfatfs pwd uname
> >>>>>>>>>
> >>>>>>>>> Builtin Apps:
> >>>>>>>>> cmocka
> >>>>>>>>> cmocka_driver_block
> >>>>>>>>> cmocka_driver_gpio
> >>>>>>>>> cmocka_driver_pm
> >>>>>>>>> cmocka_driver_pm_runtime
> >>>>>>>>> cmocka_driver_regulator
> >>>>>>>>>
> >>>>>>>>> Matteo Golin <ma...@gmail.com> 于2025年12月22日周一
> >> 10:15写道：
> >>>>>>>>>
> >>>>>>>>>> I think so. Best to leave OS test alone and slowly replace
> >> it
> >>>>> with
> >>>>>> a
> >>>>>>>> new
> >>>>>>>>>> tool that has improvements. The only reason I can see to
> >>> modify
> >>>>> OS
> >>>>>>> test
> >>>>>>>>> now
> >>>>>>>>>> would just be to have clearer output about what
> >>> passed/failed.
> >>>>>>>>>>
> >>>>>>>>>> I'll come up with an improved proof of concept for a new
> >> tool
> >>>> to
> >>>>>>> share
> >>>>>>>> in
> >>>>>>>>>> the next few days.
> >>>>>>>>>>
> >>>>>>>>>> Matteo
> >>>>>>>>>>
> >>>>>>>>>> On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <
> >> tomek@cedro.info>
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Small steps with measurable results may be the best
> >>> approach
> >>>>>> here,
> >>>>>>> to
> >>>>>>>>>>> create some new solution next to existing ostest but not
> >>>>>> replacing
> >>>>>>> it
> >>>>>>>>>>> until ready? :-)
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> >>>>>>> matteo.golin@gmail.com
> >>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I agree, I think it would take a long time to have
> >> proper
> >>>>>>> coverage
> >>>>>>>> of
> >>>>>>>>>>> NuttX
> >>>>>>>>>>>> as a whole. Perhaps even impossible, since Unity is
> >> more
> >>>> of a
> >>>>>>> unit
> >>>>>>>>>>> testing
> >>>>>>>>>>>> framework and thus there are definitely test cases for
> >>>> NuttX
> >>>>>> that
> >>>>>>>>> would
> >>>>>>>>>>> be
> >>>>>>>>>>>> outside of its realm of "applicability". There is
> >> still a
> >>>>> need
> >>>>>>> for
> >>>>>>>> a
> >>>>>>>>>>>> variety of testing applications and bench-marking
> >>>>> applications
> >>>>>>> for
> >>>>>>>>>> those
> >>>>>>>>>>>> scenarios.
> >>>>>>>>>>>>
> >>>>>>>>>>>> That makes sense, prerequisites would need to be
> >>>> considered.
> >>>>> I
> >>>>>>>> think
> >>>>>>>>>>>> targeting the low-hanging fruit of unit-testable code
> >>> would
> >>>>> be
> >>>>>>> easy
> >>>>>>>>> in
> >>>>>>>>>>> that
> >>>>>>>>>>>> case, since prerequisites are mostly encoded in the
> >>> Kconfig
> >>>>>>>>> definitions
> >>>>>>>>>>>> existing or not for features to be tested. Any complex
> >>>>>> behaviour
> >>>>>>>>>>>> (interrupts, multi-threading, etc.) might be out of the
> >>>> realm
> >>>>>> of
> >>>>>>>>> what's
> >>>>>>>>>>>> testable by Unity in the first place. We'll always have
> >>> OS
> >>>>> test
> >>>>>>> in
> >>>>>>>>> that
> >>>>>>>>>>>> case, but maybe then it's better to do what Alan
> >>> suggested
> >>>>> and
> >>>>>> at
> >>>>>>>>> least
> >>>>>>>>>>>> make its output more clear in the meantime. I suppose
> >>> RTOS
> >>>>>>> testing
> >>>>>>>>> is a
> >>>>>>>>>>>> whole field in itself.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> >>>>>>> spudaneco@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I think that the answer depends on a tradeoff between
> >>>>> effort
> >>>>>>> and
> >>>>>>>>> test
> >>>>>>>>>>>>> coverage. I could imagine that a proper job could be
> >>>>>> man-years
> >>>>>>>> of
> >>>>>>>>>>> effort.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> My only concern with a re-design is to be certain
> >> that
> >>>>> tests
> >>>>>>> are
> >>>>>>>>>>>>> arranged so that the prerequisites of later test are
> >>>> tested
> >>>>>>>> first.
> >>>>>>>>>> If
> >>>>>>>>>>>>> tests are grouped by, for example, functional area,
> >>> then
> >>>>>>> failures
> >>>>>>>>> can
> >>>>>>>>>>> be
> >>>>>>>>>>>>> difficult to understand if there is a failure in an
> >>>>>> unverified
> >>>>>>>> test
> >>>>>>>>>>>>> prerequisite.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Some of the "tests" are bogus and only exist to
> >> provide
> >>>>> some
> >>>>>>>>> minimum
> >>>>>>>>>>>>> confidence that we have the functionality to run the
> >>>> test.
> >>>>>> For
> >>>>>>>>>>>>> example: Can I start tasks? Can I pass parameters?
> >>> Can
> >>>> I
> >>>>>>>> survive
> >>>>>>>>> a
> >>>>>>>>>>>>> context switch. Some of the tests were intended only
> >>> to
> >>>>>>> support
> >>>>>>>>>> debug.
> >>>>>>>>>>>>> Some were created only to track down specific bugs.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 12/21/25 18:39, Matteo Golin wrote:
> >>>>>>>>>>>>>> LTS being the Long-Term-Stable release of NuttX?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm starting to think that it would be best to
> >> create
> >>>> new
> >>>>>>> suite
> >>>>>>>>> of
> >>>>>>>>>>> tests
> >>>>>>>>>>>>>> using Unity, using what's in OS Test as a basis and
> >>>> then
> >>>>>>>>> expanding
> >>>>>>>>>> it
> >>>>>>>>>>>>>> further. Greg has pointed out that OS test is not
> >>>>> intended
> >>>>>> to
> >>>>>>>> be
> >>>>>>>>> a
> >>>>>>>>>>>>> complete
> >>>>>>>>>>>>>> test, so maybe a more solid testing framework base
> >>> in a
> >>>>> new
> >>>>>>>> form
> >>>>>>>>>>> would be
> >>>>>>>>>>>>>> in order. At that point it would also be good to
> >>>> document
> >>>>>> and
> >>>>>>>>>> educate
> >>>>>>>>>>>>> users
> >>>>>>>>>>>>>> about other test options, since OS test is a
> >> current
> >>>>>>> favourite
> >>>>>>>>> for
> >>>>>>>>>>>>> showing
> >>>>>>>>>>>>>> PRs haven't broken anything. Maybe that's not the
> >>> best
> >>>>>> choice
> >>>>>>>>>> anymore
> >>>>>>>>>>>>> with
> >>>>>>>>>>>>>> this information.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I don't want to re-invent the wheel of what's
> >> already
> >>>> in
> >>>>>>>>>> `testsuite`,
> >>>>>>>>>>>>> but I
> >>>>>>>>>>>>>> think Unity might end up being lighter-weight in
> >> the
> >>>> long
> >>>>>>> run?
> >>>>>>>>> And
> >>>>>>>>>>> it is
> >>>>>>>>>>>>>> also not a Xiaomi owned/maintained framework so it
> >>> may
> >>>>>>> receive
> >>>>>>>>>> better
> >>>>>>>>>>>>>> external support (i.e. more users rely on it?). I
> >>> tried
> >>>>> to
> >>>>>>>>> compile
> >>>>>>>>>>> the
> >>>>>>>>>>>>>> sim:citest configuration to run the testsuite
> >> example
> >>>> and
> >>>>>> see
> >>>>>>>>> what
> >>>>>>>>>>> it was
> >>>>>>>>>>>>>> like, but I was met with a long list of C++
> >>> compilation
> >>>>>>> errors
> >>>>>>>>> that
> >>>>>>>>>>> I'd
> >>>>>>>>>>>>>> rather not deal with. There also seems to be zero
> >>> docs
> >>>>>> about
> >>>>>>> it
> >>>>>>>>> or
> >>>>>>>>>>> cmocka
> >>>>>>>>>>>>>> in our NuttX docs at the moment.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> What do you think? Start a Unity test suite, add
> >> some
> >>>>>>> switches
> >>>>>>>> to
> >>>>>>>>>>> choose
> >>>>>>>>>>>>>> which test cases get compiled into the binary, open
> >>> for
> >>>>>>>>> extensions
> >>>>>>>>>> by
> >>>>>>>>>>>>> users
> >>>>>>>>>>>>>> who know more about different subsystems? This way
> >> we
> >>>> can
> >>>>>>> keep
> >>>>>>>> OS
> >>>>>>>>>>> test
> >>>>>>>>>>>>> how
> >>>>>>>>>>>>>> it is for a quick, low-resource sanity check of
> >>> "board
> >>>>>>> bringup"
> >>>>>>>>>> while
> >>>>>>>>>>>>>> having a dedicated testing application for patches?
> >>>>> Someone
> >>>>>>>>>> changing
> >>>>>>>>>>> file
> >>>>>>>>>>>>>> system stuff can just compile to include file
> >> system
> >>>> test
> >>>>>>> cases
> >>>>>>>>> and
> >>>>>>>>>>> run
> >>>>>>>>>>>>>> that suite only as proof that their PR works. The
> >>>> output
> >>>>> is
> >>>>>>>>> really
> >>>>>>>>>>> easy
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>> glance at to determine pass/fail results.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Anyways, here's my draft demonstrating how Unity
> >>> works
> >>>>> with
> >>>>>>> the
> >>>>>>>>> FPU
> >>>>>>>>>>> test
> >>>>>>>>>>>>>> suite from OS test:
> >>>>>>>>> https://github.com/apache/nuttx-apps/pull/3259
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Matteo
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> >>>>>>> tomek@cedro.info>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> >>>>>>>>> spudaneco@gmail.com
> >>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>> (..)
> >>>>>>>>>>>>>>>> Starting with bits and pieces of something like
> >> the
> >>>> LTS
> >>>>>>> might
> >>>>>>>>>> make
> >>>>>>>>>>> more
> >>>>>>>>>>>>>>>> sense other than its licensing problems. Since
> >> it
> >>>>> would
> >>>>>>>> never
> >>>>>>>>> be
> >>>>>>>>>>>>>>>> included in a distributed binary, the licensing
> >>>> really
> >>>>>>> should
> >>>>>>>>> not
> >>>>>>>>>>>>> matter
> >>>>>>>>>>>>>>>> much in practice.
> >>>>>>>>>>>>>>> Unless its GPL and some company would like to have
> >>>> their
> >>>>>> own
> >>>>>>>>>>> internal
> >>>>>>>>>>>>>>> bits and pieces.. luckily its MIT and the whole
> >>> thing
> >>>> is
> >>>>>>>>>> open-source
> >>>>>>>>>>>>>>> anyways :-)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Yes we had many discussions about the LTS and this
> >>>> could
> >>>>>> be
> >>>>>>>>> great
> >>>>>>>>>>> start!
> >>>>>>>>>>>>>>> :-)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
Re: OS Test re-factor proposal
Posted by Pierre-Noel Bouteville <pn...@hotmail.com>.
How to unsbcribe?
Pierre-Noël Bouteville
Envoyé de mon iPhone
> Le 22 déc. 2025 à 19:13, raiden00pl <ra...@gmail.com> a écrit :
>
> Cmocka tests can run on any target. It doesn't matter if it's sim, qemu or
> real hardware.
> You just need to prepare a defconfig and have the right amount of FLASH.
>
> For small systems we can always try to reduce the size of the cmock, e.g.
> by disabling
> unnecessary prints. I doubt that the logic of running tests in cmoka is
> heavy, running
> test cases are not a complicated operation, I think most of the required
> FLASH is wasted
> on strings.
>
> NTFC supports serial port communication and sample test scenario based on
> nucleo-h743z
> with cmocka tests enabled. The only thing missing for full test automation
> (compile, flash,
> test, result) is cmake support for testsuites.
>
> pon., 22 gru 2025 o 18:08 Matteo Golin <ma...@gmail.com> napisał(a):
>
>> Hello everyone,
>>
>> Thanks for all the feedback! I'm glad I discussed this here first. I'll try
>> to address everything that was brought up.
>>
>> Thanks Guiding for the input! I must just be missing some library on my
>> machine if you're able to compile it just fine. Maybe I'll track down that
>> error and see if you can recognize what's wrong.
>>
>> I don't intend to replace OS test anymore. I think it serves its purpose as
>> a minimal sanity check. Instead, I think we need something a little more
>> involved to test PRs since OS test can't catch many issues. It is currently
>> the standard everyone uses for verifying their patches and it would be
>> better to have more complete, alternate options. I do think that we can all
>> agree that OS test needs a revision of its console output so that it is
>> much easier to see pass/fail results, but that is just a matter of print
>> statements and can be done in a different effort.
>>
>> I think the overwhelming concern here is mixing two unit testing frameworks
>> and no longer having one resource where we can go for tests. I share raiden
>> and Guiding's concern about that, which is why I'm glad I opened for
>> discussion here. On the one hand, Xiaomi has a pretty substantial
>> collection of tests in the `testsuite` application which they are using
>> internally I believe (in their CI). These run well on virtual machines,
>> which I assume is why the only configurations with that test suite included
>> is sim and rv-virt. I do not know nearly enough about cmocka to make any
>> claim that it *wouldn't* work on smaller hardware devices, but it also
>> definitely looks to be less light-weight from the feature set in the
>> documentation. On the other hand, the Xiaomi test suites are also not a
>> complete silver bullet since they seem not to be intended to run on
>> hardware, and there are *a lot *of tests that need hardware verification.
>> In PRs we often request more than just a simulator test for certain
>> changes. For instance, sim isn't able to have `/dev/rand`, so that needs to
>> be tested on a different target.
>>
>> Maybe someone more familiar with Xiaomi's testing practices can weigh in on
>> whether cmocka has been run on hardware. Going purely based off of their
>> PRs, which almost exclusively cite tests on simulators, my guess is that
>> cmocka was not designed with embedded in mind when it was written.
>>
>> As for size: unit testing can get large with Unity or cmocka (as raiden
>> mentioned) as more tests are added. Hence my effort to enable/disable
>> individual test suites from getting compiled. It might not be possible to
>> ever get good testing ability for AVR just because the devices are so
>> small. Whatever solution is used there is probably going to involve
>> flashing the device multiple times with different tests. The size goal
>> should just be to target the most NuttX devices possible.
>>
>> Then there are the pros and cons of each framework (again, I don't know
>> much about cmocka so take it with a grain of salt):
>>
>> Unity's cons:
>> - Designed for each test-suite to be a standalone binary. This is clunky on
>> NuttX, so `nest` is currently implemented as a mega-suite where test cases
>> just follow a naming convention to make it clear what is being tested
>> - setUp/tearDown can't really be leveraged because of the above
>> - I've noticed some funky behaviour using TEST_IGNORE in certain scenarios
>> but that's possibly my own configuration issue
>> - Would be in conflict with existing cmocka tests
>>
>> Unity's pros:
>> - Highly configurable, like NuttX
>> - Light-weight and works for embedded devices
>> - Significantly larger user base than cmocka
>>
>> Cmocka's cons:
>> - Possibly (?) not light-weight
>> - No documentation for existing test cases in NuttX
>> - Small user base
>>
>> Cmocka's pros:
>> - Seems to have a larger set of features, including setup/teardown
>> functions per-test case
>> - Existing tests in NuttX
>>
>> Irrespective of whether we go with Unity or cmocka, the collection of tests
>> that we write will be intended to run on the actual hardware devices
>> themselves for the purposes of testing patches and doing regression tests.
>> As a result, the tests will be separate from the Xiaomi 'testsuite'
>> application which is intended for CI testing. I think this realistically
>> nullifies choosing which framework to use on the basis of consistency,
>> since the tests are not coexisting with the cmocka tests Xiaomi has
>> written. Instead, the choice should be based on what best achieves the goal
>> from a functionality perspective. Unity seems like a really great choice
>> for this in my opinion, but I am bothered by the fact that it's designed
>> with multiple test binaries in mind.
>>
>> Let me know what you think,
>> Matteo
>>
>>> On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <ac...@gmail.com> wrote:
>>>
>>> Yes, I agree! We need data size from each solution for comparison, this
>> is
>>> the first step.
>>>
>>> Maybe the conversion could be facilitated if we create equivalent
>> functions
>>> like those used in the apps/testing/testsuites:
>>>
>>> cmocka_run_group_tests -> unity_run_group_tests
>>> cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown
>>> etc...
>>>
>>> This way we avoid end up with two unit test framework, we already have
>> the
>>> Makefile vs CMake precedent! :-D
>>>
>>> On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <ra...@gmail.com>
>> wrote:
>>>
>>>> The discussion about small systems doesn't make much sense without
>>>> data comparing resources of the same test cases for cmock and unity.
>>>>
>>>> But let's assume that Unity is better. Should we use two frameworks
>>>> to implement unit tests for OS? Or choose one framework? If we choose
>>> only
>>>> Unity, who
>>>> will migrate the current cmocka tests to Unity? Remember that cmoka
>> tests
>>>> are used in CI, so all CI tools must be also migrated.
>>>> Or should we use two different frameworks to test different things and
>>> make
>>>> even
>>>> more mess in apps/testing as it is now?
>>>>
>>>> No one is forcing anyone to do anything. This is an open-source
>> project,
>>>> everyone
>>>> should do what they feel is right. But whether the change should be
>>> merged
>>>> is
>>>> a different question. In this case, it's a very important question,
>>> because
>>>> unit testing
>>>> isn't just about test cases. It's about the entire infrastructure that
>>>> runs, captures and
>>>> handles test results later. Writing the test cases is the least of the
>>>> problems.
>>>> The infrastructure for cmoka testing is already in place, the
>>>> infrastructure for Unity
>>>> testing isn't.
>>>>
>>>> pon., 22 gru 2025 o 14:16 Alan C. Assis <ac...@gmail.com>
>> napisał(a):
>>>>
>>>>> Mateusz,
>>>>> but some chips like AVR DA have big flash, but not so much RAM.
>>>>>
>>>>> It is better to have some possible way to test on a small system,
>> using
>>>>> cmocka we are removing this option from the table.
>>>>>
>>>>> I liked the test Matteo did with Unity and I think we should let him
>>>>> continue. Forcing him to migrate to cmocka just because there's a
>> test
>>>>> suite from Xiaomi doesn't seem prudent to me.
>>>>>
>>>>> This is the advantage of an open mind open-source project: "the
>>>> possibility
>>>>> to test new ideas", if someone had said to Michal Lenc he couldn't
>>> create
>>>>> NXBoot because where is already MCUBoot integrated on NuttX, we
>> didn't
>>>> have
>>>>> the option to use NXBoot today. ;-)
>>>>>
>>>>> BR,
>>>>>
>>>>> Alan
>>>>>
>>>>> On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <ra...@gmail.com>
>>>> wrote:
>>>>>
>>>>>> Unit testing by itself isn't suitable for small systems. Ostest is
>>>>>> something
>>>>>> different (system level testing), so running it on small systems
>>> makes
>>>>>> sense.
>>>>>> Running unit tests on small systems is pointless because you won't
>>> fit
>>>> in
>>>>>> FLASH
>>>>>> either way. You can run the same on large targets (preferably on
>>> host:
>>>>> sim,
>>>>>> qemu)
>>>>>> and achieve the same result.
>>>>>>
>>>>>> pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com>
>>>> napisał(a):
>>>>>>
>>>>>>> Is it too complicated to rewrite the cmocka tests in Unity?
>>>>>>>
>>>>>>> If cmoka is not feasible for small microcontrollers, like AVRs
>> and
>>>>>> others,
>>>>>>> our test coverage will be faulty.
>>>>>>>
>>>>>>> BR,
>>>>>>>
>>>>>>> Alan
>>>>>>>
>>>>>>> On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <raiden00pl@gmail.com
>>>
>>>>> wrote:
>>>>>>>
>>>>>>>> I'm a fan of Unity and have been using it in my projects for
>>> years,
>>>>>> but I
>>>>>>>> think
>>>>>>>> we should stick with one testing framework. Currently, Cmocka
>> is
>>>> used
>>>>>> in
>>>>>>>> nuttx-apps, and we should stick with it. The only question is
>> how
>>>>> well
>>>>>>>> Cmocka
>>>>>>>> is suitable for running on small systems? Will ostest still be
>>> able
>>>>> to
>>>>>>> run
>>>>>>>> on small MCUs?
>>>>>>>> With Unity this would be quite likely, so maybe an exception
>> for
>>>>> Unity
>>>>>>> and
>>>>>>>> ostest would
>>>>>>>> be acceptable, but using it for other test cases will only
>> cause
>>>>> chaos.
>>>>>>>>
>>>>>>>> pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com>
>>>> napisał(a):
>>>>>>>>
>>>>>>>>> Hi Matteo,
>>>>>>>>>
>>>>>>>>> Good idea !
>>>>>>>>>
>>>>>>>>> But it is best to stick with a single testing framework.
>>>> Currently,
>>>>>>> many
>>>>>>>>> test cases are using cmocka, so I suggest reusing cmocka for
>>>>>>> consistency.
>>>>>>>>>
>>>>>>>>> ostest is a minimal basic test suite, typically used for CT
>>>>>> (continuous
>>>>>>>>> testing) of each patch.
>>>>>>>>> If we move ostest to cmocka, it would make ostest dependent
>> on
>>> a
>>>>>>> specific
>>>>>>>>> framework before it can be run.
>>>>>>>>> And It may increase the code size, which is not very friendly
>>> for
>>>>>>>>> configurations with small flash memory.
>>>>>>>>>
>>>>>>>>> As Greg mentioned: the answer depends on a tradeoff between
>>>> effort
>>>>>> and
>>>>>>>> test
>>>>>>>>> coverage.
>>>>>>>>>
>>>>>>>>> We need to clarify whether we define ostest as a lightweight,
>>>>>>>> ready-to-run
>>>>>>>>> test set for quick verification, or as part of the full test
>>>> suite.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The follow is my compiling for cmocka:
>>>>>>>>>
>>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh
>>>>> sim:citest
>>>>>>>> -j8*
>>>>>>>>> Copy files
>>>>>>>>> Select CONFIG_HOST_LINUX=y
>>>>>>>>> Refreshing...
>>>>>>>>> CP: arch/dummy/Kconfig to
>>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
>>>>>>>>> CP: boards/dummy/Kconfig to
>>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
>>>>>>>>> LN: include/arch to arch/sim/include
>>>>>>>>> LN: drivers/platform to
>>>>>>> /home/ligd/platform/mainline/nuttx/drivers/dummy
>>>>>>>>> LN: arch/sim/src/chip to
>>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
>>>>>>>>> LN: arch/sim/src/board to
>>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
>>>>>>>>> LN: include/arch/board to
>>>>>>>>> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
>>>>>>>>> LN: include/arch/chip to
>>>>>>>>> /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
>>>>>>>>> LN: platform/board to
>>>>>> /home/ligd/platform/mainline/apps/platform/dummy
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/audioutils
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/boot
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/canutils
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/crypto
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/database
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
>>>>>>>>> mkkconfig in
>> /home/ligd/platform/mainline/apps/examples/sotest
>>>>>>>>> mkkconfig in
>> /home/ligd/platform/mainline/apps/examples/module
>>>>>>>>> mkkconfig in
>> /home/ligd/platform/mainline/apps/examples/mcuboot
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/examples
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/fsutils
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/games
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/graphics
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/industry
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/inertial
>>>>>>>>> mkkconfig in
>>>>>> /home/ligd/platform/mainline/apps/interpreters/luamodules
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/interpreters
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/logging
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/lte
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/math
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/mlearning
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/netutils
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/sdr
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/system
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/tee
>>>>>>>>> mkkconfig in
>> /home/ligd/platform/mainline/apps/testing/drivers
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/testing
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/videoutils
>>>>>>>>> mkkconfig in
>>> /home/ligd/platform/mainline/apps/wireless/bluetooth
>>>>>>>>> mkkconfig in
>>>> /home/ligd/platform/mainline/apps/wireless/ieee802154
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps/wireless
>>>>>>>>> mkkconfig in /home/ligd/platform/mainline/apps
>>>>>>>>> Loaded configuration '.config'
>>>>>>>>> Configuration saved to '.config'
>>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
>>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ make*
>>>>>>>>> Create version.h
>>>>>>>>> Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
>>>>>>>>> /home/ligd/platform/mainline/apps/platform/dummy
>>>>>>>>> Downloading: v1.46.0.zip
>>>>>>>>> ...
>>>>>>>>> patching file
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
>>>>>>>>> Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
>>>>>>>>>
>>>>>>>>> CP:
>> /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
>>>>>>>>> CP:
>>> /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
>>>>>>>>> LD: nuttx
>>>>>>>>> Pac SIM with dynamic libs..
>>>>>>>>> '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
>>>>>>>>> '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
>>>>>> 'sim-pac/libs/libstdc++.so.6'
>>>>>>>>> '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
>>>>>>>>> '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
>>>>>>>>> '/lib/x86_64-linux-gnu/libgcc_s.so.1' ->
>>>>> 'sim-pac/libs/libgcc_s.so.1'
>>>>>>>>> '/lib64/ld-linux-x86-64.so.2' ->
>> 'sim-pac/ld-linux-x86-64.so.2'
>>>>>>>>> SIM elf with dynamic libs archive in nuttx.tgz
>>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
>>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
>>>>>>>>> ligd@BAI:~/platform/mainline/nuttx$
>>>>>>>>> *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
>>>>>>>>> CHelloWorld: Constructor: mSecret=42
>>>>>>>>> nsh: init: open failed: 2
>>>>>>>>>
>>>>>>>>> NuttShell (NSH) NuttX-12.11.0
>>>>>>>>> MOTD: username=admin password=Administrator
>>>>>>>>> nsh>
>>>>>>>>> nsh> help
>>>>>>>>> help usage: help [-v] [<cmd>]
>>>>>>>>>
>>>>>>>>> . df ifconfig mkfifo readlink
>>>>>> umount
>>>>>>>>> [ dmesg ifdown mkrd rm
>>>>> unset
>>>>>>>>> ? echo ifup mount rmdir
>>>>>> uptime
>>>>>>>>> alias env insmod mv rmmod
>>>>>> usleep
>>>>>>>>> unalias exec kill nslookup set
>>>>> watch
>>>>>>>>> basename exit pkill pidof sleep
>>>> xd
>>>>>>>>> break expr losetup pmconfig source
>>>>> wait
>>>>>>>>> cat false ln poweroff test
>>>>>>>>> cd fdinfo ls quit time
>>>>>>>>> cp free lsmod printf true
>>>>>>>>> cmp help mkdir ps truncate
>>>>>>>>> dirname hexdump mkfatfs pwd uname
>>>>>>>>>
>>>>>>>>> Builtin Apps:
>>>>>>>>> cmocka
>>>>>>>>> cmocka_driver_block
>>>>>>>>> cmocka_driver_gpio
>>>>>>>>> cmocka_driver_pm
>>>>>>>>> cmocka_driver_pm_runtime
>>>>>>>>> cmocka_driver_regulator
>>>>>>>>>
>>>>>>>>> Matteo Golin <ma...@gmail.com> 于2025年12月22日周一
>> 10:15写道：
>>>>>>>>>
>>>>>>>>>> I think so. Best to leave OS test alone and slowly replace
>> it
>>>>> with
>>>>>> a
>>>>>>>> new
>>>>>>>>>> tool that has improvements. The only reason I can see to
>>> modify
>>>>> OS
>>>>>>> test
>>>>>>>>> now
>>>>>>>>>> would just be to have clearer output about what
>>> passed/failed.
>>>>>>>>>>
>>>>>>>>>> I'll come up with an improved proof of concept for a new
>> tool
>>>> to
>>>>>>> share
>>>>>>>> in
>>>>>>>>>> the next few days.
>>>>>>>>>>
>>>>>>>>>> Matteo
>>>>>>>>>>
>>>>>>>>>> On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <
>> tomek@cedro.info>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Small steps with measurable results may be the best
>>> approach
>>>>>> here,
>>>>>>> to
>>>>>>>>>>> create some new solution next to existing ostest but not
>>>>>> replacing
>>>>>>> it
>>>>>>>>>>> until ready? :-)
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
>>>>>>> matteo.golin@gmail.com
>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I agree, I think it would take a long time to have
>> proper
>>>>>>> coverage
>>>>>>>> of
>>>>>>>>>>> NuttX
>>>>>>>>>>>> as a whole. Perhaps even impossible, since Unity is
>> more
>>>> of a
>>>>>>> unit
>>>>>>>>>>> testing
>>>>>>>>>>>> framework and thus there are definitely test cases for
>>>> NuttX
>>>>>> that
>>>>>>>>> would
>>>>>>>>>>> be
>>>>>>>>>>>> outside of its realm of "applicability". There is
>> still a
>>>>> need
>>>>>>> for
>>>>>>>> a
>>>>>>>>>>>> variety of testing applications and bench-marking
>>>>> applications
>>>>>>> for
>>>>>>>>>> those
>>>>>>>>>>>> scenarios.
>>>>>>>>>>>>
>>>>>>>>>>>> That makes sense, prerequisites would need to be
>>>> considered.
>>>>> I
>>>>>>>> think
>>>>>>>>>>>> targeting the low-hanging fruit of unit-testable code
>>> would
>>>>> be
>>>>>>> easy
>>>>>>>>> in
>>>>>>>>>>> that
>>>>>>>>>>>> case, since prerequisites are mostly encoded in the
>>> Kconfig
>>>>>>>>> definitions
>>>>>>>>>>>> existing or not for features to be tested. Any complex
>>>>>> behaviour
>>>>>>>>>>>> (interrupts, multi-threading, etc.) might be out of the
>>>> realm
>>>>>> of
>>>>>>>>> what's
>>>>>>>>>>>> testable by Unity in the first place. We'll always have
>>> OS
>>>>> test
>>>>>>> in
>>>>>>>>> that
>>>>>>>>>>>> case, but maybe then it's better to do what Alan
>>> suggested
>>>>> and
>>>>>> at
>>>>>>>>> least
>>>>>>>>>>>> make its output more clear in the meantime. I suppose
>>> RTOS
>>>>>>> testing
>>>>>>>>> is a
>>>>>>>>>>>> whole field in itself.
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
>>>>>>> spudaneco@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I think that the answer depends on a tradeoff between
>>>>> effort
>>>>>>> and
>>>>>>>>> test
>>>>>>>>>>>>> coverage. I could imagine that a proper job could be
>>>>>> man-years
>>>>>>>> of
>>>>>>>>>>> effort.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My only concern with a re-design is to be certain
>> that
>>>>> tests
>>>>>>> are
>>>>>>>>>>>>> arranged so that the prerequisites of later test are
>>>> tested
>>>>>>>> first.
>>>>>>>>>> If
>>>>>>>>>>>>> tests are grouped by, for example, functional area,
>>> then
>>>>>>> failures
>>>>>>>>> can
>>>>>>>>>>> be
>>>>>>>>>>>>> difficult to understand if there is a failure in an
>>>>>> unverified
>>>>>>>> test
>>>>>>>>>>>>> prerequisite.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Some of the "tests" are bogus and only exist to
>> provide
>>>>> some
>>>>>>>>> minimum
>>>>>>>>>>>>> confidence that we have the functionality to run the
>>>> test.
>>>>>> For
>>>>>>>>>>>>> example: Can I start tasks? Can I pass parameters?
>>> Can
>>>> I
>>>>>>>> survive
>>>>>>>>> a
>>>>>>>>>>>>> context switch. Some of the tests were intended only
>>> to
>>>>>>> support
>>>>>>>>>> debug.
>>>>>>>>>>>>> Some were created only to track down specific bugs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 12/21/25 18:39, Matteo Golin wrote:
>>>>>>>>>>>>>> LTS being the Long-Term-Stable release of NuttX?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm starting to think that it would be best to
>> create
>>>> new
>>>>>>> suite
>>>>>>>>> of
>>>>>>>>>>> tests
>>>>>>>>>>>>>> using Unity, using what's in OS Test as a basis and
>>>> then
>>>>>>>>> expanding
>>>>>>>>>> it
>>>>>>>>>>>>>> further. Greg has pointed out that OS test is not
>>>>> intended
>>>>>> to
>>>>>>>> be
>>>>>>>>> a
>>>>>>>>>>>>> complete
>>>>>>>>>>>>>> test, so maybe a more solid testing framework base
>>> in a
>>>>> new
>>>>>>>> form
>>>>>>>>>>> would be
>>>>>>>>>>>>>> in order. At that point it would also be good to
>>>> document
>>>>>> and
>>>>>>>>>> educate
>>>>>>>>>>>>> users
>>>>>>>>>>>>>> about other test options, since OS test is a
>> current
>>>>>>> favourite
>>>>>>>>> for
>>>>>>>>>>>>> showing
>>>>>>>>>>>>>> PRs haven't broken anything. Maybe that's not the
>>> best
>>>>>> choice
>>>>>>>>>> anymore
>>>>>>>>>>>>> with
>>>>>>>>>>>>>> this information.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't want to re-invent the wheel of what's
>> already
>>>> in
>>>>>>>>>> `testsuite`,
>>>>>>>>>>>>> but I
>>>>>>>>>>>>>> think Unity might end up being lighter-weight in
>> the
>>>> long
>>>>>>> run?
>>>>>>>>> And
>>>>>>>>>>> it is
>>>>>>>>>>>>>> also not a Xiaomi owned/maintained framework so it
>>> may
>>>>>>> receive
>>>>>>>>>> better
>>>>>>>>>>>>>> external support (i.e. more users rely on it?). I
>>> tried
>>>>> to
>>>>>>>>> compile
>>>>>>>>>>> the
>>>>>>>>>>>>>> sim:citest configuration to run the testsuite
>> example
>>>> and
>>>>>> see
>>>>>>>>> what
>>>>>>>>>>> it was
>>>>>>>>>>>>>> like, but I was met with a long list of C++
>>> compilation
>>>>>>> errors
>>>>>>>>> that
>>>>>>>>>>> I'd
>>>>>>>>>>>>>> rather not deal with. There also seems to be zero
>>> docs
>>>>>> about
>>>>>>> it
>>>>>>>>> or
>>>>>>>>>>> cmocka
>>>>>>>>>>>>>> in our NuttX docs at the moment.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What do you think? Start a Unity test suite, add
>> some
>>>>>>> switches
>>>>>>>> to
>>>>>>>>>>> choose
>>>>>>>>>>>>>> which test cases get compiled into the binary, open
>>> for
>>>>>>>>> extensions
>>>>>>>>>> by
>>>>>>>>>>>>> users
>>>>>>>>>>>>>> who know more about different subsystems? This way
>> we
>>>> can
>>>>>>> keep
>>>>>>>> OS
>>>>>>>>>>> test
>>>>>>>>>>>>> how
>>>>>>>>>>>>>> it is for a quick, low-resource sanity check of
>>> "board
>>>>>>> bringup"
>>>>>>>>>> while
>>>>>>>>>>>>>> having a dedicated testing application for patches?
>>>>> Someone
>>>>>>>>>> changing
>>>>>>>>>>> file
>>>>>>>>>>>>>> system stuff can just compile to include file
>> system
>>>> test
>>>>>>> cases
>>>>>>>>> and
>>>>>>>>>>> run
>>>>>>>>>>>>>> that suite only as proof that their PR works. The
>>>> output
>>>>> is
>>>>>>>>> really
>>>>>>>>>>> easy
>>>>>>>>>>>>> to
>>>>>>>>>>>>>> glance at to determine pass/fail results.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Anyways, here's my draft demonstrating how Unity
>>> works
>>>>> with
>>>>>>> the
>>>>>>>>> FPU
>>>>>>>>>>> test
>>>>>>>>>>>>>> suite from OS test:
>>>>>>>>> https://github.com/apache/nuttx-apps/pull/3259
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Matteo
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
>>>>>>> tomek@cedro.info>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
>>>>>>>>> spudaneco@gmail.com
>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> (..)
>>>>>>>>>>>>>>>> Starting with bits and pieces of something like
>> the
>>>> LTS
>>>>>>> might
>>>>>>>>>> make
>>>>>>>>>>> more
>>>>>>>>>>>>>>>> sense other than its licensing problems. Since
>> it
>>>>> would
>>>>>>>> never
>>>>>>>>> be
>>>>>>>>>>>>>>>> included in a distributed binary, the licensing
>>>> really
>>>>>>> should
>>>>>>>>> not
>>>>>>>>>>>>> matter
>>>>>>>>>>>>>>>> much in practice.
>>>>>>>>>>>>>>> Unless its GPL and some company would like to have
>>>> their
>>>>>> own
>>>>>>>>>>> internal
>>>>>>>>>>>>>>> bits and pieces.. luckily its MIT and the whole
>>> thing
>>>> is
>>>>>>>>>> open-source
>>>>>>>>>>>>>>> anyways :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes we had many discussions about the LTS and this
>>>> could
>>>>>> be
>>>>>>>>> great
>>>>>>>>>>> start!
>>>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: OS Test re-factor proposal
Posted by raiden00pl <ra...@gmail.com>.
Cmocka tests can run on any target. It doesn't matter if it's sim, qemu or
real hardware.
You just need to prepare a defconfig and have the right amount of FLASH.
For small systems we can always try to reduce the size of the cmock, e.g.
by disabling
unnecessary prints. I doubt that the logic of running tests in cmoka is
heavy, running
test cases are not a complicated operation, I think most of the required
FLASH is wasted
on strings.
NTFC supports serial port communication and sample test scenario based on
nucleo-h743z
with cmocka tests enabled. The only thing missing for full test automation
(compile, flash,
test, result) is cmake support for testsuites.
pon., 22 gru 2025 o 18:08 Matteo Golin <ma...@gmail.com> napisał(a):
> Hello everyone,
>
> Thanks for all the feedback! I'm glad I discussed this here first. I'll try
> to address everything that was brought up.
>
> Thanks Guiding for the input! I must just be missing some library on my
> machine if you're able to compile it just fine. Maybe I'll track down that
> error and see if you can recognize what's wrong.
>
> I don't intend to replace OS test anymore. I think it serves its purpose as
> a minimal sanity check. Instead, I think we need something a little more
> involved to test PRs since OS test can't catch many issues. It is currently
> the standard everyone uses for verifying their patches and it would be
> better to have more complete, alternate options. I do think that we can all
> agree that OS test needs a revision of its console output so that it is
> much easier to see pass/fail results, but that is just a matter of print
> statements and can be done in a different effort.
>
> I think the overwhelming concern here is mixing two unit testing frameworks
> and no longer having one resource where we can go for tests. I share raiden
> and Guiding's concern about that, which is why I'm glad I opened for
> discussion here. On the one hand, Xiaomi has a pretty substantial
> collection of tests in the `testsuite` application which they are using
> internally I believe (in their CI). These run well on virtual machines,
> which I assume is why the only configurations with that test suite included
> is sim and rv-virt. I do not know nearly enough about cmocka to make any
> claim that it *wouldn't* work on smaller hardware devices, but it also
> definitely looks to be less light-weight from the feature set in the
> documentation. On the other hand, the Xiaomi test suites are also not a
> complete silver bullet since they seem not to be intended to run on
> hardware, and there are *a lot *of tests that need hardware verification.
> In PRs we often request more than just a simulator test for certain
> changes. For instance, sim isn't able to have `/dev/rand`, so that needs to
> be tested on a different target.
>
> Maybe someone more familiar with Xiaomi's testing practices can weigh in on
> whether cmocka has been run on hardware. Going purely based off of their
> PRs, which almost exclusively cite tests on simulators, my guess is that
> cmocka was not designed with embedded in mind when it was written.
>
> As for size: unit testing can get large with Unity or cmocka (as raiden
> mentioned) as more tests are added. Hence my effort to enable/disable
> individual test suites from getting compiled. It might not be possible to
> ever get good testing ability for AVR just because the devices are so
> small. Whatever solution is used there is probably going to involve
> flashing the device multiple times with different tests. The size goal
> should just be to target the most NuttX devices possible.
>
> Then there are the pros and cons of each framework (again, I don't know
> much about cmocka so take it with a grain of salt):
>
> Unity's cons:
> - Designed for each test-suite to be a standalone binary. This is clunky on
> NuttX, so `nest` is currently implemented as a mega-suite where test cases
> just follow a naming convention to make it clear what is being tested
> - setUp/tearDown can't really be leveraged because of the above
> - I've noticed some funky behaviour using TEST_IGNORE in certain scenarios
> but that's possibly my own configuration issue
> - Would be in conflict with existing cmocka tests
>
> Unity's pros:
> - Highly configurable, like NuttX
> - Light-weight and works for embedded devices
> - Significantly larger user base than cmocka
>
> Cmocka's cons:
> - Possibly (?) not light-weight
> - No documentation for existing test cases in NuttX
> - Small user base
>
> Cmocka's pros:
> - Seems to have a larger set of features, including setup/teardown
> functions per-test case
> - Existing tests in NuttX
>
> Irrespective of whether we go with Unity or cmocka, the collection of tests
> that we write will be intended to run on the actual hardware devices
> themselves for the purposes of testing patches and doing regression tests.
> As a result, the tests will be separate from the Xiaomi 'testsuite'
> application which is intended for CI testing. I think this realistically
> nullifies choosing which framework to use on the basis of consistency,
> since the tests are not coexisting with the cmocka tests Xiaomi has
> written. Instead, the choice should be based on what best achieves the goal
> from a functionality perspective. Unity seems like a really great choice
> for this in my opinion, but I am bothered by the fact that it's designed
> with multiple test binaries in mind.
>
> Let me know what you think,
> Matteo
>
> On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <ac...@gmail.com> wrote:
>
> > Yes, I agree! We need data size from each solution for comparison, this
> is
> > the first step.
> >
> > Maybe the conversion could be facilitated if we create equivalent
> functions
> > like those used in the apps/testing/testsuites:
> >
> > cmocka_run_group_tests -> unity_run_group_tests
> > cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown
> > etc...
> >
> > This way we avoid end up with two unit test framework, we already have
> the
> > Makefile vs CMake precedent! :-D
> >
> > On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <ra...@gmail.com>
> wrote:
> >
> > > The discussion about small systems doesn't make much sense without
> > > data comparing resources of the same test cases for cmock and unity.
> > >
> > > But let's assume that Unity is better. Should we use two frameworks
> > > to implement unit tests for OS? Or choose one framework? If we choose
> > only
> > > Unity, who
> > > will migrate the current cmocka tests to Unity? Remember that cmoka
> tests
> > > are used in CI, so all CI tools must be also migrated.
> > > Or should we use two different frameworks to test different things and
> > make
> > > even
> > > more mess in apps/testing as it is now?
> > >
> > > No one is forcing anyone to do anything. This is an open-source
> project,
> > > everyone
> > > should do what they feel is right. But whether the change should be
> > merged
> > > is
> > > a different question. In this case, it's a very important question,
> > because
> > > unit testing
> > > isn't just about test cases. It's about the entire infrastructure that
> > > runs, captures and
> > > handles test results later. Writing the test cases is the least of the
> > > problems.
> > > The infrastructure for cmoka testing is already in place, the
> > > infrastructure for Unity
> > > testing isn't.
> > >
> > > pon., 22 gru 2025 o 14:16 Alan C. Assis <ac...@gmail.com>
> napisał(a):
> > >
> > > > Mateusz,
> > > > but some chips like AVR DA have big flash, but not so much RAM.
> > > >
> > > > It is better to have some possible way to test on a small system,
> using
> > > > cmocka we are removing this option from the table.
> > > >
> > > > I liked the test Matteo did with Unity and I think we should let him
> > > > continue. Forcing him to migrate to cmocka just because there's a
> test
> > > > suite from Xiaomi doesn't seem prudent to me.
> > > >
> > > > This is the advantage of an open mind open-source project: "the
> > > possibility
> > > > to test new ideas", if someone had said to Michal Lenc he couldn't
> > create
> > > > NXBoot because where is already MCUBoot integrated on NuttX, we
> didn't
> > > have
> > > > the option to use NXBoot today. ;-)
> > > >
> > > > BR,
> > > >
> > > > Alan
> > > >
> > > > On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <ra...@gmail.com>
> > > wrote:
> > > >
> > > > > Unit testing by itself isn't suitable for small systems. Ostest is
> > > > > something
> > > > > different (system level testing), so running it on small systems
> > makes
> > > > > sense.
> > > > > Running unit tests on small systems is pointless because you won't
> > fit
> > > in
> > > > > FLASH
> > > > > either way. You can run the same on large targets (preferably on
> > host:
> > > > sim,
> > > > > qemu)
> > > > > and achieve the same result.
> > > > >
> > > > > pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com>
> > > napisał(a):
> > > > >
> > > > > > Is it too complicated to rewrite the cmocka tests in Unity?
> > > > > >
> > > > > > If cmoka is not feasible for small microcontrollers, like AVRs
> and
> > > > > others,
> > > > > > our test coverage will be faulty.
> > > > > >
> > > > > > BR,
> > > > > >
> > > > > > Alan
> > > > > >
> > > > > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <raiden00pl@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > I'm a fan of Unity and have been using it in my projects for
> > years,
> > > > > but I
> > > > > > > think
> > > > > > > we should stick with one testing framework. Currently, Cmocka
> is
> > > used
> > > > > in
> > > > > > > nuttx-apps, and we should stick with it. The only question is
> how
> > > > well
> > > > > > > Cmocka
> > > > > > > is suitable for running on small systems? Will ostest still be
> > able
> > > > to
> > > > > > run
> > > > > > > on small MCUs?
> > > > > > > With Unity this would be quite likely, so maybe an exception
> for
> > > > Unity
> > > > > > and
> > > > > > > ostest would
> > > > > > > be acceptable, but using it for other test cases will only
> cause
> > > > chaos.
> > > > > > >
> > > > > > > pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com>
> > > napisał(a):
> > > > > > >
> > > > > > > > Hi Matteo,
> > > > > > > >
> > > > > > > > Good idea !
> > > > > > > >
> > > > > > > > But it is best to stick with a single testing framework.
> > > Currently,
> > > > > > many
> > > > > > > > test cases are using cmocka, so I suggest reusing cmocka for
> > > > > > consistency.
> > > > > > > >
> > > > > > > > ostest is a minimal basic test suite, typically used for CT
> > > > > (continuous
> > > > > > > > testing) of each patch.
> > > > > > > > If we move ostest to cmocka, it would make ostest dependent
> on
> > a
> > > > > > specific
> > > > > > > > framework before it can be run.
> > > > > > > > And It may increase the code size, which is not very friendly
> > for
> > > > > > > > configurations with small flash memory.
> > > > > > > >
> > > > > > > > As Greg mentioned: the answer depends on a tradeoff between
> > > effort
> > > > > and
> > > > > > > test
> > > > > > > > coverage.
> > > > > > > >
> > > > > > > > We need to clarify whether we define ostest as a lightweight,
> > > > > > > ready-to-run
> > > > > > > > test set for quick verification, or as part of the full test
> > > suite.
> > > > > > > >
> > > > > > > >
> > > > > > > > The follow is my compiling for cmocka:
> > > > > > > >
> > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh
> > > > sim:citest
> > > > > > > -j8*
> > > > > > > > Copy files
> > > > > > > > Select CONFIG_HOST_LINUX=y
> > > > > > > > Refreshing...
> > > > > > > > CP: arch/dummy/Kconfig to
> > > > > > > > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> > > > > > > > CP: boards/dummy/Kconfig to
> > > > > > > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> > > > > > > > LN: include/arch to arch/sim/include
> > > > > > > > LN: drivers/platform to
> > > > > > /home/ligd/platform/mainline/nuttx/drivers/dummy
> > > > > > > > LN: arch/sim/src/chip to
> > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> > > > > > > > LN: arch/sim/src/board to
> > > > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> > > > > > > > LN: include/arch/board to
> > > > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> > > > > > > > LN: include/arch/chip to
> > > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> > > > > > > > LN: platform/board to
> > > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/boot
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/canutils
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/database
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/examples/sotest
> > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/examples/module
> > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/examples/mcuboot
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/games
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/graphics
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/industry
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/inertial
> > > > > > > > mkkconfig in
> > > > > /home/ligd/platform/mainline/apps/interpreters/luamodules
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/logging
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/lte
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/math
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/netutils
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/system
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/tee
> > > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/testing/drivers
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> > > > > > > > mkkconfig in
> > /home/ligd/platform/mainline/apps/wireless/bluetooth
> > > > > > > > mkkconfig in
> > > /home/ligd/platform/mainline/apps/wireless/ieee802154
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless
> > > > > > > > mkkconfig in /home/ligd/platform/mainline/apps
> > > > > > > > Loaded configuration '.config'
> > > > > > > > Configuration saved to '.config'
> > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ make*
> > > > > > > > Create version.h
> > > > > > > > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> > > > > > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > > > > Downloading: v1.46.0.zip
> > > > > > > > ...
> > > > > > > > patching file
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> > > > > > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> > > > > > > >
> > > > > > > > CP:
> /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> > > > > > > > CP:
> > /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> > > > > > > > LD: nuttx
> > > > > > > > Pac SIM with dynamic libs..
> > > > > > > > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> > > > > > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
> > > > > 'sim-pac/libs/libstdc++.so.6'
> > > > > > > > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> > > > > > > > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> > > > > > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' ->
> > > > 'sim-pac/libs/libgcc_s.so.1'
> > > > > > > > '/lib64/ld-linux-x86-64.so.2' ->
> 'sim-pac/ld-linux-x86-64.so.2'
> > > > > > > > SIM elf with dynamic libs archive in nuttx.tgz
> > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> > > > > > > > CHelloWorld: Constructor: mSecret=42
> > > > > > > > nsh: init: open failed: 2
> > > > > > > >
> > > > > > > > NuttShell (NSH) NuttX-12.11.0
> > > > > > > > MOTD: username=admin password=Administrator
> > > > > > > > nsh>
> > > > > > > > nsh> help
> > > > > > > > help usage: help [-v] [<cmd>]
> > > > > > > >
> > > > > > > > . df ifconfig mkfifo readlink
> > > > > umount
> > > > > > > > [ dmesg ifdown mkrd rm
> > > > unset
> > > > > > > > ? echo ifup mount rmdir
> > > > > uptime
> > > > > > > > alias env insmod mv rmmod
> > > > > usleep
> > > > > > > > unalias exec kill nslookup set
> > > > watch
> > > > > > > > basename exit pkill pidof sleep
> > > xd
> > > > > > > > break expr losetup pmconfig source
> > > > wait
> > > > > > > > cat false ln poweroff test
> > > > > > > > cd fdinfo ls quit time
> > > > > > > > cp free lsmod printf true
> > > > > > > > cmp help mkdir ps truncate
> > > > > > > > dirname hexdump mkfatfs pwd uname
> > > > > > > >
> > > > > > > > Builtin Apps:
> > > > > > > > cmocka
> > > > > > > > cmocka_driver_block
> > > > > > > > cmocka_driver_gpio
> > > > > > > > cmocka_driver_pm
> > > > > > > > cmocka_driver_pm_runtime
> > > > > > > > cmocka_driver_regulator
> > > > > > > >
> > > > > > > > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一
> 10:15写道：
> > > > > > > >
> > > > > > > > > I think so. Best to leave OS test alone and slowly replace
> it
> > > > with
> > > > > a
> > > > > > > new
> > > > > > > > > tool that has improvements. The only reason I can see to
> > modify
> > > > OS
> > > > > > test
> > > > > > > > now
> > > > > > > > > would just be to have clearer output about what
> > passed/failed.
> > > > > > > > >
> > > > > > > > > I'll come up with an improved proof of concept for a new
> tool
> > > to
> > > > > > share
> > > > > > > in
> > > > > > > > > the next few days.
> > > > > > > > >
> > > > > > > > > Matteo
> > > > > > > > >
> > > > > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <
> tomek@cedro.info>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Small steps with measurable results may be the best
> > approach
> > > > > here,
> > > > > > to
> > > > > > > > > > create some new solution next to existing ostest but not
> > > > > replacing
> > > > > > it
> > > > > > > > > > until ready? :-)
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > > > >
> > > > > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> > > > > > matteo.golin@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > I agree, I think it would take a long time to have
> proper
> > > > > > coverage
> > > > > > > of
> > > > > > > > > > NuttX
> > > > > > > > > > > as a whole. Perhaps even impossible, since Unity is
> more
> > > of a
> > > > > > unit
> > > > > > > > > > testing
> > > > > > > > > > > framework and thus there are definitely test cases for
> > > NuttX
> > > > > that
> > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > > outside of its realm of "applicability". There is
> still a
> > > > need
> > > > > > for
> > > > > > > a
> > > > > > > > > > > variety of testing applications and bench-marking
> > > > applications
> > > > > > for
> > > > > > > > > those
> > > > > > > > > > > scenarios.
> > > > > > > > > > >
> > > > > > > > > > > That makes sense, prerequisites would need to be
> > > considered.
> > > > I
> > > > > > > think
> > > > > > > > > > > targeting the low-hanging fruit of unit-testable code
> > would
> > > > be
> > > > > > easy
> > > > > > > > in
> > > > > > > > > > that
> > > > > > > > > > > case, since prerequisites are mostly encoded in the
> > Kconfig
> > > > > > > > definitions
> > > > > > > > > > > existing or not for features to be tested. Any complex
> > > > > behaviour
> > > > > > > > > > > (interrupts, multi-threading, etc.) might be out of the
> > > realm
> > > > > of
> > > > > > > > what's
> > > > > > > > > > > testable by Unity in the first place. We'll always have
> > OS
> > > > test
> > > > > > in
> > > > > > > > that
> > > > > > > > > > > case, but maybe then it's better to do what Alan
> > suggested
> > > > and
> > > > > at
> > > > > > > > least
> > > > > > > > > > > make its output more clear in the meantime. I suppose
> > RTOS
> > > > > > testing
> > > > > > > > is a
> > > > > > > > > > > whole field in itself.
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> > > > > > spudaneco@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I think that the answer depends on a tradeoff between
> > > > effort
> > > > > > and
> > > > > > > > test
> > > > > > > > > > > > coverage. I could imagine that a proper job could be
> > > > > man-years
> > > > > > > of
> > > > > > > > > > effort.
> > > > > > > > > > > >
> > > > > > > > > > > > My only concern with a re-design is to be certain
> that
> > > > tests
> > > > > > are
> > > > > > > > > > > > arranged so that the prerequisites of later test are
> > > tested
> > > > > > > first.
> > > > > > > > > If
> > > > > > > > > > > > tests are grouped by, for example, functional area,
> > then
> > > > > > failures
> > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > > > difficult to understand if there is a failure in an
> > > > > unverified
> > > > > > > test
> > > > > > > > > > > > prerequisite.
> > > > > > > > > > > >
> > > > > > > > > > > > Some of the "tests" are bogus and only exist to
> provide
> > > > some
> > > > > > > > minimum
> > > > > > > > > > > > confidence that we have the functionality to run the
> > > test.
> > > > > For
> > > > > > > > > > > > example: Can I start tasks? Can I pass parameters?
> > Can
> > > I
> > > > > > > survive
> > > > > > > > a
> > > > > > > > > > > > context switch. Some of the tests were intended only
> > to
> > > > > > support
> > > > > > > > > debug.
> > > > > > > > > > > > Some were created only to track down specific bugs.
> > > > > > > > > > > >
> > > > > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > > > > > > > > > LTS being the Long-Term-Stable release of NuttX?
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm starting to think that it would be best to
> create
> > > new
> > > > > > suite
> > > > > > > > of
> > > > > > > > > > tests
> > > > > > > > > > > > > using Unity, using what's in OS Test as a basis and
> > > then
> > > > > > > > expanding
> > > > > > > > > it
> > > > > > > > > > > > > further. Greg has pointed out that OS test is not
> > > > intended
> > > > > to
> > > > > > > be
> > > > > > > > a
> > > > > > > > > > > > complete
> > > > > > > > > > > > > test, so maybe a more solid testing framework base
> > in a
> > > > new
> > > > > > > form
> > > > > > > > > > would be
> > > > > > > > > > > > > in order. At that point it would also be good to
> > > document
> > > > > and
> > > > > > > > > educate
> > > > > > > > > > > > users
> > > > > > > > > > > > > about other test options, since OS test is a
> current
> > > > > > favourite
> > > > > > > > for
> > > > > > > > > > > > showing
> > > > > > > > > > > > > PRs haven't broken anything. Maybe that's not the
> > best
> > > > > choice
> > > > > > > > > anymore
> > > > > > > > > > > > with
> > > > > > > > > > > > > this information.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I don't want to re-invent the wheel of what's
> already
> > > in
> > > > > > > > > `testsuite`,
> > > > > > > > > > > > but I
> > > > > > > > > > > > > think Unity might end up being lighter-weight in
> the
> > > long
> > > > > > run?
> > > > > > > > And
> > > > > > > > > > it is
> > > > > > > > > > > > > also not a Xiaomi owned/maintained framework so it
> > may
> > > > > > receive
> > > > > > > > > better
> > > > > > > > > > > > > external support (i.e. more users rely on it?). I
> > tried
> > > > to
> > > > > > > > compile
> > > > > > > > > > the
> > > > > > > > > > > > > sim:citest configuration to run the testsuite
> example
> > > and
> > > > > see
> > > > > > > > what
> > > > > > > > > > it was
> > > > > > > > > > > > > like, but I was met with a long list of C++
> > compilation
> > > > > > errors
> > > > > > > > that
> > > > > > > > > > I'd
> > > > > > > > > > > > > rather not deal with. There also seems to be zero
> > docs
> > > > > about
> > > > > > it
> > > > > > > > or
> > > > > > > > > > cmocka
> > > > > > > > > > > > > in our NuttX docs at the moment.
> > > > > > > > > > > > >
> > > > > > > > > > > > > What do you think? Start a Unity test suite, add
> some
> > > > > > switches
> > > > > > > to
> > > > > > > > > > choose
> > > > > > > > > > > > > which test cases get compiled into the binary, open
> > for
> > > > > > > > extensions
> > > > > > > > > by
> > > > > > > > > > > > users
> > > > > > > > > > > > > who know more about different subsystems? This way
> we
> > > can
> > > > > > keep
> > > > > > > OS
> > > > > > > > > > test
> > > > > > > > > > > > how
> > > > > > > > > > > > > it is for a quick, low-resource sanity check of
> > "board
> > > > > > bringup"
> > > > > > > > > while
> > > > > > > > > > > > > having a dedicated testing application for patches?
> > > > Someone
> > > > > > > > > changing
> > > > > > > > > > file
> > > > > > > > > > > > > system stuff can just compile to include file
> system
> > > test
> > > > > > cases
> > > > > > > > and
> > > > > > > > > > run
> > > > > > > > > > > > > that suite only as proof that their PR works. The
> > > output
> > > > is
> > > > > > > > really
> > > > > > > > > > easy
> > > > > > > > > > > > to
> > > > > > > > > > > > > glance at to determine pass/fail results.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Anyways, here's my draft demonstrating how Unity
> > works
> > > > with
> > > > > > the
> > > > > > > > FPU
> > > > > > > > > > test
> > > > > > > > > > > > > suite from OS test:
> > > > > > > > https://github.com/apache/nuttx-apps/pull/3259
> > > > > > > > > > > > >
> > > > > > > > > > > > > Matteo
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> > > > > > tomek@cedro.info>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> > > > > > > > spudaneco@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >>> (..)
> > > > > > > > > > > > >>> Starting with bits and pieces of something like
> the
> > > LTS
> > > > > > might
> > > > > > > > > make
> > > > > > > > > > more
> > > > > > > > > > > > >>> sense other than its licensing problems. Since
> it
> > > > would
> > > > > > > never
> > > > > > > > be
> > > > > > > > > > > > >>> included in a distributed binary, the licensing
> > > really
> > > > > > should
> > > > > > > > not
> > > > > > > > > > > > matter
> > > > > > > > > > > > >>> much in practice.
> > > > > > > > > > > > >> Unless its GPL and some company would like to have
> > > their
> > > > > own
> > > > > > > > > > internal
> > > > > > > > > > > > >> bits and pieces.. luckily its MIT and the whole
> > thing
> > > is
> > > > > > > > > open-source
> > > > > > > > > > > > >> anyways :-)
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Yes we had many discussions about the LTS and this
> > > could
> > > > > be
> > > > > > > > great
> > > > > > > > > > start!
> > > > > > > > > > > > >> :-)
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> --
> > > > > > > > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
Hello everyone,
Thanks for all the feedback! I'm glad I discussed this here first. I'll try
to address everything that was brought up.
Thanks Guiding for the input! I must just be missing some library on my
machine if you're able to compile it just fine. Maybe I'll track down that
error and see if you can recognize what's wrong.
I don't intend to replace OS test anymore. I think it serves its purpose as
a minimal sanity check. Instead, I think we need something a little more
involved to test PRs since OS test can't catch many issues. It is currently
the standard everyone uses for verifying their patches and it would be
better to have more complete, alternate options. I do think that we can all
agree that OS test needs a revision of its console output so that it is
much easier to see pass/fail results, but that is just a matter of print
statements and can be done in a different effort.
I think the overwhelming concern here is mixing two unit testing frameworks
and no longer having one resource where we can go for tests. I share raiden
and Guiding's concern about that, which is why I'm glad I opened for
discussion here. On the one hand, Xiaomi has a pretty substantial
collection of tests in the `testsuite` application which they are using
internally I believe (in their CI). These run well on virtual machines,
which I assume is why the only configurations with that test suite included
is sim and rv-virt. I do not know nearly enough about cmocka to make any
claim that it *wouldn't* work on smaller hardware devices, but it also
definitely looks to be less light-weight from the feature set in the
documentation. On the other hand, the Xiaomi test suites are also not a
complete silver bullet since they seem not to be intended to run on
hardware, and there are *a lot *of tests that need hardware verification.
In PRs we often request more than just a simulator test for certain
changes. For instance, sim isn't able to have `/dev/rand`, so that needs to
be tested on a different target.
Maybe someone more familiar with Xiaomi's testing practices can weigh in on
whether cmocka has been run on hardware. Going purely based off of their
PRs, which almost exclusively cite tests on simulators, my guess is that
cmocka was not designed with embedded in mind when it was written.
As for size: unit testing can get large with Unity or cmocka (as raiden
mentioned) as more tests are added. Hence my effort to enable/disable
individual test suites from getting compiled. It might not be possible to
ever get good testing ability for AVR just because the devices are so
small. Whatever solution is used there is probably going to involve
flashing the device multiple times with different tests. The size goal
should just be to target the most NuttX devices possible.
Then there are the pros and cons of each framework (again, I don't know
much about cmocka so take it with a grain of salt):
Unity's cons:
- Designed for each test-suite to be a standalone binary. This is clunky on
NuttX, so `nest` is currently implemented as a mega-suite where test cases
just follow a naming convention to make it clear what is being tested
- setUp/tearDown can't really be leveraged because of the above
- I've noticed some funky behaviour using TEST_IGNORE in certain scenarios
but that's possibly my own configuration issue
- Would be in conflict with existing cmocka tests
Unity's pros:
- Highly configurable, like NuttX
- Light-weight and works for embedded devices
- Significantly larger user base than cmocka
Cmocka's cons:
- Possibly (?) not light-weight
- No documentation for existing test cases in NuttX
- Small user base
Cmocka's pros:
- Seems to have a larger set of features, including setup/teardown
functions per-test case
- Existing tests in NuttX
Irrespective of whether we go with Unity or cmocka, the collection of tests
that we write will be intended to run on the actual hardware devices
themselves for the purposes of testing patches and doing regression tests.
As a result, the tests will be separate from the Xiaomi 'testsuite'
application which is intended for CI testing. I think this realistically
nullifies choosing which framework to use on the basis of consistency,
since the tests are not coexisting with the cmocka tests Xiaomi has
written. Instead, the choice should be based on what best achieves the goal
from a functionality perspective. Unity seems like a really great choice
for this in my opinion, but I am bothered by the fact that it's designed
with multiple test binaries in mind.
Let me know what you think,
Matteo
On Mon, Dec 22, 2025 at 11:02 AM Alan C. Assis <ac...@gmail.com> wrote:
> Yes, I agree! We need data size from each solution for comparison, this is
> the first step.
>
> Maybe the conversion could be facilitated if we create equivalent functions
> like those used in the apps/testing/testsuites:
>
> cmocka_run_group_tests -> unity_run_group_tests
> cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown
> etc...
>
> This way we avoid end up with two unit test framework, we already have the
> Makefile vs CMake precedent! :-D
>
> On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <ra...@gmail.com> wrote:
>
> > The discussion about small systems doesn't make much sense without
> > data comparing resources of the same test cases for cmock and unity.
> >
> > But let's assume that Unity is better. Should we use two frameworks
> > to implement unit tests for OS? Or choose one framework? If we choose
> only
> > Unity, who
> > will migrate the current cmocka tests to Unity? Remember that cmoka tests
> > are used in CI, so all CI tools must be also migrated.
> > Or should we use two different frameworks to test different things and
> make
> > even
> > more mess in apps/testing as it is now?
> >
> > No one is forcing anyone to do anything. This is an open-source project,
> > everyone
> > should do what they feel is right. But whether the change should be
> merged
> > is
> > a different question. In this case, it's a very important question,
> because
> > unit testing
> > isn't just about test cases. It's about the entire infrastructure that
> > runs, captures and
> > handles test results later. Writing the test cases is the least of the
> > problems.
> > The infrastructure for cmoka testing is already in place, the
> > infrastructure for Unity
> > testing isn't.
> >
> > pon., 22 gru 2025 o 14:16 Alan C. Assis <ac...@gmail.com> napisał(a):
> >
> > > Mateusz,
> > > but some chips like AVR DA have big flash, but not so much RAM.
> > >
> > > It is better to have some possible way to test on a small system, using
> > > cmocka we are removing this option from the table.
> > >
> > > I liked the test Matteo did with Unity and I think we should let him
> > > continue. Forcing him to migrate to cmocka just because there's a test
> > > suite from Xiaomi doesn't seem prudent to me.
> > >
> > > This is the advantage of an open mind open-source project: "the
> > possibility
> > > to test new ideas", if someone had said to Michal Lenc he couldn't
> create
> > > NXBoot because where is already MCUBoot integrated on NuttX, we didn't
> > have
> > > the option to use NXBoot today. ;-)
> > >
> > > BR,
> > >
> > > Alan
> > >
> > > On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <ra...@gmail.com>
> > wrote:
> > >
> > > > Unit testing by itself isn't suitable for small systems. Ostest is
> > > > something
> > > > different (system level testing), so running it on small systems
> makes
> > > > sense.
> > > > Running unit tests on small systems is pointless because you won't
> fit
> > in
> > > > FLASH
> > > > either way. You can run the same on large targets (preferably on
> host:
> > > sim,
> > > > qemu)
> > > > and achieve the same result.
> > > >
> > > > pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com>
> > napisał(a):
> > > >
> > > > > Is it too complicated to rewrite the cmocka tests in Unity?
> > > > >
> > > > > If cmoka is not feasible for small microcontrollers, like AVRs and
> > > > others,
> > > > > our test coverage will be faulty.
> > > > >
> > > > > BR,
> > > > >
> > > > > Alan
> > > > >
> > > > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <ra...@gmail.com>
> > > wrote:
> > > > >
> > > > > > I'm a fan of Unity and have been using it in my projects for
> years,
> > > > but I
> > > > > > think
> > > > > > we should stick with one testing framework. Currently, Cmocka is
> > used
> > > > in
> > > > > > nuttx-apps, and we should stick with it. The only question is how
> > > well
> > > > > > Cmocka
> > > > > > is suitable for running on small systems? Will ostest still be
> able
> > > to
> > > > > run
> > > > > > on small MCUs?
> > > > > > With Unity this would be quite likely, so maybe an exception for
> > > Unity
> > > > > and
> > > > > > ostest would
> > > > > > be acceptable, but using it for other test cases will only cause
> > > chaos.
> > > > > >
> > > > > > pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com>
> > napisał(a):
> > > > > >
> > > > > > > Hi Matteo,
> > > > > > >
> > > > > > > Good idea !
> > > > > > >
> > > > > > > But it is best to stick with a single testing framework.
> > Currently,
> > > > > many
> > > > > > > test cases are using cmocka, so I suggest reusing cmocka for
> > > > > consistency.
> > > > > > >
> > > > > > > ostest is a minimal basic test suite, typically used for CT
> > > > (continuous
> > > > > > > testing) of each patch.
> > > > > > > If we move ostest to cmocka, it would make ostest dependent on
> a
> > > > > specific
> > > > > > > framework before it can be run.
> > > > > > > And It may increase the code size, which is not very friendly
> for
> > > > > > > configurations with small flash memory.
> > > > > > >
> > > > > > > As Greg mentioned: the answer depends on a tradeoff between
> > effort
> > > > and
> > > > > > test
> > > > > > > coverage.
> > > > > > >
> > > > > > > We need to clarify whether we define ostest as a lightweight,
> > > > > > ready-to-run
> > > > > > > test set for quick verification, or as part of the full test
> > suite.
> > > > > > >
> > > > > > >
> > > > > > > The follow is my compiling for cmocka:
> > > > > > >
> > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh
> > > sim:citest
> > > > > > -j8*
> > > > > > > Copy files
> > > > > > > Select CONFIG_HOST_LINUX=y
> > > > > > > Refreshing...
> > > > > > > CP: arch/dummy/Kconfig to
> > > > > > > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> > > > > > > CP: boards/dummy/Kconfig to
> > > > > > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> > > > > > > LN: include/arch to arch/sim/include
> > > > > > > LN: drivers/platform to
> > > > > /home/ligd/platform/mainline/nuttx/drivers/dummy
> > > > > > > LN: arch/sim/src/chip to
> > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> > > > > > > LN: arch/sim/src/board to
> > > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> > > > > > > LN: include/arch/board to
> > > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> > > > > > > LN: include/arch/chip to
> > > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> > > > > > > LN: platform/board to
> > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/boot
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/canutils
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/database
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/module
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/games
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/graphics
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/industry
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/inertial
> > > > > > > mkkconfig in
> > > > /home/ligd/platform/mainline/apps/interpreters/luamodules
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/logging
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/lte
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/math
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/netutils
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/system
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/tee
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> > > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/wireless/bluetooth
> > > > > > > mkkconfig in
> > /home/ligd/platform/mainline/apps/wireless/ieee802154
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless
> > > > > > > mkkconfig in /home/ligd/platform/mainline/apps
> > > > > > > Loaded configuration '.config'
> > > > > > > Configuration saved to '.config'
> > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ make*
> > > > > > > Create version.h
> > > > > > > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> > > > > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > > > Downloading: v1.46.0.zip
> > > > > > > ...
> > > > > > > patching file
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> > > > > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> > > > > > >
> > > > > > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> > > > > > > CP:
> /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> > > > > > > LD: nuttx
> > > > > > > Pac SIM with dynamic libs..
> > > > > > > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> > > > > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
> > > > 'sim-pac/libs/libstdc++.so.6'
> > > > > > > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> > > > > > > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> > > > > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' ->
> > > 'sim-pac/libs/libgcc_s.so.1'
> > > > > > > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
> > > > > > > SIM elf with dynamic libs archive in nuttx.tgz
> > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> > > > > > > CHelloWorld: Constructor: mSecret=42
> > > > > > > nsh: init: open failed: 2
> > > > > > >
> > > > > > > NuttShell (NSH) NuttX-12.11.0
> > > > > > > MOTD: username=admin password=Administrator
> > > > > > > nsh>
> > > > > > > nsh> help
> > > > > > > help usage: help [-v] [<cmd>]
> > > > > > >
> > > > > > > . df ifconfig mkfifo readlink
> > > > umount
> > > > > > > [ dmesg ifdown mkrd rm
> > > unset
> > > > > > > ? echo ifup mount rmdir
> > > > uptime
> > > > > > > alias env insmod mv rmmod
> > > > usleep
> > > > > > > unalias exec kill nslookup set
> > > watch
> > > > > > > basename exit pkill pidof sleep
> > xd
> > > > > > > break expr losetup pmconfig source
> > > wait
> > > > > > > cat false ln poweroff test
> > > > > > > cd fdinfo ls quit time
> > > > > > > cp free lsmod printf true
> > > > > > > cmp help mkdir ps truncate
> > > > > > > dirname hexdump mkfatfs pwd uname
> > > > > > >
> > > > > > > Builtin Apps:
> > > > > > > cmocka
> > > > > > > cmocka_driver_block
> > > > > > > cmocka_driver_gpio
> > > > > > > cmocka_driver_pm
> > > > > > > cmocka_driver_pm_runtime
> > > > > > > cmocka_driver_regulator
> > > > > > >
> > > > > > > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一 10:15写道：
> > > > > > >
> > > > > > > > I think so. Best to leave OS test alone and slowly replace it
> > > with
> > > > a
> > > > > > new
> > > > > > > > tool that has improvements. The only reason I can see to
> modify
> > > OS
> > > > > test
> > > > > > > now
> > > > > > > > would just be to have clearer output about what
> passed/failed.
> > > > > > > >
> > > > > > > > I'll come up with an improved proof of concept for a new tool
> > to
> > > > > share
> > > > > > in
> > > > > > > > the next few days.
> > > > > > > >
> > > > > > > > Matteo
> > > > > > > >
> > > > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <to...@cedro.info>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Small steps with measurable results may be the best
> approach
> > > > here,
> > > > > to
> > > > > > > > > create some new solution next to existing ostest but not
> > > > replacing
> > > > > it
> > > > > > > > > until ready? :-)
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > > >
> > > > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> > > > > matteo.golin@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > I agree, I think it would take a long time to have proper
> > > > > coverage
> > > > > > of
> > > > > > > > > NuttX
> > > > > > > > > > as a whole. Perhaps even impossible, since Unity is more
> > of a
> > > > > unit
> > > > > > > > > testing
> > > > > > > > > > framework and thus there are definitely test cases for
> > NuttX
> > > > that
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > > outside of its realm of "applicability". There is still a
> > > need
> > > > > for
> > > > > > a
> > > > > > > > > > variety of testing applications and bench-marking
> > > applications
> > > > > for
> > > > > > > > those
> > > > > > > > > > scenarios.
> > > > > > > > > >
> > > > > > > > > > That makes sense, prerequisites would need to be
> > considered.
> > > I
> > > > > > think
> > > > > > > > > > targeting the low-hanging fruit of unit-testable code
> would
> > > be
> > > > > easy
> > > > > > > in
> > > > > > > > > that
> > > > > > > > > > case, since prerequisites are mostly encoded in the
> Kconfig
> > > > > > > definitions
> > > > > > > > > > existing or not for features to be tested. Any complex
> > > > behaviour
> > > > > > > > > > (interrupts, multi-threading, etc.) might be out of the
> > realm
> > > > of
> > > > > > > what's
> > > > > > > > > > testable by Unity in the first place. We'll always have
> OS
> > > test
> > > > > in
> > > > > > > that
> > > > > > > > > > case, but maybe then it's better to do what Alan
> suggested
> > > and
> > > > at
> > > > > > > least
> > > > > > > > > > make its output more clear in the meantime. I suppose
> RTOS
> > > > > testing
> > > > > > > is a
> > > > > > > > > > whole field in itself.
> > > > > > > > > >
> > > > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> > > > > spudaneco@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I think that the answer depends on a tradeoff between
> > > effort
> > > > > and
> > > > > > > test
> > > > > > > > > > > coverage. I could imagine that a proper job could be
> > > > man-years
> > > > > > of
> > > > > > > > > effort.
> > > > > > > > > > >
> > > > > > > > > > > My only concern with a re-design is to be certain that
> > > tests
> > > > > are
> > > > > > > > > > > arranged so that the prerequisites of later test are
> > tested
> > > > > > first.
> > > > > > > > If
> > > > > > > > > > > tests are grouped by, for example, functional area,
> then
> > > > > failures
> > > > > > > can
> > > > > > > > > be
> > > > > > > > > > > difficult to understand if there is a failure in an
> > > > unverified
> > > > > > test
> > > > > > > > > > > prerequisite.
> > > > > > > > > > >
> > > > > > > > > > > Some of the "tests" are bogus and only exist to provide
> > > some
> > > > > > > minimum
> > > > > > > > > > > confidence that we have the functionality to run the
> > test.
> > > > For
> > > > > > > > > > > example: Can I start tasks? Can I pass parameters?
> Can
> > I
> > > > > > survive
> > > > > > > a
> > > > > > > > > > > context switch. Some of the tests were intended only
> to
> > > > > support
> > > > > > > > debug.
> > > > > > > > > > > Some were created only to track down specific bugs.
> > > > > > > > > > >
> > > > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > > > > > > > > LTS being the Long-Term-Stable release of NuttX?
> > > > > > > > > > > >
> > > > > > > > > > > > I'm starting to think that it would be best to create
> > new
> > > > > suite
> > > > > > > of
> > > > > > > > > tests
> > > > > > > > > > > > using Unity, using what's in OS Test as a basis and
> > then
> > > > > > > expanding
> > > > > > > > it
> > > > > > > > > > > > further. Greg has pointed out that OS test is not
> > > intended
> > > > to
> > > > > > be
> > > > > > > a
> > > > > > > > > > > complete
> > > > > > > > > > > > test, so maybe a more solid testing framework base
> in a
> > > new
> > > > > > form
> > > > > > > > > would be
> > > > > > > > > > > > in order. At that point it would also be good to
> > document
> > > > and
> > > > > > > > educate
> > > > > > > > > > > users
> > > > > > > > > > > > about other test options, since OS test is a current
> > > > > favourite
> > > > > > > for
> > > > > > > > > > > showing
> > > > > > > > > > > > PRs haven't broken anything. Maybe that's not the
> best
> > > > choice
> > > > > > > > anymore
> > > > > > > > > > > with
> > > > > > > > > > > > this information.
> > > > > > > > > > > >
> > > > > > > > > > > > I don't want to re-invent the wheel of what's already
> > in
> > > > > > > > `testsuite`,
> > > > > > > > > > > but I
> > > > > > > > > > > > think Unity might end up being lighter-weight in the
> > long
> > > > > run?
> > > > > > > And
> > > > > > > > > it is
> > > > > > > > > > > > also not a Xiaomi owned/maintained framework so it
> may
> > > > > receive
> > > > > > > > better
> > > > > > > > > > > > external support (i.e. more users rely on it?). I
> tried
> > > to
> > > > > > > compile
> > > > > > > > > the
> > > > > > > > > > > > sim:citest configuration to run the testsuite example
> > and
> > > > see
> > > > > > > what
> > > > > > > > > it was
> > > > > > > > > > > > like, but I was met with a long list of C++
> compilation
> > > > > errors
> > > > > > > that
> > > > > > > > > I'd
> > > > > > > > > > > > rather not deal with. There also seems to be zero
> docs
> > > > about
> > > > > it
> > > > > > > or
> > > > > > > > > cmocka
> > > > > > > > > > > > in our NuttX docs at the moment.
> > > > > > > > > > > >
> > > > > > > > > > > > What do you think? Start a Unity test suite, add some
> > > > > switches
> > > > > > to
> > > > > > > > > choose
> > > > > > > > > > > > which test cases get compiled into the binary, open
> for
> > > > > > > extensions
> > > > > > > > by
> > > > > > > > > > > users
> > > > > > > > > > > > who know more about different subsystems? This way we
> > can
> > > > > keep
> > > > > > OS
> > > > > > > > > test
> > > > > > > > > > > how
> > > > > > > > > > > > it is for a quick, low-resource sanity check of
> "board
> > > > > bringup"
> > > > > > > > while
> > > > > > > > > > > > having a dedicated testing application for patches?
> > > Someone
> > > > > > > > changing
> > > > > > > > > file
> > > > > > > > > > > > system stuff can just compile to include file system
> > test
> > > > > cases
> > > > > > > and
> > > > > > > > > run
> > > > > > > > > > > > that suite only as proof that their PR works. The
> > output
> > > is
> > > > > > > really
> > > > > > > > > easy
> > > > > > > > > > > to
> > > > > > > > > > > > glance at to determine pass/fail results.
> > > > > > > > > > > >
> > > > > > > > > > > > Anyways, here's my draft demonstrating how Unity
> works
> > > with
> > > > > the
> > > > > > > FPU
> > > > > > > > > test
> > > > > > > > > > > > suite from OS test:
> > > > > > > https://github.com/apache/nuttx-apps/pull/3259
> > > > > > > > > > > >
> > > > > > > > > > > > Matteo
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> > > > > tomek@cedro.info>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> > > > > > > spudaneco@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >>> (..)
> > > > > > > > > > > >>> Starting with bits and pieces of something like the
> > LTS
> > > > > might
> > > > > > > > make
> > > > > > > > > more
> > > > > > > > > > > >>> sense other than its licensing problems. Since it
> > > would
> > > > > > never
> > > > > > > be
> > > > > > > > > > > >>> included in a distributed binary, the licensing
> > really
> > > > > should
> > > > > > > not
> > > > > > > > > > > matter
> > > > > > > > > > > >>> much in practice.
> > > > > > > > > > > >> Unless its GPL and some company would like to have
> > their
> > > > own
> > > > > > > > > internal
> > > > > > > > > > > >> bits and pieces.. luckily its MIT and the whole
> thing
> > is
> > > > > > > > open-source
> > > > > > > > > > > >> anyways :-)
> > > > > > > > > > > >>
> > > > > > > > > > > >> Yes we had many discussions about the LTS and this
> > could
> > > > be
> > > > > > > great
> > > > > > > > > start!
> > > > > > > > > > > >> :-)
> > > > > > > > > > > >>
> > > > > > > > > > > >> --
> > > > > > > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: OS Test re-factor proposal
Posted by "Alan C. Assis" <ac...@gmail.com>.
Yes, I agree! We need data size from each solution for comparison, this is
the first step.
Maybe the conversion could be facilitated if we create equivalent functions
like those used in the apps/testing/testsuites:
cmocka_run_group_tests -> unity_run_group_tests
cmocka_unit_test_setup_teardown -> unity_unit_test_setup_teardown
etc...
This way we avoid end up with two unit test framework, we already have the
Makefile vs CMake precedent! :-D
On Mon, Dec 22, 2025 at 11:27 AM raiden00pl <ra...@gmail.com> wrote:
> The discussion about small systems doesn't make much sense without
> data comparing resources of the same test cases for cmock and unity.
>
> But let's assume that Unity is better. Should we use two frameworks
> to implement unit tests for OS? Or choose one framework? If we choose only
> Unity, who
> will migrate the current cmocka tests to Unity? Remember that cmoka tests
> are used in CI, so all CI tools must be also migrated.
> Or should we use two different frameworks to test different things and make
> even
> more mess in apps/testing as it is now?
>
> No one is forcing anyone to do anything. This is an open-source project,
> everyone
> should do what they feel is right. But whether the change should be merged
> is
> a different question. In this case, it's a very important question, because
> unit testing
> isn't just about test cases. It's about the entire infrastructure that
> runs, captures and
> handles test results later. Writing the test cases is the least of the
> problems.
> The infrastructure for cmoka testing is already in place, the
> infrastructure for Unity
> testing isn't.
>
> pon., 22 gru 2025 o 14:16 Alan C. Assis <ac...@gmail.com> napisał(a):
>
> > Mateusz,
> > but some chips like AVR DA have big flash, but not so much RAM.
> >
> > It is better to have some possible way to test on a small system, using
> > cmocka we are removing this option from the table.
> >
> > I liked the test Matteo did with Unity and I think we should let him
> > continue. Forcing him to migrate to cmocka just because there's a test
> > suite from Xiaomi doesn't seem prudent to me.
> >
> > This is the advantage of an open mind open-source project: "the
> possibility
> > to test new ideas", if someone had said to Michal Lenc he couldn't create
> > NXBoot because where is already MCUBoot integrated on NuttX, we didn't
> have
> > the option to use NXBoot today. ;-)
> >
> > BR,
> >
> > Alan
> >
> > On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <ra...@gmail.com>
> wrote:
> >
> > > Unit testing by itself isn't suitable for small systems. Ostest is
> > > something
> > > different (system level testing), so running it on small systems makes
> > > sense.
> > > Running unit tests on small systems is pointless because you won't fit
> in
> > > FLASH
> > > either way. You can run the same on large targets (preferably on host:
> > sim,
> > > qemu)
> > > and achieve the same result.
> > >
> > > pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com>
> napisał(a):
> > >
> > > > Is it too complicated to rewrite the cmocka tests in Unity?
> > > >
> > > > If cmoka is not feasible for small microcontrollers, like AVRs and
> > > others,
> > > > our test coverage will be faulty.
> > > >
> > > > BR,
> > > >
> > > > Alan
> > > >
> > > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <ra...@gmail.com>
> > wrote:
> > > >
> > > > > I'm a fan of Unity and have been using it in my projects for years,
> > > but I
> > > > > think
> > > > > we should stick with one testing framework. Currently, Cmocka is
> used
> > > in
> > > > > nuttx-apps, and we should stick with it. The only question is how
> > well
> > > > > Cmocka
> > > > > is suitable for running on small systems? Will ostest still be able
> > to
> > > > run
> > > > > on small MCUs?
> > > > > With Unity this would be quite likely, so maybe an exception for
> > Unity
> > > > and
> > > > > ostest would
> > > > > be acceptable, but using it for other test cases will only cause
> > chaos.
> > > > >
> > > > > pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com>
> napisał(a):
> > > > >
> > > > > > Hi Matteo,
> > > > > >
> > > > > > Good idea !
> > > > > >
> > > > > > But it is best to stick with a single testing framework.
> Currently,
> > > > many
> > > > > > test cases are using cmocka, so I suggest reusing cmocka for
> > > > consistency.
> > > > > >
> > > > > > ostest is a minimal basic test suite, typically used for CT
> > > (continuous
> > > > > > testing) of each patch.
> > > > > > If we move ostest to cmocka, it would make ostest dependent on a
> > > > specific
> > > > > > framework before it can be run.
> > > > > > And It may increase the code size, which is not very friendly for
> > > > > > configurations with small flash memory.
> > > > > >
> > > > > > As Greg mentioned: the answer depends on a tradeoff between
> effort
> > > and
> > > > > test
> > > > > > coverage.
> > > > > >
> > > > > > We need to clarify whether we define ostest as a lightweight,
> > > > > ready-to-run
> > > > > > test set for quick verification, or as part of the full test
> suite.
> > > > > >
> > > > > >
> > > > > > The follow is my compiling for cmocka:
> > > > > >
> > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh
> > sim:citest
> > > > > -j8*
> > > > > > Copy files
> > > > > > Select CONFIG_HOST_LINUX=y
> > > > > > Refreshing...
> > > > > > CP: arch/dummy/Kconfig to
> > > > > > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> > > > > > CP: boards/dummy/Kconfig to
> > > > > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> > > > > > LN: include/arch to arch/sim/include
> > > > > > LN: drivers/platform to
> > > > /home/ligd/platform/mainline/nuttx/drivers/dummy
> > > > > > LN: arch/sim/src/chip to
> > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> > > > > > LN: arch/sim/src/board to
> > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> > > > > > LN: include/arch/board to
> > > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> > > > > > LN: include/arch/chip to
> > > > > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> > > > > > LN: platform/board to
> > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/boot
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/canutils
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/database
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/module
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/games
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/graphics
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/industry
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/inertial
> > > > > > mkkconfig in
> > > /home/ligd/platform/mainline/apps/interpreters/luamodules
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/logging
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/lte
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/math
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/netutils
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/system
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/tee
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth
> > > > > > mkkconfig in
> /home/ligd/platform/mainline/apps/wireless/ieee802154
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless
> > > > > > mkkconfig in /home/ligd/platform/mainline/apps
> > > > > > Loaded configuration '.config'
> > > > > > Configuration saved to '.config'
> > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > *ligd@BAI:~/platform/mainline/nuttx$ make*
> > > > > > Create version.h
> > > > > > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> > > > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > > Downloading: v1.46.0.zip
> > > > > > ...
> > > > > > patching file
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> > > > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> > > > > >
> > > > > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> > > > > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> > > > > > LD: nuttx
> > > > > > Pac SIM with dynamic libs..
> > > > > > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> > > > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
> > > 'sim-pac/libs/libstdc++.so.6'
> > > > > > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> > > > > > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> > > > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' ->
> > 'sim-pac/libs/libgcc_s.so.1'
> > > > > > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
> > > > > > SIM elf with dynamic libs archive in nuttx.tgz
> > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> > > > > > CHelloWorld: Constructor: mSecret=42
> > > > > > nsh: init: open failed: 2
> > > > > >
> > > > > > NuttShell (NSH) NuttX-12.11.0
> > > > > > MOTD: username=admin password=Administrator
> > > > > > nsh>
> > > > > > nsh> help
> > > > > > help usage: help [-v] [<cmd>]
> > > > > >
> > > > > > . df ifconfig mkfifo readlink
> > > umount
> > > > > > [ dmesg ifdown mkrd rm
> > unset
> > > > > > ? echo ifup mount rmdir
> > > uptime
> > > > > > alias env insmod mv rmmod
> > > usleep
> > > > > > unalias exec kill nslookup set
> > watch
> > > > > > basename exit pkill pidof sleep
> xd
> > > > > > break expr losetup pmconfig source
> > wait
> > > > > > cat false ln poweroff test
> > > > > > cd fdinfo ls quit time
> > > > > > cp free lsmod printf true
> > > > > > cmp help mkdir ps truncate
> > > > > > dirname hexdump mkfatfs pwd uname
> > > > > >
> > > > > > Builtin Apps:
> > > > > > cmocka
> > > > > > cmocka_driver_block
> > > > > > cmocka_driver_gpio
> > > > > > cmocka_driver_pm
> > > > > > cmocka_driver_pm_runtime
> > > > > > cmocka_driver_regulator
> > > > > >
> > > > > > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一 10:15写道：
> > > > > >
> > > > > > > I think so. Best to leave OS test alone and slowly replace it
> > with
> > > a
> > > > > new
> > > > > > > tool that has improvements. The only reason I can see to modify
> > OS
> > > > test
> > > > > > now
> > > > > > > would just be to have clearer output about what passed/failed.
> > > > > > >
> > > > > > > I'll come up with an improved proof of concept for a new tool
> to
> > > > share
> > > > > in
> > > > > > > the next few days.
> > > > > > >
> > > > > > > Matteo
> > > > > > >
> > > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <to...@cedro.info>
> > > wrote:
> > > > > > >
> > > > > > > > Small steps with measurable results may be the best approach
> > > here,
> > > > to
> > > > > > > > create some new solution next to existing ostest but not
> > > replacing
> > > > it
> > > > > > > > until ready? :-)
> > > > > > > >
> > > > > > > > --
> > > > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > >
> > > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> > > > matteo.golin@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > I agree, I think it would take a long time to have proper
> > > > coverage
> > > > > of
> > > > > > > > NuttX
> > > > > > > > > as a whole. Perhaps even impossible, since Unity is more
> of a
> > > > unit
> > > > > > > > testing
> > > > > > > > > framework and thus there are definitely test cases for
> NuttX
> > > that
> > > > > > would
> > > > > > > > be
> > > > > > > > > outside of its realm of "applicability". There is still a
> > need
> > > > for
> > > > > a
> > > > > > > > > variety of testing applications and bench-marking
> > applications
> > > > for
> > > > > > > those
> > > > > > > > > scenarios.
> > > > > > > > >
> > > > > > > > > That makes sense, prerequisites would need to be
> considered.
> > I
> > > > > think
> > > > > > > > > targeting the low-hanging fruit of unit-testable code would
> > be
> > > > easy
> > > > > > in
> > > > > > > > that
> > > > > > > > > case, since prerequisites are mostly encoded in the Kconfig
> > > > > > definitions
> > > > > > > > > existing or not for features to be tested. Any complex
> > > behaviour
> > > > > > > > > (interrupts, multi-threading, etc.) might be out of the
> realm
> > > of
> > > > > > what's
> > > > > > > > > testable by Unity in the first place. We'll always have OS
> > test
> > > > in
> > > > > > that
> > > > > > > > > case, but maybe then it's better to do what Alan suggested
> > and
> > > at
> > > > > > least
> > > > > > > > > make its output more clear in the meantime. I suppose RTOS
> > > > testing
> > > > > > is a
> > > > > > > > > whole field in itself.
> > > > > > > > >
> > > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> > > > spudaneco@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I think that the answer depends on a tradeoff between
> > effort
> > > > and
> > > > > > test
> > > > > > > > > > coverage. I could imagine that a proper job could be
> > > man-years
> > > > > of
> > > > > > > > effort.
> > > > > > > > > >
> > > > > > > > > > My only concern with a re-design is to be certain that
> > tests
> > > > are
> > > > > > > > > > arranged so that the prerequisites of later test are
> tested
> > > > > first.
> > > > > > > If
> > > > > > > > > > tests are grouped by, for example, functional area, then
> > > > failures
> > > > > > can
> > > > > > > > be
> > > > > > > > > > difficult to understand if there is a failure in an
> > > unverified
> > > > > test
> > > > > > > > > > prerequisite.
> > > > > > > > > >
> > > > > > > > > > Some of the "tests" are bogus and only exist to provide
> > some
> > > > > > minimum
> > > > > > > > > > confidence that we have the functionality to run the
> test.
> > > For
> > > > > > > > > > example: Can I start tasks? Can I pass parameters? Can
> I
> > > > > survive
> > > > > > a
> > > > > > > > > > context switch. Some of the tests were intended only to
> > > > support
> > > > > > > debug.
> > > > > > > > > > Some were created only to track down specific bugs.
> > > > > > > > > >
> > > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > > > > > > > LTS being the Long-Term-Stable release of NuttX?
> > > > > > > > > > >
> > > > > > > > > > > I'm starting to think that it would be best to create
> new
> > > > suite
> > > > > > of
> > > > > > > > tests
> > > > > > > > > > > using Unity, using what's in OS Test as a basis and
> then
> > > > > > expanding
> > > > > > > it
> > > > > > > > > > > further. Greg has pointed out that OS test is not
> > intended
> > > to
> > > > > be
> > > > > > a
> > > > > > > > > > complete
> > > > > > > > > > > test, so maybe a more solid testing framework base in a
> > new
> > > > > form
> > > > > > > > would be
> > > > > > > > > > > in order. At that point it would also be good to
> document
> > > and
> > > > > > > educate
> > > > > > > > > > users
> > > > > > > > > > > about other test options, since OS test is a current
> > > > favourite
> > > > > > for
> > > > > > > > > > showing
> > > > > > > > > > > PRs haven't broken anything. Maybe that's not the best
> > > choice
> > > > > > > anymore
> > > > > > > > > > with
> > > > > > > > > > > this information.
> > > > > > > > > > >
> > > > > > > > > > > I don't want to re-invent the wheel of what's already
> in
> > > > > > > `testsuite`,
> > > > > > > > > > but I
> > > > > > > > > > > think Unity might end up being lighter-weight in the
> long
> > > > run?
> > > > > > And
> > > > > > > > it is
> > > > > > > > > > > also not a Xiaomi owned/maintained framework so it may
> > > > receive
> > > > > > > better
> > > > > > > > > > > external support (i.e. more users rely on it?). I tried
> > to
> > > > > > compile
> > > > > > > > the
> > > > > > > > > > > sim:citest configuration to run the testsuite example
> and
> > > see
> > > > > > what
> > > > > > > > it was
> > > > > > > > > > > like, but I was met with a long list of C++ compilation
> > > > errors
> > > > > > that
> > > > > > > > I'd
> > > > > > > > > > > rather not deal with. There also seems to be zero docs
> > > about
> > > > it
> > > > > > or
> > > > > > > > cmocka
> > > > > > > > > > > in our NuttX docs at the moment.
> > > > > > > > > > >
> > > > > > > > > > > What do you think? Start a Unity test suite, add some
> > > > switches
> > > > > to
> > > > > > > > choose
> > > > > > > > > > > which test cases get compiled into the binary, open for
> > > > > > extensions
> > > > > > > by
> > > > > > > > > > users
> > > > > > > > > > > who know more about different subsystems? This way we
> can
> > > > keep
> > > > > OS
> > > > > > > > test
> > > > > > > > > > how
> > > > > > > > > > > it is for a quick, low-resource sanity check of "board
> > > > bringup"
> > > > > > > while
> > > > > > > > > > > having a dedicated testing application for patches?
> > Someone
> > > > > > > changing
> > > > > > > > file
> > > > > > > > > > > system stuff can just compile to include file system
> test
> > > > cases
> > > > > > and
> > > > > > > > run
> > > > > > > > > > > that suite only as proof that their PR works. The
> output
> > is
> > > > > > really
> > > > > > > > easy
> > > > > > > > > > to
> > > > > > > > > > > glance at to determine pass/fail results.
> > > > > > > > > > >
> > > > > > > > > > > Anyways, here's my draft demonstrating how Unity works
> > with
> > > > the
> > > > > > FPU
> > > > > > > > test
> > > > > > > > > > > suite from OS test:
> > > > > > https://github.com/apache/nuttx-apps/pull/3259
> > > > > > > > > > >
> > > > > > > > > > > Matteo
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> > > > tomek@cedro.info>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> > > > > > spudaneco@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >>> (..)
> > > > > > > > > > >>> Starting with bits and pieces of something like the
> LTS
> > > > might
> > > > > > > make
> > > > > > > > more
> > > > > > > > > > >>> sense other than its licensing problems. Since it
> > would
> > > > > never
> > > > > > be
> > > > > > > > > > >>> included in a distributed binary, the licensing
> really
> > > > should
> > > > > > not
> > > > > > > > > > matter
> > > > > > > > > > >>> much in practice.
> > > > > > > > > > >> Unless its GPL and some company would like to have
> their
> > > own
> > > > > > > > internal
> > > > > > > > > > >> bits and pieces.. luckily its MIT and the whole thing
> is
> > > > > > > open-source
> > > > > > > > > > >> anyways :-)
> > > > > > > > > > >>
> > > > > > > > > > >> Yes we had many discussions about the LTS and this
> could
> > > be
> > > > > > great
> > > > > > > > start!
> > > > > > > > > > >> :-)
> > > > > > > > > > >>
> > > > > > > > > > >> --
> > > > > > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: OS Test re-factor proposal
Posted by raiden00pl <ra...@gmail.com>.
The discussion about small systems doesn't make much sense without
data comparing resources of the same test cases for cmock and unity.
But let's assume that Unity is better. Should we use two frameworks
to implement unit tests for OS? Or choose one framework? If we choose only
Unity, who
will migrate the current cmocka tests to Unity? Remember that cmoka tests
are used in CI, so all CI tools must be also migrated.
Or should we use two different frameworks to test different things and make
even
more mess in apps/testing as it is now?
No one is forcing anyone to do anything. This is an open-source project,
everyone
should do what they feel is right. But whether the change should be merged
is
a different question. In this case, it's a very important question, because
unit testing
isn't just about test cases. It's about the entire infrastructure that
runs, captures and
handles test results later. Writing the test cases is the least of the
problems.
The infrastructure for cmoka testing is already in place, the
infrastructure for Unity
testing isn't.
pon., 22 gru 2025 o 14:16 Alan C. Assis <ac...@gmail.com> napisał(a):
> Mateusz,
> but some chips like AVR DA have big flash, but not so much RAM.
>
> It is better to have some possible way to test on a small system, using
> cmocka we are removing this option from the table.
>
> I liked the test Matteo did with Unity and I think we should let him
> continue. Forcing him to migrate to cmocka just because there's a test
> suite from Xiaomi doesn't seem prudent to me.
>
> This is the advantage of an open mind open-source project: "the possibility
> to test new ideas", if someone had said to Michal Lenc he couldn't create
> NXBoot because where is already MCUBoot integrated on NuttX, we didn't have
> the option to use NXBoot today. ;-)
>
> BR,
>
> Alan
>
> On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <ra...@gmail.com> wrote:
>
> > Unit testing by itself isn't suitable for small systems. Ostest is
> > something
> > different (system level testing), so running it on small systems makes
> > sense.
> > Running unit tests on small systems is pointless because you won't fit in
> > FLASH
> > either way. You can run the same on large targets (preferably on host:
> sim,
> > qemu)
> > and achieve the same result.
> >
> > pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com> napisał(a):
> >
> > > Is it too complicated to rewrite the cmocka tests in Unity?
> > >
> > > If cmoka is not feasible for small microcontrollers, like AVRs and
> > others,
> > > our test coverage will be faulty.
> > >
> > > BR,
> > >
> > > Alan
> > >
> > > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <ra...@gmail.com>
> wrote:
> > >
> > > > I'm a fan of Unity and have been using it in my projects for years,
> > but I
> > > > think
> > > > we should stick with one testing framework. Currently, Cmocka is used
> > in
> > > > nuttx-apps, and we should stick with it. The only question is how
> well
> > > > Cmocka
> > > > is suitable for running on small systems? Will ostest still be able
> to
> > > run
> > > > on small MCUs?
> > > > With Unity this would be quite likely, so maybe an exception for
> Unity
> > > and
> > > > ostest would
> > > > be acceptable, but using it for other test cases will only cause
> chaos.
> > > >
> > > > pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com> napisał(a):
> > > >
> > > > > Hi Matteo,
> > > > >
> > > > > Good idea !
> > > > >
> > > > > But it is best to stick with a single testing framework. Currently,
> > > many
> > > > > test cases are using cmocka, so I suggest reusing cmocka for
> > > consistency.
> > > > >
> > > > > ostest is a minimal basic test suite, typically used for CT
> > (continuous
> > > > > testing) of each patch.
> > > > > If we move ostest to cmocka, it would make ostest dependent on a
> > > specific
> > > > > framework before it can be run.
> > > > > And It may increase the code size, which is not very friendly for
> > > > > configurations with small flash memory.
> > > > >
> > > > > As Greg mentioned: the answer depends on a tradeoff between effort
> > and
> > > > test
> > > > > coverage.
> > > > >
> > > > > We need to clarify whether we define ostest as a lightweight,
> > > > ready-to-run
> > > > > test set for quick verification, or as part of the full test suite.
> > > > >
> > > > >
> > > > > The follow is my compiling for cmocka:
> > > > >
> > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh
> sim:citest
> > > > -j8*
> > > > > Copy files
> > > > > Select CONFIG_HOST_LINUX=y
> > > > > Refreshing...
> > > > > CP: arch/dummy/Kconfig to
> > > > > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> > > > > CP: boards/dummy/Kconfig to
> > > > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> > > > > LN: include/arch to arch/sim/include
> > > > > LN: drivers/platform to
> > > /home/ligd/platform/mainline/nuttx/drivers/dummy
> > > > > LN: arch/sim/src/chip to
> > > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> > > > > LN: arch/sim/src/board to
> > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> > > > > LN: include/arch/board to
> > > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> > > > > LN: include/arch/chip to
> > > > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> > > > > LN: platform/board to
> > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/boot
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/canutils
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/database
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/module
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/examples
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/games
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/graphics
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/industry
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/inertial
> > > > > mkkconfig in
> > /home/ligd/platform/mainline/apps/interpreters/luamodules
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/logging
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/lte
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/math
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/netutils
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/system
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/tee
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/testing
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154
> > > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless
> > > > > mkkconfig in /home/ligd/platform/mainline/apps
> > > > > Loaded configuration '.config'
> > > > > Configuration saved to '.config'
> > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > *ligd@BAI:~/platform/mainline/nuttx$ make*
> > > > > Create version.h
> > > > > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> > > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > > Downloading: v1.46.0.zip
> > > > > ...
> > > > > patching file
> > > > >
> > > > >
> > > >
> > >
> >
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> > > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> > > > >
> > > > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> > > > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> > > > > LD: nuttx
> > > > > Pac SIM with dynamic libs..
> > > > > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> > > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
> > 'sim-pac/libs/libstdc++.so.6'
> > > > > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> > > > > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> > > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' ->
> 'sim-pac/libs/libgcc_s.so.1'
> > > > > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
> > > > > SIM elf with dynamic libs archive in nuttx.tgz
> > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> > > > > CHelloWorld: Constructor: mSecret=42
> > > > > nsh: init: open failed: 2
> > > > >
> > > > > NuttShell (NSH) NuttX-12.11.0
> > > > > MOTD: username=admin password=Administrator
> > > > > nsh>
> > > > > nsh> help
> > > > > help usage: help [-v] [<cmd>]
> > > > >
> > > > > . df ifconfig mkfifo readlink
> > umount
> > > > > [ dmesg ifdown mkrd rm
> unset
> > > > > ? echo ifup mount rmdir
> > uptime
> > > > > alias env insmod mv rmmod
> > usleep
> > > > > unalias exec kill nslookup set
> watch
> > > > > basename exit pkill pidof sleep xd
> > > > > break expr losetup pmconfig source
> wait
> > > > > cat false ln poweroff test
> > > > > cd fdinfo ls quit time
> > > > > cp free lsmod printf true
> > > > > cmp help mkdir ps truncate
> > > > > dirname hexdump mkfatfs pwd uname
> > > > >
> > > > > Builtin Apps:
> > > > > cmocka
> > > > > cmocka_driver_block
> > > > > cmocka_driver_gpio
> > > > > cmocka_driver_pm
> > > > > cmocka_driver_pm_runtime
> > > > > cmocka_driver_regulator
> > > > >
> > > > > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一 10:15写道：
> > > > >
> > > > > > I think so. Best to leave OS test alone and slowly replace it
> with
> > a
> > > > new
> > > > > > tool that has improvements. The only reason I can see to modify
> OS
> > > test
> > > > > now
> > > > > > would just be to have clearer output about what passed/failed.
> > > > > >
> > > > > > I'll come up with an improved proof of concept for a new tool to
> > > share
> > > > in
> > > > > > the next few days.
> > > > > >
> > > > > > Matteo
> > > > > >
> > > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <to...@cedro.info>
> > wrote:
> > > > > >
> > > > > > > Small steps with measurable results may be the best approach
> > here,
> > > to
> > > > > > > create some new solution next to existing ostest but not
> > replacing
> > > it
> > > > > > > until ready? :-)
> > > > > > >
> > > > > > > --
> > > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > >
> > > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> > > matteo.golin@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > I agree, I think it would take a long time to have proper
> > > coverage
> > > > of
> > > > > > > NuttX
> > > > > > > > as a whole. Perhaps even impossible, since Unity is more of a
> > > unit
> > > > > > > testing
> > > > > > > > framework and thus there are definitely test cases for NuttX
> > that
> > > > > would
> > > > > > > be
> > > > > > > > outside of its realm of "applicability". There is still a
> need
> > > for
> > > > a
> > > > > > > > variety of testing applications and bench-marking
> applications
> > > for
> > > > > > those
> > > > > > > > scenarios.
> > > > > > > >
> > > > > > > > That makes sense, prerequisites would need to be considered.
> I
> > > > think
> > > > > > > > targeting the low-hanging fruit of unit-testable code would
> be
> > > easy
> > > > > in
> > > > > > > that
> > > > > > > > case, since prerequisites are mostly encoded in the Kconfig
> > > > > definitions
> > > > > > > > existing or not for features to be tested. Any complex
> > behaviour
> > > > > > > > (interrupts, multi-threading, etc.) might be out of the realm
> > of
> > > > > what's
> > > > > > > > testable by Unity in the first place. We'll always have OS
> test
> > > in
> > > > > that
> > > > > > > > case, but maybe then it's better to do what Alan suggested
> and
> > at
> > > > > least
> > > > > > > > make its output more clear in the meantime. I suppose RTOS
> > > testing
> > > > > is a
> > > > > > > > whole field in itself.
> > > > > > > >
> > > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> > > spudaneco@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I think that the answer depends on a tradeoff between
> effort
> > > and
> > > > > test
> > > > > > > > > coverage. I could imagine that a proper job could be
> > man-years
> > > > of
> > > > > > > effort.
> > > > > > > > >
> > > > > > > > > My only concern with a re-design is to be certain that
> tests
> > > are
> > > > > > > > > arranged so that the prerequisites of later test are tested
> > > > first.
> > > > > > If
> > > > > > > > > tests are grouped by, for example, functional area, then
> > > failures
> > > > > can
> > > > > > > be
> > > > > > > > > difficult to understand if there is a failure in an
> > unverified
> > > > test
> > > > > > > > > prerequisite.
> > > > > > > > >
> > > > > > > > > Some of the "tests" are bogus and only exist to provide
> some
> > > > > minimum
> > > > > > > > > confidence that we have the functionality to run the test.
> > For
> > > > > > > > > example: Can I start tasks? Can I pass parameters? Can I
> > > > survive
> > > > > a
> > > > > > > > > context switch. Some of the tests were intended only to
> > > support
> > > > > > debug.
> > > > > > > > > Some were created only to track down specific bugs.
> > > > > > > > >
> > > > > > > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > > > > > > LTS being the Long-Term-Stable release of NuttX?
> > > > > > > > > >
> > > > > > > > > > I'm starting to think that it would be best to create new
> > > suite
> > > > > of
> > > > > > > tests
> > > > > > > > > > using Unity, using what's in OS Test as a basis and then
> > > > > expanding
> > > > > > it
> > > > > > > > > > further. Greg has pointed out that OS test is not
> intended
> > to
> > > > be
> > > > > a
> > > > > > > > > complete
> > > > > > > > > > test, so maybe a more solid testing framework base in a
> new
> > > > form
> > > > > > > would be
> > > > > > > > > > in order. At that point it would also be good to document
> > and
> > > > > > educate
> > > > > > > > > users
> > > > > > > > > > about other test options, since OS test is a current
> > > favourite
> > > > > for
> > > > > > > > > showing
> > > > > > > > > > PRs haven't broken anything. Maybe that's not the best
> > choice
> > > > > > anymore
> > > > > > > > > with
> > > > > > > > > > this information.
> > > > > > > > > >
> > > > > > > > > > I don't want to re-invent the wheel of what's already in
> > > > > > `testsuite`,
> > > > > > > > > but I
> > > > > > > > > > think Unity might end up being lighter-weight in the long
> > > run?
> > > > > And
> > > > > > > it is
> > > > > > > > > > also not a Xiaomi owned/maintained framework so it may
> > > receive
> > > > > > better
> > > > > > > > > > external support (i.e. more users rely on it?). I tried
> to
> > > > > compile
> > > > > > > the
> > > > > > > > > > sim:citest configuration to run the testsuite example and
> > see
> > > > > what
> > > > > > > it was
> > > > > > > > > > like, but I was met with a long list of C++ compilation
> > > errors
> > > > > that
> > > > > > > I'd
> > > > > > > > > > rather not deal with. There also seems to be zero docs
> > about
> > > it
> > > > > or
> > > > > > > cmocka
> > > > > > > > > > in our NuttX docs at the moment.
> > > > > > > > > >
> > > > > > > > > > What do you think? Start a Unity test suite, add some
> > > switches
> > > > to
> > > > > > > choose
> > > > > > > > > > which test cases get compiled into the binary, open for
> > > > > extensions
> > > > > > by
> > > > > > > > > users
> > > > > > > > > > who know more about different subsystems? This way we can
> > > keep
> > > > OS
> > > > > > > test
> > > > > > > > > how
> > > > > > > > > > it is for a quick, low-resource sanity check of "board
> > > bringup"
> > > > > > while
> > > > > > > > > > having a dedicated testing application for patches?
> Someone
> > > > > > changing
> > > > > > > file
> > > > > > > > > > system stuff can just compile to include file system test
> > > cases
> > > > > and
> > > > > > > run
> > > > > > > > > > that suite only as proof that their PR works. The output
> is
> > > > > really
> > > > > > > easy
> > > > > > > > > to
> > > > > > > > > > glance at to determine pass/fail results.
> > > > > > > > > >
> > > > > > > > > > Anyways, here's my draft demonstrating how Unity works
> with
> > > the
> > > > > FPU
> > > > > > > test
> > > > > > > > > > suite from OS test:
> > > > > https://github.com/apache/nuttx-apps/pull/3259
> > > > > > > > > >
> > > > > > > > > > Matteo
> > > > > > > > > >
> > > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> > > tomek@cedro.info>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> > > > > spudaneco@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >>> (..)
> > > > > > > > > >>> Starting with bits and pieces of something like the LTS
> > > might
> > > > > > make
> > > > > > > more
> > > > > > > > > >>> sense other than its licensing problems. Since it
> would
> > > > never
> > > > > be
> > > > > > > > > >>> included in a distributed binary, the licensing really
> > > should
> > > > > not
> > > > > > > > > matter
> > > > > > > > > >>> much in practice.
> > > > > > > > > >> Unless its GPL and some company would like to have their
> > own
> > > > > > > internal
> > > > > > > > > >> bits and pieces.. luckily its MIT and the whole thing is
> > > > > > open-source
> > > > > > > > > >> anyways :-)
> > > > > > > > > >>
> > > > > > > > > >> Yes we had many discussions about the LTS and this could
> > be
> > > > > great
> > > > > > > start!
> > > > > > > > > >> :-)
> > > > > > > > > >>
> > > > > > > > > >> --
> > > > > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: OS Test re-factor proposal
Posted by "Alan C. Assis" <ac...@gmail.com>.
Mateusz,
but some chips like AVR DA have big flash, but not so much RAM.
It is better to have some possible way to test on a small system, using
cmocka we are removing this option from the table.
I liked the test Matteo did with Unity and I think we should let him
continue. Forcing him to migrate to cmocka just because there's a test
suite from Xiaomi doesn't seem prudent to me.
This is the advantage of an open mind open-source project: "the possibility
to test new ideas", if someone had said to Michal Lenc he couldn't create
NXBoot because where is already MCUBoot integrated on NuttX, we didn't have
the option to use NXBoot today. ;-)
BR,
Alan
On Mon, Dec 22, 2025 at 10:02 AM raiden00pl <ra...@gmail.com> wrote:
> Unit testing by itself isn't suitable for small systems. Ostest is
> something
> different (system level testing), so running it on small systems makes
> sense.
> Running unit tests on small systems is pointless because you won't fit in
> FLASH
> either way. You can run the same on large targets (preferably on host: sim,
> qemu)
> and achieve the same result.
>
> pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com> napisał(a):
>
> > Is it too complicated to rewrite the cmocka tests in Unity?
> >
> > If cmoka is not feasible for small microcontrollers, like AVRs and
> others,
> > our test coverage will be faulty.
> >
> > BR,
> >
> > Alan
> >
> > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <ra...@gmail.com> wrote:
> >
> > > I'm a fan of Unity and have been using it in my projects for years,
> but I
> > > think
> > > we should stick with one testing framework. Currently, Cmocka is used
> in
> > > nuttx-apps, and we should stick with it. The only question is how well
> > > Cmocka
> > > is suitable for running on small systems? Will ostest still be able to
> > run
> > > on small MCUs?
> > > With Unity this would be quite likely, so maybe an exception for Unity
> > and
> > > ostest would
> > > be acceptable, but using it for other test cases will only cause chaos.
> > >
> > > pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com> napisał(a):
> > >
> > > > Hi Matteo,
> > > >
> > > > Good idea !
> > > >
> > > > But it is best to stick with a single testing framework. Currently,
> > many
> > > > test cases are using cmocka, so I suggest reusing cmocka for
> > consistency.
> > > >
> > > > ostest is a minimal basic test suite, typically used for CT
> (continuous
> > > > testing) of each patch.
> > > > If we move ostest to cmocka, it would make ostest dependent on a
> > specific
> > > > framework before it can be run.
> > > > And It may increase the code size, which is not very friendly for
> > > > configurations with small flash memory.
> > > >
> > > > As Greg mentioned: the answer depends on a tradeoff between effort
> and
> > > test
> > > > coverage.
> > > >
> > > > We need to clarify whether we define ostest as a lightweight,
> > > ready-to-run
> > > > test set for quick verification, or as part of the full test suite.
> > > >
> > > >
> > > > The follow is my compiling for cmocka:
> > > >
> > > > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh sim:citest
> > > -j8*
> > > > Copy files
> > > > Select CONFIG_HOST_LINUX=y
> > > > Refreshing...
> > > > CP: arch/dummy/Kconfig to
> > > > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> > > > CP: boards/dummy/Kconfig to
> > > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> > > > LN: include/arch to arch/sim/include
> > > > LN: drivers/platform to
> > /home/ligd/platform/mainline/nuttx/drivers/dummy
> > > > LN: arch/sim/src/chip to
> > > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> > > > LN: arch/sim/src/board to
> > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> > > > LN: include/arch/board to
> > > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> > > > LN: include/arch/chip to
> > > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> > > > LN: platform/board to
> /home/ligd/platform/mainline/apps/platform/dummy
> > > > mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> > > > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> > > > mkkconfig in /home/ligd/platform/mainline/apps/boot
> > > > mkkconfig in /home/ligd/platform/mainline/apps/canutils
> > > > mkkconfig in /home/ligd/platform/mainline/apps/crypto
> > > > mkkconfig in /home/ligd/platform/mainline/apps/database
> > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest
> > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/module
> > > > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot
> > > > mkkconfig in /home/ligd/platform/mainline/apps/examples
> > > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> > > > mkkconfig in /home/ligd/platform/mainline/apps/games
> > > > mkkconfig in /home/ligd/platform/mainline/apps/graphics
> > > > mkkconfig in /home/ligd/platform/mainline/apps/industry
> > > > mkkconfig in /home/ligd/platform/mainline/apps/inertial
> > > > mkkconfig in
> /home/ligd/platform/mainline/apps/interpreters/luamodules
> > > > mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> > > > mkkconfig in /home/ligd/platform/mainline/apps/logging
> > > > mkkconfig in /home/ligd/platform/mainline/apps/lte
> > > > mkkconfig in /home/ligd/platform/mainline/apps/math
> > > > mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> > > > mkkconfig in /home/ligd/platform/mainline/apps/netutils
> > > > mkkconfig in /home/ligd/platform/mainline/apps/sdr
> > > > mkkconfig in /home/ligd/platform/mainline/apps/system
> > > > mkkconfig in /home/ligd/platform/mainline/apps/tee
> > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers
> > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> > > > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> > > > mkkconfig in /home/ligd/platform/mainline/apps/testing
> > > > mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth
> > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154
> > > > mkkconfig in /home/ligd/platform/mainline/apps/wireless
> > > > mkkconfig in /home/ligd/platform/mainline/apps
> > > > Loaded configuration '.config'
> > > > Configuration saved to '.config'
> > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > *ligd@BAI:~/platform/mainline/nuttx$ make*
> > > > Create version.h
> > > > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> > > > /home/ligd/platform/mainline/apps/platform/dummy
> > > > Downloading: v1.46.0.zip
> > > > ...
> > > > patching file
> > > >
> > > >
> > >
> >
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> > > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> > > >
> > > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> > > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> > > > LD: nuttx
> > > > Pac SIM with dynamic libs..
> > > > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> > > > '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
> 'sim-pac/libs/libstdc++.so.6'
> > > > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> > > > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> > > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> 'sim-pac/libs/libgcc_s.so.1'
> > > > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
> > > > SIM elf with dynamic libs archive in nuttx.tgz
> > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > ligd@BAI:~/platform/mainline/nuttx$
> > > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> > > > CHelloWorld: Constructor: mSecret=42
> > > > nsh: init: open failed: 2
> > > >
> > > > NuttShell (NSH) NuttX-12.11.0
> > > > MOTD: username=admin password=Administrator
> > > > nsh>
> > > > nsh> help
> > > > help usage: help [-v] [<cmd>]
> > > >
> > > > . df ifconfig mkfifo readlink
> umount
> > > > [ dmesg ifdown mkrd rm unset
> > > > ? echo ifup mount rmdir
> uptime
> > > > alias env insmod mv rmmod
> usleep
> > > > unalias exec kill nslookup set watch
> > > > basename exit pkill pidof sleep xd
> > > > break expr losetup pmconfig source wait
> > > > cat false ln poweroff test
> > > > cd fdinfo ls quit time
> > > > cp free lsmod printf true
> > > > cmp help mkdir ps truncate
> > > > dirname hexdump mkfatfs pwd uname
> > > >
> > > > Builtin Apps:
> > > > cmocka
> > > > cmocka_driver_block
> > > > cmocka_driver_gpio
> > > > cmocka_driver_pm
> > > > cmocka_driver_pm_runtime
> > > > cmocka_driver_regulator
> > > >
> > > > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一 10:15写道：
> > > >
> > > > > I think so. Best to leave OS test alone and slowly replace it with
> a
> > > new
> > > > > tool that has improvements. The only reason I can see to modify OS
> > test
> > > > now
> > > > > would just be to have clearer output about what passed/failed.
> > > > >
> > > > > I'll come up with an improved proof of concept for a new tool to
> > share
> > > in
> > > > > the next few days.
> > > > >
> > > > > Matteo
> > > > >
> > > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <to...@cedro.info>
> wrote:
> > > > >
> > > > > > Small steps with measurable results may be the best approach
> here,
> > to
> > > > > > create some new solution next to existing ostest but not
> replacing
> > it
> > > > > > until ready? :-)
> > > > > >
> > > > > > --
> > > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > >
> > > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> > matteo.golin@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > I agree, I think it would take a long time to have proper
> > coverage
> > > of
> > > > > > NuttX
> > > > > > > as a whole. Perhaps even impossible, since Unity is more of a
> > unit
> > > > > > testing
> > > > > > > framework and thus there are definitely test cases for NuttX
> that
> > > > would
> > > > > > be
> > > > > > > outside of its realm of "applicability". There is still a need
> > for
> > > a
> > > > > > > variety of testing applications and bench-marking applications
> > for
> > > > > those
> > > > > > > scenarios.
> > > > > > >
> > > > > > > That makes sense, prerequisites would need to be considered. I
> > > think
> > > > > > > targeting the low-hanging fruit of unit-testable code would be
> > easy
> > > > in
> > > > > > that
> > > > > > > case, since prerequisites are mostly encoded in the Kconfig
> > > > definitions
> > > > > > > existing or not for features to be tested. Any complex
> behaviour
> > > > > > > (interrupts, multi-threading, etc.) might be out of the realm
> of
> > > > what's
> > > > > > > testable by Unity in the first place. We'll always have OS test
> > in
> > > > that
> > > > > > > case, but maybe then it's better to do what Alan suggested and
> at
> > > > least
> > > > > > > make its output more clear in the meantime. I suppose RTOS
> > testing
> > > > is a
> > > > > > > whole field in itself.
> > > > > > >
> > > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> > spudaneco@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > I think that the answer depends on a tradeoff between effort
> > and
> > > > test
> > > > > > > > coverage. I could imagine that a proper job could be
> man-years
> > > of
> > > > > > effort.
> > > > > > > >
> > > > > > > > My only concern with a re-design is to be certain that tests
> > are
> > > > > > > > arranged so that the prerequisites of later test are tested
> > > first.
> > > > > If
> > > > > > > > tests are grouped by, for example, functional area, then
> > failures
> > > > can
> > > > > > be
> > > > > > > > difficult to understand if there is a failure in an
> unverified
> > > test
> > > > > > > > prerequisite.
> > > > > > > >
> > > > > > > > Some of the "tests" are bogus and only exist to provide some
> > > > minimum
> > > > > > > > confidence that we have the functionality to run the test.
> For
> > > > > > > > example: Can I start tasks? Can I pass parameters? Can I
> > > survive
> > > > a
> > > > > > > > context switch. Some of the tests were intended only to
> > support
> > > > > debug.
> > > > > > > > Some were created only to track down specific bugs.
> > > > > > > >
> > > > > > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > > > > > LTS being the Long-Term-Stable release of NuttX?
> > > > > > > > >
> > > > > > > > > I'm starting to think that it would be best to create new
> > suite
> > > > of
> > > > > > tests
> > > > > > > > > using Unity, using what's in OS Test as a basis and then
> > > > expanding
> > > > > it
> > > > > > > > > further. Greg has pointed out that OS test is not intended
> to
> > > be
> > > > a
> > > > > > > > complete
> > > > > > > > > test, so maybe a more solid testing framework base in a new
> > > form
> > > > > > would be
> > > > > > > > > in order. At that point it would also be good to document
> and
> > > > > educate
> > > > > > > > users
> > > > > > > > > about other test options, since OS test is a current
> > favourite
> > > > for
> > > > > > > > showing
> > > > > > > > > PRs haven't broken anything. Maybe that's not the best
> choice
> > > > > anymore
> > > > > > > > with
> > > > > > > > > this information.
> > > > > > > > >
> > > > > > > > > I don't want to re-invent the wheel of what's already in
> > > > > `testsuite`,
> > > > > > > > but I
> > > > > > > > > think Unity might end up being lighter-weight in the long
> > run?
> > > > And
> > > > > > it is
> > > > > > > > > also not a Xiaomi owned/maintained framework so it may
> > receive
> > > > > better
> > > > > > > > > external support (i.e. more users rely on it?). I tried to
> > > > compile
> > > > > > the
> > > > > > > > > sim:citest configuration to run the testsuite example and
> see
> > > > what
> > > > > > it was
> > > > > > > > > like, but I was met with a long list of C++ compilation
> > errors
> > > > that
> > > > > > I'd
> > > > > > > > > rather not deal with. There also seems to be zero docs
> about
> > it
> > > > or
> > > > > > cmocka
> > > > > > > > > in our NuttX docs at the moment.
> > > > > > > > >
> > > > > > > > > What do you think? Start a Unity test suite, add some
> > switches
> > > to
> > > > > > choose
> > > > > > > > > which test cases get compiled into the binary, open for
> > > > extensions
> > > > > by
> > > > > > > > users
> > > > > > > > > who know more about different subsystems? This way we can
> > keep
> > > OS
> > > > > > test
> > > > > > > > how
> > > > > > > > > it is for a quick, low-resource sanity check of "board
> > bringup"
> > > > > while
> > > > > > > > > having a dedicated testing application for patches? Someone
> > > > > changing
> > > > > > file
> > > > > > > > > system stuff can just compile to include file system test
> > cases
> > > > and
> > > > > > run
> > > > > > > > > that suite only as proof that their PR works. The output is
> > > > really
> > > > > > easy
> > > > > > > > to
> > > > > > > > > glance at to determine pass/fail results.
> > > > > > > > >
> > > > > > > > > Anyways, here's my draft demonstrating how Unity works with
> > the
> > > > FPU
> > > > > > test
> > > > > > > > > suite from OS test:
> > > > https://github.com/apache/nuttx-apps/pull/3259
> > > > > > > > >
> > > > > > > > > Matteo
> > > > > > > > >
> > > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> > tomek@cedro.info>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> > > > spudaneco@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >>> (..)
> > > > > > > > >>> Starting with bits and pieces of something like the LTS
> > might
> > > > > make
> > > > > > more
> > > > > > > > >>> sense other than its licensing problems. Since it would
> > > never
> > > > be
> > > > > > > > >>> included in a distributed binary, the licensing really
> > should
> > > > not
> > > > > > > > matter
> > > > > > > > >>> much in practice.
> > > > > > > > >> Unless its GPL and some company would like to have their
> own
> > > > > > internal
> > > > > > > > >> bits and pieces.. luckily its MIT and the whole thing is
> > > > > open-source
> > > > > > > > >> anyways :-)
> > > > > > > > >>
> > > > > > > > >> Yes we had many discussions about the LTS and this could
> be
> > > > great
> > > > > > start!
> > > > > > > > >> :-)
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > > >>
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: OS Test re-factor proposal
Posted by raiden00pl <ra...@gmail.com>.
Unit testing by itself isn't suitable for small systems. Ostest is something
different (system level testing), so running it on small systems makes
sense.
Running unit tests on small systems is pointless because you won't fit in
FLASH
either way. You can run the same on large targets (preferably on host: sim,
qemu)
and achieve the same result.
pon., 22 gru 2025 o 13:35 Alan C. Assis <ac...@gmail.com> napisał(a):
> Is it too complicated to rewrite the cmocka tests in Unity?
>
> If cmoka is not feasible for small microcontrollers, like AVRs and others,
> our test coverage will be faulty.
>
> BR,
>
> Alan
>
> On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <ra...@gmail.com> wrote:
>
> > I'm a fan of Unity and have been using it in my projects for years, but I
> > think
> > we should stick with one testing framework. Currently, Cmocka is used in
> > nuttx-apps, and we should stick with it. The only question is how well
> > Cmocka
> > is suitable for running on small systems? Will ostest still be able to
> run
> > on small MCUs?
> > With Unity this would be quite likely, so maybe an exception for Unity
> and
> > ostest would
> > be acceptable, but using it for other test cases will only cause chaos.
> >
> > pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com> napisał(a):
> >
> > > Hi Matteo,
> > >
> > > Good idea !
> > >
> > > But it is best to stick with a single testing framework. Currently,
> many
> > > test cases are using cmocka, so I suggest reusing cmocka for
> consistency.
> > >
> > > ostest is a minimal basic test suite, typically used for CT (continuous
> > > testing) of each patch.
> > > If we move ostest to cmocka, it would make ostest dependent on a
> specific
> > > framework before it can be run.
> > > And It may increase the code size, which is not very friendly for
> > > configurations with small flash memory.
> > >
> > > As Greg mentioned: the answer depends on a tradeoff between effort and
> > test
> > > coverage.
> > >
> > > We need to clarify whether we define ostest as a lightweight,
> > ready-to-run
> > > test set for quick verification, or as part of the full test suite.
> > >
> > >
> > > The follow is my compiling for cmocka:
> > >
> > > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh sim:citest
> > -j8*
> > > Copy files
> > > Select CONFIG_HOST_LINUX=y
> > > Refreshing...
> > > CP: arch/dummy/Kconfig to
> > > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> > > CP: boards/dummy/Kconfig to
> > > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> > > LN: include/arch to arch/sim/include
> > > LN: drivers/platform to
> /home/ligd/platform/mainline/nuttx/drivers/dummy
> > > LN: arch/sim/src/chip to
> > > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> > > LN: arch/sim/src/board to
> > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> > > LN: include/arch/board to
> > > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> > > LN: include/arch/chip to
> > > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> > > LN: platform/board to /home/ligd/platform/mainline/apps/platform/dummy
> > > mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> > > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> > > mkkconfig in /home/ligd/platform/mainline/apps/boot
> > > mkkconfig in /home/ligd/platform/mainline/apps/canutils
> > > mkkconfig in /home/ligd/platform/mainline/apps/crypto
> > > mkkconfig in /home/ligd/platform/mainline/apps/database
> > > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> > > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest
> > > mkkconfig in /home/ligd/platform/mainline/apps/examples/module
> > > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot
> > > mkkconfig in /home/ligd/platform/mainline/apps/examples
> > > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> > > mkkconfig in /home/ligd/platform/mainline/apps/games
> > > mkkconfig in /home/ligd/platform/mainline/apps/graphics
> > > mkkconfig in /home/ligd/platform/mainline/apps/industry
> > > mkkconfig in /home/ligd/platform/mainline/apps/inertial
> > > mkkconfig in /home/ligd/platform/mainline/apps/interpreters/luamodules
> > > mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> > > mkkconfig in /home/ligd/platform/mainline/apps/logging
> > > mkkconfig in /home/ligd/platform/mainline/apps/lte
> > > mkkconfig in /home/ligd/platform/mainline/apps/math
> > > mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> > > mkkconfig in /home/ligd/platform/mainline/apps/netutils
> > > mkkconfig in /home/ligd/platform/mainline/apps/sdr
> > > mkkconfig in /home/ligd/platform/mainline/apps/system
> > > mkkconfig in /home/ligd/platform/mainline/apps/tee
> > > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers
> > > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> > > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> > > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> > > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> > > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> > > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> > > mkkconfig in /home/ligd/platform/mainline/apps/testing
> > > mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> > > mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth
> > > mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154
> > > mkkconfig in /home/ligd/platform/mainline/apps/wireless
> > > mkkconfig in /home/ligd/platform/mainline/apps
> > > Loaded configuration '.config'
> > > Configuration saved to '.config'
> > > ligd@BAI:~/platform/mainline/nuttx$
> > > *ligd@BAI:~/platform/mainline/nuttx$ make*
> > > Create version.h
> > > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> > > /home/ligd/platform/mainline/apps/platform/dummy
> > > Downloading: v1.46.0.zip
> > > ...
> > > patching file
> > >
> > >
> >
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> > > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> > >
> > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> > > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> > > LD: nuttx
> > > Pac SIM with dynamic libs..
> > > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> > > '/lib/x86_64-linux-gnu/libstdc++.so.6' -> 'sim-pac/libs/libstdc++.so.6'
> > > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> > > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> > > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> 'sim-pac/libs/libgcc_s.so.1'
> > > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
> > > SIM elf with dynamic libs archive in nuttx.tgz
> > > ligd@BAI:~/platform/mainline/nuttx$
> > > ligd@BAI:~/platform/mainline/nuttx$
> > > ligd@BAI:~/platform/mainline/nuttx$
> > > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> > > CHelloWorld: Constructor: mSecret=42
> > > nsh: init: open failed: 2
> > >
> > > NuttShell (NSH) NuttX-12.11.0
> > > MOTD: username=admin password=Administrator
> > > nsh>
> > > nsh> help
> > > help usage: help [-v] [<cmd>]
> > >
> > > . df ifconfig mkfifo readlink umount
> > > [ dmesg ifdown mkrd rm unset
> > > ? echo ifup mount rmdir uptime
> > > alias env insmod mv rmmod usleep
> > > unalias exec kill nslookup set watch
> > > basename exit pkill pidof sleep xd
> > > break expr losetup pmconfig source wait
> > > cat false ln poweroff test
> > > cd fdinfo ls quit time
> > > cp free lsmod printf true
> > > cmp help mkdir ps truncate
> > > dirname hexdump mkfatfs pwd uname
> > >
> > > Builtin Apps:
> > > cmocka
> > > cmocka_driver_block
> > > cmocka_driver_gpio
> > > cmocka_driver_pm
> > > cmocka_driver_pm_runtime
> > > cmocka_driver_regulator
> > >
> > > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一 10:15写道：
> > >
> > > > I think so. Best to leave OS test alone and slowly replace it with a
> > new
> > > > tool that has improvements. The only reason I can see to modify OS
> test
> > > now
> > > > would just be to have clearer output about what passed/failed.
> > > >
> > > > I'll come up with an improved proof of concept for a new tool to
> share
> > in
> > > > the next few days.
> > > >
> > > > Matteo
> > > >
> > > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <to...@cedro.info> wrote:
> > > >
> > > > > Small steps with measurable results may be the best approach here,
> to
> > > > > create some new solution next to existing ostest but not replacing
> it
> > > > > until ready? :-)
> > > > >
> > > > > --
> > > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > >
> > > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> matteo.golin@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > I agree, I think it would take a long time to have proper
> coverage
> > of
> > > > > NuttX
> > > > > > as a whole. Perhaps even impossible, since Unity is more of a
> unit
> > > > > testing
> > > > > > framework and thus there are definitely test cases for NuttX that
> > > would
> > > > > be
> > > > > > outside of its realm of "applicability". There is still a need
> for
> > a
> > > > > > variety of testing applications and bench-marking applications
> for
> > > > those
> > > > > > scenarios.
> > > > > >
> > > > > > That makes sense, prerequisites would need to be considered. I
> > think
> > > > > > targeting the low-hanging fruit of unit-testable code would be
> easy
> > > in
> > > > > that
> > > > > > case, since prerequisites are mostly encoded in the Kconfig
> > > definitions
> > > > > > existing or not for features to be tested. Any complex behaviour
> > > > > > (interrupts, multi-threading, etc.) might be out of the realm of
> > > what's
> > > > > > testable by Unity in the first place. We'll always have OS test
> in
> > > that
> > > > > > case, but maybe then it's better to do what Alan suggested and at
> > > least
> > > > > > make its output more clear in the meantime. I suppose RTOS
> testing
> > > is a
> > > > > > whole field in itself.
> > > > > >
> > > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> spudaneco@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > I think that the answer depends on a tradeoff between effort
> and
> > > test
> > > > > > > coverage. I could imagine that a proper job could be man-years
> > of
> > > > > effort.
> > > > > > >
> > > > > > > My only concern with a re-design is to be certain that tests
> are
> > > > > > > arranged so that the prerequisites of later test are tested
> > first.
> > > > If
> > > > > > > tests are grouped by, for example, functional area, then
> failures
> > > can
> > > > > be
> > > > > > > difficult to understand if there is a failure in an unverified
> > test
> > > > > > > prerequisite.
> > > > > > >
> > > > > > > Some of the "tests" are bogus and only exist to provide some
> > > minimum
> > > > > > > confidence that we have the functionality to run the test. For
> > > > > > > example: Can I start tasks? Can I pass parameters? Can I
> > survive
> > > a
> > > > > > > context switch. Some of the tests were intended only to
> support
> > > > debug.
> > > > > > > Some were created only to track down specific bugs.
> > > > > > >
> > > > > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > > > > LTS being the Long-Term-Stable release of NuttX?
> > > > > > > >
> > > > > > > > I'm starting to think that it would be best to create new
> suite
> > > of
> > > > > tests
> > > > > > > > using Unity, using what's in OS Test as a basis and then
> > > expanding
> > > > it
> > > > > > > > further. Greg has pointed out that OS test is not intended to
> > be
> > > a
> > > > > > > complete
> > > > > > > > test, so maybe a more solid testing framework base in a new
> > form
> > > > > would be
> > > > > > > > in order. At that point it would also be good to document and
> > > > educate
> > > > > > > users
> > > > > > > > about other test options, since OS test is a current
> favourite
> > > for
> > > > > > > showing
> > > > > > > > PRs haven't broken anything. Maybe that's not the best choice
> > > > anymore
> > > > > > > with
> > > > > > > > this information.
> > > > > > > >
> > > > > > > > I don't want to re-invent the wheel of what's already in
> > > > `testsuite`,
> > > > > > > but I
> > > > > > > > think Unity might end up being lighter-weight in the long
> run?
> > > And
> > > > > it is
> > > > > > > > also not a Xiaomi owned/maintained framework so it may
> receive
> > > > better
> > > > > > > > external support (i.e. more users rely on it?). I tried to
> > > compile
> > > > > the
> > > > > > > > sim:citest configuration to run the testsuite example and see
> > > what
> > > > > it was
> > > > > > > > like, but I was met with a long list of C++ compilation
> errors
> > > that
> > > > > I'd
> > > > > > > > rather not deal with. There also seems to be zero docs about
> it
> > > or
> > > > > cmocka
> > > > > > > > in our NuttX docs at the moment.
> > > > > > > >
> > > > > > > > What do you think? Start a Unity test suite, add some
> switches
> > to
> > > > > choose
> > > > > > > > which test cases get compiled into the binary, open for
> > > extensions
> > > > by
> > > > > > > users
> > > > > > > > who know more about different subsystems? This way we can
> keep
> > OS
> > > > > test
> > > > > > > how
> > > > > > > > it is for a quick, low-resource sanity check of "board
> bringup"
> > > > while
> > > > > > > > having a dedicated testing application for patches? Someone
> > > > changing
> > > > > file
> > > > > > > > system stuff can just compile to include file system test
> cases
> > > and
> > > > > run
> > > > > > > > that suite only as proof that their PR works. The output is
> > > really
> > > > > easy
> > > > > > > to
> > > > > > > > glance at to determine pass/fail results.
> > > > > > > >
> > > > > > > > Anyways, here's my draft demonstrating how Unity works with
> the
> > > FPU
> > > > > test
> > > > > > > > suite from OS test:
> > > https://github.com/apache/nuttx-apps/pull/3259
> > > > > > > >
> > > > > > > > Matteo
> > > > > > > >
> > > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> tomek@cedro.info>
> > > > > wrote:
> > > > > > > >
> > > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> > > spudaneco@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >>> (..)
> > > > > > > >>> Starting with bits and pieces of something like the LTS
> might
> > > > make
> > > > > more
> > > > > > > >>> sense other than its licensing problems. Since it would
> > never
> > > be
> > > > > > > >>> included in a distributed binary, the licensing really
> should
> > > not
> > > > > > > matter
> > > > > > > >>> much in practice.
> > > > > > > >> Unless its GPL and some company would like to have their own
> > > > > internal
> > > > > > > >> bits and pieces.. luckily its MIT and the whole thing is
> > > > open-source
> > > > > > > >> anyways :-)
> > > > > > > >>
> > > > > > > >> Yes we had many discussions about the LTS and this could be
> > > great
> > > > > start!
> > > > > > > >> :-)
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > > >>
> > > > > > >
> > > > >
> > > >
> > >
> >
>
Re: OS Test re-factor proposal
Posted by Gregory Nutt <sp...@gmail.com>.
On 12/25/25 11:33, Matteo Golin wrote:
> I wonder if the 8 bit architectures use smaller integer sizes for things
> like the timespec structs. I wonder if there are any other parts of NuttX
> that would complain during compilation seeing that ostest seems to make the
> assumption that tv_nsec is big enough for that operation. I guess its
> infrequent that things are tested on 8-bit MCUs now.
>
> As someone who is regularly developing for AVR, do you think it would be a
> benefit to have a test suite that is able to run fully on AVR itself? Even
> if only a couple tests can be included in an image at once due to flash
> size limits?
Most POSIX interfaces use fixed width integer type from stdtpe.h to
assure portabilityu in all places where integer width matters. But POSIX
will use int or unsigned for small integers whose storage size does not
matter. The ANSII spec guarantees that the size of an integer is at
least 16 bits. Linux and Windows treat an integer as 32 bits. Standard
integer types should be used if a width can be more than that.
The actual width of type int is determined by the compiler and you would
have to look at the ABI to determine which size is generated by the
compiler.
Re: OS Test re-factor proposal
Posted by Tomek CEDRO <to...@cedro.info>.
On the other hand AVR is still very popular and widely available so it
could be NuttX ostest/benchmark for 8-bit platforms.. I have some
small/medium/big AVR MCUs somewhere around so I could find them and
help in testing/verification :-)
--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
On Fri, Dec 26, 2025 at 12:06 PM <kr...@kerogit.eu> wrote:
>
> With avr-gcc, int is int16_t. Numeric literals are treated as int so
> every compile-time calculation larger than 16 bits gets truncated unless
> one of the operands is marked as longer type. tv_nsec is big enough to
> fit the value (sizeof(long) == 4) but the assignment to it only happens
> after the 16 bit truncation. Timing constants section in
> include/nuttx/clock.h is an example of what's needed to make such
> assignments to behave correctly.
>
> As for the test suite - it would be nice to have of course but
> considering that (I guess) there are not very many people who use NuttX
> on AVR, the time spent on making that happen would probably be better
> used on something with larger reach.
>
> (On a related note - there are other similar pitfalls when it comes to
> NuttX on AVR and probably on other 8 bit MCUs in general. For example,
> any code that assumes that arithmetical/logical operations on int
> operand are atomic is faulty in this setting. You can see commit
> a3f8b55143ad for possible implications of this. As far as I can see,
> struct circbuf_s has similar issue and I wonder if something like this
> can be found in core code like the scheduler. Because if so, then having
> tests is nice but not that useful since they should always fail under
> the circumstances.)
>
> On 2025-12-25 18:33, Matteo Golin wrote:
> > I wonder if the 8 bit architectures use smaller integer sizes for
> > things
> > like the timespec structs. I wonder if there are any other parts of
> > NuttX
> > that would complain during compilation seeing that ostest seems to make
> > the
> > assumption that tv_nsec is big enough for that operation. I guess its
> > infrequent that things are tested on 8-bit MCUs now.
> >
> > As someone who is regularly developing for AVR, do you think it would
> > be a
> > benefit to have a test suite that is able to run fully on AVR itself?
> > Even
> > if only a couple tests can be included in an image at once due to flash
> > size limits?
> >
> > Merry Christmas,
> > Matteo
> >
> > On Thu, Dec 25, 2025 at 8:18 AM <kr...@kerogit.eu> wrote:
> >
> >> I was wondering if ostest is even supposed to work on AVR so I gave it
> >> a
> >> try... it didn't do much, the only result I got when I ran ostest from
> >> NSH was:
> >>
> >> stdio_test: write fd=1
> >> stdio_test: write fd=2
> >>
> >> And that's it.
> >>
> >> (There may be something wrong with printf when the application gets
> >> executed through NSH though - just noticed that my other testing apps
> >> also print nothing when executed from NSH while being able to printf
> >> when executed as init entrypoint. Will try to update apps to something
> >> more recent but if anyone has any idea what may be causing this, it
> >> would be appreciated.)
> >>
> >> Anyway, the compilation also throws a lot of warnings, for example
> >>
> >> semtimed.c:87:32: warning: integer overflow in expression of type
> >> ‘int’
> >> results in ‘16960’ [-Woverflow]
> >> 87 | tp->tv_nsec >= 1000 * 1000 * 1000)
> >>
> >> and there are also warnings regarding to size mismatch in
> >> pointer/integer conversion.
> >>
> >> So it may be that ostest doesn't work well for AVR (and probably for
> >> other 8 bit MCUs) already. It also already barely fits the flash of
> >> the
> >> largest AVR DA chip.
> >>
> >> (Not suggesting to make it even more faulty, just mentioning it FYI.)
> >>
> >> On 2025-12-22 13:35, Alan C. Assis wrote:
> >> > Is it too complicated to rewrite the cmocka tests in Unity?
> >> >
> >> > If cmoka is not feasible for small microcontrollers, like AVRs and
> >> > others,
> >> > our test coverage will be faulty.
> >> >
> >> > BR,
> >> >
> >> > Alan
> >> >
> >> > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <ra...@gmail.com>
> >> > wrote:
> >> >
> >> >> I'm a fan of Unity and have been using it in my projects for years,
> >> >> but I
> >> >> think
> >> >> we should stick with one testing framework. Currently, Cmocka is used
> >> >> in
> >> >> nuttx-apps, and we should stick with it. The only question is how well
> >> >> Cmocka
> >> >> is suitable for running on small systems? Will ostest still be able to
> >> >> run
> >> >> on small MCUs?
> >> >> With Unity this would be quite likely, so maybe an exception for Unity
> >> >> and
> >> >> ostest would
> >> >> be acceptable, but using it for other test cases will only cause
> >> >> chaos.
> >> >>
> >> >> pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com> napisał(a):
> >> >>
> >> >> > Hi Matteo,
> >> >> >
> >> >> > Good idea !
> >> >> >
> >> >> > But it is best to stick with a single testing framework. Currently,
> >> many
> >> >> > test cases are using cmocka, so I suggest reusing cmocka for
> >> consistency.
> >> >> >
> >> >> > ostest is a minimal basic test suite, typically used for CT
> >> (continuous
> >> >> > testing) of each patch.
> >> >> > If we move ostest to cmocka, it would make ostest dependent on a
> >> specific
> >> >> > framework before it can be run.
> >> >> > And It may increase the code size, which is not very friendly for
> >> >> > configurations with small flash memory.
> >> >> >
> >> >> > As Greg mentioned: the answer depends on a tradeoff between effort and
> >> >> test
> >> >> > coverage.
> >> >> >
> >> >> > We need to clarify whether we define ostest as a lightweight,
> >> >> ready-to-run
> >> >> > test set for quick verification, or as part of the full test suite.
> >> >> >
> >> >> >
> >> >> > The follow is my compiling for cmocka:
> >> >> >
> >> >> > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh sim:citest
> >> >> -j8*
> >> >> > Copy files
> >> >> > Select CONFIG_HOST_LINUX=y
> >> >> > Refreshing...
> >> >> > CP: arch/dummy/Kconfig to
> >> >> > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> >> >> > CP: boards/dummy/Kconfig to
> >> >> > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> >> >> > LN: include/arch to arch/sim/include
> >> >> > LN: drivers/platform to
> >> /home/ligd/platform/mainline/nuttx/drivers/dummy
> >> >> > LN: arch/sim/src/chip to
> >> >> > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> >> >> > LN: arch/sim/src/board to
> >> >> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> >> >> > LN: include/arch/board to
> >> >> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> >> >> > LN: include/arch/chip to
> >> >> > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> >> >> > LN: platform/board to /home/ligd/platform/mainline/apps/platform/dummy
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/boot
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/canutils
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/crypto
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/database
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/module
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/games
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/graphics
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/industry
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/inertial
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/interpreters/luamodules
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/logging
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/lte
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/math
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/netutils
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/sdr
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/system
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/tee
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless
> >> >> > mkkconfig in /home/ligd/platform/mainline/apps
> >> >> > Loaded configuration '.config'
> >> >> > Configuration saved to '.config'
> >> >> > ligd@BAI:~/platform/mainline/nuttx$
> >> >> > *ligd@BAI:~/platform/mainline/nuttx$ make*
> >> >> > Create version.h
> >> >> > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> >> >> > /home/ligd/platform/mainline/apps/platform/dummy
> >> >> > Downloading: v1.46.0.zip
> >> >> > ...
> >> >> > patching file
> >> >> >
> >> >> >
> >> >>
> >> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> >> >> > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> >> >> >
> >> >> > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> >> >> > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> >> >> > LD: nuttx
> >> >> > Pac SIM with dynamic libs..
> >> >> > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> >> >> > '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
> >> 'sim-pac/libs/libstdc++.so.6'
> >> >> > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> >> >> > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> >> >> > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> 'sim-pac/libs/libgcc_s.so.1'
> >> >> > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
> >> >> > SIM elf with dynamic libs archive in nuttx.tgz
> >> >> > ligd@BAI:~/platform/mainline/nuttx$
> >> >> > ligd@BAI:~/platform/mainline/nuttx$
> >> >> > ligd@BAI:~/platform/mainline/nuttx$
> >> >> > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> >> >> > CHelloWorld: Constructor: mSecret=42
> >> >> > nsh: init: open failed: 2
> >> >> >
> >> >> > NuttShell (NSH) NuttX-12.11.0
> >> >> > MOTD: username=admin password=Administrator
> >> >> > nsh>
> >> >> > nsh> help
> >> >> > help usage: help [-v] [<cmd>]
> >> >> >
> >> >> > . df ifconfig mkfifo readlink umount
> >> >> > [ dmesg ifdown mkrd rm unset
> >> >> > ? echo ifup mount rmdir uptime
> >> >> > alias env insmod mv rmmod usleep
> >> >> > unalias exec kill nslookup set watch
> >> >> > basename exit pkill pidof sleep xd
> >> >> > break expr losetup pmconfig source wait
> >> >> > cat false ln poweroff test
> >> >> > cd fdinfo ls quit time
> >> >> > cp free lsmod printf true
> >> >> > cmp help mkdir ps truncate
> >> >> > dirname hexdump mkfatfs pwd uname
> >> >> >
> >> >> > Builtin Apps:
> >> >> > cmocka
> >> >> > cmocka_driver_block
> >> >> > cmocka_driver_gpio
> >> >> > cmocka_driver_pm
> >> >> > cmocka_driver_pm_runtime
> >> >> > cmocka_driver_regulator
> >> >> >
> >> >> > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一 10:15写道：
> >> >> >
> >> >> > > I think so. Best to leave OS test alone and slowly replace it with a
> >> >> new
> >> >> > > tool that has improvements. The only reason I can see to modify OS
> >> test
> >> >> > now
> >> >> > > would just be to have clearer output about what passed/failed.
> >> >> > >
> >> >> > > I'll come up with an improved proof of concept for a new tool to
> >> share
> >> >> in
> >> >> > > the next few days.
> >> >> > >
> >> >> > > Matteo
> >> >> > >
> >> >> > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <to...@cedro.info> wrote:
> >> >> > >
> >> >> > > > Small steps with measurable results may be the best approach
> >> here, to
> >> >> > > > create some new solution next to existing ostest but not
> >> replacing it
> >> >> > > > until ready? :-)
> >> >> > > >
> >> >> > > > --
> >> >> > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >> >> > > >
> >> >> > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> >> matteo.golin@gmail.com
> >> >> >
> >> >> > > > wrote:
> >> >> > > > >
> >> >> > > > > I agree, I think it would take a long time to have proper
> >> coverage
> >> >> of
> >> >> > > > NuttX
> >> >> > > > > as a whole. Perhaps even impossible, since Unity is more of a
> >> unit
> >> >> > > > testing
> >> >> > > > > framework and thus there are definitely test cases for NuttX
> >> that
> >> >> > would
> >> >> > > > be
> >> >> > > > > outside of its realm of "applicability". There is still a need
> >> for
> >> >> a
> >> >> > > > > variety of testing applications and bench-marking applications
> >> for
> >> >> > > those
> >> >> > > > > scenarios.
> >> >> > > > >
> >> >> > > > > That makes sense, prerequisites would need to be considered. I
> >> >> think
> >> >> > > > > targeting the low-hanging fruit of unit-testable code would be
> >> easy
> >> >> > in
> >> >> > > > that
> >> >> > > > > case, since prerequisites are mostly encoded in the Kconfig
> >> >> > definitions
> >> >> > > > > existing or not for features to be tested. Any complex behaviour
> >> >> > > > > (interrupts, multi-threading, etc.) might be out of the realm of
> >> >> > what's
> >> >> > > > > testable by Unity in the first place. We'll always have OS test
> >> in
> >> >> > that
> >> >> > > > > case, but maybe then it's better to do what Alan suggested and
> >> at
> >> >> > least
> >> >> > > > > make its output more clear in the meantime. I suppose RTOS
> >> testing
> >> >> > is a
> >> >> > > > > whole field in itself.
> >> >> > > > >
> >> >> > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> >> spudaneco@gmail.com>
> >> >> > > > wrote:
> >> >> > > > >
> >> >> > > > > > I think that the answer depends on a tradeoff between effort
> >> and
> >> >> > test
> >> >> > > > > > coverage. I could imagine that a proper job could be
> >> man-years
> >> >> of
> >> >> > > > effort.
> >> >> > > > > >
> >> >> > > > > > My only concern with a re-design is to be certain that tests
> >> are
> >> >> > > > > > arranged so that the prerequisites of later test are tested
> >> >> first.
> >> >> > > If
> >> >> > > > > > tests are grouped by, for example, functional area, then
> >> failures
> >> >> > can
> >> >> > > > be
> >> >> > > > > > difficult to understand if there is a failure in an unverified
> >> >> test
> >> >> > > > > > prerequisite.
> >> >> > > > > >
> >> >> > > > > > Some of the "tests" are bogus and only exist to provide some
> >> >> > minimum
> >> >> > > > > > confidence that we have the functionality to run the test.
> >> For
> >> >> > > > > > example: Can I start tasks? Can I pass parameters? Can I
> >> >> survive
> >> >> > a
> >> >> > > > > > context switch. Some of the tests were intended only to
> >> support
> >> >> > > debug.
> >> >> > > > > > Some were created only to track down specific bugs.
> >> >> > > > > >
> >> >> > > > > > On 12/21/25 18:39, Matteo Golin wrote:
> >> >> > > > > > > LTS being the Long-Term-Stable release of NuttX?
> >> >> > > > > > >
> >> >> > > > > > > I'm starting to think that it would be best to create new
> >> suite
> >> >> > of
> >> >> > > > tests
> >> >> > > > > > > using Unity, using what's in OS Test as a basis and then
> >> >> > expanding
> >> >> > > it
> >> >> > > > > > > further. Greg has pointed out that OS test is not intended
> >> to
> >> >> be
> >> >> > a
> >> >> > > > > > complete
> >> >> > > > > > > test, so maybe a more solid testing framework base in a new
> >> >> form
> >> >> > > > would be
> >> >> > > > > > > in order. At that point it would also be good to document
> >> and
> >> >> > > educate
> >> >> > > > > > users
> >> >> > > > > > > about other test options, since OS test is a current
> >> favourite
> >> >> > for
> >> >> > > > > > showing
> >> >> > > > > > > PRs haven't broken anything. Maybe that's not the best
> >> choice
> >> >> > > anymore
> >> >> > > > > > with
> >> >> > > > > > > this information.
> >> >> > > > > > >
> >> >> > > > > > > I don't want to re-invent the wheel of what's already in
> >> >> > > `testsuite`,
> >> >> > > > > > but I
> >> >> > > > > > > think Unity might end up being lighter-weight in the long
> >> run?
> >> >> > And
> >> >> > > > it is
> >> >> > > > > > > also not a Xiaomi owned/maintained framework so it may
> >> receive
> >> >> > > better
> >> >> > > > > > > external support (i.e. more users rely on it?). I tried to
> >> >> > compile
> >> >> > > > the
> >> >> > > > > > > sim:citest configuration to run the testsuite example and
> >> see
> >> >> > what
> >> >> > > > it was
> >> >> > > > > > > like, but I was met with a long list of C++ compilation
> >> errors
> >> >> > that
> >> >> > > > I'd
> >> >> > > > > > > rather not deal with. There also seems to be zero docs
> >> about it
> >> >> > or
> >> >> > > > cmocka
> >> >> > > > > > > in our NuttX docs at the moment.
> >> >> > > > > > >
> >> >> > > > > > > What do you think? Start a Unity test suite, add some
> >> switches
> >> >> to
> >> >> > > > choose
> >> >> > > > > > > which test cases get compiled into the binary, open for
> >> >> > extensions
> >> >> > > by
> >> >> > > > > > users
> >> >> > > > > > > who know more about different subsystems? This way we can
> >> keep
> >> >> OS
> >> >> > > > test
> >> >> > > > > > how
> >> >> > > > > > > it is for a quick, low-resource sanity check of "board
> >> bringup"
> >> >> > > while
> >> >> > > > > > > having a dedicated testing application for patches? Someone
> >> >> > > changing
> >> >> > > > file
> >> >> > > > > > > system stuff can just compile to include file system test
> >> cases
> >> >> > and
> >> >> > > > run
> >> >> > > > > > > that suite only as proof that their PR works. The output is
> >> >> > really
> >> >> > > > easy
> >> >> > > > > > to
> >> >> > > > > > > glance at to determine pass/fail results.
> >> >> > > > > > >
> >> >> > > > > > > Anyways, here's my draft demonstrating how Unity works with
> >> the
> >> >> > FPU
> >> >> > > > test
> >> >> > > > > > > suite from OS test:
> >> >> > https://github.com/apache/nuttx-apps/pull/3259
> >> >> > > > > > >
> >> >> > > > > > > Matteo
> >> >> > > > > > >
> >> >> > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> >> tomek@cedro.info>
> >> >> > > > wrote:
> >> >> > > > > > >
> >> >> > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> >> >> > spudaneco@gmail.com
> >> >> > > >
> >> >> > > > > > wrote:
> >> >> > > > > > >>> (..)
> >> >> > > > > > >>> Starting with bits and pieces of something like the LTS
> >> might
> >> >> > > make
> >> >> > > > more
> >> >> > > > > > >>> sense other than its licensing problems. Since it would
> >> >> never
> >> >> > be
> >> >> > > > > > >>> included in a distributed binary, the licensing really
> >> should
> >> >> > not
> >> >> > > > > > matter
> >> >> > > > > > >>> much in practice.
> >> >> > > > > > >> Unless its GPL and some company would like to have their
> >> own
> >> >> > > > internal
> >> >> > > > > > >> bits and pieces.. luckily its MIT and the whole thing is
> >> >> > > open-source
> >> >> > > > > > >> anyways :-)
> >> >> > > > > > >>
> >> >> > > > > > >> Yes we had many discussions about the LTS and this could be
> >> >> > great
> >> >> > > > start!
> >> >> > > > > > >> :-)
> >> >> > > > > > >>
> >> >> > > > > > >> --
> >> >> > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >> >> > > > > > >>
> >> >> > > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
Re: OS Test re-factor proposal
Posted by kr...@kerogit.eu.
On 2025-12-26 13:44, Gregory Nutt wrote:
> And from what I gather, int16_t accesses are not atomic. Most 8-bit
> CPUs
> offer some hardware support for atomicity (AVR has shadow registers).
To my knowledge - and unless we're talking about different AVR families
- the shadow registers are not relevant here, those are only available
for access to 16 bit I/O registers (for example current count in 16 bit
timer/counter.)
For generic memory access, the only option to make the access to
multibyte values atomic is disabling interrupts.
> This is not really a NuttX-specific problem, but would effect all
> real-time
> systems that use and ANSI C. I suppose you could define type int to be
> type int-atomic and then exploit this hardware support in a
> cpu-specific
> way.
In the serial driver, the chosen solution was to simply reduce maximum
buffer size to fit uint8_t because branching the code with #ifdefs to
only disable interrupts if needed would make readability worse and the
only way I know to make a MAKE_THIS_BLOCK_ATOMIC_IF_NEEDED {} macro is
IIRC invalid with C89. Not sure if this solution will be applicable for
circbuf_s though because that is used throughout the source tree and at
least pipe driver allows buffers up to 64kB size.
Re: OS Test re-factor proposal
Posted by Gregory Nutt <sp...@gmail.com>.
On Fri, Dec 26, 2025 at 5:06 AM <kr...@kerogit.eu> wrote:
> With avr-gcc, int is int16_t. ...
And from what I gather, int16_t accesses are not atomic. Most 8-bit CPUs
offer some hardware support for atomicity (AVR has shadow registers).
This is not really a NuttX-specific problem, but would effect all real-time
systems that use and ANSI C. I suppose you could define type int to be
type int-atomic and then exploit this hardware support in a cpu-specific
way.
Re: OS Test re-factor proposal
Posted by kr...@kerogit.eu.
With avr-gcc, int is int16_t. Numeric literals are treated as int so
every compile-time calculation larger than 16 bits gets truncated unless
one of the operands is marked as longer type. tv_nsec is big enough to
fit the value (sizeof(long) == 4) but the assignment to it only happens
after the 16 bit truncation. Timing constants section in
include/nuttx/clock.h is an example of what's needed to make such
assignments to behave correctly.
As for the test suite - it would be nice to have of course but
considering that (I guess) there are not very many people who use NuttX
on AVR, the time spent on making that happen would probably be better
used on something with larger reach.
(On a related note - there are other similar pitfalls when it comes to
NuttX on AVR and probably on other 8 bit MCUs in general. For example,
any code that assumes that arithmetical/logical operations on int
operand are atomic is faulty in this setting. You can see commit
a3f8b55143ad for possible implications of this. As far as I can see,
struct circbuf_s has similar issue and I wonder if something like this
can be found in core code like the scheduler. Because if so, then having
tests is nice but not that useful since they should always fail under
the circumstances.)
On 2025-12-25 18:33, Matteo Golin wrote:
> I wonder if the 8 bit architectures use smaller integer sizes for
> things
> like the timespec structs. I wonder if there are any other parts of
> NuttX
> that would complain during compilation seeing that ostest seems to make
> the
> assumption that tv_nsec is big enough for that operation. I guess its
> infrequent that things are tested on 8-bit MCUs now.
>
> As someone who is regularly developing for AVR, do you think it would
> be a
> benefit to have a test suite that is able to run fully on AVR itself?
> Even
> if only a couple tests can be included in an image at once due to flash
> size limits?
>
> Merry Christmas,
> Matteo
>
> On Thu, Dec 25, 2025 at 8:18 AM <kr...@kerogit.eu> wrote:
>
>> I was wondering if ostest is even supposed to work on AVR so I gave it
>> a
>> try... it didn't do much, the only result I got when I ran ostest from
>> NSH was:
>>
>> stdio_test: write fd=1
>> stdio_test: write fd=2
>>
>> And that's it.
>>
>> (There may be something wrong with printf when the application gets
>> executed through NSH though - just noticed that my other testing apps
>> also print nothing when executed from NSH while being able to printf
>> when executed as init entrypoint. Will try to update apps to something
>> more recent but if anyone has any idea what may be causing this, it
>> would be appreciated.)
>>
>> Anyway, the compilation also throws a lot of warnings, for example
>>
>> semtimed.c:87:32: warning: integer overflow in expression of type
>> ‘int’
>> results in ‘16960’ [-Woverflow]
>> 87 | tp->tv_nsec >= 1000 * 1000 * 1000)
>>
>> and there are also warnings regarding to size mismatch in
>> pointer/integer conversion.
>>
>> So it may be that ostest doesn't work well for AVR (and probably for
>> other 8 bit MCUs) already. It also already barely fits the flash of
>> the
>> largest AVR DA chip.
>>
>> (Not suggesting to make it even more faulty, just mentioning it FYI.)
>>
>> On 2025-12-22 13:35, Alan C. Assis wrote:
>> > Is it too complicated to rewrite the cmocka tests in Unity?
>> >
>> > If cmoka is not feasible for small microcontrollers, like AVRs and
>> > others,
>> > our test coverage will be faulty.
>> >
>> > BR,
>> >
>> > Alan
>> >
>> > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <ra...@gmail.com>
>> > wrote:
>> >
>> >> I'm a fan of Unity and have been using it in my projects for years,
>> >> but I
>> >> think
>> >> we should stick with one testing framework. Currently, Cmocka is used
>> >> in
>> >> nuttx-apps, and we should stick with it. The only question is how well
>> >> Cmocka
>> >> is suitable for running on small systems? Will ostest still be able to
>> >> run
>> >> on small MCUs?
>> >> With Unity this would be quite likely, so maybe an exception for Unity
>> >> and
>> >> ostest would
>> >> be acceptable, but using it for other test cases will only cause
>> >> chaos.
>> >>
>> >> pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com> napisał(a):
>> >>
>> >> > Hi Matteo,
>> >> >
>> >> > Good idea !
>> >> >
>> >> > But it is best to stick with a single testing framework. Currently,
>> many
>> >> > test cases are using cmocka, so I suggest reusing cmocka for
>> consistency.
>> >> >
>> >> > ostest is a minimal basic test suite, typically used for CT
>> (continuous
>> >> > testing) of each patch.
>> >> > If we move ostest to cmocka, it would make ostest dependent on a
>> specific
>> >> > framework before it can be run.
>> >> > And It may increase the code size, which is not very friendly for
>> >> > configurations with small flash memory.
>> >> >
>> >> > As Greg mentioned: the answer depends on a tradeoff between effort and
>> >> test
>> >> > coverage.
>> >> >
>> >> > We need to clarify whether we define ostest as a lightweight,
>> >> ready-to-run
>> >> > test set for quick verification, or as part of the full test suite.
>> >> >
>> >> >
>> >> > The follow is my compiling for cmocka:
>> >> >
>> >> > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh sim:citest
>> >> -j8*
>> >> > Copy files
>> >> > Select CONFIG_HOST_LINUX=y
>> >> > Refreshing...
>> >> > CP: arch/dummy/Kconfig to
>> >> > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
>> >> > CP: boards/dummy/Kconfig to
>> >> > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
>> >> > LN: include/arch to arch/sim/include
>> >> > LN: drivers/platform to
>> /home/ligd/platform/mainline/nuttx/drivers/dummy
>> >> > LN: arch/sim/src/chip to
>> >> > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
>> >> > LN: arch/sim/src/board to
>> >> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
>> >> > LN: include/arch/board to
>> >> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
>> >> > LN: include/arch/chip to
>> >> > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
>> >> > LN: platform/board to /home/ligd/platform/mainline/apps/platform/dummy
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/audioutils
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/boot
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/canutils
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/crypto
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/database
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/module
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/games
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/graphics
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/industry
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/inertial
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/interpreters/luamodules
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/interpreters
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/logging
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/lte
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/math
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/mlearning
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/netutils
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/sdr
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/system
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/tee
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/videoutils
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154
>> >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless
>> >> > mkkconfig in /home/ligd/platform/mainline/apps
>> >> > Loaded configuration '.config'
>> >> > Configuration saved to '.config'
>> >> > ligd@BAI:~/platform/mainline/nuttx$
>> >> > *ligd@BAI:~/platform/mainline/nuttx$ make*
>> >> > Create version.h
>> >> > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
>> >> > /home/ligd/platform/mainline/apps/platform/dummy
>> >> > Downloading: v1.46.0.zip
>> >> > ...
>> >> > patching file
>> >> >
>> >> >
>> >>
>> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
>> >> > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
>> >> >
>> >> > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
>> >> > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
>> >> > LD: nuttx
>> >> > Pac SIM with dynamic libs..
>> >> > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
>> >> > '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
>> 'sim-pac/libs/libstdc++.so.6'
>> >> > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
>> >> > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
>> >> > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> 'sim-pac/libs/libgcc_s.so.1'
>> >> > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
>> >> > SIM elf with dynamic libs archive in nuttx.tgz
>> >> > ligd@BAI:~/platform/mainline/nuttx$
>> >> > ligd@BAI:~/platform/mainline/nuttx$
>> >> > ligd@BAI:~/platform/mainline/nuttx$
>> >> > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
>> >> > CHelloWorld: Constructor: mSecret=42
>> >> > nsh: init: open failed: 2
>> >> >
>> >> > NuttShell (NSH) NuttX-12.11.0
>> >> > MOTD: username=admin password=Administrator
>> >> > nsh>
>> >> > nsh> help
>> >> > help usage: help [-v] [<cmd>]
>> >> >
>> >> > . df ifconfig mkfifo readlink umount
>> >> > [ dmesg ifdown mkrd rm unset
>> >> > ? echo ifup mount rmdir uptime
>> >> > alias env insmod mv rmmod usleep
>> >> > unalias exec kill nslookup set watch
>> >> > basename exit pkill pidof sleep xd
>> >> > break expr losetup pmconfig source wait
>> >> > cat false ln poweroff test
>> >> > cd fdinfo ls quit time
>> >> > cp free lsmod printf true
>> >> > cmp help mkdir ps truncate
>> >> > dirname hexdump mkfatfs pwd uname
>> >> >
>> >> > Builtin Apps:
>> >> > cmocka
>> >> > cmocka_driver_block
>> >> > cmocka_driver_gpio
>> >> > cmocka_driver_pm
>> >> > cmocka_driver_pm_runtime
>> >> > cmocka_driver_regulator
>> >> >
>> >> > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一 10:15写道：
>> >> >
>> >> > > I think so. Best to leave OS test alone and slowly replace it with a
>> >> new
>> >> > > tool that has improvements. The only reason I can see to modify OS
>> test
>> >> > now
>> >> > > would just be to have clearer output about what passed/failed.
>> >> > >
>> >> > > I'll come up with an improved proof of concept for a new tool to
>> share
>> >> in
>> >> > > the next few days.
>> >> > >
>> >> > > Matteo
>> >> > >
>> >> > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <to...@cedro.info> wrote:
>> >> > >
>> >> > > > Small steps with measurable results may be the best approach
>> here, to
>> >> > > > create some new solution next to existing ostest but not
>> replacing it
>> >> > > > until ready? :-)
>> >> > > >
>> >> > > > --
>> >> > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>> >> > > >
>> >> > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
>> matteo.golin@gmail.com
>> >> >
>> >> > > > wrote:
>> >> > > > >
>> >> > > > > I agree, I think it would take a long time to have proper
>> coverage
>> >> of
>> >> > > > NuttX
>> >> > > > > as a whole. Perhaps even impossible, since Unity is more of a
>> unit
>> >> > > > testing
>> >> > > > > framework and thus there are definitely test cases for NuttX
>> that
>> >> > would
>> >> > > > be
>> >> > > > > outside of its realm of "applicability". There is still a need
>> for
>> >> a
>> >> > > > > variety of testing applications and bench-marking applications
>> for
>> >> > > those
>> >> > > > > scenarios.
>> >> > > > >
>> >> > > > > That makes sense, prerequisites would need to be considered. I
>> >> think
>> >> > > > > targeting the low-hanging fruit of unit-testable code would be
>> easy
>> >> > in
>> >> > > > that
>> >> > > > > case, since prerequisites are mostly encoded in the Kconfig
>> >> > definitions
>> >> > > > > existing or not for features to be tested. Any complex behaviour
>> >> > > > > (interrupts, multi-threading, etc.) might be out of the realm of
>> >> > what's
>> >> > > > > testable by Unity in the first place. We'll always have OS test
>> in
>> >> > that
>> >> > > > > case, but maybe then it's better to do what Alan suggested and
>> at
>> >> > least
>> >> > > > > make its output more clear in the meantime. I suppose RTOS
>> testing
>> >> > is a
>> >> > > > > whole field in itself.
>> >> > > > >
>> >> > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
>> spudaneco@gmail.com>
>> >> > > > wrote:
>> >> > > > >
>> >> > > > > > I think that the answer depends on a tradeoff between effort
>> and
>> >> > test
>> >> > > > > > coverage. I could imagine that a proper job could be
>> man-years
>> >> of
>> >> > > > effort.
>> >> > > > > >
>> >> > > > > > My only concern with a re-design is to be certain that tests
>> are
>> >> > > > > > arranged so that the prerequisites of later test are tested
>> >> first.
>> >> > > If
>> >> > > > > > tests are grouped by, for example, functional area, then
>> failures
>> >> > can
>> >> > > > be
>> >> > > > > > difficult to understand if there is a failure in an unverified
>> >> test
>> >> > > > > > prerequisite.
>> >> > > > > >
>> >> > > > > > Some of the "tests" are bogus and only exist to provide some
>> >> > minimum
>> >> > > > > > confidence that we have the functionality to run the test.
>> For
>> >> > > > > > example: Can I start tasks? Can I pass parameters? Can I
>> >> survive
>> >> > a
>> >> > > > > > context switch. Some of the tests were intended only to
>> support
>> >> > > debug.
>> >> > > > > > Some were created only to track down specific bugs.
>> >> > > > > >
>> >> > > > > > On 12/21/25 18:39, Matteo Golin wrote:
>> >> > > > > > > LTS being the Long-Term-Stable release of NuttX?
>> >> > > > > > >
>> >> > > > > > > I'm starting to think that it would be best to create new
>> suite
>> >> > of
>> >> > > > tests
>> >> > > > > > > using Unity, using what's in OS Test as a basis and then
>> >> > expanding
>> >> > > it
>> >> > > > > > > further. Greg has pointed out that OS test is not intended
>> to
>> >> be
>> >> > a
>> >> > > > > > complete
>> >> > > > > > > test, so maybe a more solid testing framework base in a new
>> >> form
>> >> > > > would be
>> >> > > > > > > in order. At that point it would also be good to document
>> and
>> >> > > educate
>> >> > > > > > users
>> >> > > > > > > about other test options, since OS test is a current
>> favourite
>> >> > for
>> >> > > > > > showing
>> >> > > > > > > PRs haven't broken anything. Maybe that's not the best
>> choice
>> >> > > anymore
>> >> > > > > > with
>> >> > > > > > > this information.
>> >> > > > > > >
>> >> > > > > > > I don't want to re-invent the wheel of what's already in
>> >> > > `testsuite`,
>> >> > > > > > but I
>> >> > > > > > > think Unity might end up being lighter-weight in the long
>> run?
>> >> > And
>> >> > > > it is
>> >> > > > > > > also not a Xiaomi owned/maintained framework so it may
>> receive
>> >> > > better
>> >> > > > > > > external support (i.e. more users rely on it?). I tried to
>> >> > compile
>> >> > > > the
>> >> > > > > > > sim:citest configuration to run the testsuite example and
>> see
>> >> > what
>> >> > > > it was
>> >> > > > > > > like, but I was met with a long list of C++ compilation
>> errors
>> >> > that
>> >> > > > I'd
>> >> > > > > > > rather not deal with. There also seems to be zero docs
>> about it
>> >> > or
>> >> > > > cmocka
>> >> > > > > > > in our NuttX docs at the moment.
>> >> > > > > > >
>> >> > > > > > > What do you think? Start a Unity test suite, add some
>> switches
>> >> to
>> >> > > > choose
>> >> > > > > > > which test cases get compiled into the binary, open for
>> >> > extensions
>> >> > > by
>> >> > > > > > users
>> >> > > > > > > who know more about different subsystems? This way we can
>> keep
>> >> OS
>> >> > > > test
>> >> > > > > > how
>> >> > > > > > > it is for a quick, low-resource sanity check of "board
>> bringup"
>> >> > > while
>> >> > > > > > > having a dedicated testing application for patches? Someone
>> >> > > changing
>> >> > > > file
>> >> > > > > > > system stuff can just compile to include file system test
>> cases
>> >> > and
>> >> > > > run
>> >> > > > > > > that suite only as proof that their PR works. The output is
>> >> > really
>> >> > > > easy
>> >> > > > > > to
>> >> > > > > > > glance at to determine pass/fail results.
>> >> > > > > > >
>> >> > > > > > > Anyways, here's my draft demonstrating how Unity works with
>> the
>> >> > FPU
>> >> > > > test
>> >> > > > > > > suite from OS test:
>> >> > https://github.com/apache/nuttx-apps/pull/3259
>> >> > > > > > >
>> >> > > > > > > Matteo
>> >> > > > > > >
>> >> > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
>> tomek@cedro.info>
>> >> > > > wrote:
>> >> > > > > > >
>> >> > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
>> >> > spudaneco@gmail.com
>> >> > > >
>> >> > > > > > wrote:
>> >> > > > > > >>> (..)
>> >> > > > > > >>> Starting with bits and pieces of something like the LTS
>> might
>> >> > > make
>> >> > > > more
>> >> > > > > > >>> sense other than its licensing problems. Since it would
>> >> never
>> >> > be
>> >> > > > > > >>> included in a distributed binary, the licensing really
>> should
>> >> > not
>> >> > > > > > matter
>> >> > > > > > >>> much in practice.
>> >> > > > > > >> Unless its GPL and some company would like to have their
>> own
>> >> > > > internal
>> >> > > > > > >> bits and pieces.. luckily its MIT and the whole thing is
>> >> > > open-source
>> >> > > > > > >> anyways :-)
>> >> > > > > > >>
>> >> > > > > > >> Yes we had many discussions about the LTS and this could be
>> >> > great
>> >> > > > start!
>> >> > > > > > >> :-)
>> >> > > > > > >>
>> >> > > > > > >> --
>> >> > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>> >> > > > > > >>
>> >> > > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>>
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
I wonder if the 8 bit architectures use smaller integer sizes for things
like the timespec structs. I wonder if there are any other parts of NuttX
that would complain during compilation seeing that ostest seems to make the
assumption that tv_nsec is big enough for that operation. I guess its
infrequent that things are tested on 8-bit MCUs now.
As someone who is regularly developing for AVR, do you think it would be a
benefit to have a test suite that is able to run fully on AVR itself? Even
if only a couple tests can be included in an image at once due to flash
size limits?
Merry Christmas,
Matteo
On Thu, Dec 25, 2025 at 8:18 AM <kr...@kerogit.eu> wrote:
> I was wondering if ostest is even supposed to work on AVR so I gave it a
> try... it didn't do much, the only result I got when I ran ostest from
> NSH was:
>
> stdio_test: write fd=1
> stdio_test: write fd=2
>
> And that's it.
>
> (There may be something wrong with printf when the application gets
> executed through NSH though - just noticed that my other testing apps
> also print nothing when executed from NSH while being able to printf
> when executed as init entrypoint. Will try to update apps to something
> more recent but if anyone has any idea what may be causing this, it
> would be appreciated.)
>
> Anyway, the compilation also throws a lot of warnings, for example
>
> semtimed.c:87:32: warning: integer overflow in expression of type ‘int’
> results in ‘16960’ [-Woverflow]
> 87 | tp->tv_nsec >= 1000 * 1000 * 1000)
>
> and there are also warnings regarding to size mismatch in
> pointer/integer conversion.
>
> So it may be that ostest doesn't work well for AVR (and probably for
> other 8 bit MCUs) already. It also already barely fits the flash of the
> largest AVR DA chip.
>
> (Not suggesting to make it even more faulty, just mentioning it FYI.)
>
> On 2025-12-22 13:35, Alan C. Assis wrote:
> > Is it too complicated to rewrite the cmocka tests in Unity?
> >
> > If cmoka is not feasible for small microcontrollers, like AVRs and
> > others,
> > our test coverage will be faulty.
> >
> > BR,
> >
> > Alan
> >
> > On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <ra...@gmail.com>
> > wrote:
> >
> >> I'm a fan of Unity and have been using it in my projects for years,
> >> but I
> >> think
> >> we should stick with one testing framework. Currently, Cmocka is used
> >> in
> >> nuttx-apps, and we should stick with it. The only question is how well
> >> Cmocka
> >> is suitable for running on small systems? Will ostest still be able to
> >> run
> >> on small MCUs?
> >> With Unity this would be quite likely, so maybe an exception for Unity
> >> and
> >> ostest would
> >> be acceptable, but using it for other test cases will only cause
> >> chaos.
> >>
> >> pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com> napisał(a):
> >>
> >> > Hi Matteo,
> >> >
> >> > Good idea !
> >> >
> >> > But it is best to stick with a single testing framework. Currently,
> many
> >> > test cases are using cmocka, so I suggest reusing cmocka for
> consistency.
> >> >
> >> > ostest is a minimal basic test suite, typically used for CT
> (continuous
> >> > testing) of each patch.
> >> > If we move ostest to cmocka, it would make ostest dependent on a
> specific
> >> > framework before it can be run.
> >> > And It may increase the code size, which is not very friendly for
> >> > configurations with small flash memory.
> >> >
> >> > As Greg mentioned: the answer depends on a tradeoff between effort and
> >> test
> >> > coverage.
> >> >
> >> > We need to clarify whether we define ostest as a lightweight,
> >> ready-to-run
> >> > test set for quick verification, or as part of the full test suite.
> >> >
> >> >
> >> > The follow is my compiling for cmocka:
> >> >
> >> > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh sim:citest
> >> -j8*
> >> > Copy files
> >> > Select CONFIG_HOST_LINUX=y
> >> > Refreshing...
> >> > CP: arch/dummy/Kconfig to
> >> > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> >> > CP: boards/dummy/Kconfig to
> >> > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> >> > LN: include/arch to arch/sim/include
> >> > LN: drivers/platform to
> /home/ligd/platform/mainline/nuttx/drivers/dummy
> >> > LN: arch/sim/src/chip to
> >> > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> >> > LN: arch/sim/src/board to
> >> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> >> > LN: include/arch/board to
> >> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> >> > LN: include/arch/chip to
> >> > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> >> > LN: platform/board to /home/ligd/platform/mainline/apps/platform/dummy
> >> > mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> >> > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> >> > mkkconfig in /home/ligd/platform/mainline/apps/boot
> >> > mkkconfig in /home/ligd/platform/mainline/apps/canutils
> >> > mkkconfig in /home/ligd/platform/mainline/apps/crypto
> >> > mkkconfig in /home/ligd/platform/mainline/apps/database
> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest
> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/module
> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot
> >> > mkkconfig in /home/ligd/platform/mainline/apps/examples
> >> > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> >> > mkkconfig in /home/ligd/platform/mainline/apps/games
> >> > mkkconfig in /home/ligd/platform/mainline/apps/graphics
> >> > mkkconfig in /home/ligd/platform/mainline/apps/industry
> >> > mkkconfig in /home/ligd/platform/mainline/apps/inertial
> >> > mkkconfig in /home/ligd/platform/mainline/apps/interpreters/luamodules
> >> > mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> >> > mkkconfig in /home/ligd/platform/mainline/apps/logging
> >> > mkkconfig in /home/ligd/platform/mainline/apps/lte
> >> > mkkconfig in /home/ligd/platform/mainline/apps/math
> >> > mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> >> > mkkconfig in /home/ligd/platform/mainline/apps/netutils
> >> > mkkconfig in /home/ligd/platform/mainline/apps/sdr
> >> > mkkconfig in /home/ligd/platform/mainline/apps/system
> >> > mkkconfig in /home/ligd/platform/mainline/apps/tee
> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers
> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> >> > mkkconfig in /home/ligd/platform/mainline/apps/testing
> >> > mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth
> >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154
> >> > mkkconfig in /home/ligd/platform/mainline/apps/wireless
> >> > mkkconfig in /home/ligd/platform/mainline/apps
> >> > Loaded configuration '.config'
> >> > Configuration saved to '.config'
> >> > ligd@BAI:~/platform/mainline/nuttx$
> >> > *ligd@BAI:~/platform/mainline/nuttx$ make*
> >> > Create version.h
> >> > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> >> > /home/ligd/platform/mainline/apps/platform/dummy
> >> > Downloading: v1.46.0.zip
> >> > ...
> >> > patching file
> >> >
> >> >
> >>
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> >> > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> >> >
> >> > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> >> > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> >> > LD: nuttx
> >> > Pac SIM with dynamic libs..
> >> > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> >> > '/lib/x86_64-linux-gnu/libstdc++.so.6' ->
> 'sim-pac/libs/libstdc++.so.6'
> >> > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> >> > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> >> > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> 'sim-pac/libs/libgcc_s.so.1'
> >> > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
> >> > SIM elf with dynamic libs archive in nuttx.tgz
> >> > ligd@BAI:~/platform/mainline/nuttx$
> >> > ligd@BAI:~/platform/mainline/nuttx$
> >> > ligd@BAI:~/platform/mainline/nuttx$
> >> > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> >> > CHelloWorld: Constructor: mSecret=42
> >> > nsh: init: open failed: 2
> >> >
> >> > NuttShell (NSH) NuttX-12.11.0
> >> > MOTD: username=admin password=Administrator
> >> > nsh>
> >> > nsh> help
> >> > help usage: help [-v] [<cmd>]
> >> >
> >> > . df ifconfig mkfifo readlink umount
> >> > [ dmesg ifdown mkrd rm unset
> >> > ? echo ifup mount rmdir uptime
> >> > alias env insmod mv rmmod usleep
> >> > unalias exec kill nslookup set watch
> >> > basename exit pkill pidof sleep xd
> >> > break expr losetup pmconfig source wait
> >> > cat false ln poweroff test
> >> > cd fdinfo ls quit time
> >> > cp free lsmod printf true
> >> > cmp help mkdir ps truncate
> >> > dirname hexdump mkfatfs pwd uname
> >> >
> >> > Builtin Apps:
> >> > cmocka
> >> > cmocka_driver_block
> >> > cmocka_driver_gpio
> >> > cmocka_driver_pm
> >> > cmocka_driver_pm_runtime
> >> > cmocka_driver_regulator
> >> >
> >> > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一 10:15写道：
> >> >
> >> > > I think so. Best to leave OS test alone and slowly replace it with a
> >> new
> >> > > tool that has improvements. The only reason I can see to modify OS
> test
> >> > now
> >> > > would just be to have clearer output about what passed/failed.
> >> > >
> >> > > I'll come up with an improved proof of concept for a new tool to
> share
> >> in
> >> > > the next few days.
> >> > >
> >> > > Matteo
> >> > >
> >> > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <to...@cedro.info> wrote:
> >> > >
> >> > > > Small steps with measurable results may be the best approach
> here, to
> >> > > > create some new solution next to existing ostest but not
> replacing it
> >> > > > until ready? :-)
> >> > > >
> >> > > > --
> >> > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >> > > >
> >> > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <
> matteo.golin@gmail.com
> >> >
> >> > > > wrote:
> >> > > > >
> >> > > > > I agree, I think it would take a long time to have proper
> coverage
> >> of
> >> > > > NuttX
> >> > > > > as a whole. Perhaps even impossible, since Unity is more of a
> unit
> >> > > > testing
> >> > > > > framework and thus there are definitely test cases for NuttX
> that
> >> > would
> >> > > > be
> >> > > > > outside of its realm of "applicability". There is still a need
> for
> >> a
> >> > > > > variety of testing applications and bench-marking applications
> for
> >> > > those
> >> > > > > scenarios.
> >> > > > >
> >> > > > > That makes sense, prerequisites would need to be considered. I
> >> think
> >> > > > > targeting the low-hanging fruit of unit-testable code would be
> easy
> >> > in
> >> > > > that
> >> > > > > case, since prerequisites are mostly encoded in the Kconfig
> >> > definitions
> >> > > > > existing or not for features to be tested. Any complex behaviour
> >> > > > > (interrupts, multi-threading, etc.) might be out of the realm of
> >> > what's
> >> > > > > testable by Unity in the first place. We'll always have OS test
> in
> >> > that
> >> > > > > case, but maybe then it's better to do what Alan suggested and
> at
> >> > least
> >> > > > > make its output more clear in the meantime. I suppose RTOS
> testing
> >> > is a
> >> > > > > whole field in itself.
> >> > > > >
> >> > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <
> spudaneco@gmail.com>
> >> > > > wrote:
> >> > > > >
> >> > > > > > I think that the answer depends on a tradeoff between effort
> and
> >> > test
> >> > > > > > coverage. I could imagine that a proper job could be
> man-years
> >> of
> >> > > > effort.
> >> > > > > >
> >> > > > > > My only concern with a re-design is to be certain that tests
> are
> >> > > > > > arranged so that the prerequisites of later test are tested
> >> first.
> >> > > If
> >> > > > > > tests are grouped by, for example, functional area, then
> failures
> >> > can
> >> > > > be
> >> > > > > > difficult to understand if there is a failure in an unverified
> >> test
> >> > > > > > prerequisite.
> >> > > > > >
> >> > > > > > Some of the "tests" are bogus and only exist to provide some
> >> > minimum
> >> > > > > > confidence that we have the functionality to run the test.
> For
> >> > > > > > example: Can I start tasks? Can I pass parameters? Can I
> >> survive
> >> > a
> >> > > > > > context switch. Some of the tests were intended only to
> support
> >> > > debug.
> >> > > > > > Some were created only to track down specific bugs.
> >> > > > > >
> >> > > > > > On 12/21/25 18:39, Matteo Golin wrote:
> >> > > > > > > LTS being the Long-Term-Stable release of NuttX?
> >> > > > > > >
> >> > > > > > > I'm starting to think that it would be best to create new
> suite
> >> > of
> >> > > > tests
> >> > > > > > > using Unity, using what's in OS Test as a basis and then
> >> > expanding
> >> > > it
> >> > > > > > > further. Greg has pointed out that OS test is not intended
> to
> >> be
> >> > a
> >> > > > > > complete
> >> > > > > > > test, so maybe a more solid testing framework base in a new
> >> form
> >> > > > would be
> >> > > > > > > in order. At that point it would also be good to document
> and
> >> > > educate
> >> > > > > > users
> >> > > > > > > about other test options, since OS test is a current
> favourite
> >> > for
> >> > > > > > showing
> >> > > > > > > PRs haven't broken anything. Maybe that's not the best
> choice
> >> > > anymore
> >> > > > > > with
> >> > > > > > > this information.
> >> > > > > > >
> >> > > > > > > I don't want to re-invent the wheel of what's already in
> >> > > `testsuite`,
> >> > > > > > but I
> >> > > > > > > think Unity might end up being lighter-weight in the long
> run?
> >> > And
> >> > > > it is
> >> > > > > > > also not a Xiaomi owned/maintained framework so it may
> receive
> >> > > better
> >> > > > > > > external support (i.e. more users rely on it?). I tried to
> >> > compile
> >> > > > the
> >> > > > > > > sim:citest configuration to run the testsuite example and
> see
> >> > what
> >> > > > it was
> >> > > > > > > like, but I was met with a long list of C++ compilation
> errors
> >> > that
> >> > > > I'd
> >> > > > > > > rather not deal with. There also seems to be zero docs
> about it
> >> > or
> >> > > > cmocka
> >> > > > > > > in our NuttX docs at the moment.
> >> > > > > > >
> >> > > > > > > What do you think? Start a Unity test suite, add some
> switches
> >> to
> >> > > > choose
> >> > > > > > > which test cases get compiled into the binary, open for
> >> > extensions
> >> > > by
> >> > > > > > users
> >> > > > > > > who know more about different subsystems? This way we can
> keep
> >> OS
> >> > > > test
> >> > > > > > how
> >> > > > > > > it is for a quick, low-resource sanity check of "board
> bringup"
> >> > > while
> >> > > > > > > having a dedicated testing application for patches? Someone
> >> > > changing
> >> > > > file
> >> > > > > > > system stuff can just compile to include file system test
> cases
> >> > and
> >> > > > run
> >> > > > > > > that suite only as proof that their PR works. The output is
> >> > really
> >> > > > easy
> >> > > > > > to
> >> > > > > > > glance at to determine pass/fail results.
> >> > > > > > >
> >> > > > > > > Anyways, here's my draft demonstrating how Unity works with
> the
> >> > FPU
> >> > > > test
> >> > > > > > > suite from OS test:
> >> > https://github.com/apache/nuttx-apps/pull/3259
> >> > > > > > >
> >> > > > > > > Matteo
> >> > > > > > >
> >> > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <
> tomek@cedro.info>
> >> > > > wrote:
> >> > > > > > >
> >> > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> >> > spudaneco@gmail.com
> >> > > >
> >> > > > > > wrote:
> >> > > > > > >>> (..)
> >> > > > > > >>> Starting with bits and pieces of something like the LTS
> might
> >> > > make
> >> > > > more
> >> > > > > > >>> sense other than its licensing problems. Since it would
> >> never
> >> > be
> >> > > > > > >>> included in a distributed binary, the licensing really
> should
> >> > not
> >> > > > > > matter
> >> > > > > > >>> much in practice.
> >> > > > > > >> Unless its GPL and some company would like to have their
> own
> >> > > > internal
> >> > > > > > >> bits and pieces.. luckily its MIT and the whole thing is
> >> > > open-source
> >> > > > > > >> anyways :-)
> >> > > > > > >>
> >> > > > > > >> Yes we had many discussions about the LTS and this could be
> >> > great
> >> > > > start!
> >> > > > > > >> :-)
> >> > > > > > >>
> >> > > > > > >> --
> >> > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >> > > > > > >>
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
>
Re: OS Test re-factor proposal
Posted by kr...@kerogit.eu.
I was wondering if ostest is even supposed to work on AVR so I gave it a
try... it didn't do much, the only result I got when I ran ostest from
NSH was:
stdio_test: write fd=1
stdio_test: write fd=2
And that's it.
(There may be something wrong with printf when the application gets
executed through NSH though - just noticed that my other testing apps
also print nothing when executed from NSH while being able to printf
when executed as init entrypoint. Will try to update apps to something
more recent but if anyone has any idea what may be causing this, it
would be appreciated.)
Anyway, the compilation also throws a lot of warnings, for example
semtimed.c:87:32: warning: integer overflow in expression of type ‘int’
results in ‘16960’ [-Woverflow]
87 | tp->tv_nsec >= 1000 * 1000 * 1000)
and there are also warnings regarding to size mismatch in
pointer/integer conversion.
So it may be that ostest doesn't work well for AVR (and probably for
other 8 bit MCUs) already. It also already barely fits the flash of the
largest AVR DA chip.
(Not suggesting to make it even more faulty, just mentioning it FYI.)
On 2025-12-22 13:35, Alan C. Assis wrote:
> Is it too complicated to rewrite the cmocka tests in Unity?
>
> If cmoka is not feasible for small microcontrollers, like AVRs and
> others,
> our test coverage will be faulty.
>
> BR,
>
> Alan
>
> On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <ra...@gmail.com>
> wrote:
>
>> I'm a fan of Unity and have been using it in my projects for years,
>> but I
>> think
>> we should stick with one testing framework. Currently, Cmocka is used
>> in
>> nuttx-apps, and we should stick with it. The only question is how well
>> Cmocka
>> is suitable for running on small systems? Will ostest still be able to
>> run
>> on small MCUs?
>> With Unity this would be quite likely, so maybe an exception for Unity
>> and
>> ostest would
>> be acceptable, but using it for other test cases will only cause
>> chaos.
>>
>> pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com> napisał(a):
>>
>> > Hi Matteo,
>> >
>> > Good idea !
>> >
>> > But it is best to stick with a single testing framework. Currently, many
>> > test cases are using cmocka, so I suggest reusing cmocka for consistency.
>> >
>> > ostest is a minimal basic test suite, typically used for CT (continuous
>> > testing) of each patch.
>> > If we move ostest to cmocka, it would make ostest dependent on a specific
>> > framework before it can be run.
>> > And It may increase the code size, which is not very friendly for
>> > configurations with small flash memory.
>> >
>> > As Greg mentioned: the answer depends on a tradeoff between effort and
>> test
>> > coverage.
>> >
>> > We need to clarify whether we define ostest as a lightweight,
>> ready-to-run
>> > test set for quick verification, or as part of the full test suite.
>> >
>> >
>> > The follow is my compiling for cmocka:
>> >
>> > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh sim:citest
>> -j8*
>> > Copy files
>> > Select CONFIG_HOST_LINUX=y
>> > Refreshing...
>> > CP: arch/dummy/Kconfig to
>> > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
>> > CP: boards/dummy/Kconfig to
>> > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
>> > LN: include/arch to arch/sim/include
>> > LN: drivers/platform to /home/ligd/platform/mainline/nuttx/drivers/dummy
>> > LN: arch/sim/src/chip to
>> > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
>> > LN: arch/sim/src/board to
>> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
>> > LN: include/arch/board to
>> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
>> > LN: include/arch/chip to
>> > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
>> > LN: platform/board to /home/ligd/platform/mainline/apps/platform/dummy
>> > mkkconfig in /home/ligd/platform/mainline/apps/audioutils
>> > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
>> > mkkconfig in /home/ligd/platform/mainline/apps/boot
>> > mkkconfig in /home/ligd/platform/mainline/apps/canutils
>> > mkkconfig in /home/ligd/platform/mainline/apps/crypto
>> > mkkconfig in /home/ligd/platform/mainline/apps/database
>> > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
>> > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest
>> > mkkconfig in /home/ligd/platform/mainline/apps/examples/module
>> > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot
>> > mkkconfig in /home/ligd/platform/mainline/apps/examples
>> > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
>> > mkkconfig in /home/ligd/platform/mainline/apps/games
>> > mkkconfig in /home/ligd/platform/mainline/apps/graphics
>> > mkkconfig in /home/ligd/platform/mainline/apps/industry
>> > mkkconfig in /home/ligd/platform/mainline/apps/inertial
>> > mkkconfig in /home/ligd/platform/mainline/apps/interpreters/luamodules
>> > mkkconfig in /home/ligd/platform/mainline/apps/interpreters
>> > mkkconfig in /home/ligd/platform/mainline/apps/logging
>> > mkkconfig in /home/ligd/platform/mainline/apps/lte
>> > mkkconfig in /home/ligd/platform/mainline/apps/math
>> > mkkconfig in /home/ligd/platform/mainline/apps/mlearning
>> > mkkconfig in /home/ligd/platform/mainline/apps/netutils
>> > mkkconfig in /home/ligd/platform/mainline/apps/sdr
>> > mkkconfig in /home/ligd/platform/mainline/apps/system
>> > mkkconfig in /home/ligd/platform/mainline/apps/tee
>> > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers
>> > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
>> > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
>> > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
>> > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
>> > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
>> > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
>> > mkkconfig in /home/ligd/platform/mainline/apps/testing
>> > mkkconfig in /home/ligd/platform/mainline/apps/videoutils
>> > mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth
>> > mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154
>> > mkkconfig in /home/ligd/platform/mainline/apps/wireless
>> > mkkconfig in /home/ligd/platform/mainline/apps
>> > Loaded configuration '.config'
>> > Configuration saved to '.config'
>> > ligd@BAI:~/platform/mainline/nuttx$
>> > *ligd@BAI:~/platform/mainline/nuttx$ make*
>> > Create version.h
>> > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
>> > /home/ligd/platform/mainline/apps/platform/dummy
>> > Downloading: v1.46.0.zip
>> > ...
>> > patching file
>> >
>> >
>> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
>> > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
>> >
>> > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
>> > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
>> > LD: nuttx
>> > Pac SIM with dynamic libs..
>> > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
>> > '/lib/x86_64-linux-gnu/libstdc++.so.6' -> 'sim-pac/libs/libstdc++.so.6'
>> > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
>> > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
>> > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> 'sim-pac/libs/libgcc_s.so.1'
>> > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
>> > SIM elf with dynamic libs archive in nuttx.tgz
>> > ligd@BAI:~/platform/mainline/nuttx$
>> > ligd@BAI:~/platform/mainline/nuttx$
>> > ligd@BAI:~/platform/mainline/nuttx$
>> > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
>> > CHelloWorld: Constructor: mSecret=42
>> > nsh: init: open failed: 2
>> >
>> > NuttShell (NSH) NuttX-12.11.0
>> > MOTD: username=admin password=Administrator
>> > nsh>
>> > nsh> help
>> > help usage: help [-v] [<cmd>]
>> >
>> > . df ifconfig mkfifo readlink umount
>> > [ dmesg ifdown mkrd rm unset
>> > ? echo ifup mount rmdir uptime
>> > alias env insmod mv rmmod usleep
>> > unalias exec kill nslookup set watch
>> > basename exit pkill pidof sleep xd
>> > break expr losetup pmconfig source wait
>> > cat false ln poweroff test
>> > cd fdinfo ls quit time
>> > cp free lsmod printf true
>> > cmp help mkdir ps truncate
>> > dirname hexdump mkfatfs pwd uname
>> >
>> > Builtin Apps:
>> > cmocka
>> > cmocka_driver_block
>> > cmocka_driver_gpio
>> > cmocka_driver_pm
>> > cmocka_driver_pm_runtime
>> > cmocka_driver_regulator
>> >
>> > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一 10:15写道：
>> >
>> > > I think so. Best to leave OS test alone and slowly replace it with a
>> new
>> > > tool that has improvements. The only reason I can see to modify OS test
>> > now
>> > > would just be to have clearer output about what passed/failed.
>> > >
>> > > I'll come up with an improved proof of concept for a new tool to share
>> in
>> > > the next few days.
>> > >
>> > > Matteo
>> > >
>> > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <to...@cedro.info> wrote:
>> > >
>> > > > Small steps with measurable results may be the best approach here, to
>> > > > create some new solution next to existing ostest but not replacing it
>> > > > until ready? :-)
>> > > >
>> > > > --
>> > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>> > > >
>> > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <matteo.golin@gmail.com
>> >
>> > > > wrote:
>> > > > >
>> > > > > I agree, I think it would take a long time to have proper coverage
>> of
>> > > > NuttX
>> > > > > as a whole. Perhaps even impossible, since Unity is more of a unit
>> > > > testing
>> > > > > framework and thus there are definitely test cases for NuttX that
>> > would
>> > > > be
>> > > > > outside of its realm of "applicability". There is still a need for
>> a
>> > > > > variety of testing applications and bench-marking applications for
>> > > those
>> > > > > scenarios.
>> > > > >
>> > > > > That makes sense, prerequisites would need to be considered. I
>> think
>> > > > > targeting the low-hanging fruit of unit-testable code would be easy
>> > in
>> > > > that
>> > > > > case, since prerequisites are mostly encoded in the Kconfig
>> > definitions
>> > > > > existing or not for features to be tested. Any complex behaviour
>> > > > > (interrupts, multi-threading, etc.) might be out of the realm of
>> > what's
>> > > > > testable by Unity in the first place. We'll always have OS test in
>> > that
>> > > > > case, but maybe then it's better to do what Alan suggested and at
>> > least
>> > > > > make its output more clear in the meantime. I suppose RTOS testing
>> > is a
>> > > > > whole field in itself.
>> > > > >
>> > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <sp...@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > > I think that the answer depends on a tradeoff between effort and
>> > test
>> > > > > > coverage. I could imagine that a proper job could be man-years
>> of
>> > > > effort.
>> > > > > >
>> > > > > > My only concern with a re-design is to be certain that tests are
>> > > > > > arranged so that the prerequisites of later test are tested
>> first.
>> > > If
>> > > > > > tests are grouped by, for example, functional area, then failures
>> > can
>> > > > be
>> > > > > > difficult to understand if there is a failure in an unverified
>> test
>> > > > > > prerequisite.
>> > > > > >
>> > > > > > Some of the "tests" are bogus and only exist to provide some
>> > minimum
>> > > > > > confidence that we have the functionality to run the test. For
>> > > > > > example: Can I start tasks? Can I pass parameters? Can I
>> survive
>> > a
>> > > > > > context switch. Some of the tests were intended only to support
>> > > debug.
>> > > > > > Some were created only to track down specific bugs.
>> > > > > >
>> > > > > > On 12/21/25 18:39, Matteo Golin wrote:
>> > > > > > > LTS being the Long-Term-Stable release of NuttX?
>> > > > > > >
>> > > > > > > I'm starting to think that it would be best to create new suite
>> > of
>> > > > tests
>> > > > > > > using Unity, using what's in OS Test as a basis and then
>> > expanding
>> > > it
>> > > > > > > further. Greg has pointed out that OS test is not intended to
>> be
>> > a
>> > > > > > complete
>> > > > > > > test, so maybe a more solid testing framework base in a new
>> form
>> > > > would be
>> > > > > > > in order. At that point it would also be good to document and
>> > > educate
>> > > > > > users
>> > > > > > > about other test options, since OS test is a current favourite
>> > for
>> > > > > > showing
>> > > > > > > PRs haven't broken anything. Maybe that's not the best choice
>> > > anymore
>> > > > > > with
>> > > > > > > this information.
>> > > > > > >
>> > > > > > > I don't want to re-invent the wheel of what's already in
>> > > `testsuite`,
>> > > > > > but I
>> > > > > > > think Unity might end up being lighter-weight in the long run?
>> > And
>> > > > it is
>> > > > > > > also not a Xiaomi owned/maintained framework so it may receive
>> > > better
>> > > > > > > external support (i.e. more users rely on it?). I tried to
>> > compile
>> > > > the
>> > > > > > > sim:citest configuration to run the testsuite example and see
>> > what
>> > > > it was
>> > > > > > > like, but I was met with a long list of C++ compilation errors
>> > that
>> > > > I'd
>> > > > > > > rather not deal with. There also seems to be zero docs about it
>> > or
>> > > > cmocka
>> > > > > > > in our NuttX docs at the moment.
>> > > > > > >
>> > > > > > > What do you think? Start a Unity test suite, add some switches
>> to
>> > > > choose
>> > > > > > > which test cases get compiled into the binary, open for
>> > extensions
>> > > by
>> > > > > > users
>> > > > > > > who know more about different subsystems? This way we can keep
>> OS
>> > > > test
>> > > > > > how
>> > > > > > > it is for a quick, low-resource sanity check of "board bringup"
>> > > while
>> > > > > > > having a dedicated testing application for patches? Someone
>> > > changing
>> > > > file
>> > > > > > > system stuff can just compile to include file system test cases
>> > and
>> > > > run
>> > > > > > > that suite only as proof that their PR works. The output is
>> > really
>> > > > easy
>> > > > > > to
>> > > > > > > glance at to determine pass/fail results.
>> > > > > > >
>> > > > > > > Anyways, here's my draft demonstrating how Unity works with the
>> > FPU
>> > > > test
>> > > > > > > suite from OS test:
>> > https://github.com/apache/nuttx-apps/pull/3259
>> > > > > > >
>> > > > > > > Matteo
>> > > > > > >
>> > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <to...@cedro.info>
>> > > > wrote:
>> > > > > > >
>> > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
>> > spudaneco@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > > >>> (..)
>> > > > > > >>> Starting with bits and pieces of something like the LTS might
>> > > make
>> > > > more
>> > > > > > >>> sense other than its licensing problems. Since it would
>> never
>> > be
>> > > > > > >>> included in a distributed binary, the licensing really should
>> > not
>> > > > > > matter
>> > > > > > >>> much in practice.
>> > > > > > >> Unless its GPL and some company would like to have their own
>> > > > internal
>> > > > > > >> bits and pieces.. luckily its MIT and the whole thing is
>> > > open-source
>> > > > > > >> anyways :-)
>> > > > > > >>
>> > > > > > >> Yes we had many discussions about the LTS and this could be
>> > great
>> > > > start!
>> > > > > > >> :-)
>> > > > > > >>
>> > > > > > >> --
>> > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>> > > > > > >>
>> > > > > >
>> > > >
>> > >
>> >
>>
Re: OS Test re-factor proposal
Posted by "Alan C. Assis" <ac...@gmail.com>.
Is it too complicated to rewrite the cmocka tests in Unity?
If cmoka is not feasible for small microcontrollers, like AVRs and others,
our test coverage will be faulty.
BR,
Alan
On Mon, Dec 22, 2025 at 6:14 AM raiden00pl <ra...@gmail.com> wrote:
> I'm a fan of Unity and have been using it in my projects for years, but I
> think
> we should stick with one testing framework. Currently, Cmocka is used in
> nuttx-apps, and we should stick with it. The only question is how well
> Cmocka
> is suitable for running on small systems? Will ostest still be able to run
> on small MCUs?
> With Unity this would be quite likely, so maybe an exception for Unity and
> ostest would
> be acceptable, but using it for other test cases will only cause chaos.
>
> pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com> napisał(a):
>
> > Hi Matteo,
> >
> > Good idea !
> >
> > But it is best to stick with a single testing framework. Currently, many
> > test cases are using cmocka, so I suggest reusing cmocka for consistency.
> >
> > ostest is a minimal basic test suite, typically used for CT (continuous
> > testing) of each patch.
> > If we move ostest to cmocka, it would make ostest dependent on a specific
> > framework before it can be run.
> > And It may increase the code size, which is not very friendly for
> > configurations with small flash memory.
> >
> > As Greg mentioned: the answer depends on a tradeoff between effort and
> test
> > coverage.
> >
> > We need to clarify whether we define ostest as a lightweight,
> ready-to-run
> > test set for quick verification, or as part of the full test suite.
> >
> >
> > The follow is my compiling for cmocka:
> >
> > *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh sim:citest
> -j8*
> > Copy files
> > Select CONFIG_HOST_LINUX=y
> > Refreshing...
> > CP: arch/dummy/Kconfig to
> > /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> > CP: boards/dummy/Kconfig to
> > /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> > LN: include/arch to arch/sim/include
> > LN: drivers/platform to /home/ligd/platform/mainline/nuttx/drivers/dummy
> > LN: arch/sim/src/chip to
> > /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> > LN: arch/sim/src/board to
> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> > LN: include/arch/board to
> > /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> > LN: include/arch/chip to
> > /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> > LN: platform/board to /home/ligd/platform/mainline/apps/platform/dummy
> > mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> > mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> > mkkconfig in /home/ligd/platform/mainline/apps/boot
> > mkkconfig in /home/ligd/platform/mainline/apps/canutils
> > mkkconfig in /home/ligd/platform/mainline/apps/crypto
> > mkkconfig in /home/ligd/platform/mainline/apps/database
> > mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> > mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest
> > mkkconfig in /home/ligd/platform/mainline/apps/examples/module
> > mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot
> > mkkconfig in /home/ligd/platform/mainline/apps/examples
> > mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> > mkkconfig in /home/ligd/platform/mainline/apps/games
> > mkkconfig in /home/ligd/platform/mainline/apps/graphics
> > mkkconfig in /home/ligd/platform/mainline/apps/industry
> > mkkconfig in /home/ligd/platform/mainline/apps/inertial
> > mkkconfig in /home/ligd/platform/mainline/apps/interpreters/luamodules
> > mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> > mkkconfig in /home/ligd/platform/mainline/apps/logging
> > mkkconfig in /home/ligd/platform/mainline/apps/lte
> > mkkconfig in /home/ligd/platform/mainline/apps/math
> > mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> > mkkconfig in /home/ligd/platform/mainline/apps/netutils
> > mkkconfig in /home/ligd/platform/mainline/apps/sdr
> > mkkconfig in /home/ligd/platform/mainline/apps/system
> > mkkconfig in /home/ligd/platform/mainline/apps/tee
> > mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers
> > mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> > mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> > mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> > mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> > mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> > mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> > mkkconfig in /home/ligd/platform/mainline/apps/testing
> > mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> > mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth
> > mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154
> > mkkconfig in /home/ligd/platform/mainline/apps/wireless
> > mkkconfig in /home/ligd/platform/mainline/apps
> > Loaded configuration '.config'
> > Configuration saved to '.config'
> > ligd@BAI:~/platform/mainline/nuttx$
> > *ligd@BAI:~/platform/mainline/nuttx$ make*
> > Create version.h
> > Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> > /home/ligd/platform/mainline/apps/platform/dummy
> > Downloading: v1.46.0.zip
> > ...
> > patching file
> >
> >
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> > Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
> >
> > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> > CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> > LD: nuttx
> > Pac SIM with dynamic libs..
> > '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> > '/lib/x86_64-linux-gnu/libstdc++.so.6' -> 'sim-pac/libs/libstdc++.so.6'
> > '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> > '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> > '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> 'sim-pac/libs/libgcc_s.so.1'
> > '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
> > SIM elf with dynamic libs archive in nuttx.tgz
> > ligd@BAI:~/platform/mainline/nuttx$
> > ligd@BAI:~/platform/mainline/nuttx$
> > ligd@BAI:~/platform/mainline/nuttx$
> > *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> > CHelloWorld: Constructor: mSecret=42
> > nsh: init: open failed: 2
> >
> > NuttShell (NSH) NuttX-12.11.0
> > MOTD: username=admin password=Administrator
> > nsh>
> > nsh> help
> > help usage: help [-v] [<cmd>]
> >
> > . df ifconfig mkfifo readlink umount
> > [ dmesg ifdown mkrd rm unset
> > ? echo ifup mount rmdir uptime
> > alias env insmod mv rmmod usleep
> > unalias exec kill nslookup set watch
> > basename exit pkill pidof sleep xd
> > break expr losetup pmconfig source wait
> > cat false ln poweroff test
> > cd fdinfo ls quit time
> > cp free lsmod printf true
> > cmp help mkdir ps truncate
> > dirname hexdump mkfatfs pwd uname
> >
> > Builtin Apps:
> > cmocka
> > cmocka_driver_block
> > cmocka_driver_gpio
> > cmocka_driver_pm
> > cmocka_driver_pm_runtime
> > cmocka_driver_regulator
> >
> > Matteo Golin <ma...@gmail.com> 于2025年12月22日周一 10:15写道：
> >
> > > I think so. Best to leave OS test alone and slowly replace it with a
> new
> > > tool that has improvements. The only reason I can see to modify OS test
> > now
> > > would just be to have clearer output about what passed/failed.
> > >
> > > I'll come up with an improved proof of concept for a new tool to share
> in
> > > the next few days.
> > >
> > > Matteo
> > >
> > > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <to...@cedro.info> wrote:
> > >
> > > > Small steps with measurable results may be the best approach here, to
> > > > create some new solution next to existing ostest but not replacing it
> > > > until ready? :-)
> > > >
> > > > --
> > > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > >
> > > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <matteo.golin@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > I agree, I think it would take a long time to have proper coverage
> of
> > > > NuttX
> > > > > as a whole. Perhaps even impossible, since Unity is more of a unit
> > > > testing
> > > > > framework and thus there are definitely test cases for NuttX that
> > would
> > > > be
> > > > > outside of its realm of "applicability". There is still a need for
> a
> > > > > variety of testing applications and bench-marking applications for
> > > those
> > > > > scenarios.
> > > > >
> > > > > That makes sense, prerequisites would need to be considered. I
> think
> > > > > targeting the low-hanging fruit of unit-testable code would be easy
> > in
> > > > that
> > > > > case, since prerequisites are mostly encoded in the Kconfig
> > definitions
> > > > > existing or not for features to be tested. Any complex behaviour
> > > > > (interrupts, multi-threading, etc.) might be out of the realm of
> > what's
> > > > > testable by Unity in the first place. We'll always have OS test in
> > that
> > > > > case, but maybe then it's better to do what Alan suggested and at
> > least
> > > > > make its output more clear in the meantime. I suppose RTOS testing
> > is a
> > > > > whole field in itself.
> > > > >
> > > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <sp...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > I think that the answer depends on a tradeoff between effort and
> > test
> > > > > > coverage. I could imagine that a proper job could be man-years
> of
> > > > effort.
> > > > > >
> > > > > > My only concern with a re-design is to be certain that tests are
> > > > > > arranged so that the prerequisites of later test are tested
> first.
> > > If
> > > > > > tests are grouped by, for example, functional area, then failures
> > can
> > > > be
> > > > > > difficult to understand if there is a failure in an unverified
> test
> > > > > > prerequisite.
> > > > > >
> > > > > > Some of the "tests" are bogus and only exist to provide some
> > minimum
> > > > > > confidence that we have the functionality to run the test. For
> > > > > > example: Can I start tasks? Can I pass parameters? Can I
> survive
> > a
> > > > > > context switch. Some of the tests were intended only to support
> > > debug.
> > > > > > Some were created only to track down specific bugs.
> > > > > >
> > > > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > > > LTS being the Long-Term-Stable release of NuttX?
> > > > > > >
> > > > > > > I'm starting to think that it would be best to create new suite
> > of
> > > > tests
> > > > > > > using Unity, using what's in OS Test as a basis and then
> > expanding
> > > it
> > > > > > > further. Greg has pointed out that OS test is not intended to
> be
> > a
> > > > > > complete
> > > > > > > test, so maybe a more solid testing framework base in a new
> form
> > > > would be
> > > > > > > in order. At that point it would also be good to document and
> > > educate
> > > > > > users
> > > > > > > about other test options, since OS test is a current favourite
> > for
> > > > > > showing
> > > > > > > PRs haven't broken anything. Maybe that's not the best choice
> > > anymore
> > > > > > with
> > > > > > > this information.
> > > > > > >
> > > > > > > I don't want to re-invent the wheel of what's already in
> > > `testsuite`,
> > > > > > but I
> > > > > > > think Unity might end up being lighter-weight in the long run?
> > And
> > > > it is
> > > > > > > also not a Xiaomi owned/maintained framework so it may receive
> > > better
> > > > > > > external support (i.e. more users rely on it?). I tried to
> > compile
> > > > the
> > > > > > > sim:citest configuration to run the testsuite example and see
> > what
> > > > it was
> > > > > > > like, but I was met with a long list of C++ compilation errors
> > that
> > > > I'd
> > > > > > > rather not deal with. There also seems to be zero docs about it
> > or
> > > > cmocka
> > > > > > > in our NuttX docs at the moment.
> > > > > > >
> > > > > > > What do you think? Start a Unity test suite, add some switches
> to
> > > > choose
> > > > > > > which test cases get compiled into the binary, open for
> > extensions
> > > by
> > > > > > users
> > > > > > > who know more about different subsystems? This way we can keep
> OS
> > > > test
> > > > > > how
> > > > > > > it is for a quick, low-resource sanity check of "board bringup"
> > > while
> > > > > > > having a dedicated testing application for patches? Someone
> > > changing
> > > > file
> > > > > > > system stuff can just compile to include file system test cases
> > and
> > > > run
> > > > > > > that suite only as proof that their PR works. The output is
> > really
> > > > easy
> > > > > > to
> > > > > > > glance at to determine pass/fail results.
> > > > > > >
> > > > > > > Anyways, here's my draft demonstrating how Unity works with the
> > FPU
> > > > test
> > > > > > > suite from OS test:
> > https://github.com/apache/nuttx-apps/pull/3259
> > > > > > >
> > > > > > > Matteo
> > > > > > >
> > > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <to...@cedro.info>
> > > > wrote:
> > > > > > >
> > > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> > spudaneco@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >>> (..)
> > > > > > >>> Starting with bits and pieces of something like the LTS might
> > > make
> > > > more
> > > > > > >>> sense other than its licensing problems. Since it would
> never
> > be
> > > > > > >>> included in a distributed binary, the licensing really should
> > not
> > > > > > matter
> > > > > > >>> much in practice.
> > > > > > >> Unless its GPL and some company would like to have their own
> > > > internal
> > > > > > >> bits and pieces.. luckily its MIT and the whole thing is
> > > open-source
> > > > > > >> anyways :-)
> > > > > > >>
> > > > > > >> Yes we had many discussions about the LTS and this could be
> > great
> > > > start!
> > > > > > >> :-)
> > > > > > >>
> > > > > > >> --
> > > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > > >>
> > > > > >
> > > >
> > >
> >
>
Re: OS Test re-factor proposal
Posted by raiden00pl <ra...@gmail.com>.
I'm a fan of Unity and have been using it in my projects for years, but I
think
we should stick with one testing framework. Currently, Cmocka is used in
nuttx-apps, and we should stick with it. The only question is how well
Cmocka
is suitable for running on small systems? Will ostest still be able to run
on small MCUs?
With Unity this would be quite likely, so maybe an exception for Unity and
ostest would
be acceptable, but using it for other test cases will only cause chaos.
pon., 22 gru 2025 o 10:05 Guiding Li <li...@gmail.com> napisał(a):
> Hi Matteo,
>
> Good idea !
>
> But it is best to stick with a single testing framework. Currently, many
> test cases are using cmocka, so I suggest reusing cmocka for consistency.
>
> ostest is a minimal basic test suite, typically used for CT (continuous
> testing) of each patch.
> If we move ostest to cmocka, it would make ostest dependent on a specific
> framework before it can be run.
> And It may increase the code size, which is not very friendly for
> configurations with small flash memory.
>
> As Greg mentioned: the answer depends on a tradeoff between effort and test
> coverage.
>
> We need to clarify whether we define ostest as a lightweight, ready-to-run
> test set for quick verification, or as part of the full test suite.
>
>
> The follow is my compiling for cmocka:
>
> *ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh sim:citest -j8*
> Copy files
> Select CONFIG_HOST_LINUX=y
> Refreshing...
> CP: arch/dummy/Kconfig to
> /home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
> CP: boards/dummy/Kconfig to
> /home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
> LN: include/arch to arch/sim/include
> LN: drivers/platform to /home/ligd/platform/mainline/nuttx/drivers/dummy
> LN: arch/sim/src/chip to
> /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
> LN: arch/sim/src/board to
> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
> LN: include/arch/board to
> /home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
> LN: include/arch/chip to
> /home/ligd/platform/mainline/nuttx/arch/sim/include/sim
> LN: platform/board to /home/ligd/platform/mainline/apps/platform/dummy
> mkkconfig in /home/ligd/platform/mainline/apps/audioutils
> mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
> mkkconfig in /home/ligd/platform/mainline/apps/boot
> mkkconfig in /home/ligd/platform/mainline/apps/canutils
> mkkconfig in /home/ligd/platform/mainline/apps/crypto
> mkkconfig in /home/ligd/platform/mainline/apps/database
> mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
> mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest
> mkkconfig in /home/ligd/platform/mainline/apps/examples/module
> mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot
> mkkconfig in /home/ligd/platform/mainline/apps/examples
> mkkconfig in /home/ligd/platform/mainline/apps/fsutils
> mkkconfig in /home/ligd/platform/mainline/apps/games
> mkkconfig in /home/ligd/platform/mainline/apps/graphics
> mkkconfig in /home/ligd/platform/mainline/apps/industry
> mkkconfig in /home/ligd/platform/mainline/apps/inertial
> mkkconfig in /home/ligd/platform/mainline/apps/interpreters/luamodules
> mkkconfig in /home/ligd/platform/mainline/apps/interpreters
> mkkconfig in /home/ligd/platform/mainline/apps/logging
> mkkconfig in /home/ligd/platform/mainline/apps/lte
> mkkconfig in /home/ligd/platform/mainline/apps/math
> mkkconfig in /home/ligd/platform/mainline/apps/mlearning
> mkkconfig in /home/ligd/platform/mainline/apps/netutils
> mkkconfig in /home/ligd/platform/mainline/apps/sdr
> mkkconfig in /home/ligd/platform/mainline/apps/system
> mkkconfig in /home/ligd/platform/mainline/apps/tee
> mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers
> mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
> mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
> mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
> mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
> mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
> mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
> mkkconfig in /home/ligd/platform/mainline/apps/testing
> mkkconfig in /home/ligd/platform/mainline/apps/videoutils
> mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth
> mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154
> mkkconfig in /home/ligd/platform/mainline/apps/wireless
> mkkconfig in /home/ligd/platform/mainline/apps
> Loaded configuration '.config'
> Configuration saved to '.config'
> ligd@BAI:~/platform/mainline/nuttx$
> *ligd@BAI:~/platform/mainline/nuttx$ make*
> Create version.h
> Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
> /home/ligd/platform/mainline/apps/platform/dummy
> Downloading: v1.46.0.zip
> ...
> patching file
>
> testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
> Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
>
> CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
> CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
> LD: nuttx
> Pac SIM with dynamic libs..
> '/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
> '/lib/x86_64-linux-gnu/libstdc++.so.6' -> 'sim-pac/libs/libstdc++.so.6'
> '/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
> '/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
> '/lib/x86_64-linux-gnu/libgcc_s.so.1' -> 'sim-pac/libs/libgcc_s.so.1'
> '/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
> SIM elf with dynamic libs archive in nuttx.tgz
> ligd@BAI:~/platform/mainline/nuttx$
> ligd@BAI:~/platform/mainline/nuttx$
> ligd@BAI:~/platform/mainline/nuttx$
> *ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
> CHelloWorld: Constructor: mSecret=42
> nsh: init: open failed: 2
>
> NuttShell (NSH) NuttX-12.11.0
> MOTD: username=admin password=Administrator
> nsh>
> nsh> help
> help usage: help [-v] [<cmd>]
>
> . df ifconfig mkfifo readlink umount
> [ dmesg ifdown mkrd rm unset
> ? echo ifup mount rmdir uptime
> alias env insmod mv rmmod usleep
> unalias exec kill nslookup set watch
> basename exit pkill pidof sleep xd
> break expr losetup pmconfig source wait
> cat false ln poweroff test
> cd fdinfo ls quit time
> cp free lsmod printf true
> cmp help mkdir ps truncate
> dirname hexdump mkfatfs pwd uname
>
> Builtin Apps:
> cmocka
> cmocka_driver_block
> cmocka_driver_gpio
> cmocka_driver_pm
> cmocka_driver_pm_runtime
> cmocka_driver_regulator
>
> Matteo Golin <ma...@gmail.com> 于2025年12月22日周一 10:15写道：
>
> > I think so. Best to leave OS test alone and slowly replace it with a new
> > tool that has improvements. The only reason I can see to modify OS test
> now
> > would just be to have clearer output about what passed/failed.
> >
> > I'll come up with an improved proof of concept for a new tool to share in
> > the next few days.
> >
> > Matteo
> >
> > On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <to...@cedro.info> wrote:
> >
> > > Small steps with measurable results may be the best approach here, to
> > > create some new solution next to existing ostest but not replacing it
> > > until ready? :-)
> > >
> > > --
> > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > >
> > > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <ma...@gmail.com>
> > > wrote:
> > > >
> > > > I agree, I think it would take a long time to have proper coverage of
> > > NuttX
> > > > as a whole. Perhaps even impossible, since Unity is more of a unit
> > > testing
> > > > framework and thus there are definitely test cases for NuttX that
> would
> > > be
> > > > outside of its realm of "applicability". There is still a need for a
> > > > variety of testing applications and bench-marking applications for
> > those
> > > > scenarios.
> > > >
> > > > That makes sense, prerequisites would need to be considered. I think
> > > > targeting the low-hanging fruit of unit-testable code would be easy
> in
> > > that
> > > > case, since prerequisites are mostly encoded in the Kconfig
> definitions
> > > > existing or not for features to be tested. Any complex behaviour
> > > > (interrupts, multi-threading, etc.) might be out of the realm of
> what's
> > > > testable by Unity in the first place. We'll always have OS test in
> that
> > > > case, but maybe then it's better to do what Alan suggested and at
> least
> > > > make its output more clear in the meantime. I suppose RTOS testing
> is a
> > > > whole field in itself.
> > > >
> > > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <sp...@gmail.com>
> > > wrote:
> > > >
> > > > > I think that the answer depends on a tradeoff between effort and
> test
> > > > > coverage. I could imagine that a proper job could be man-years of
> > > effort.
> > > > >
> > > > > My only concern with a re-design is to be certain that tests are
> > > > > arranged so that the prerequisites of later test are tested first.
> > If
> > > > > tests are grouped by, for example, functional area, then failures
> can
> > > be
> > > > > difficult to understand if there is a failure in an unverified test
> > > > > prerequisite.
> > > > >
> > > > > Some of the "tests" are bogus and only exist to provide some
> minimum
> > > > > confidence that we have the functionality to run the test. For
> > > > > example: Can I start tasks? Can I pass parameters? Can I survive
> a
> > > > > context switch. Some of the tests were intended only to support
> > debug.
> > > > > Some were created only to track down specific bugs.
> > > > >
> > > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > > LTS being the Long-Term-Stable release of NuttX?
> > > > > >
> > > > > > I'm starting to think that it would be best to create new suite
> of
> > > tests
> > > > > > using Unity, using what's in OS Test as a basis and then
> expanding
> > it
> > > > > > further. Greg has pointed out that OS test is not intended to be
> a
> > > > > complete
> > > > > > test, so maybe a more solid testing framework base in a new form
> > > would be
> > > > > > in order. At that point it would also be good to document and
> > educate
> > > > > users
> > > > > > about other test options, since OS test is a current favourite
> for
> > > > > showing
> > > > > > PRs haven't broken anything. Maybe that's not the best choice
> > anymore
> > > > > with
> > > > > > this information.
> > > > > >
> > > > > > I don't want to re-invent the wheel of what's already in
> > `testsuite`,
> > > > > but I
> > > > > > think Unity might end up being lighter-weight in the long run?
> And
> > > it is
> > > > > > also not a Xiaomi owned/maintained framework so it may receive
> > better
> > > > > > external support (i.e. more users rely on it?). I tried to
> compile
> > > the
> > > > > > sim:citest configuration to run the testsuite example and see
> what
> > > it was
> > > > > > like, but I was met with a long list of C++ compilation errors
> that
> > > I'd
> > > > > > rather not deal with. There also seems to be zero docs about it
> or
> > > cmocka
> > > > > > in our NuttX docs at the moment.
> > > > > >
> > > > > > What do you think? Start a Unity test suite, add some switches to
> > > choose
> > > > > > which test cases get compiled into the binary, open for
> extensions
> > by
> > > > > users
> > > > > > who know more about different subsystems? This way we can keep OS
> > > test
> > > > > how
> > > > > > it is for a quick, low-resource sanity check of "board bringup"
> > while
> > > > > > having a dedicated testing application for patches? Someone
> > changing
> > > file
> > > > > > system stuff can just compile to include file system test cases
> and
> > > run
> > > > > > that suite only as proof that their PR works. The output is
> really
> > > easy
> > > > > to
> > > > > > glance at to determine pass/fail results.
> > > > > >
> > > > > > Anyways, here's my draft demonstrating how Unity works with the
> FPU
> > > test
> > > > > > suite from OS test:
> https://github.com/apache/nuttx-apps/pull/3259
> > > > > >
> > > > > > Matteo
> > > > > >
> > > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <to...@cedro.info>
> > > wrote:
> > > > > >
> > > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <
> spudaneco@gmail.com
> > >
> > > > > wrote:
> > > > > >>> (..)
> > > > > >>> Starting with bits and pieces of something like the LTS might
> > make
> > > more
> > > > > >>> sense other than its licensing problems. Since it would never
> be
> > > > > >>> included in a distributed binary, the licensing really should
> not
> > > > > matter
> > > > > >>> much in practice.
> > > > > >> Unless its GPL and some company would like to have their own
> > > internal
> > > > > >> bits and pieces.. luckily its MIT and the whole thing is
> > open-source
> > > > > >> anyways :-)
> > > > > >>
> > > > > >> Yes we had many discussions about the LTS and this could be
> great
> > > start!
> > > > > >> :-)
> > > > > >>
> > > > > >> --
> > > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > > >>
> > > > >
> > >
> >
>
Re: OS Test re-factor proposal
Posted by Guiding Li <li...@gmail.com>.
Hi Matteo,
Good idea !
But it is best to stick with a single testing framework. Currently, many
test cases are using cmocka, so I suggest reusing cmocka for consistency.
ostest is a minimal basic test suite, typically used for CT (continuous
testing) of each patch.
If we move ostest to cmocka, it would make ostest dependent on a specific
framework before it can be run.
And It may increase the code size, which is not very friendly for
configurations with small flash memory.
As Greg mentioned: the answer depends on a tradeoff between effort and test
coverage.
We need to clarify whether we define ostest as a lightweight, ready-to-run
test set for quick verification, or as part of the full test suite.
The follow is my compiling for cmocka:
*ligd@BAI:~/platform/mainline/nuttx$ ./tools/configure.sh sim:citest -j8*
Copy files
Select CONFIG_HOST_LINUX=y
Refreshing...
CP: arch/dummy/Kconfig to
/home/ligd/platform/mainline/nuttx/arch/dummy/dummy_kconfig
CP: boards/dummy/Kconfig to
/home/ligd/platform/mainline/nuttx/boards/dummy/dummy_kconfig
LN: include/arch to arch/sim/include
LN: drivers/platform to /home/ligd/platform/mainline/nuttx/drivers/dummy
LN: arch/sim/src/chip to /home/ligd/platform/mainline/nuttx/arch/sim/src/sim
LN: arch/sim/src/board to
/home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/src
LN: include/arch/board to
/home/ligd/platform/mainline/nuttx/boards/sim/sim/sim/include
LN: include/arch/chip to
/home/ligd/platform/mainline/nuttx/arch/sim/include/sim
LN: platform/board to /home/ligd/platform/mainline/apps/platform/dummy
mkkconfig in /home/ligd/platform/mainline/apps/audioutils
mkkconfig in /home/ligd/platform/mainline/apps/benchmarks
mkkconfig in /home/ligd/platform/mainline/apps/boot
mkkconfig in /home/ligd/platform/mainline/apps/canutils
mkkconfig in /home/ligd/platform/mainline/apps/crypto
mkkconfig in /home/ligd/platform/mainline/apps/database
mkkconfig in /home/ligd/platform/mainline/apps/examples/rust
mkkconfig in /home/ligd/platform/mainline/apps/examples/sotest
mkkconfig in /home/ligd/platform/mainline/apps/examples/module
mkkconfig in /home/ligd/platform/mainline/apps/examples/mcuboot
mkkconfig in /home/ligd/platform/mainline/apps/examples
mkkconfig in /home/ligd/platform/mainline/apps/fsutils
mkkconfig in /home/ligd/platform/mainline/apps/games
mkkconfig in /home/ligd/platform/mainline/apps/graphics
mkkconfig in /home/ligd/platform/mainline/apps/industry
mkkconfig in /home/ligd/platform/mainline/apps/inertial
mkkconfig in /home/ligd/platform/mainline/apps/interpreters/luamodules
mkkconfig in /home/ligd/platform/mainline/apps/interpreters
mkkconfig in /home/ligd/platform/mainline/apps/logging
mkkconfig in /home/ligd/platform/mainline/apps/lte
mkkconfig in /home/ligd/platform/mainline/apps/math
mkkconfig in /home/ligd/platform/mainline/apps/mlearning
mkkconfig in /home/ligd/platform/mainline/apps/netutils
mkkconfig in /home/ligd/platform/mainline/apps/sdr
mkkconfig in /home/ligd/platform/mainline/apps/system
mkkconfig in /home/ligd/platform/mainline/apps/tee
mkkconfig in /home/ligd/platform/mainline/apps/testing/drivers
mkkconfig in /home/ligd/platform/mainline/apps/testing/sched
mkkconfig in /home/ligd/platform/mainline/apps/testing/fs
mkkconfig in /home/ligd/platform/mainline/apps/testing/mm
mkkconfig in /home/ligd/platform/mainline/apps/testing/libc
mkkconfig in /home/ligd/platform/mainline/apps/testing/arch
mkkconfig in /home/ligd/platform/mainline/apps/testing/cxx
mkkconfig in /home/ligd/platform/mainline/apps/testing
mkkconfig in /home/ligd/platform/mainline/apps/videoutils
mkkconfig in /home/ligd/platform/mainline/apps/wireless/bluetooth
mkkconfig in /home/ligd/platform/mainline/apps/wireless/ieee802154
mkkconfig in /home/ligd/platform/mainline/apps/wireless
mkkconfig in /home/ligd/platform/mainline/apps
Loaded configuration '.config'
Configuration saved to '.config'
ligd@BAI:~/platform/mainline/nuttx$
*ligd@BAI:~/platform/mainline/nuttx$ make*
Create version.h
Downloading: libcxx-17.0.6.src.tar.xz LN: platform/board to
/home/ligd/platform/mainline/apps/platform/dummy
Downloading: v1.46.0.zip
...
patching file
testcases/open_posix_testsuite/conformance/interfaces/pthread_sigmask/18-1.c
Hunk #1 succeeded at 160 with fuzz 2 (offset -8 lines).
CP: /home/ligd/platform/mainline/nuttx/include/nuttx/config.h
CP: /home/ligd/platform/mainline/nuttx/include/nuttx/fs/hostfs.h
LD: nuttx
Pac SIM with dynamic libs..
'/lib/x86_64-linux-gnu/libz.so.1' -> 'sim-pac/libs/libz.so.1'
'/lib/x86_64-linux-gnu/libstdc++.so.6' -> 'sim-pac/libs/libstdc++.so.6'
'/lib/x86_64-linux-gnu/libc.so.6' -> 'sim-pac/libs/libc.so.6'
'/lib/x86_64-linux-gnu/libm.so.6' -> 'sim-pac/libs/libm.so.6'
'/lib/x86_64-linux-gnu/libgcc_s.so.1' -> 'sim-pac/libs/libgcc_s.so.1'
'/lib64/ld-linux-x86-64.so.2' -> 'sim-pac/ld-linux-x86-64.so.2'
SIM elf with dynamic libs archive in nuttx.tgz
ligd@BAI:~/platform/mainline/nuttx$
ligd@BAI:~/platform/mainline/nuttx$
ligd@BAI:~/platform/mainline/nuttx$
*ligd@BAI:~/platform/mainline/nuttx$ ./nuttx *
CHelloWorld: Constructor: mSecret=42
nsh: init: open failed: 2
NuttShell (NSH) NuttX-12.11.0
MOTD: username=admin password=Administrator
nsh>
nsh> help
help usage: help [-v] [<cmd>]
. df ifconfig mkfifo readlink umount
[ dmesg ifdown mkrd rm unset
? echo ifup mount rmdir uptime
alias env insmod mv rmmod usleep
unalias exec kill nslookup set watch
basename exit pkill pidof sleep xd
break expr losetup pmconfig source wait
cat false ln poweroff test
cd fdinfo ls quit time
cp free lsmod printf true
cmp help mkdir ps truncate
dirname hexdump mkfatfs pwd uname
Builtin Apps:
cmocka
cmocka_driver_block
cmocka_driver_gpio
cmocka_driver_pm
cmocka_driver_pm_runtime
cmocka_driver_regulator
Matteo Golin <ma...@gmail.com> 于2025年12月22日周一 10:15写道：
> I think so. Best to leave OS test alone and slowly replace it with a new
> tool that has improvements. The only reason I can see to modify OS test now
> would just be to have clearer output about what passed/failed.
>
> I'll come up with an improved proof of concept for a new tool to share in
> the next few days.
>
> Matteo
>
> On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <to...@cedro.info> wrote:
>
> > Small steps with measurable results may be the best approach here, to
> > create some new solution next to existing ostest but not replacing it
> > until ready? :-)
> >
> > --
> > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >
> > On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <ma...@gmail.com>
> > wrote:
> > >
> > > I agree, I think it would take a long time to have proper coverage of
> > NuttX
> > > as a whole. Perhaps even impossible, since Unity is more of a unit
> > testing
> > > framework and thus there are definitely test cases for NuttX that would
> > be
> > > outside of its realm of "applicability". There is still a need for a
> > > variety of testing applications and bench-marking applications for
> those
> > > scenarios.
> > >
> > > That makes sense, prerequisites would need to be considered. I think
> > > targeting the low-hanging fruit of unit-testable code would be easy in
> > that
> > > case, since prerequisites are mostly encoded in the Kconfig definitions
> > > existing or not for features to be tested. Any complex behaviour
> > > (interrupts, multi-threading, etc.) might be out of the realm of what's
> > > testable by Unity in the first place. We'll always have OS test in that
> > > case, but maybe then it's better to do what Alan suggested and at least
> > > make its output more clear in the meantime. I suppose RTOS testing is a
> > > whole field in itself.
> > >
> > > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <sp...@gmail.com>
> > wrote:
> > >
> > > > I think that the answer depends on a tradeoff between effort and test
> > > > coverage. I could imagine that a proper job could be man-years of
> > effort.
> > > >
> > > > My only concern with a re-design is to be certain that tests are
> > > > arranged so that the prerequisites of later test are tested first.
> If
> > > > tests are grouped by, for example, functional area, then failures can
> > be
> > > > difficult to understand if there is a failure in an unverified test
> > > > prerequisite.
> > > >
> > > > Some of the "tests" are bogus and only exist to provide some minimum
> > > > confidence that we have the functionality to run the test. For
> > > > example: Can I start tasks? Can I pass parameters? Can I survive a
> > > > context switch. Some of the tests were intended only to support
> debug.
> > > > Some were created only to track down specific bugs.
> > > >
> > > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > > LTS being the Long-Term-Stable release of NuttX?
> > > > >
> > > > > I'm starting to think that it would be best to create new suite of
> > tests
> > > > > using Unity, using what's in OS Test as a basis and then expanding
> it
> > > > > further. Greg has pointed out that OS test is not intended to be a
> > > > complete
> > > > > test, so maybe a more solid testing framework base in a new form
> > would be
> > > > > in order. At that point it would also be good to document and
> educate
> > > > users
> > > > > about other test options, since OS test is a current favourite for
> > > > showing
> > > > > PRs haven't broken anything. Maybe that's not the best choice
> anymore
> > > > with
> > > > > this information.
> > > > >
> > > > > I don't want to re-invent the wheel of what's already in
> `testsuite`,
> > > > but I
> > > > > think Unity might end up being lighter-weight in the long run? And
> > it is
> > > > > also not a Xiaomi owned/maintained framework so it may receive
> better
> > > > > external support (i.e. more users rely on it?). I tried to compile
> > the
> > > > > sim:citest configuration to run the testsuite example and see what
> > it was
> > > > > like, but I was met with a long list of C++ compilation errors that
> > I'd
> > > > > rather not deal with. There also seems to be zero docs about it or
> > cmocka
> > > > > in our NuttX docs at the moment.
> > > > >
> > > > > What do you think? Start a Unity test suite, add some switches to
> > choose
> > > > > which test cases get compiled into the binary, open for extensions
> by
> > > > users
> > > > > who know more about different subsystems? This way we can keep OS
> > test
> > > > how
> > > > > it is for a quick, low-resource sanity check of "board bringup"
> while
> > > > > having a dedicated testing application for patches? Someone
> changing
> > file
> > > > > system stuff can just compile to include file system test cases and
> > run
> > > > > that suite only as proof that their PR works. The output is really
> > easy
> > > > to
> > > > > glance at to determine pass/fail results.
> > > > >
> > > > > Anyways, here's my draft demonstrating how Unity works with the FPU
> > test
> > > > > suite from OS test: https://github.com/apache/nuttx-apps/pull/3259
> > > > >
> > > > > Matteo
> > > > >
> > > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <to...@cedro.info>
> > wrote:
> > > > >
> > > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <spudaneco@gmail.com
> >
> > > > wrote:
> > > > >>> (..)
> > > > >>> Starting with bits and pieces of something like the LTS might
> make
> > more
> > > > >>> sense other than its licensing problems. Since it would never be
> > > > >>> included in a distributed binary, the licensing really should not
> > > > matter
> > > > >>> much in practice.
> > > > >> Unless its GPL and some company would like to have their own
> > internal
> > > > >> bits and pieces.. luckily its MIT and the whole thing is
> open-source
> > > > >> anyways :-)
> > > > >>
> > > > >> Yes we had many discussions about the LTS and this could be great
> > start!
> > > > >> :-)
> > > > >>
> > > > >> --
> > > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > > >>
> > > >
> >
>
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
I think so. Best to leave OS test alone and slowly replace it with a new
tool that has improvements. The only reason I can see to modify OS test now
would just be to have clearer output about what passed/failed.
I'll come up with an improved proof of concept for a new tool to share in
the next few days.
Matteo
On Sun, Dec 21, 2025, 9:13 PM Tomek CEDRO <to...@cedro.info> wrote:
> Small steps with measurable results may be the best approach here, to
> create some new solution next to existing ostest but not replacing it
> until ready? :-)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>
> On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <ma...@gmail.com>
> wrote:
> >
> > I agree, I think it would take a long time to have proper coverage of
> NuttX
> > as a whole. Perhaps even impossible, since Unity is more of a unit
> testing
> > framework and thus there are definitely test cases for NuttX that would
> be
> > outside of its realm of "applicability". There is still a need for a
> > variety of testing applications and bench-marking applications for those
> > scenarios.
> >
> > That makes sense, prerequisites would need to be considered. I think
> > targeting the low-hanging fruit of unit-testable code would be easy in
> that
> > case, since prerequisites are mostly encoded in the Kconfig definitions
> > existing or not for features to be tested. Any complex behaviour
> > (interrupts, multi-threading, etc.) might be out of the realm of what's
> > testable by Unity in the first place. We'll always have OS test in that
> > case, but maybe then it's better to do what Alan suggested and at least
> > make its output more clear in the meantime. I suppose RTOS testing is a
> > whole field in itself.
> >
> > On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <sp...@gmail.com>
> wrote:
> >
> > > I think that the answer depends on a tradeoff between effort and test
> > > coverage. I could imagine that a proper job could be man-years of
> effort.
> > >
> > > My only concern with a re-design is to be certain that tests are
> > > arranged so that the prerequisites of later test are tested first. If
> > > tests are grouped by, for example, functional area, then failures can
> be
> > > difficult to understand if there is a failure in an unverified test
> > > prerequisite.
> > >
> > > Some of the "tests" are bogus and only exist to provide some minimum
> > > confidence that we have the functionality to run the test. For
> > > example: Can I start tasks? Can I pass parameters? Can I survive a
> > > context switch. Some of the tests were intended only to support debug.
> > > Some were created only to track down specific bugs.
> > >
> > > On 12/21/25 18:39, Matteo Golin wrote:
> > > > LTS being the Long-Term-Stable release of NuttX?
> > > >
> > > > I'm starting to think that it would be best to create new suite of
> tests
> > > > using Unity, using what's in OS Test as a basis and then expanding it
> > > > further. Greg has pointed out that OS test is not intended to be a
> > > complete
> > > > test, so maybe a more solid testing framework base in a new form
> would be
> > > > in order. At that point it would also be good to document and educate
> > > users
> > > > about other test options, since OS test is a current favourite for
> > > showing
> > > > PRs haven't broken anything. Maybe that's not the best choice anymore
> > > with
> > > > this information.
> > > >
> > > > I don't want to re-invent the wheel of what's already in `testsuite`,
> > > but I
> > > > think Unity might end up being lighter-weight in the long run? And
> it is
> > > > also not a Xiaomi owned/maintained framework so it may receive better
> > > > external support (i.e. more users rely on it?). I tried to compile
> the
> > > > sim:citest configuration to run the testsuite example and see what
> it was
> > > > like, but I was met with a long list of C++ compilation errors that
> I'd
> > > > rather not deal with. There also seems to be zero docs about it or
> cmocka
> > > > in our NuttX docs at the moment.
> > > >
> > > > What do you think? Start a Unity test suite, add some switches to
> choose
> > > > which test cases get compiled into the binary, open for extensions by
> > > users
> > > > who know more about different subsystems? This way we can keep OS
> test
> > > how
> > > > it is for a quick, low-resource sanity check of "board bringup" while
> > > > having a dedicated testing application for patches? Someone changing
> file
> > > > system stuff can just compile to include file system test cases and
> run
> > > > that suite only as proof that their PR works. The output is really
> easy
> > > to
> > > > glance at to determine pass/fail results.
> > > >
> > > > Anyways, here's my draft demonstrating how Unity works with the FPU
> test
> > > > suite from OS test: https://github.com/apache/nuttx-apps/pull/3259
> > > >
> > > > Matteo
> > > >
> > > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <to...@cedro.info>
> wrote:
> > > >
> > > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <sp...@gmail.com>
> > > wrote:
> > > >>> (..)
> > > >>> Starting with bits and pieces of something like the LTS might make
> more
> > > >>> sense other than its licensing problems. Since it would never be
> > > >>> included in a distributed binary, the licensing really should not
> > > matter
> > > >>> much in practice.
> > > >> Unless its GPL and some company would like to have their own
> internal
> > > >> bits and pieces.. luckily its MIT and the whole thing is open-source
> > > >> anyways :-)
> > > >>
> > > >> Yes we had many discussions about the LTS and this could be great
> start!
> > > >> :-)
> > > >>
> > > >> --
> > > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > > >>
> > >
>
Re: OS Test re-factor proposal
Posted by Tomek CEDRO <to...@cedro.info>.
Small steps with measurable results may be the best approach here, to
create some new solution next to existing ostest but not replacing it
until ready? :-)
--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
On Mon, Dec 22, 2025 at 3:06 AM Matteo Golin <ma...@gmail.com> wrote:
>
> I agree, I think it would take a long time to have proper coverage of NuttX
> as a whole. Perhaps even impossible, since Unity is more of a unit testing
> framework and thus there are definitely test cases for NuttX that would be
> outside of its realm of "applicability". There is still a need for a
> variety of testing applications and bench-marking applications for those
> scenarios.
>
> That makes sense, prerequisites would need to be considered. I think
> targeting the low-hanging fruit of unit-testable code would be easy in that
> case, since prerequisites are mostly encoded in the Kconfig definitions
> existing or not for features to be tested. Any complex behaviour
> (interrupts, multi-threading, etc.) might be out of the realm of what's
> testable by Unity in the first place. We'll always have OS test in that
> case, but maybe then it's better to do what Alan suggested and at least
> make its output more clear in the meantime. I suppose RTOS testing is a
> whole field in itself.
>
> On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <sp...@gmail.com> wrote:
>
> > I think that the answer depends on a tradeoff between effort and test
> > coverage. I could imagine that a proper job could be man-years of effort.
> >
> > My only concern with a re-design is to be certain that tests are
> > arranged so that the prerequisites of later test are tested first. If
> > tests are grouped by, for example, functional area, then failures can be
> > difficult to understand if there is a failure in an unverified test
> > prerequisite.
> >
> > Some of the "tests" are bogus and only exist to provide some minimum
> > confidence that we have the functionality to run the test. For
> > example: Can I start tasks? Can I pass parameters? Can I survive a
> > context switch. Some of the tests were intended only to support debug.
> > Some were created only to track down specific bugs.
> >
> > On 12/21/25 18:39, Matteo Golin wrote:
> > > LTS being the Long-Term-Stable release of NuttX?
> > >
> > > I'm starting to think that it would be best to create new suite of tests
> > > using Unity, using what's in OS Test as a basis and then expanding it
> > > further. Greg has pointed out that OS test is not intended to be a
> > complete
> > > test, so maybe a more solid testing framework base in a new form would be
> > > in order. At that point it would also be good to document and educate
> > users
> > > about other test options, since OS test is a current favourite for
> > showing
> > > PRs haven't broken anything. Maybe that's not the best choice anymore
> > with
> > > this information.
> > >
> > > I don't want to re-invent the wheel of what's already in `testsuite`,
> > but I
> > > think Unity might end up being lighter-weight in the long run? And it is
> > > also not a Xiaomi owned/maintained framework so it may receive better
> > > external support (i.e. more users rely on it?). I tried to compile the
> > > sim:citest configuration to run the testsuite example and see what it was
> > > like, but I was met with a long list of C++ compilation errors that I'd
> > > rather not deal with. There also seems to be zero docs about it or cmocka
> > > in our NuttX docs at the moment.
> > >
> > > What do you think? Start a Unity test suite, add some switches to choose
> > > which test cases get compiled into the binary, open for extensions by
> > users
> > > who know more about different subsystems? This way we can keep OS test
> > how
> > > it is for a quick, low-resource sanity check of "board bringup" while
> > > having a dedicated testing application for patches? Someone changing file
> > > system stuff can just compile to include file system test cases and run
> > > that suite only as proof that their PR works. The output is really easy
> > to
> > > glance at to determine pass/fail results.
> > >
> > > Anyways, here's my draft demonstrating how Unity works with the FPU test
> > > suite from OS test: https://github.com/apache/nuttx-apps/pull/3259
> > >
> > > Matteo
> > >
> > > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <to...@cedro.info> wrote:
> > >
> > >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <sp...@gmail.com>
> > wrote:
> > >>> (..)
> > >>> Starting with bits and pieces of something like the LTS might make more
> > >>> sense other than its licensing problems. Since it would never be
> > >>> included in a distributed binary, the licensing really should not
> > matter
> > >>> much in practice.
> > >> Unless its GPL and some company would like to have their own internal
> > >> bits and pieces.. luckily its MIT and the whole thing is open-source
> > >> anyways :-)
> > >>
> > >> Yes we had many discussions about the LTS and this could be great start!
> > >> :-)
> > >>
> > >> --
> > >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > >>
> >
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
I agree, I think it would take a long time to have proper coverage of NuttX
as a whole. Perhaps even impossible, since Unity is more of a unit testing
framework and thus there are definitely test cases for NuttX that would be
outside of its realm of "applicability". There is still a need for a
variety of testing applications and bench-marking applications for those
scenarios.
That makes sense, prerequisites would need to be considered. I think
targeting the low-hanging fruit of unit-testable code would be easy in that
case, since prerequisites are mostly encoded in the Kconfig definitions
existing or not for features to be tested. Any complex behaviour
(interrupts, multi-threading, etc.) might be out of the realm of what's
testable by Unity in the first place. We'll always have OS test in that
case, but maybe then it's better to do what Alan suggested and at least
make its output more clear in the meantime. I suppose RTOS testing is a
whole field in itself.
On Sun, Dec 21, 2025 at 8:40 PM Gregory Nutt <sp...@gmail.com> wrote:
> I think that the answer depends on a tradeoff between effort and test
> coverage. I could imagine that a proper job could be man-years of effort.
>
> My only concern with a re-design is to be certain that tests are
> arranged so that the prerequisites of later test are tested first. If
> tests are grouped by, for example, functional area, then failures can be
> difficult to understand if there is a failure in an unverified test
> prerequisite.
>
> Some of the "tests" are bogus and only exist to provide some minimum
> confidence that we have the functionality to run the test. For
> example: Can I start tasks? Can I pass parameters? Can I survive a
> context switch. Some of the tests were intended only to support debug.
> Some were created only to track down specific bugs.
>
> On 12/21/25 18:39, Matteo Golin wrote:
> > LTS being the Long-Term-Stable release of NuttX?
> >
> > I'm starting to think that it would be best to create new suite of tests
> > using Unity, using what's in OS Test as a basis and then expanding it
> > further. Greg has pointed out that OS test is not intended to be a
> complete
> > test, so maybe a more solid testing framework base in a new form would be
> > in order. At that point it would also be good to document and educate
> users
> > about other test options, since OS test is a current favourite for
> showing
> > PRs haven't broken anything. Maybe that's not the best choice anymore
> with
> > this information.
> >
> > I don't want to re-invent the wheel of what's already in `testsuite`,
> but I
> > think Unity might end up being lighter-weight in the long run? And it is
> > also not a Xiaomi owned/maintained framework so it may receive better
> > external support (i.e. more users rely on it?). I tried to compile the
> > sim:citest configuration to run the testsuite example and see what it was
> > like, but I was met with a long list of C++ compilation errors that I'd
> > rather not deal with. There also seems to be zero docs about it or cmocka
> > in our NuttX docs at the moment.
> >
> > What do you think? Start a Unity test suite, add some switches to choose
> > which test cases get compiled into the binary, open for extensions by
> users
> > who know more about different subsystems? This way we can keep OS test
> how
> > it is for a quick, low-resource sanity check of "board bringup" while
> > having a dedicated testing application for patches? Someone changing file
> > system stuff can just compile to include file system test cases and run
> > that suite only as proof that their PR works. The output is really easy
> to
> > glance at to determine pass/fail results.
> >
> > Anyways, here's my draft demonstrating how Unity works with the FPU test
> > suite from OS test: https://github.com/apache/nuttx-apps/pull/3259
> >
> > Matteo
> >
> > On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <to...@cedro.info> wrote:
> >
> >> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <sp...@gmail.com>
> wrote:
> >>> (..)
> >>> Starting with bits and pieces of something like the LTS might make more
> >>> sense other than its licensing problems. Since it would never be
> >>> included in a distributed binary, the licensing really should not
> matter
> >>> much in practice.
> >> Unless its GPL and some company would like to have their own internal
> >> bits and pieces.. luckily its MIT and the whole thing is open-source
> >> anyways :-)
> >>
> >> Yes we had many discussions about the LTS and this could be great start!
> >> :-)
> >>
> >> --
> >> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >>
>
Re: OS Test re-factor proposal
Posted by Gregory Nutt <sp...@gmail.com>.
I think that the answer depends on a tradeoff between effort and test
coverage. I could imagine that a proper job could be man-years of effort.
My only concern with a re-design is to be certain that tests are
arranged so that the prerequisites of later test are tested first. If
tests are grouped by, for example, functional area, then failures can be
difficult to understand if there is a failure in an unverified test
prerequisite.
Some of the "tests" are bogus and only exist to provide some minimum
confidence that we have the functionality to run the test. For
example: Can I start tasks? Can I pass parameters? Can I survive a
context switch. Some of the tests were intended only to support debug.
Some were created only to track down specific bugs.
On 12/21/25 18:39, Matteo Golin wrote:
> LTS being the Long-Term-Stable release of NuttX?
>
> I'm starting to think that it would be best to create new suite of tests
> using Unity, using what's in OS Test as a basis and then expanding it
> further. Greg has pointed out that OS test is not intended to be a complete
> test, so maybe a more solid testing framework base in a new form would be
> in order. At that point it would also be good to document and educate users
> about other test options, since OS test is a current favourite for showing
> PRs haven't broken anything. Maybe that's not the best choice anymore with
> this information.
>
> I don't want to re-invent the wheel of what's already in `testsuite`, but I
> think Unity might end up being lighter-weight in the long run? And it is
> also not a Xiaomi owned/maintained framework so it may receive better
> external support (i.e. more users rely on it?). I tried to compile the
> sim:citest configuration to run the testsuite example and see what it was
> like, but I was met with a long list of C++ compilation errors that I'd
> rather not deal with. There also seems to be zero docs about it or cmocka
> in our NuttX docs at the moment.
>
> What do you think? Start a Unity test suite, add some switches to choose
> which test cases get compiled into the binary, open for extensions by users
> who know more about different subsystems? This way we can keep OS test how
> it is for a quick, low-resource sanity check of "board bringup" while
> having a dedicated testing application for patches? Someone changing file
> system stuff can just compile to include file system test cases and run
> that suite only as proof that their PR works. The output is really easy to
> glance at to determine pass/fail results.
>
> Anyways, here's my draft demonstrating how Unity works with the FPU test
> suite from OS test: https://github.com/apache/nuttx-apps/pull/3259
>
> Matteo
>
> On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <to...@cedro.info> wrote:
>
>> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <sp...@gmail.com> wrote:
>>> (..)
>>> Starting with bits and pieces of something like the LTS might make more
>>> sense other than its licensing problems. Since it would never be
>>> included in a distributed binary, the licensing really should not matter
>>> much in practice.
>> Unless its GPL and some company would like to have their own internal
>> bits and pieces.. luckily its MIT and the whole thing is open-source
>> anyways :-)
>>
>> Yes we had many discussions about the LTS and this could be great start!
>> :-)
>>
>> --
>> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>>
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
LTS being the Long-Term-Stable release of NuttX?
I'm starting to think that it would be best to create new suite of tests
using Unity, using what's in OS Test as a basis and then expanding it
further. Greg has pointed out that OS test is not intended to be a complete
test, so maybe a more solid testing framework base in a new form would be
in order. At that point it would also be good to document and educate users
about other test options, since OS test is a current favourite for showing
PRs haven't broken anything. Maybe that's not the best choice anymore with
this information.
I don't want to re-invent the wheel of what's already in `testsuite`, but I
think Unity might end up being lighter-weight in the long run? And it is
also not a Xiaomi owned/maintained framework so it may receive better
external support (i.e. more users rely on it?). I tried to compile the
sim:citest configuration to run the testsuite example and see what it was
like, but I was met with a long list of C++ compilation errors that I'd
rather not deal with. There also seems to be zero docs about it or cmocka
in our NuttX docs at the moment.
What do you think? Start a Unity test suite, add some switches to choose
which test cases get compiled into the binary, open for extensions by users
who know more about different subsystems? This way we can keep OS test how
it is for a quick, low-resource sanity check of "board bringup" while
having a dedicated testing application for patches? Someone changing file
system stuff can just compile to include file system test cases and run
that suite only as proof that their PR works. The output is really easy to
glance at to determine pass/fail results.
Anyways, here's my draft demonstrating how Unity works with the FPU test
suite from OS test: https://github.com/apache/nuttx-apps/pull/3259
Matteo
On Sun, Dec 21, 2025 at 7:13 PM Tomek CEDRO <to...@cedro.info> wrote:
> On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <sp...@gmail.com> wrote:
> > (..)
> > Starting with bits and pieces of something like the LTS might make more
> > sense other than its licensing problems. Since it would never be
> > included in a distributed binary, the licensing really should not matter
> > much in practice.
>
> Unless its GPL and some company would like to have their own internal
> bits and pieces.. luckily its MIT and the whole thing is open-source
> anyways :-)
>
> Yes we had many discussions about the LTS and this could be great start!
> :-)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>
Re: OS Test re-factor proposal
Posted by Tomek CEDRO <to...@cedro.info>.
On Mon, Dec 22, 2025 at 1:02 AM Gregory Nutt <sp...@gmail.com> wrote:
> (..)
> Starting with bits and pieces of something like the LTS might make more
> sense other than its licensing problems. Since it would never be
> included in a distributed binary, the licensing really should not matter
> much in practice.
Unless its GPL and some company would like to have their own internal
bits and pieces.. luckily its MIT and the whole thing is open-source
anyways :-)
Yes we had many discussions about the LTS and this could be great start! :-)
--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
Re: OS Test re-factor proposal
Posted by Gregory Nutt <sp...@gmail.com>.
Just a not about the history of OS test...
OS test was never designed to be a complete functional test. It was
developed originally as a small test tool to help me develop and debug
new OS features. It tests only basic functionality and would require a
LOT or work to become credible test of the OS with good feature coverage.
These little test tools were accumulated into one big test program to
check for regressions and became a substantial but poorly designed
program. It is still useful for regression testing if your test
expectations are low.
Starting with bits and pieces of something like the LTS might make more
sense other than its licensing problems. Since it would never be
included in a distributed binary, the licensing really should not matter
much in practice.
Re: OS Test re-factor proposal
Posted by Tomek CEDRO <to...@cedro.info>.
Question 3 was about test-sets/suites that you already mentioned thanks! :-)
Yes some simple existing ostest would be great as poc so we can compare :-)
I can see cmocka framework already present in the nuttx-apps how would
unity relate to that is it something different or similar? :-)
https://github.com/apache/nuttx-apps/tree/master/testing/cmocka
Thanks! :-)
Tomek
--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
On Sun, Dec 21, 2025 at 9:14 PM Matteo Golin <ma...@gmail.com> wrote:
>
> 1. From my understanding, very stable. I also believe that we fix the
> version in NuttX so we'd be in control when/if we upgrade the version used.
>
> 2. Yes, it has PASS/FAIL for certain. I'm not sure about SKIP, but I can
> check that. I also seem to remember a feature where you can choose to run
> single tests, suites or all test cases with names matching a certain
> pattern, but don't quote me on that. I will refer back to their docs and
> send an update here for those questions.
>
> 3. Can you clarify what you mean?
>
> I could definitely make a proof of concept to demonstrate first. Would you
> suggest just a dummy application with fake tests, or creating a new demo
> app where a few test cases from OS test are adapted to use Unity?
>
> Matteo
>
> On Sun, Dec 21, 2025, 3:06 PM Tomek CEDRO <to...@cedro.info> wrote:
>
> > Would it be possible to create some proof-of-concept to show how it
> > would look and work? :-)
> >
> > --
> > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >
> > On Sun, Dec 21, 2025 at 8:58 PM Tomek CEDRO <to...@cedro.info> wrote:
> > >
> > > Sounds great Linguini :-)
> > >
> > > 1. How stable is the API of that Unity framework? Can this impact
> > > self-containment or long term self compatibility problems?
> > >
> > > 2. Would that allow marking status of each step like FAIL, PASS, SKIP,
> > > ERROR? We miss that now, only errors are reported but
> > >
> > > 3. Can this enable test sets?
> > >
> > > Thanks! :-)
> > > Tomek
> > >
> > > --
> > > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> > >
> > > On Sun, Dec 21, 2025 at 8:38 PM Matteo Golin <ma...@gmail.com>
> > wrote:
> > > >
> > > > Hello everyone,
> > > >
> > > > I wanted to get some feedback on my proposal for re-factor of OS test:
> > https://github.com/apache/nuttx-apps/issues/3258
> > > >
> > > > Since it's the holidays and I have some time on my hands, I figured
> > that OS test could use some improvements, especially
> > > > since it is the primary test used to check for regressions/correct
> > functionality of kernel logic on NuttX.
> > > >
> > > > My proposal basically aims to use the Unity test framework to organize
> > all of the existing tests in OS test so that
> > > > there is a logical code structure, and also logical console output
> > which makes it much clearer to tell if a test has
> > > > passed/failed. The goal is that a) this should make reviewing test
> > results much easier in PRs and b) the improved code
> > > > readability/extendability should make it easier for contributors to
> > identify gaps in the testing and also add their own
> > > > suggested tests.
> > > >
> > > > I have used the Unity support on NuttX for embedded testing for
> > rocketry and really enjoyed it; I think it would be a
> > > > huge improvement for NuttX.
> > > >
> > > > If this refactor is accepted by the community, once it is done I plan
> > on completing more documentation for the OS test
> > > > on the NuttX website. There is currently a lack of information about
> > all of the test suites that exist for the OS and
> > > > what exactly is being verified.
> > > >
> > > > Please let me know what you think!
> > > >
> > > > --
> > > > Matteo Golin
> >
Re: OS Test re-factor proposal
Posted by Matteo Golin <ma...@gmail.com>.
1. From my understanding, very stable. I also believe that we fix the
version in NuttX so we'd be in control when/if we upgrade the version used.
2. Yes, it has PASS/FAIL for certain. I'm not sure about SKIP, but I can
check that. I also seem to remember a feature where you can choose to run
single tests, suites or all test cases with names matching a certain
pattern, but don't quote me on that. I will refer back to their docs and
send an update here for those questions.
3. Can you clarify what you mean?
I could definitely make a proof of concept to demonstrate first. Would you
suggest just a dummy application with fake tests, or creating a new demo
app where a few test cases from OS test are adapted to use Unity?
Matteo
On Sun, Dec 21, 2025, 3:06 PM Tomek CEDRO <to...@cedro.info> wrote:
> Would it be possible to create some proof-of-concept to show how it
> would look and work? :-)
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>
> On Sun, Dec 21, 2025 at 8:58 PM Tomek CEDRO <to...@cedro.info> wrote:
> >
> > Sounds great Linguini :-)
> >
> > 1. How stable is the API of that Unity framework? Can this impact
> > self-containment or long term self compatibility problems?
> >
> > 2. Would that allow marking status of each step like FAIL, PASS, SKIP,
> > ERROR? We miss that now, only errors are reported but
> >
> > 3. Can this enable test sets?
> >
> > Thanks! :-)
> > Tomek
> >
> > --
> > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> >
> > On Sun, Dec 21, 2025 at 8:38 PM Matteo Golin <ma...@gmail.com>
> wrote:
> > >
> > > Hello everyone,
> > >
> > > I wanted to get some feedback on my proposal for re-factor of OS test:
> https://github.com/apache/nuttx-apps/issues/3258
> > >
> > > Since it's the holidays and I have some time on my hands, I figured
> that OS test could use some improvements, especially
> > > since it is the primary test used to check for regressions/correct
> functionality of kernel logic on NuttX.
> > >
> > > My proposal basically aims to use the Unity test framework to organize
> all of the existing tests in OS test so that
> > > there is a logical code structure, and also logical console output
> which makes it much clearer to tell if a test has
> > > passed/failed. The goal is that a) this should make reviewing test
> results much easier in PRs and b) the improved code
> > > readability/extendability should make it easier for contributors to
> identify gaps in the testing and also add their own
> > > suggested tests.
> > >
> > > I have used the Unity support on NuttX for embedded testing for
> rocketry and really enjoyed it; I think it would be a
> > > huge improvement for NuttX.
> > >
> > > If this refactor is accepted by the community, once it is done I plan
> on completing more documentation for the OS test
> > > on the NuttX website. There is currently a lack of information about
> all of the test suites that exist for the OS and
> > > what exactly is being verified.
> > >
> > > Please let me know what you think!
> > >
> > > --
> > > Matteo Golin
>
Re: OS Test re-factor proposal
Posted by Tomek CEDRO <to...@cedro.info>.
Would it be possible to create some proof-of-concept to show how it
would look and work? :-)
--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
On Sun, Dec 21, 2025 at 8:58 PM Tomek CEDRO <to...@cedro.info> wrote:
>
> Sounds great Linguini :-)
>
> 1. How stable is the API of that Unity framework? Can this impact
> self-containment or long term self compatibility problems?
>
> 2. Would that allow marking status of each step like FAIL, PASS, SKIP,
> ERROR? We miss that now, only errors are reported but
>
> 3. Can this enable test sets?
>
> Thanks! :-)
> Tomek
>
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
>
> On Sun, Dec 21, 2025 at 8:38 PM Matteo Golin <ma...@gmail.com> wrote:
> >
> > Hello everyone,
> >
> > I wanted to get some feedback on my proposal for re-factor of OS test: https://github.com/apache/nuttx-apps/issues/3258
> >
> > Since it's the holidays and I have some time on my hands, I figured that OS test could use some improvements, especially
> > since it is the primary test used to check for regressions/correct functionality of kernel logic on NuttX.
> >
> > My proposal basically aims to use the Unity test framework to organize all of the existing tests in OS test so that
> > there is a logical code structure, and also logical console output which makes it much clearer to tell if a test has
> > passed/failed. The goal is that a) this should make reviewing test results much easier in PRs and b) the improved code
> > readability/extendability should make it easier for contributors to identify gaps in the testing and also add their own
> > suggested tests.
> >
> > I have used the Unity support on NuttX for embedded testing for rocketry and really enjoyed it; I think it would be a
> > huge improvement for NuttX.
> >
> > If this refactor is accepted by the community, once it is done I plan on completing more documentation for the OS test
> > on the NuttX website. There is currently a lack of information about all of the test suites that exist for the OS and
> > what exactly is being verified.
> >
> > Please let me know what you think!
> >
> > --
> > Matteo Golin
Re: OS Test re-factor proposal
Posted by Tomek CEDRO <to...@cedro.info>.
Sounds great Linguini :-)
1. How stable is the API of that Unity framework? Can this impact
self-containment or long term self compatibility problems?
2. Would that allow marking status of each step like FAIL, PASS, SKIP,
ERROR? We miss that now, only errors are reported but
3. Can this enable test sets?
Thanks! :-)
Tomek
--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
On Sun, Dec 21, 2025 at 8:38 PM Matteo Golin <ma...@gmail.com> wrote:
>
> Hello everyone,
>
> I wanted to get some feedback on my proposal for re-factor of OS test: https://github.com/apache/nuttx-apps/issues/3258
>
> Since it's the holidays and I have some time on my hands, I figured that OS test could use some improvements, especially
> since it is the primary test used to check for regressions/correct functionality of kernel logic on NuttX.
>
> My proposal basically aims to use the Unity test framework to organize all of the existing tests in OS test so that
> there is a logical code structure, and also logical console output which makes it much clearer to tell if a test has
> passed/failed. The goal is that a) this should make reviewing test results much easier in PRs and b) the improved code
> readability/extendability should make it easier for contributors to identify gaps in the testing and also add their own
> suggested tests.
>
> I have used the Unity support on NuttX for embedded testing for rocketry and really enjoyed it; I think it would be a
> huge improvement for NuttX.
>
> If this refactor is accepted by the community, once it is done I plan on completing more documentation for the OS test
> on the NuttX website. There is currently a lack of information about all of the test suites that exist for the OS and
> what exactly is being verified.
>
> Please let me know what you think!
>
> --
> Matteo Golin