You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nuttx.apache.org by GitBox <gi...@apache.org> on 2021/04/22 14:20:02 UTC
[GitHub] [incubator-nuttx] v01d opened a new pull request #3589: Migrate build system to CMake
v01d opened a new pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589
## Summary
This is an incomplete work in progress, please do not perform a detailed review. I've been working quite hard on this last few weeks and my interest at this point is to gain visibility to how I structured this, gain some feedback to continue the work and hopefully get some help to complete migration and testing.
Note that this depends on accompanying apps/ change (which I have in my fork's cmake branch: [master..cmake compare link](https://github.com/apache/incubator-nuttx-apps/compare/master...v01d:cmake)).
I will continue migration until complete and then rebase to incorporate relevant changes that happened in between.
For colaboration, I can either accept PRs to my fork or we could setup a new branch. Either way, if someone is willing to help, we should coordinate.
## Some changes part of migration
1. Since the goal is to allow for out-of-tree builds, the context step does not create symlinks inside source tree, but in build directory. Include path is extended to expose this as expected
2. Debug/Release flags are handled via CMAKE_BUILD_TYPE as expected, not from config options
3. APPDIR is taken from cmake command line, not from config
4. Each arch should have a cmake/ directory preparing the required toolchain and relevant options
5. Some specialty functions are defined:
* nuttx_add_application: defines an application as expected. it adds an OBJECT library for each app, which is then built into the libapps.a static lib as expected
* nuttx_add_library: adds a top-level static library which is linked into final nuttx binary (libfs, libnet, etc.)
* nuttx_add_user_library: similar to add_application but not defining an application, but a "library"
6. apps/Kconfig is no longer build-time generated but it is now a static file
7. sim's up_head.c was giving some issues and realized it had a mix of host and nuttx headers. it was split into host-only + up_internal.h API and nuttx calls (which went into sim's up_syslog.c). This most likely could be backported to current NuttX if wanted.
Similar to 2 and 3, most likely options defining the host platform should not be part of config file.
## TODO
The following are not yet addressed
- two-phase build: how to handle this?
- revise toolchain handling: I based my changes on @dagar but I'm not entirely sure if there's a 1:1 arch-toolchain mapping or not
- extend to other arch/boards: only sim and one board supported right now
- verify all Make.defs/Makefile against CMakeLists.txt once all Make.defs are complete
- support custom boards (currently disabled inclusion of boards/dummy/Kconfig)
- Similar to 2 and 3 points above, remove host/toolchain definition from Kconfig, this should be a config time setting
- Remove control of optimization from Kconfig, this is handled via CMAKE_BUILD_TYPE
- Support module building (extend nuttx_add_application to take a symbol?)
- Module handling in sim build
- export: probably needs redesign using cmake, maybe even not needed?
- Add a flash target (debug also?), and other useful ones
- Avoid having to place .config on source tree (kconfig utils supposeddly takes the path to this file and the Kconfig)
- examples/bridge, examples/flowc, examples/nettest?
They include host code, this will probably require creating a subproject as we cannot have a mixed build as is.
- examples/module: builds a character device?? Is this right?
- how to support building apps with external code?
- FetchContent: if child project is CMake, it automatically configures it (not always wanted, see libaudiosrc)
- ExternalProject: delays child configure to build stage, does not allow adding sources to parent
- file(DOWNLOAD ...): manual process
- Update GitHub workflow: I've done this but I'm not sure if it works yet
- Perform extensive testing on other target and host platforms
- Document the build system under Documentation/ (design and how to add board/arch/app)
### Low-priority (could be in later PR)
- Make applications conform to <app>/include/<app> and <app>/src/ so that a general include/
is not needed (external apps can also split implementation from interface and expose the latter).
- remove unneeded support for renaming most applications, not real use, only adds Kconfig options
- do we need setting priority via Kconfig of most applications? shouldn't this be done on runtime?
## Conversion checklist:
So far I, this is how far I got the conversion, which allows for sim:nsh to build. You fill find some CMakeLists.txt there from @dagar which need adapting (you'll see I'm using `target_sources` and `CONFIG` variables are not evaluated with `STREQUAL` on the ones I converted). This is quite a tedious and manual process so it will take some time to complete (maybe someone wants to lend a hand? I can give some instructions). I will definitely need help for supporting and testing other boards and also other platforms such as mac and windows.
### Apps
- [x] apps/builtin
- [ ] apps/canutils
- [ ] apps/canutils/libcanard
- [x] rest
- [ ] apps/examples
- [ ] apps/examples/audio_rttl: handle SDK
- [ ] apps/examples/bridge
- [ ] apps/examples/elf
- [ ] apps/examples/flowc
- [ ] apps/examples/lvgldemo
- [ ] apps/examples/module
- [ ] apps/examples/nxflat: tests
- [ ] apps/examples/posix_spawn
- [ ] apps/examples/romfs
- [ ] apps/fsutils
- [ ] apps/fsutils/inih
- [x] rest
- [ ] apps/gpsutils
- [ ] apps/import
- [ ] apps/industry
- [ ] apps/interpreters
- [ ] apps/modbus
- [ ] apps/netutils
- [x] apps/nshlib
- [ ] rest
- [ ] apps/platform
- [ ] apps/system
- [x] apps/system/nsh
- [ ] rest
- [ ] apps/tools
- [ ] apps/wireless
### NuttX
- [ ] nuttx/arch
- [x] nuttx/audio
- [x] nuttx/binfmt
- [ ] nuttx/boards
- [x] nuttx/boards/sim/sim/sim (tested: nsh)
- [x] nuttx/boards/arm/stm32/stm32f103-minimum (built: nsh)
- [ ] rest
- [x] nuttx/crypto
- [x] nuttx/drivers
- [x] nuttx/fs
- [x] nuttx/graphics
- [ ] nuttx/libs
- [ ] nuttx/libs/libc
- [x] nuttx/libs/libc/audio/libsrc: testing file(DOWNLOAD), should not use master version (build issues)
- [ ] nuttx/libs/libc/zoneinfo/ (romfs)
- [x] rest
- [x] nuttx/libs/libdsp
- [ ] nuttx/libs/libnx: only need to finish font generation
- [ ] nuttx/libs/libxx
- [x] nuttx/mm
- [x] nuttx/net
- [ ] nuttx/openamp
- [x] nuttx/sched
- [ ] nuttx/syscall
- [ ] nuttx/tools: only basic binaries for now
- [x] nuttx/video
- [x] nuttx/wireless
## Impact
Major, this should probably be part of a major NuttX release.
## Testing
Basic sim:nsh on Linux.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] davids5 edited a comment on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
davids5 edited a comment on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-825168361
> @antmerlino @davids5 what are your though on the integration of NuttX with CMake as part of a larger project? Would the top-level project simply use ExternalProject to nest NuttX build system? I'm also thinking about current "export" feature of Make-based system. Do we still need this?
I think we should take it slow to not force the migration on our users base all at once.
But ultimately NuttX can be a library. We should allow the consuming project to be able to configure build options. Things like `-Werror` etc. There are also some compile time settings that need to be equivalent in both builds. Like `CONFIG_ARMV7M_STACKCHECK` -> drives `FLAGS += -finstrument-functions -ffixed-r10` [see this for example](https://github.com/PX4/PX4-Autopilot/blob/437825418297c1ff43681acc6cf870e2f51d1dcb/platforms/nuttx/NuttX/Make.defs.in#L120)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] btashton commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
btashton commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-839380351
We could just move this work to a branch which would let all the CI stuff work just like we do on release branches which might make this effort a lot easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d edited a comment on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d edited a comment on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-827992528
I made some more progress and I realize I now need to correctly handle `__KERNEL__`. So far I used this for all static libs created from top-level directories but I realize this is not correct. My understanding is that this should not be defined for subdirectories of `libs`, `mm` nor for applications. Any other OS library should have it defined. Is that correct?
I realize this ties into the two-pass build (which I haven't fully looked into yet) and kernel/protected mode. Do both of these modes use the two-pass build? Is there any documentation and how the build system works in this case? I'm looking at the makefiles but it isn't very clear to me how it all works together.
cc: @patacongo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-825074169
BTW, I'm currently trying to solve an issue with sim: my understanding is that up_head call's the host's setjmp/longjmp, but when inspecting this symbol from Make build, it appears it is renamed to NXsetjmp so it is using internal definition of setjmp. Isn't up_head.c the host interface to the simulator? Why is it using the simulated setjmp? Is the host's setjmp not usable for up_head? (I understand that the simulated nuttx internally provides its own mechanism for setjmp)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] btashton commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
btashton commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-825146205
> Also, what are your thoughts about what we should use for downloading external project built inside NuttX? For example, littlefs, lvgl, etc. I mentioned three strategies above, they each have pros/cons or at least different use cases.
I think we should try to keep this close to what we have today and then iterate on some of the more external options later. I suspect we will not be able to make use of the existing projects cmake configuration is most cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] patacongo commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
patacongo commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-828078710
> No, your understanding is not quite correct. In the FLAT build,
> only one copy of the libraries is built. The OS and the
> applications link to the same libraries. For example, libs/libc
> builds libc.a for both the applications and the OS code.
>
> Sorry, I didn't clarify: my question regarding *KERNEL* was for FLAT
> mode. I noticed looking at |LibTargets.mk| that libs, mm and libapps
> do not ever receive the KDEFINE flags, while other libs do.
>
I don't think __KERNEL__ does anything in the FLAT build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] btashton edited a comment on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
btashton edited a comment on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-839851148
> Thanks!
>
> We can merge to a branch as Brennan suggests in the meantime. That would keep that branch point stable and I will be able to test Mac as well.
>
> How do I create a branch via GitHub? Ideally I would like it to be at the branch point of the branch in my fork so CI runs OK. Is that possible?
You have commit access do you can just push the branch to the remote.
To work with the CI system in a clean way where you build with your branch on the app side I would recommend using `releases/cmake` as the branch name for both repos as that will do the correct pattern matching.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-825140778
Ok, no up_head.c is handled correctly and it solved the crash I had.
@btashton It seems cmake is not available in the docker image, can I make it available in this PR somehow? Or does this require pushing a new image? I would like to have sim:nsh on linux and macos on CI to start.
Also, I would need this to build against my fork's "cmake" apps branch. I think I have modified it to do so (there's a separate commit with that change here). Could you see if it makes sense?
@antmerlino @davids5 what are your thoughs on the integration of NuttX with CMake as part of a larger project? Would the top-level project simply use ExternalProject to nest NuttX build system? I'm also thinking about current "export" feature of Make-based system. Do we still need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] btashton commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
btashton commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-825153316
It's in the tools build image but not the final one. Only what is here is actually used in CI. This helps keep the image smaller.
https://github.com/apache/incubator-nuttx/blob/fecdd27df37baa843001b0d34cfdfd3516e96424/tools/ci/docker/linux/Dockerfile#L174
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] patacongo commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
patacongo commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-828071751
No, your understanding is not quite correct.
In the FLAT build, only one copy of the libraries is built. The OS and
the applications link to the same libraries. For example, libs/libc
builds libc.a for both the applications and the OS code.
Only a single libc.a is needed because only one "blob" is built.
In the PROTECTED and KERNEL builds, the applications and OS are
separately linked: two blobs for the PROTECTED case and many for the
KERNEL case. And two different libraries are built: libuc.a for the
user/application space blob and libkc.a for the kernel-space OS blob.
These blobs are different because applications and the OS use different
different interfaces. The application, for example, never calls
directly into the OS but gets into the OS via system calls (call gates
that switch from user to supervisor modes).
When building the application space library, libuc.a, __KERNEL__ must
not be defined. This tells the system that we are building the
application space version of the library. For libkc.a, __KERNEL__ must
be defined. This tells the build that we are building for code that
runs inside the OS.
This is done via a two pass build. For the PROTECTED mode, the
application blob is built first and linked with the system call
proxies. In the second pass, the OS blob is built and linked with the
system call stubs.
KERNEL mode is a little different because the applications are build as
executable that run from a file system, much like with Linux. Part of
the build is manual. That is description in board README's like
boards/arm/sama5/sama5d4-ek/README.txt for knsh cconfiguration.
The simulator also depends on a 2-pass build, but for different
reasons. See boards/sim REAME file.
Please be careful and don't break. This is pretty complex and delicate
logic and critical to the future of the OS.
Greg
On 4/27/2021 4:41 PM, Matias N. wrote:
>
> I made some more progress and I realize I now need to correctly handle
> |__KERNEL__|. So far I used this for all static libs created from
> top-level directories but I realize this is not correct. My
> understanding is that this should not be defined for subdirectories of
> |libs|, |mm| nor for applications. Any other OS library should have it
> defined. Is that correct?
>
> I realize this ties into the two-pass build (which I haven't fully
> looked into yet) and kernel/protected mode. Do both of these modes use
> the two-pass build? Is there any documentation and how the build
> system works in this case? I'm looking at the makefiles but it isn't
> very clear to me how it all works together.
>
> cc: @patacongo <https://github.com/patacongo>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-827992528>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABFUG6WGONHLFS4YVXSHQR3TK44PZANCNFSM43MSFWZQ>.
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-825191064
> > @antmerlino @davids5 what are your though on the integration of NuttX with CMake as part of a larger project? Would the top-level project simply use ExternalProject to nest NuttX build system? I'm also thinking about current "export" feature of Make-based system. Do we still need this?
>
> I think we should take it slow to not force the migration on our users base all at once.
You're right. I will look into adding an export target.
> But ultimately NuttX can be a library. We should allow the consuming project to be able to configure build options. Things like `-Werror` etc. There are also some compile time settings that need to be equivalent in both builds. Like `CONFIG_ARMV7M_STACKCHECK` -> drives `FLAGS += -finstrument-functions -ffixed-r10` [see this for example](https://github.com/PX4/PX4-Autopilot/blob/437825418297c1ff43681acc6cf870e2f51d1dcb/platforms/nuttx/NuttX/Make.defs.in#L120)
I'm not sure what specific things to have in mind to achieve this. But I guess we can leave it for later on once things are more settled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-825150164
> > Also, what are your thoughts about what we should use for downloading external project built inside NuttX? For example, littlefs, lvgl, etc. I mentioned three strategies above, they each have pros/cons or at least different use cases.
>
> I think we should try to keep this close to what we have today and then iterate on some of the more external options later. I suspect we will not be able to make use of the existing projects cmake configuration is most cases.
So most likely I will have to fetch manually with file(DOWNLOAD) and so on. When the child project is CMake FetchContent() forcibly runs CMake on it. I will create a wrapper function for this step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-825081367
> > BTW, I'm currently trying to solve an issue with sim: my understanding is that up_head call's the host's setjmp/longjmp, but when inspecting this symbol from Make build, it appears it is renamed to NXsetjmp so it is using internal definition of setjmp. Isn't up_head.c the host interface to the simulator? Why is it using the simulated setjmp? Is the host's setjmp not usable for up_head? (I understand that the simulated nuttx internally provides its own mechanism for setjmp)
>
> I tried that many years ago. It would crash immediately. I forget the details, but as I recall, there are a couple of issues with with the host longjmp/setjmp for this usage. For one, the stack is encrypted for security reasons. I forget why this caused the crash, try it and see.
>
> Another problem is the the GNU/Linux setjmp/longjmp is tied into Linux signal handling in a way that makes it unsuitable for use in the simulation.
>
> For the simulation, you need a simple setup/longjmp that just saves the and restores the critical registers.
Great, thanks for the explanation!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] davids5 commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
davids5 commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-839672735
@v01d - This is great progress. I have a lot going on at the moment and I will need to carve out some time to help on this over the next couple of weeks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] patacongo commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
patacongo commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-828084853
>
>
> > I don't think **KERNEL** does anything in the FLAT build.
>
> Oh ok. In my case I had an issue with these lines in `stdlib.h` when building the C++ library:
>
> ```c
> /* System() command is not implemented in the NuttX libc because it is so
> * entangled with shell logic. There is an experimental version at
> * apps/system/system. system() is prototyped here, however, for
> * standards compatibility.
> */
>
> #ifndef __KERNEL__
> int system(FAR const char *cmd);
> #endif
> ```
Yes, this is a case where __KERNEL__ "kind of" does something in the FLAT build.
system() is a standard interface defined in OpenGroup.org, but in the case of NuttX, it is NOT part of the OS. It is part of apps/ provided at apps/system/system. NuttX code should never depend on anything in apps/. That would be bad architecture: applications are built on top the OS and the OS must be independent of the application.
Linux really has the same issue, but they mitigate the coupling via the file system. /bin/sh is the shell used by system() (I think, I should check) and /bin/sh might be a symbolic link to the real shell binary.
So in this case, the __KERNEL__ conditioning is a feeble attempt at restricting things so that nothing under nuttx/ can call system() without at least a warning. But this is so feeble as to be useless and the conditioning could just as well be removed.
From what you said above, this might actually provide some protection is __KERNEL__ is defined for all OS libraries under nuttx/ and not defined for libapps.a. But it is still pretty lame.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d edited a comment on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d edited a comment on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-836749777
Last week I continued work on this and got apps/ mostly there (still some specific cases not covered yet) and other major nuttx subdirectories.
I would like to ask for some help to tackle the following major features:
- loadable module support
- protected + kernel build
- export
- support other arch's/boards
I have some progress on the loadable part but not yet completed it (at least for kernel module, loadable apps are not yet handled). I tried to look into protected + kernel build but I don't have much experience with this part so it would be great to get some help here. I haven't used the export feature either so, while it doesn't seem complex, that is another area where help would be appreciated.
For the last point, I focused my testing on sim platform (I think I can build 95% of all configs) but only verified it builds (run nsh config though). This is also only on Linux, so building on other platforms would be great. And of course, eventually the goal is to extend to other arch's.
I'm also looking for some help since right now this is not runnable in CI since GitHub rebases the PR branch to master and this breaks due to many changes from the fork point. To have it runnable it would require manually porting all changes affecting the build system. Since this becomes a larger problem every day, it would be great to get some help and complete this before going through the rebase process.
So, anyone with some knowledge of CMake and those areas of interest I mentioned? @davids5 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d edited a comment on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d edited a comment on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-825140778
Ok, now up_head.c is handled correctly and it solved the crash I had.
@btashton It seems cmake is not available in the docker image, can I make it available in this PR somehow? Or does this require pushing a new image? I would like to have sim:nsh on linux and macos on CI to start.
Also, I would need this to build against my fork's "cmake" apps branch. I think I have modified it to do so (there's a separate commit with that change here). Could you see if it makes sense?
@antmerlino @davids5 what are your thoughs on the integration of NuttX with CMake as part of a larger project? Would the top-level project simply use ExternalProject to nest NuttX build system? I'm also thinking about current "export" feature of Make-based system. Do we still need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-827992528
I made some more progress and I realize I now need to correctly handle __KERNEL__. So far I used this for all static libs created from top-level directories but I realize this is not correct. My understanding is that this should not be defined for subdirectories of `libs`, `mm` nor for applications. Any other OS library should have it defined. Is that correct?
I realize this ties into the two-pass build (which I haven't fully looked into yet) and kernel/protected mode. Do both of these modes use the two-pass build? Is there any documentation and how the build system works in this case? I'm looking at the makefiles but it isn't very clear to me how it all works together.
cc: @patacongo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d edited a comment on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d edited a comment on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-828076813
> No, your understanding is not quite correct. In the FLAT build, only one copy of the libraries is built. The OS and the applications link to the same libraries. For example, libs/libc builds libc.a for both the applications and the OS code.
Sorry, I didn't clarify: my question regarding `__KERNEL__` was for FLAT mode. I noticed looking at `LibTargets.mk` that libs, mm and libapps do not ever receive the KDEFINE flags, while other libs do.
> In the PROTECTED and KERNEL builds, the applications and OS are separately linked: two blobs for the PROTECTED case and many for the KERNEL case. And two different libraries are built: libuc.a for the user/application space blob and libkc.a for the kernel-space OS blob. These blobs are different because applications and the OS use different different interfaces. The application, for example, never calls directly into the OS but gets into the OS via system calls (call gates that switch from user to supervisor modes). When building the application space library, libuc.a, __KERNEL__ must not be defined. This tells the system that we are building the application space version of the library. For libkc.a, __KERNEL__ must be defined. This tells the build that we are building for code that runs inside the OS. This is done via a two pass build. For the PROTECTED mode, the application blob is built first and linked with the system call proxies. In the second pass, the OS blob i
s built and linked with the system call stubs. KERNEL mode is a little different because the applications are build as executable that run from a file system, much like with Linux. Part of the build is manual. That is description in board README's like boards/arm/sama5/sama5d4-ek/README.txt for knsh cconfiguration.
Thanks for the explanation, I will look further into how the build is structured with this information.
> The simulator also depends on a 2-pass build, but for different reasons. See boards/sim REAME file.
Yes, I already have the nuttx.rel + nuttx handled for sim.
> Please be careful and don't break. This is pretty complex and delicate logic and critical to the future of the OS.
Of course. This will require a great deal of testing and also manual validation of correct build flags being used. There's quite a lot to do still.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-839879623
I found an easy way to create the branch from GitHub (I have my remote set to a bogus "no_push" to avoid accidental pushes to main repo): https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-and-deleting-branches-within-your-repository
I recreated the PR in #3704
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d edited a comment on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d edited a comment on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-828079445
> I don't think __KERNEL__ does anything in the FLAT build.
Oh ok. In my case I had an issue with these lines in `stdlib.h` when building the C++ library:
```c
/* System() command is not implemented in the NuttX libc because it is so
* entangled with shell logic. There is an experimental version at
* apps/system/system. system() is prototyped here, however, for
* standards compatibility.
*/
#ifndef __KERNEL__
int system(FAR const char *cmd);
#endif
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] btashton commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
btashton commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-825143579
> @btashton It seems cmake is not available in the docker image, can I make it available in this PR somehow? Or does this require pushing a new image? I would like to have sim:nsh on linux and macos on CI to start.
Yeah just go ahead and add it the dockerfile on the master branch and create a PR. Once that is merged it will be in the image for Linux. For macOS you will need to update the cibuild.sh script but that can live on your branch.
https://github.com/apache/incubator-nuttx/blob/master/tools/ci/cibuild.sh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-825151910
> > @btashton It seems cmake is not available in the docker image, can I make it available in this PR somehow? Or does this require pushing a new image? I would like to have sim:nsh on linux and macos on CI to start.
>
> Yeah just go ahead and add it the dockerfile on the master branch and create a PR. Once that is merged it will be in the image for Linux. For macOS you will need to update the cibuild.sh script but that can live on your branch.
>
> https://github.com/apache/incubator-nuttx/blob/master/tools/ci/cibuild.sh
Actually I just checked and cmake appears to be added to the base image. However, I'm getting "command not found" for cmake (https://github.com/apache/incubator-nuttx/pull/3589/checks?check_run_id=2413832642#step:7:84).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] btashton commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
btashton commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-839851148
> Thanks!
>
> We can merge to a branch as Brennan suggests in the meantime. That would keep that branch point stable and I will be able to test Mac as well.
>
> How do I create a branch via GitHub? Ideally I would like it to be at the branch point of the branch in my fork so CI runs OK. Is that possible?
You have commit access do you can just push the branch to the remote.
To work with the CI system in a clean way where you build with your branch on the app side I would recommend using the `releases/cmake` for both repos as that will do the correct pattern matching.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-828076813
> No, your understanding is not quite correct. In the FLAT build, only one copy of the libraries is built. The OS and the applications link to the same libraries. For example, libs/libc builds libc.a for both the applications and the OS code.
Sorry, I didn't clarify: my question regarding __KERNEL__ was for FLAT mode. I noticed looking at `LibTargets.mk` that libs, mm and libapps do not ever receive the KDEFINE flags, while other libs do.
> In the PROTECTED and KERNEL builds, the applications and OS are separately linked: two blobs for the PROTECTED case and many for the KERNEL case. And two different libraries are built: libuc.a for the user/application space blob and libkc.a for the kernel-space OS blob. These blobs are different because applications and the OS use different different interfaces. The application, for example, never calls directly into the OS but gets into the OS via system calls (call gates that switch from user to supervisor modes). When building the application space library, libuc.a, __KERNEL__ must not be defined. This tells the system that we are building the application space version of the library. For libkc.a, __KERNEL__ must be defined. This tells the build that we are building for code that runs inside the OS. This is done via a two pass build. For the PROTECTED mode, the application blob is built first and linked with the system call proxies. In the second pass, the OS blob i
s built and linked with the system call stubs. KERNEL mode is a little different because the applications are build as executable that run from a file system, much like with Linux. Part of the build is manual. That is description in board README's like boards/arm/sama5/sama5d4-ek/README.txt for knsh cconfiguration.
Thanks for the explanation, I will look further into how the build is structured with this information.
> The simulator also depends on a 2-pass build, but for different reasons. See boards/sim REAME file.
Yes, I already have the nuttx.rel + nuttx handled for sim.
> Please be careful and don't break. This is pretty complex and delicate logic and critical to the future of the OS.
Of course. This will require a great deal of testing and also manual validation of correct build flags being used. There's quite a lot to do still.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-839872969
Great, thanks. I'll create the branch and recreate this PR to point it there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-839378197
Any takers? @dagar ? @antmerlino ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-839836922
Thanks!
We can merge to a branch as Brennan suggests in the meantime. That would keep that branch point stable and I will be able to test Mac as well.
How do I create a branch via GitHub? Ideally I would like it to be at the branch point of the branch in my fork so CI runs OK. Is that possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-836749777
Last week I continued work on this and go apps/ mostly there (still some specific cases not covered yet) and other major nuttx subdirectories.
I would like to ask for some help to tackle the following major features:
- loadable module support
- protected + kernel build
- export
- support other arch's/boards
I have some progress on the loadable part but not yet completed it (at least for kernel module, loadable apps are not yet handled). I tried to look into protected + kernel build but I don't have much experience with this part so it would be great to get some help here. I haven't used the export feature either so, while it doesn't seem complex, that is another area where help would be appreciated.
For the last point, I focused my testing on sim platform (I think I can build 95% of all configs) but only verified it builds (run nsh config though). This is also only on Linux, so building on other platforms would be great. And of course, eventually the goal is to extend to other arch's.
I'm also looking for some help since right now this is not runnable in CI since GitHub rebases the PR branch to master and this breaks due to many changes from the fork point. To have it runnable it would require manually porting all changes affecting the build system. Since this becomes a larger problem every day, it would be great to get some help and complete this before going through the rebase process.
So, anyone with some knowledge of CMake and those areas of interest I mentioned? @davids5 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d closed pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d closed pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] antmerlino commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
antmerlino commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-839833532
@v01d
I don't use loadable modules or kernel/protected modes at the moment - so I'm not sure I'd understand the subtleties as much as others might.
@btashton suggestion to move to a branch for CI/development should make things a little easier to collaborate.
Let me see if I can pull the branch and at least catch up on where you are at. Full disclosure, I'm not sure I'll have time to dig into this, but I'll try and do what I can - I really appreciate the effort you are putting into this. It will be a really good improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-828079445
> No, your understanding is not quite correct. In the FLAT build, only one copy of the libraries is built. The OS and the applications link to the same libraries. For example, libs/libc builds libc.a for both the applications and the OS code. Sorry, I didn't clarify: my question regarding *KERNEL* was for FLAT mode. I noticed looking at |LibTargets.mk| that libs, mm and libapps do not ever receive the KDEFINE flags, while other libs do.
> I don't think __KERNEL__ does anything in the FLAT build.
Oh ok. In my case I had an issue with these lines in `stdlib.h` when building the C++ library:
```c
/* System() command is not implemented in the NuttX libc because it is so
* entangled with shell logic. There is an experimental version at
* apps/system/system. system() is prototyped here, however, for
* standards compatibility.
*/
#ifndef __KERNEL__
int system(FAR const char *cmd);
#endif
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] davids5 commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
davids5 commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-825168361
> @antmerlino @davids5 what are your though on the integration of NuttX with CMake as part of a larger project? Would the top-level project simply use ExternalProject to nest NuttX build system? I'm also thinking about current "export" feature of Make-based system. Do we still need this?
I think we should take it slow to not force the migration on our users base all at once.
But ultimately NuttX can be a library. We should allow the consuming project to be able to configure build options. Things like -Werror etc. There are also some compile time settings that need to be equivalent in both builds. Like `CONFIG_ARMV7M_STACKCHECK` -> drives `FLAGS += -finstrument-functions -ffixed-r10` [see this for example](https://github.com/PX4/PX4-Autopilot/blob/437825418297c1ff43681acc6cf870e2f51d1dcb/platforms/nuttx/NuttX/Make.defs.in#L120)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] patacongo commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
patacongo commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-825080290
>
>
> BTW, I'm currently trying to solve an issue with sim: my understanding is that up_head call's the host's setjmp/longjmp, but when inspecting this symbol from Make build, it appears it is renamed to NXsetjmp so it is using internal definition of setjmp. Isn't up_head.c the host interface to the simulator? Why is it using the simulated setjmp? Is the host's setjmp not usable for up_head? (I understand that the simulated nuttx internally provides its own mechanism for setjmp)
I tried that many years ago. It would crash immediately. I forget the details, but as I recall, there are a couple of issues with with the host longjmp/setjmp for this usage. For one, the stack is encrypted for security reasons. I forget why this caused the crash, try it and see.
Another problem is the the GNU/Linux setjmp/longjmp is tied into Linux signal handling in a way that makes it unsuitable for use in the simulation.
For the simulation, you need a simple setup/longjmp that just saves the and restores the critical registers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] v01d commented on pull request #3589: Migrate build system to CMake
Posted by GitBox <gi...@apache.org>.
v01d commented on pull request #3589:
URL: https://github.com/apache/incubator-nuttx/pull/3589#issuecomment-825142281
Also, what are your thoughts about what we should use for downloading external project built inside NuttX? For example, littlefs, lvgl, etc. I mentioned three strategies above, they each have pros/cons or at least different use cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org