You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mynewt.apache.org by Sterling Hughes <st...@apache.org> on 2016/03/11 02:14:59 UTC
New "Newt"
Howdy,
This is a long mail meant to describe the rationale behind a number of
the changes in newt in the develop branch. Feel free to read at your
leisure and comment on it with improvements or likes/dislikes.
It's turned into somewhat more of a description of the newt tool, which
I think was needed anyway. Description of changes at the end in the
section "Changes from Previous Newt."
Cheers,
Sterling
Introduction
Newt is a combination of build and package management system for
embedded contexts. The idea was to build a single tool that did both
source package management and build, debug and install.
There are a couple of reasons for this:
- In order to architect an operating system that works well for
constrained environments across the many different types of
microcontroller applications(from doorbells to medical devices to power
grids), you need a system that lets you select which packages to install
and which packages to build.
- The build systems for embedded devices are often fairly complicated
and not well served. Autoconf is more designed for detecting system
compatibility issues, but not well suited when it comes to tasks like:
- Building for multiple targets
- Deciding what to build in and what not to build in
- Managing dependencies between components
When looking at solving this problem, what we realized is that the
embedded problem is one very similar to what's been solved with source
package management systems in higher level languages such as Javascript
(Node), Go, PHP and Ruby. We decided to fuse their source management
systems with a make system built for embedded systems.
Thus begat newt.
Build System
For the past mos the majority of focus on newt has been on making the
build system support a lot of the common things you might want to do
when developing embedded applications. This includes:
- Generating full flash images
- Downloading debug images to a target board using a debugger
- Conditionally compiling libraries & code based upon build settings
In order to accomplish this, newt has a fairly smart package manager
that can read a directory tree, build a dependency tree, and emit the
right build artifacts. An example newt source tree is in
incubator-mynewt-blinky/develop:
$ tree -L 3
.
├── DISCLAIMER
├── LICENSE
├── NOTICE
├── README.md
├── apps
│ └── blinky
│ ├── pkg.yml
│ └── src
├── project.yml
└── targets
├── my_blinky_sim
│ ├── pkg.yml
│ └── target.yml
└── unittest
├── pkg.yml
└── target.yml
6 directories, 10 files
When newt sees a directory tree that contains a "project.yml" file.
Newt knows that its in the base directory of a project, and
automatically builds a package tree.
Here, you can see that there are two package directories, "apps" and
"targets." Apps is where applications are stored, and applications are
where the main() function is contained. They represent the top-level of
the build tree, and define the dependencies and features for the rest of
the system.
An example of blinky's app.yml file is:
$ more apps/blinky/pkg.yml
<snip>
pkg.name: apps/blinky
pkg.vers: 0.8.0
pkg.description: Basic example application which blinks an LED.
pkg.author: "Apache Mynewt <de...@mynewt.incubator.apache.org>"
pkg.homepage: "http://mynewt.apache.org/"
pkg.repository:
pkg.keywords:
pkg.deps:
- "@apache-mynewt-core/libs/os"
- "@apache-mynewt-core/hw/hal"
- "@apache-mynewt-core/libs/console/full"
This file says that the name of the package is apps/blinky, and it
depends on libs/os, hw/hal and libs/console/full packages.
NOTE: @apache-mynewt-core is a repository descriptor, and this will be
covered in the "repository" section.
Now, when newt is told to build this project, it will:
- Find the top-level project.yml file
- Recurse the packages in the package tree, and build a list of all
source packages
Newt then looks at the target that the user set, for example, blinky_sim:
$ more targets/my_blinky_sim/
pkg.yml target.yml
$ more targets/my_blinky_sim/target.yml
### Target: targets/my_blinky_sim
target.app: "apps/blinky"
target.bsp: "@apache-mynewt-core/hw/bsp/native"
target.build_profile: "debug"
The target specifies two major things:
- Application (target.app): The application to build
- Board Support Package (target.bsp): The board support package to build
along with that application.
These two packages represent the top of the build dependency tree. Newt
then goes and builds the dependency tree specified by all the packages.
While building this tree, it does a few other things:
- Any package that depends on another package, automatically gets the
includes from the package it includes. Include directories in the
newt structure must always be prefix by the package name:
i.e. libs/os has the following include tree:
$ tree
.
├── include
│ └── shell
│ └── shell.h
├── pkg.yml
└── src
├── shell.c
├── shell_os.c
└── shell_priv.h
include contains the package name "shell" before any header files. This
is in order to avoid any header file conflicts.
- API requirements are validated. Packages can export APIs they
implement, (i.e. pkg.api: hw-hal-impl), and other packages can require
those APIs (i.e. pkg.req_api: hw-hal-impl).
- Features options are supported. Packages can change what dependencies
they have, or what Cflags they are using based upon what features are
enabled in the system. As an example, many packages will add additional
software, based on whether the shell package is present. To do this,
they can overwrite cflags or deps based upon the shell "feature."
pkg.cflags.SHELL: -DSHELL_PRESENT
In order to properly resolve all dependencies in the build system, newt
recursively processes the package dependencies until there are no new
dependencies or features (because features can add dependencies.) And
it builds a big list of all the packages that need to be build.
Newt then goes through this package list, and builds every package into
an archive file.
NOTE: The newt tool generates compiler dependencies for all of these
packages, and only rebuilds the packages whose dependencies have not
changed. Changes in package & project dependencies are also taken into
account.
Once newt has built all the archive files, it then links the archive
files together. The linkerscript to use is specified by the board
support package (BSP.)
NOTE: One common use of the "features" option above is to overwrite
which linkerscript is used, based upon whether or not the BSP is being
build for a raw image, bootable image or bootloader itself.
The newt tool places all of it's artifacts into the bin/ directory at
the top-level of the project, prefixed by the target name being built,
for example:
$ tree -L 4 bin/
bin/
└── my_blinky_sim
├── apps
│ └── blinky
│ ├── blinky.a
│ ├── blinky.a.cmd
│ ├── blinky.elf
│ ├── blinky.elf.cmd
│ ├── blinky.elf.dSYM
│ ├── blinky.elf.lst
│ ├── main.d
│ ├── main.o
│ └── main.o.cmd
├── hw
│ ├── bsp
│ │ └── native
│ ├── hal
│ │ ├── flash_map.d
│ │ ├── flash_map.o
<snip>
As you can see, a number of files are generated:
- Archive File
- *.cmd: The command use to generate the object or archive file
- *.lst: The list file where symbols are located
- *.o The object files that get put into the archive file
Download/Debug Support
Once a target has been build, there are a number of helper functions
that work on the target. These are:
download Download built target to board
debug Open debugger session to target
size Size of target components
create-image Add image header to target binary
Download and debug handles driving GDB and the system debugger. These
commands call out to scripts that are defined by the BSP.
$ more repos/apache-mynewt-core/hw/bsp/nrf52pdk/nrf52pdk_debug.sh
<snip>
#
if [ $# -lt 1 ]; then
echo "Need binary to download"
exit 1
fi
FILE_NAME=$2.elf
GDB_CMD_FILE=.gdb_cmds
echo "Debugging" $FILE_NAME
# Monitor mode. Background process gets it's own process group.
set -m
JLinkGDBServer -device nRF52 -speed 4000 -if SWD -port 3333 -singlerun &
set +m
echo "target remote localhost:3333" > $GDB_CMD_FILE
arm-none-eabi-gdb -x $GDB_CMD_FILE $FILE_NAME
rm $GDB_CMD_FILE
The idea is that every BSP will add support for the debugger environment
for that board. That way common tools can be used across various
development boards and kits.
NOTE: Both for compiler definitions and debugger scripts, the plan is to
create Dockerizable containers of these toolchains. This should make
things much easier to support across Mac OS X, Linux and Windows. Newt
will know how to call out to Docker to perform these processes.
Source Management
First, congrats for making it here. You are a dedicated reader, I wrote
this mail for you.
The other major element of the newt tool is the ability to create
reusable source distributions from a collection of code. The first
question we had to answer is what is a reusable container of source
code: a package or a project.
In our development process, we have a number of packages that we really
think should (if for nothing else than convenience) should be released
together. It makes sense to release the RTOS core, and filesystem APIs
and networking stack together -- so that releases of the Mynewt OS have
some cohesion to them and are not drastically different.
Therefore, the decision was made to provide versioning and
redistribution of the project, and not the individual packages within
those projects.
A project that has been made redistributable is known as a repository.
Repositories can be added to your local project by adding them into your
project.yml file. Here is an example of the blinky project's yml file,
which relies on apache-mynewt-core:
$ more project.yml
<snip>
project.repositories:
- apache-mynewt-core
# Use github's distribution mechanism for core ASF libraries.
# This provides mirroring automatically for us.
#
repository.apache-mynewt-core:
type: github
vers: 0-latest
user: apache
repo: incubator-mynewt-core
When you specify this repository in the blinky's project file, you can
then use newt to install dependencies:
$ newt install
Downloading repository description for apache-mynewt-core... success!
Downloading repository incubator-mynewt-core (branch: develop) at
https://github.com/apache/incubator-mynewt-core.git
Cloning into
'/var/folders/7l/7b3w9m4n2mg3sqmgw2q1b9p80000gn/T/newt-repo814721459'...
remote: Counting objects: 17601, done.
remote: Compressing objects: 100% (300/300), done.
remote: Total 17601 (delta 142), reused 0 (delta 0), pack-reused 17284
Receiving objects: 100% (17601/17601), 6.09 MiB | 3.17 MiB/s, done.
Resolving deltas: 100% (10347/10347), done.
Checking connectivity... done.
Repos successfully installed
Newt will install this repository in the <project>/repos directory. In
the case of blinky, the directory structure ends up looking like:
$ tree -L 2
.
├── DISCLAIMER
├── LICENSE
├── NOTICE
├── README.md
├── apps
│ └── blinky
├── project.state
├── project.yml
├── repos
│ └── apache-mynewt-core
└── targets
├── my_blinky_sim
└── unittest
In order to reference the installed repositories in packages, the "@"
notation should be specified in the repository specifier. As an
example, the apps/blinky application has the following dependencies in
it's pkg.yml file:
$ more apps/blinky/pkg.yml
<snip>
pkg.deps:
- "@apache-mynewt-core/libs/os"
- "@apache-mynewt-core/hw/hal"
- "@apache-mynewt-core/libs/console/full"
This tells the build system to look in the base directory of
repos/apache-mynewt-core for the "libs/os" package.
In order to create a repository out of a project, all you need to do is
create a repository.yml file, and check it into the master branch of
your project.
NOTE: Currently only github is supported by our package management
system, but straight git will be added soon.
The repository.yml defines all versions of this repository and the
corresponding source control tags that these versions correspond to. As
an example, the repository.yml file has the following contents:
$ more repository.yml
repo.name: apache-mynewt-core
repo.versions:
"0.0.0": "develop"
"0-latest": "0.0.0"
There is one version of the apache-mynewt-core operating system
available, which is 0.0.0 (we haven't released yet! :-) This version
corresponds to the "develop" branch in this repository.
In addition to the 0.0.0 branch, there is a holding version 0-latest,
which specifies the latest version of the 0.0.0 release (which is 0.)
In many cases, most people who are maintaining dependencies to the
repository will likely provide some form of major/minor and the holding
branch (e.g. 0.8-stable, 0.8-latest). These map to specific versions in
the repository.yml file, and get finally resolved into the branch name.
Repositories can also have dependencies on other repositories. These
dependencies should be listed out on a per-tag basis. So, for example,
if apache-mynewt-core were to depend on sterlys-little-repo, you might
have the following directives in the repository.yml:
develop.repositories:
sterlys-little-repo:
type: github
vers: 0.8-latest
user: sterlinghughes
repo: sterlys-little-repo
This would tell newt that for anything that resolves to the develop
branch, this repository requires the sterlys-little-repo repository.
Dependencies are resolved circularly by the newt tool, and every
dependent repository is placed as a sibling in the repos directory.
Currently, if two repositories have the same name, they will conflict
and bad things will happen.
When a repository is installed to the repos/ directory, the current
version of that repository is written to the "project.state" file. The
project state file contains the currently installed version of any given
repository. This way, the current set of repositories can be recreated
from the project.state file reliably, whereas the project.yml file can
have higher level directives (i.e. include 0.8-stable.)
In order to upgrade a previously installed repository, the "newt
upgrade" command should be issued:
$ newt upgrade
Newt upgrade will look at the current desired version in project.yml,
and compare it to the version in project.state. If these two differ, it
will upgrade the dependency. Upgrade works not just for the dependency
in project.yml, but for all the sub-dependencies that they might have.
A NOTE ON DEPENDENCY RESOLUTION:
At the moment, all dependencies must match, otherwise newt will provide
an error. As an example, if you have a set of dependencies such that:
apache-mynewt-core depends on sterlys-little-repo 0.6-stable
apache-mynewt-core depends on sterlys-big-repo 0.5.1
sterlys-big-repo-0.5.1 depends on sterlys-little-repo 0.6.2
Where 0.6-stable is 0.6.3
The newt tool will try and resolve the dependency to
sterlys-little-repo. It will notice that there are two conflicting
versions of the repository, and not perform installation.
In the future newt will be smarter about loading in all dependencies,
and then looking to satisfy those dependencies to the best match of all
potential options.
Changes from Previous Newt
The original newt was always written as "write one to throw it away."
We wanted to get familiar with and test the concepts behind the tool.
As such, the code wasn't very pretty. Combine that with our naming
packages, "eggs" and our base directory "larva" and multiple stages of
renaming without refactoring.
Oh, and we learned Go while writing it. Eek.
Very little has changed on the build side of the newt tool in terms of
functionality. Identities have been renamed to features, capabilities
to APIs, and helper commands have been added, but the cleanups have been
mostly internal here.
The repository management has been designed and written from the ground
up, and is therefore a bit rawer, and more susceptible to change. I
believe we have the right approach here now, and so it will stabilize
much quicker.
For reference, the previous package installation and search was built on
the concept that newt would search for individual packages within git
repositories and place them in the local directory without any prefixing
or version history. In addition to the difficulty of maintaining a
branching strategy for this, it ended up being confusing as to what were
system packages and what were application packages.
To add pain, the package installation and upgrade system used the build
system's notion of package trees. The requirements for the two are
somewhat at odds (build tree very specific, and is a true tree, whereas
install & upgrade is more of a DAG.)
This has all been cleaned up in the new implementation.
Known Issues & Potential Future Changes
- Not 100% sure if packages/repositories shouldn't be rename to
libraries/packages.
- Dependency resolution is not as smart as it should be. It needs to
build a graph of all supported dependencies and then resolve the best
match from that.
- Missing helper commands to generate repository.yml and project.yml files
- Missing templates for common use cases (applications, drivers, etc.)
- Missing any ability to search for 3rd party packages
Re: New "Newt"
Posted by James Pace <pa...@runtime.io>.
>
> +1. I'm not touching the other connotations here, load makes sense.
>
> sterling
Presumably you’re referring to getting sued by Cisco for having a command that’s used in iOS.
Re: New "Newt"
Posted by Sterling Hughes <st...@apache.org>.
On 3/14/16 12:21 PM, ray suorsa wrote:
>> On Mar 14, 2016, at 11:49 AM, marko kiiskila <ma...@runtime.io> wrote:
>>
>>
>>> On Mar 11, 2016, at 6:28 PM, Christopher Collins <cc...@apache.org> wrote:
>>>
>>> On Sat, Mar 12, 2016 at 10:31:12AM +1100, Justin Mclean wrote:
>>>>> download Download built target to board
>>>>
>>>> Would install rather than download be better? download to me means download something to your computer not upload to the target device.
>>>
>>> I agree that "download" might not be the best name for this command.
>>> There is already a command called "install", though. Perhaps the
>>> command should be renamed to "upload"?
>>>
>>
>> I’m not a big fan of “upload”; that feels like sending something out to Internet.
>> I’d be ok with “load”, or maybe “program”.
>>
>> How about ‘load’
>
> ‘load’ makes sense to me.
>
+1. I'm not touching the other connotations here, load makes sense.
sterling
Re: New "Newt"
Posted by ray suorsa <re...@runtime.io>.
> On Mar 14, 2016, at 11:49 AM, marko kiiskila <ma...@runtime.io> wrote:
>
>
>> On Mar 11, 2016, at 6:28 PM, Christopher Collins <cc...@apache.org> wrote:
>>
>> On Sat, Mar 12, 2016 at 10:31:12AM +1100, Justin Mclean wrote:
>>>> download Download built target to board
>>>
>>> Would install rather than download be better? download to me means download something to your computer not upload to the target device.
>>
>> I agree that "download" might not be the best name for this command.
>> There is already a command called "install", though. Perhaps the
>> command should be renamed to "upload"?
>>
>
> I’m not a big fan of “upload”; that feels like sending something out to Internet.
> I’d be ok with “load”, or maybe “program”.
>
> How about ‘load’
‘load’ makes sense to me.
Re: New "Newt"
Posted by marko kiiskila <ma...@runtime.io>.
> On Mar 11, 2016, at 6:28 PM, Christopher Collins <cc...@apache.org> wrote:
>
> On Sat, Mar 12, 2016 at 10:31:12AM +1100, Justin Mclean wrote:
>>> download Download built target to board
>>
>> Would install rather than download be better? download to me means download something to your computer not upload to the target device.
>
> I agree that "download" might not be the best name for this command.
> There is already a command called "install", though. Perhaps the
> command should be renamed to "upload"?
>
I’m not a big fan of “upload”; that feels like sending something out to Internet.
I’d be ok with “load”, or maybe “program”.
How about ‘load’?
Re: New "Newt"
Posted by Christopher Collins <cc...@apache.org>.
On Sat, Mar 12, 2016 at 10:31:12AM +1100, Justin Mclean wrote:
> > download Download built target to board
>
> Would install rather than download be better? download to me means download something to your computer not upload to the target device.
I agree that "download" might not be the best name for this command.
There is already a command called "install", though. Perhaps the
command should be renamed to "upload"?
Chris
Re: New "Newt"
Posted by Sterling Hughes <st...@apache.org>.
>> $ more repository.yml
>> repo.name: apache-mynewt-core
>> repo.versions:
>> "0.0.0": "develop"
>> "0-latest": "0.0.0"
>>
>> There is one version of the apache-mynewt-core operating system available, which is 0.0.0 (we haven't released yet! :-) This version corresponds to the "develop" branch in this repository.
>
> Given we have released a version why isn’t that being used? Also I’d feel more conformable if the the branch was the release tag rather than develop there less chance of user using un voted / un released code that way.
>
The v0.8b1 version we've released is not compatible with the new newt,
due to the fairly significant changes we've made to how the package
management system works.
I totally agree with the branch, it's just on develop for the moment so
that everyone can test it. When we cut the release, this will look like:
$ more repository.yml
<snip>
repo.name: apache-mynewt-core
repo.versions:
"0.8.0": "apache-mynewt-tag-v-0-8-0",
"0-latest": "0.8.0"
With apache-mynewt-tag-v-0-8-0 representing whatever we choose to call
the tag.
NOTE: One key part of this scheme is the the repository.yml must reside
in the master branch on git. In our current branching scheme, we are
going to keep this branch stable.
I was on two minds of this:
- If we keep the repository.yml on master, it's going to be easy to keep
it up to date, and easy for people to know where it is. And we want
people to distribute tons of these packages on GH, therefore making this
simple is good.
- However, there is a risk that you mess up the repository.yml on
master. Forcing people to put this on a special branch
"repository-yml"? Would make this activity a bit more protected then
messing with the master branch.
- That said, I think our current approach is more consistent with other
package systems that work with git. Which is probably the way we want
to go.
>> In order to upgrade a previously installed repository, the "newt upgrade" command should be issued:
>>
>> $ newt upgrade
>
> Perhaps “update” rather than upgrade? Would it be useful to have a "newt outdated” to tell you what it will update?
>
I prefer newt upgrade, because I'm more familiar with aptitude (Debian
apt-get.) "update" used to update the package repository descriptions,
and upgrade "upgraded" the packages. In newt, we update the package
repositories every time we do an operation (at least for now.)
Bandwidth is cheaper these days. :)
newt outdated would be super useful. I will add this prior to the next
release checkpoint (after the current one.) There are a bunch of
package management helper functions we need to add. But I wrote the
current system _very_ quickly, so I want to take some time to think
about it prior to adding too many functions to it.
I'll do this prior to the next (subsequent) release.
>> - Not 100% sure if packages/repositories shouldn't be rename to libraries/packages.
>
> May be some confusion with repositories as it also refer to the github repo where the code lives.
>
Yes. Although, that's also a nice thing with calling it repository: we
have a 1-1 relationship between our notion of a repository, and git's
notion of a repository.
Sterling
Re: New "Newt"
Posted by Justin Mclean <ju...@classsoftware.com>.
Hi,
> Once a target has been build, there are a number of helper functions that work on the target. These are:
>
> download Download built target to board
Would install rather than download be better? download to me means download something to your computer not upload to the target device.
> $ more repository.yml
> repo.name: apache-mynewt-core
> repo.versions:
> "0.0.0": "develop"
> "0-latest": "0.0.0"
>
> There is one version of the apache-mynewt-core operating system available, which is 0.0.0 (we haven't released yet! :-) This version corresponds to the "develop" branch in this repository.
Given we have released a version why isn’t that being used? Also I’d feel more conformable if the the branch was the release tag rather than develop there less chance of user using un voted / un released code that way.
> In order to upgrade a previously installed repository, the "newt upgrade" command should be issued:
>
> $ newt upgrade
Perhaps “update” rather than upgrade? Would it be useful to have a "newt outdated” to tell you what it will update?
> - Not 100% sure if packages/repositories shouldn't be rename to libraries/packages.
May be some confusion with repositories as it also refer to the github repo where the code lives.
Thanks,
Justin
Re: New "Newt"
Posted by "paul@wrada.com" <pa...@wrada.com>.
Thanks Sterling
I see this as really great. It's much clearer how to build my own project around this model.
Also, it's great to support the repos so naturally.
It's going to take trying this out tomorrow to really solidify this in my mind. Two comments after reading
1) when I think of package management I think of the repository as a place to hold many different versioned packages (brew, maven, yum etc). But in newt each repository there is only one version more like source code repos.
So it's not clear if repository is the right name. Wonder if we look at source packages from these other systems for some name ideas (they distribute buildable bundles of source as well).
2) it's going to help the community if there are a few common ways to find repositories and share their info with the community. I really hope newt can focus on solving this sooner than later. It's a good time to build this foundation to encourage sharing and community.
> On Mar 10, 2016, at 5:15 PM, Sterling Hughes <st...@apache.org> wrote:
>
> Howdy,
>
> This is a long mail meant to describe the rationale behind a number of the changes in newt in the develop branch. Feel free to read at your leisure and comment on it with improvements or likes/dislikes.
>
> It's turned into somewhat more of a description of the newt tool, which I think was needed anyway. Description of changes at the end in the section "Changes from Previous Newt."
>
> Cheers,
> Sterling
>
>
> Introduction
>
> Newt is a combination of build and package management system for embedded contexts. The idea was to build a single tool that did both source package management and build, debug and install.
>
> There are a couple of reasons for this:
>
> - In order to architect an operating system that works well for constrained environments across the many different types of microcontroller applications(from doorbells to medical devices to power grids), you need a system that lets you select which packages to install and which packages to build.
>
> - The build systems for embedded devices are often fairly complicated and not well served. Autoconf is more designed for detecting system compatibility issues, but not well suited when it comes to tasks like:
> - Building for multiple targets
> - Deciding what to build in and what not to build in
> - Managing dependencies between components
>
> When looking at solving this problem, what we realized is that the embedded problem is one very similar to what's been solved with source package management systems in higher level languages such as Javascript (Node), Go, PHP and Ruby. We decided to fuse their source management systems with a make system built for embedded systems.
>
> Thus begat newt.
>
> Build System
>
> For the past mos the majority of focus on newt has been on making the build system support a lot of the common things you might want to do when developing embedded applications. This includes:
>
> - Generating full flash images
> - Downloading debug images to a target board using a debugger
> - Conditionally compiling libraries & code based upon build settings
>
> In order to accomplish this, newt has a fairly smart package manager that can read a directory tree, build a dependency tree, and emit the right build artifacts. An example newt source tree is in incubator-mynewt-blinky/develop:
>
> $ tree -L 3
> .
> ├── DISCLAIMER
> ├── LICENSE
> ├── NOTICE
> ├── README.md
> ├── apps
> │ └── blinky
> │ ├── pkg.yml
> │ └── src
> ├── project.yml
> └── targets
> ├── my_blinky_sim
> │ ├── pkg.yml
> │ └── target.yml
> └── unittest
> ├── pkg.yml
> └── target.yml
>
> 6 directories, 10 files
>
> When newt sees a directory tree that contains a "project.yml" file. Newt knows that its in the base directory of a project, and automatically builds a package tree.
>
> Here, you can see that there are two package directories, "apps" and "targets." Apps is where applications are stored, and applications are where the main() function is contained. They represent the top-level of the build tree, and define the dependencies and features for the rest of the system.
>
> An example of blinky's app.yml file is:
>
> $ more apps/blinky/pkg.yml
> <snip>
> pkg.name: apps/blinky
> pkg.vers: 0.8.0
> pkg.description: Basic example application which blinks an LED.
> pkg.author: "Apache Mynewt <de...@mynewt.incubator.apache.org>"
> pkg.homepage: "http://mynewt.apache.org/"
> pkg.repository:
> pkg.keywords:
>
> pkg.deps:
> - "@apache-mynewt-core/libs/os"
> - "@apache-mynewt-core/hw/hal"
> - "@apache-mynewt-core/libs/console/full"
>
> This file says that the name of the package is apps/blinky, and it depends on libs/os, hw/hal and libs/console/full packages.
>
> NOTE: @apache-mynewt-core is a repository descriptor, and this will be covered in the "repository" section.
>
> Now, when newt is told to build this project, it will:
>
> - Find the top-level project.yml file
>
> - Recurse the packages in the package tree, and build a list of all source packages
>
> Newt then looks at the target that the user set, for example, blinky_sim:
>
> $ more targets/my_blinky_sim/
> pkg.yml target.yml
> $ more targets/my_blinky_sim/target.yml
> ### Target: targets/my_blinky_sim
> target.app: "apps/blinky"
> target.bsp: "@apache-mynewt-core/hw/bsp/native"
> target.build_profile: "debug"
>
> The target specifies two major things:
>
> - Application (target.app): The application to build
> - Board Support Package (target.bsp): The board support package to build along with that application.
>
> These two packages represent the top of the build dependency tree. Newt then goes and builds the dependency tree specified by all the packages. While building this tree, it does a few other things:
>
> - Any package that depends on another package, automatically gets the includes from the package it includes. Include directories in the
> newt structure must always be prefix by the package name:
>
> i.e. libs/os has the following include tree:
>
> $ tree
> .
> ├── include
> │ └── shell
> │ └── shell.h
> ├── pkg.yml
> └── src
> ├── shell.c
> ├── shell_os.c
> └── shell_priv.h
>
> include contains the package name "shell" before any header files. This is in order to avoid any header file conflicts.
>
> - API requirements are validated. Packages can export APIs they implement, (i.e. pkg.api: hw-hal-impl), and other packages can require those APIs (i.e. pkg.req_api: hw-hal-impl).
>
> - Features options are supported. Packages can change what dependencies they have, or what Cflags they are using based upon what features are enabled in the system. As an example, many packages will add additional software, based on whether the shell package is present. To do this, they can overwrite cflags or deps based upon the shell "feature."
>
> pkg.cflags.SHELL: -DSHELL_PRESENT
>
> In order to properly resolve all dependencies in the build system, newt recursively processes the package dependencies until there are no new dependencies or features (because features can add dependencies.) And it builds a big list of all the packages that need to be build.
>
> Newt then goes through this package list, and builds every package into an archive file.
>
> NOTE: The newt tool generates compiler dependencies for all of these packages, and only rebuilds the packages whose dependencies have not changed. Changes in package & project dependencies are also taken into account.
>
> Once newt has built all the archive files, it then links the archive files together. The linkerscript to use is specified by the board support package (BSP.)
>
> NOTE: One common use of the "features" option above is to overwrite which linkerscript is used, based upon whether or not the BSP is being build for a raw image, bootable image or bootloader itself.
>
> The newt tool places all of it's artifacts into the bin/ directory at the top-level of the project, prefixed by the target name being built, for example:
>
> $ tree -L 4 bin/
> bin/
> └── my_blinky_sim
> ├── apps
> │ └── blinky
> │ ├── blinky.a
> │ ├── blinky.a.cmd
> │ ├── blinky.elf
> │ ├── blinky.elf.cmd
> │ ├── blinky.elf.dSYM
> │ ├── blinky.elf.lst
> │ ├── main.d
> │ ├── main.o
> │ └── main.o.cmd
> ├── hw
> │ ├── bsp
> │ │ └── native
> │ ├── hal
> │ │ ├── flash_map.d
> │ │ ├── flash_map.o
> <snip>
>
> As you can see, a number of files are generated:
>
> - Archive File
> - *.cmd: The command use to generate the object or archive file
> - *.lst: The list file where symbols are located
> - *.o The object files that get put into the archive file
>
> Download/Debug Support
>
> Once a target has been build, there are a number of helper functions that work on the target. These are:
>
> download Download built target to board
> debug Open debugger session to target
> size Size of target components
> create-image Add image header to target binary
>
> Download and debug handles driving GDB and the system debugger. These commands call out to scripts that are defined by the BSP.
>
> $ more repos/apache-mynewt-core/hw/bsp/nrf52pdk/nrf52pdk_debug.sh
> <snip>
> #
> if [ $# -lt 1 ]; then
> echo "Need binary to download"
> exit 1
> fi
>
> FILE_NAME=$2.elf
> GDB_CMD_FILE=.gdb_cmds
>
> echo "Debugging" $FILE_NAME
>
> # Monitor mode. Background process gets it's own process group.
> set -m
> JLinkGDBServer -device nRF52 -speed 4000 -if SWD -port 3333 -singlerun &
> set +m
>
> echo "target remote localhost:3333" > $GDB_CMD_FILE
>
> arm-none-eabi-gdb -x $GDB_CMD_FILE $FILE_NAME
>
> rm $GDB_CMD_FILE
>
> The idea is that every BSP will add support for the debugger environment for that board. That way common tools can be used across various development boards and kits.
>
> NOTE: Both for compiler definitions and debugger scripts, the plan is to create Dockerizable containers of these toolchains. This should make things much easier to support across Mac OS X, Linux and Windows. Newt will know how to call out to Docker to perform these processes.
>
> Source Management
>
> First, congrats for making it here. You are a dedicated reader, I wrote this mail for you.
>
> The other major element of the newt tool is the ability to create reusable source distributions from a collection of code. The first question we had to answer is what is a reusable container of source code: a package or a project.
>
> In our development process, we have a number of packages that we really think should (if for nothing else than convenience) should be released together. It makes sense to release the RTOS core, and filesystem APIs and networking stack together -- so that releases of the Mynewt OS have some cohesion to them and are not drastically different.
>
> Therefore, the decision was made to provide versioning and redistribution of the project, and not the individual packages within those projects.
>
> A project that has been made redistributable is known as a repository. Repositories can be added to your local project by adding them into your project.yml file. Here is an example of the blinky project's yml file, which relies on apache-mynewt-core:
>
> $ more project.yml
> <snip>
> project.repositories:
> - apache-mynewt-core
>
> # Use github's distribution mechanism for core ASF libraries.
> # This provides mirroring automatically for us.
> #
> repository.apache-mynewt-core:
> type: github
> vers: 0-latest
> user: apache
> repo: incubator-mynewt-core
>
>
> When you specify this repository in the blinky's project file, you can then use newt to install dependencies:
>
> $ newt install
> Downloading repository description for apache-mynewt-core... success!
> Downloading repository incubator-mynewt-core (branch: develop) at https://github.com/apache/incubator-mynewt-core.git
> Cloning into '/var/folders/7l/7b3w9m4n2mg3sqmgw2q1b9p80000gn/T/newt-repo814721459'...
> remote: Counting objects: 17601, done.
> remote: Compressing objects: 100% (300/300), done.
> remote: Total 17601 (delta 142), reused 0 (delta 0), pack-reused 17284
> Receiving objects: 100% (17601/17601), 6.09 MiB | 3.17 MiB/s, done.
> Resolving deltas: 100% (10347/10347), done.
> Checking connectivity... done.
> Repos successfully installed
>
> Newt will install this repository in the <project>/repos directory. In the case of blinky, the directory structure ends up looking like:
>
> $ tree -L 2
> .
> ├── DISCLAIMER
> ├── LICENSE
> ├── NOTICE
> ├── README.md
> ├── apps
> │ └── blinky
> ├── project.state
> ├── project.yml
> ├── repos
> │ └── apache-mynewt-core
> └── targets
> ├── my_blinky_sim
> └── unittest
>
> In order to reference the installed repositories in packages, the "@" notation should be specified in the repository specifier. As an example, the apps/blinky application has the following dependencies in it's pkg.yml file:
>
> $ more apps/blinky/pkg.yml
> <snip>
> pkg.deps:
> - "@apache-mynewt-core/libs/os"
> - "@apache-mynewt-core/hw/hal"
> - "@apache-mynewt-core/libs/console/full"
>
>
> This tells the build system to look in the base directory of repos/apache-mynewt-core for the "libs/os" package.
>
> In order to create a repository out of a project, all you need to do is create a repository.yml file, and check it into the master branch of your project.
>
> NOTE: Currently only github is supported by our package management system, but straight git will be added soon.
>
> The repository.yml defines all versions of this repository and the corresponding source control tags that these versions correspond to. As an example, the repository.yml file has the following contents:
>
> $ more repository.yml
> repo.name: apache-mynewt-core
> repo.versions:
> "0.0.0": "develop"
> "0-latest": "0.0.0"
>
> There is one version of the apache-mynewt-core operating system available, which is 0.0.0 (we haven't released yet! :-) This version corresponds to the "develop" branch in this repository.
>
> In addition to the 0.0.0 branch, there is a holding version 0-latest, which specifies the latest version of the 0.0.0 release (which is 0.) In many cases, most people who are maintaining dependencies to the repository will likely provide some form of major/minor and the holding branch (e.g. 0.8-stable, 0.8-latest). These map to specific versions in the repository.yml file, and get finally resolved into the branch name.
>
> Repositories can also have dependencies on other repositories. These dependencies should be listed out on a per-tag basis. So, for example, if apache-mynewt-core were to depend on sterlys-little-repo, you might have the following directives in the repository.yml:
>
> develop.repositories:
> sterlys-little-repo:
> type: github
> vers: 0.8-latest
> user: sterlinghughes
> repo: sterlys-little-repo
>
>
> This would tell newt that for anything that resolves to the develop branch, this repository requires the sterlys-little-repo repository.
>
> Dependencies are resolved circularly by the newt tool, and every dependent repository is placed as a sibling in the repos directory. Currently, if two repositories have the same name, they will conflict and bad things will happen.
>
> When a repository is installed to the repos/ directory, the current version of that repository is written to the "project.state" file. The project state file contains the currently installed version of any given repository. This way, the current set of repositories can be recreated from the project.state file reliably, whereas the project.yml file can have higher level directives (i.e. include 0.8-stable.)
>
> In order to upgrade a previously installed repository, the "newt upgrade" command should be issued:
>
> $ newt upgrade
>
> Newt upgrade will look at the current desired version in project.yml, and compare it to the version in project.state. If these two differ, it will upgrade the dependency. Upgrade works not just for the dependency in project.yml, but for all the sub-dependencies that they might have.
>
> A NOTE ON DEPENDENCY RESOLUTION:
>
> At the moment, all dependencies must match, otherwise newt will provide an error. As an example, if you have a set of dependencies such that:
>
> apache-mynewt-core depends on sterlys-little-repo 0.6-stable
> apache-mynewt-core depends on sterlys-big-repo 0.5.1
> sterlys-big-repo-0.5.1 depends on sterlys-little-repo 0.6.2
>
> Where 0.6-stable is 0.6.3
>
> The newt tool will try and resolve the dependency to sterlys-little-repo. It will notice that there are two conflicting versions of the repository, and not perform installation.
>
> In the future newt will be smarter about loading in all dependencies, and then looking to satisfy those dependencies to the best match of all potential options.
>
> Changes from Previous Newt
>
> The original newt was always written as "write one to throw it away." We wanted to get familiar with and test the concepts behind the tool. As such, the code wasn't very pretty. Combine that with our naming packages, "eggs" and our base directory "larva" and multiple stages of renaming without refactoring.
>
> Oh, and we learned Go while writing it. Eek.
>
> Very little has changed on the build side of the newt tool in terms of functionality. Identities have been renamed to features, capabilities to APIs, and helper commands have been added, but the cleanups have been mostly internal here.
>
> The repository management has been designed and written from the ground up, and is therefore a bit rawer, and more susceptible to change. I believe we have the right approach here now, and so it will stabilize much quicker.
>
> For reference, the previous package installation and search was built on the concept that newt would search for individual packages within git repositories and place them in the local directory without any prefixing or version history. In addition to the difficulty of maintaining a branching strategy for this, it ended up being confusing as to what were system packages and what were application packages.
>
> To add pain, the package installation and upgrade system used the build system's notion of package trees. The requirements for the two are somewhat at odds (build tree very specific, and is a true tree, whereas install & upgrade is more of a DAG.)
>
> This has all been cleaned up in the new implementation.
>
> Known Issues & Potential Future Changes
>
> - Not 100% sure if packages/repositories shouldn't be rename to libraries/packages.
>
> - Dependency resolution is not as smart as it should be. It needs to build a graph of all supported dependencies and then resolve the best match from that.
>
> - Missing helper commands to generate repository.yml and project.yml files
>
> - Missing templates for common use cases (applications, drivers, etc.)
>
> - Missing any ability to search for 3rd party packages