Newt feature: run custom commands at build time

[Christopher Collins <ch...@runtime.io>]

Hello all,

I have implemented a feature in newt: the ability to run custom shell
commands at build time.

<https://github.com/apache/mynewt-newt/pull/335>

Is there any extra functionality that you'd like to see here?  All
comments are appreciated.  I am duplicating the PR text here, but the
formatting is a little nicer in the PR.

Thanks,
Chris

---

A package specifies custom commands in its `pkg.yml` file.  There are
two types of commands: 1) pre_cmds (run before the build), and 2)
post_cmds (run after the build). 

### EXAMPLE

Example (apps/blinky/pkg.yml):

    pkg.pre_cmds:
        scripts/pre_build1.sh: 100
        scripts/pre_build2.sh: 200

    pkg.post_cmds:
        scripts/post_build.sh: 100

For each command, the string on the left specifies the command to run.
The number on the right indicates the command's relative ordering.

When newt builds this example, it performs the following sequence:

    scripts/pre_build1.sh
    scripts/pre_build2.sh
    [compile]
    [link]
    scripts/post_build.sh

If other packages specify custom commands, those commands would also be
executed during the above sequence.  For example, if another package
specifies a pre command with an ordering of 150, that command would run
immediately after `pre_build1.sh`.  In the case of a tie, the commands
are run in lexicographic order.

All commands are run from the project's base directory.  In the above
example, the `scripts` directory would be a sibling of `targets`.

### CUSTOM BUILD INPUTS

A custom pre-build command can produce files that get fed into the
current build.  A command can generate any of the following:

1) .c files for newt to compile.
2) .a files for newt to link.
3) .h files that any package can include.

.c and .a files should be written to "$MYNEWT_USER_SRC_DIR" (or any
subdirectory within). 

.h files should be written to "$MYNEWT_USER_INCLUDE_DIR".  The
directory structure used here is directly reflected by the includer.
E.g., if a script writes to:

    $MYNEWT_USER_INCLUDE_DIR/foo/bar.h

then a source file can include this header with:

    #include "foo/bar.h"

### DETAILS

1. Environment variables

In addition to the usual environment variables defined for debug and
download scripts, newt defines the following env vars for custom
commands:

* MYNEWT_USER_SRC_DIR:     Path where build inputs get written.
* MYNEWT_USER_INCLUDE_DIR: Path where globally-accessible headers get
                               written.
* MYNEWT_PKG_NAME:         The full name of the package that specifies
                               the command being executed.
* MYNEWT_APP_BIN_DIR:      The path where the current target's binary
                               gets written.

These environment variables are defined for each processes that a
custom command runs in.  They are *not* defined in the newt process
itself.  So, the following snippet will not produce the expected
output:

BAD Example (apps/blinky/pkg.yml):

    pkg.pre_cmds:
        'echo $MYNEWT_USER_SRC_DIR': 100

You can execute `sh` here instead if you need access to the environment
variables, but it is probably saner to just use a script.

2. Detect changes in custom build inputs

To avoid unnecessary rebuilds, newt detects if custom build inputs have
changed since the previous build.  If none of the inputs have changed,
then they do not get rebuilt.  If any of them of them have changed,
they all get rebuilt.

The $MYNEWT_USER_[...] base is actually a temp directory.  After the
pre-build commands have run, newt compares the contents of the temp
directory with those of the actual user directory.  If any differences
are detected, newt replaces the user directory with the temp directory,
triggering a rebuild of its contents.

3. Paths

Custom build inputs get written to the following directories:

* bin/targets/<target>/user/src
* bin/targets/<target>/user/include

Custom commands should not write to these directories.  They should use
the $MYNEWT_USER_[...] environment variables instead.

Re: Newt feature: run custom commands at build time

[Christopher Collins <ch...@runtime.io>]

> I do not have a strong opinion on this, we can keep it as is, however... I
> expected that these paths are relative to package root but seems like they
> are relative to project root. Is this intended behavior? I did not find any
> way to address script in a package other that using full path, i.e.
> 'repos/.../.../script.sh' which is counter-intuitive tbh. Perhaps I
> misunderstood description, but my understanding was that cwd is set to
> project root, but newt will still look for a script in package root - this
> would make more sense I think.

Ah, I see what you mean.  You're right - the script path should be
relative to the package directory, not to the project.  I will change
this.

The user may also want to run scripts relative to the project root.
There is an environment variable containing the project root
(`MYNEWT_PROJECT_ROOT`).  This variable is not set in the newt process
itself, but I think it would be useful.  I will change newt so that it
defines this setting in its own process.

Thanks,
Chris

Re: Newt feature: run custom commands at build time

[Andrzej Kaczmarek <an...@codecoup.pl>]

Hi Chris,

On Tue, Oct 1, 2019 at 3:21 AM Christopher Collins <ch...@runtime.io> wrote:

> Hi Andrzej,
>
> On Thu, Sep 26, 2019 at 07:24:54PM +0200, Andrzej Kaczmarek wrote:
> > This looks very good! I was thinking if it would be possible to reference
> > other targets (i.e. artifacts) from scripts but with the latest addition
> of
> > shared folder this does not seem to be a problem since it can be also
> > shared with another newt build invoked from script and we can copy/write
> > data there. I did not yet check how this work in practice but will give
> it
> > a try and perhaps then I'll have some extra ideas.
>
> Thanks for taking a look!
>
> > > post_cmds (run after the build).
> > >
> > > ### EXAMPLE
> > >
> > > Example (apps/blinky/pkg.yml):
> > >
> > >     pkg.pre_cmds:
> > >         scripts/pre_build1.sh: 100
> > >         scripts/pre_build2.sh: 200
> > >
> > >     pkg.post_cmds:
> > >         scripts/post_build.sh: 100
> > >
> >
> > I assume these are relative to package root so perhaps we could assume
> > there is scripts/ subdir and execute from there by default? Just the same
> > as we have src/ and include/.
>
> I'm reluctant to use an implicit path here.  I think it is good to be
> explicit so that there is no confusion about where a script is located.
>
> We use an implicit "targets" path, but I feel like that is easier to
> justify because it saves the user from constantly typing the same thing.
> I don't think this custom command feature will be used very often at
> all, so I am not sure an implicit path would add much in the way of
> convenience.
>
> It's easy to add an implicit path later, but impossible to remove it.
> Unless you have a strong opinion on this, I suggest we give the feature
> some time without the implicit path and make the decision later.
>

I do not have a strong opinion on this, we can keep it as is, however... I
expected that these paths are relative to package root but seems like they
are relative to project root. Is this intended behavior? I did not find any
way to address script in a package other that using full path, i.e.
'repos/.../.../script.sh' which is counter-intuitive tbh. Perhaps I
misunderstood description, but my understanding was that cwd is set to
project root, but newt will still look for a script in package root - this
would make more sense I think.


> Thanks,
> Chris
>

Best,
Andrzej

Re: Newt feature: run custom commands at build time

[Christopher Collins <ch...@runtime.io>]

Hi Andrzej,

On Thu, Sep 26, 2019 at 07:24:54PM +0200, Andrzej Kaczmarek wrote:
> This looks very good! I was thinking if it would be possible to reference
> other targets (i.e. artifacts) from scripts but with the latest addition of
> shared folder this does not seem to be a problem since it can be also
> shared with another newt build invoked from script and we can copy/write
> data there. I did not yet check how this work in practice but will give it
> a try and perhaps then I'll have some extra ideas.

Thanks for taking a look!

> > post_cmds (run after the build).
> >
> > ### EXAMPLE
> >
> > Example (apps/blinky/pkg.yml):
> >
> >     pkg.pre_cmds:
> >         scripts/pre_build1.sh: 100
> >         scripts/pre_build2.sh: 200
> >
> >     pkg.post_cmds:
> >         scripts/post_build.sh: 100
> >
> 
> I assume these are relative to package root so perhaps we could assume
> there is scripts/ subdir and execute from there by default? Just the same
> as we have src/ and include/.

I'm reluctant to use an implicit path here.  I think it is good to be
explicit so that there is no confusion about where a script is located.

We use an implicit "targets" path, but I feel like that is easier to
justify because it saves the user from constantly typing the same thing.
I don't think this custom command feature will be used very often at
all, so I am not sure an implicit path would add much in the way of
convenience.

It's easy to add an implicit path later, but impossible to remove it.
Unless you have a strong opinion on this, I suggest we give the feature
some time without the implicit path and make the decision later.

Thanks,
Chris

Re: Newt feature: run custom commands at build time

[Andrzej Kaczmarek <an...@codecoup.pl>]

Hi Chris,

On Tue, Sep 24, 2019 at 2:48 AM Christopher Collins <ch...@runtime.io>
wrote:

> Hello all,
>
> I have implemented a feature in newt: the ability to run custom shell
> commands at build time.
>
> <https://github.com/apache/mynewt-newt/pull/335>
>
> Is there any extra functionality that you'd like to see here?  All
> comments are appreciated.  I am duplicating the PR text here, but the
> formatting is a little nicer in the PR.
>

This looks very good! I was thinking if it would be possible to reference
other targets (i.e. artifacts) from scripts but with the latest addition of
shared folder this does not seem to be a problem since it can be also
shared with another newt build invoked from script and we can copy/write
data there. I did not yet check how this work in practice but will give it
a try and perhaps then I'll have some extra ideas.


> Thanks,
> Chris
>
> ---
>
> A package specifies custom commands in its `pkg.yml` file.  There are
> two types of commands: 1) pre_cmds (run before the build), and 2)
> post_cmds (run after the build).
>
> ### EXAMPLE
>
> Example (apps/blinky/pkg.yml):
>
>     pkg.pre_cmds:
>         scripts/pre_build1.sh: 100
>         scripts/pre_build2.sh: 200
>
>     pkg.post_cmds:
>         scripts/post_build.sh: 100
>

I assume these are relative to package root so perhaps we could assume
there is scripts/ subdir and execute from there by default? Just the same
as we have src/ and include/.

Best,
Andrzej

Re: Newt feature: run custom commands at build time

[Christopher Collins <ch...@runtime.io>]

On Wed, Sep 25, 2019 at 06:29:19PM +0300, marko kiiskila wrote:
> Sounds good. If we need to add more, we can do that later.

Thanks.  I have updated the PR with the discussed changes:
<https://github.com/apache/mynewt-newt/pull/335>

Chris

Re: Newt feature: run custom commands at build time

[marko kiiskila <ma...@runtime.io>]

> On 24 Sep 2019, at 20.14, Christopher Collins <ch...@runtime.io> wrote:
> 
> Hi Marko,
> 
> On Tue, Sep 24, 2019 at 03:19:24PM +0300, marko kiiskila wrote:
>> Thanks, this is a very useful feature.
>> 
>>> On 24 Sep 2019, at 3.50, Christopher Collins <ch...@runtime.io> wrote:
>>> 
>> ...
>>> 
>>> A package specifies custom commands in its `pkg.yml` file.  There are
>>> two types of commands: 1) pre_cmds (run before the build), and 2)
>>> post_cmds (run after the build). 
>>> 
>>> ### EXAMPLE
>>> 
>>> Example (apps/blinky/pkg.yml):
>>> 
>>>   pkg.pre_cmds:
>>>       scripts/pre_build1.sh: 100
>>>       scripts/pre_build2.sh: 200
>>> 
>>>   pkg.post_cmds:
>>>       scripts/post_build.sh: 100
>>> 
>>> For each command, the string on the left specifies the command to run.
>>> The number on the right indicates the command's relative ordering.
>> 
>> I wasn’t sure about need for numbering, but I can see a use if execution
>> of custom commands depends on a syscfg variable being set.
> 
> I also wasn't sure about using numbered stages.  It would be simpler if
> each package specified a sequence of commands rather than a mapping.
> 
> I was envisioning a case where package A generates some intermediate
> file, then package B reads this file and produces the final output.  In
> this scenario, B's scripts must run after A's scripts.  Assigning stages
> to each command allows the user to enforce this ordering.  Yes you are
> right - packages would need to use syscfg settings to define the stages
> for this to be useful.
> 
> Another abstract use case: several scripts emit strings to some temp
> files, then a final script would gather these files and generate a .c /
> .h pair.  Admittedly, I can't think of a concrete example where this
> would be needed.  
> 
> My feeling is we should continue using numeric stages here because a)
> they could conceivably be useful, and b) it's already implemented this
> way :).  I'm really uncertain about this though, so if you (or anyone
> else) has a strong opinion about this, please just let me know and I'll
> go with it.

I do not have strong opinion about this one. I’m fine with the scheme
you have described.

> 
>> I think one useful addition would be to have another class of commands
>> for linking phase. Often there’s a need to convert the binary to some
>> other format, or slap in another header (like with DA1469x bootloader).
>> Maybe have such an option for BSP packages? pkg.post_cmds would
>> get executed after generating object files, and maybe pkg.post_link_cmds
>> would get executed after linking?
> 
> That is a good idea.  How about this naming scheme:
> 
>    * pre_build_cmds
>    * pre_link_cmds
>    * post_build_cmds
> 
> ?

That seems reasonable.

> 
> I also think we would need some an additional environment variables to
> make this useful:
> 
>    * MYNEWT_PKG_BIN_DIR    (dir where .o / .a files get written)
>    * MYNEWT_PKG_BIN_FILE   (full path of .a file)
> 
> While we are at it, might as well add one more:
> 
>    * MYNEWT_USER_WORK_DIR  (temp dir shared by all scripts)
> 
> It might be useful for scripts that feed input to other scripts.
> 

Sounds good. If we need to add more, we can do that later.

—
M

Re: Newt feature: run custom commands at build time

[Christopher Collins <ch...@runtime.io>]

Hi Marko,

On Tue, Sep 24, 2019 at 03:19:24PM +0300, marko kiiskila wrote:
> Thanks, this is a very useful feature.
> 
> > On 24 Sep 2019, at 3.50, Christopher Collins <ch...@runtime.io> wrote:
> > 
> ...
> > 
> > A package specifies custom commands in its `pkg.yml` file.  There are
> > two types of commands: 1) pre_cmds (run before the build), and 2)
> > post_cmds (run after the build). 
> > 
> > ### EXAMPLE
> > 
> > Example (apps/blinky/pkg.yml):
> > 
> >    pkg.pre_cmds:
> >        scripts/pre_build1.sh: 100
> >        scripts/pre_build2.sh: 200
> > 
> >    pkg.post_cmds:
> >        scripts/post_build.sh: 100
> > 
> > For each command, the string on the left specifies the command to run.
> > The number on the right indicates the command's relative ordering.
> 
> I wasn’t sure about need for numbering, but I can see a use if execution
> of custom commands depends on a syscfg variable being set.

I also wasn't sure about using numbered stages.  It would be simpler if
each package specified a sequence of commands rather than a mapping.

I was envisioning a case where package A generates some intermediate
file, then package B reads this file and produces the final output.  In
this scenario, B's scripts must run after A's scripts.  Assigning stages
to each command allows the user to enforce this ordering.  Yes you are
right - packages would need to use syscfg settings to define the stages
for this to be useful.

Another abstract use case: several scripts emit strings to some temp
files, then a final script would gather these files and generate a .c /
.h pair.  Admittedly, I can't think of a concrete example where this
would be needed.  

My feeling is we should continue using numeric stages here because a)
they could conceivably be useful, and b) it's already implemented this
way :).  I'm really uncertain about this though, so if you (or anyone
else) has a strong opinion about this, please just let me know and I'll
go with it.

> I think one useful addition would be to have another class of commands
> for linking phase. Often there’s a need to convert the binary to some
> other format, or slap in another header (like with DA1469x bootloader).
> Maybe have such an option for BSP packages? pkg.post_cmds would
> get executed after generating object files, and maybe pkg.post_link_cmds
> would get executed after linking?

That is a good idea.  How about this naming scheme:

    * pre_build_cmds
    * pre_link_cmds
    * post_build_cmds

?

I also think we would need some an additional environment variables to
make this useful:

    * MYNEWT_PKG_BIN_DIR    (dir where .o / .a files get written)
    * MYNEWT_PKG_BIN_FILE   (full path of .a file)

While we are at it, might as well add one more:

    * MYNEWT_USER_WORK_DIR  (temp dir shared by all scripts)

It might be useful for scripts that feed input to other scripts.

Chris

Re: Newt feature: run custom commands at build time

[marko kiiskila <ma...@runtime.io>]

Thanks, this is a very useful feature.

> On 24 Sep 2019, at 3.50, Christopher Collins <ch...@runtime.io> wrote:
> 
...
> 
> A package specifies custom commands in its `pkg.yml` file.  There are
> two types of commands: 1) pre_cmds (run before the build), and 2)
> post_cmds (run after the build). 
> 
> ### EXAMPLE
> 
> Example (apps/blinky/pkg.yml):
> 
>    pkg.pre_cmds:
>        scripts/pre_build1.sh: 100
>        scripts/pre_build2.sh: 200
> 
>    pkg.post_cmds:
>        scripts/post_build.sh: 100
> 
> For each command, the string on the left specifies the command to run.
> The number on the right indicates the command's relative ordering.

I wasn’t sure about need for numbering, but I can see a use if execution
of custom commands depends on a syscfg variable being set.

I think one useful addition would be to have another class of commands
for linking phase. Often there’s a need to convert the binary to some
other format, or slap in another header (like with DA1469x bootloader).
Maybe have such an option for BSP packages? pkg.post_cmds would
get executed after generating object files, and maybe pkg.post_link_cmds
would get executed after linking?