You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nuttx.apache.org by Haitao Liu <li...@gmail.com> on 2020/02/04 04:50:24 UTC
Re: Jenkins CI
Hi, Greg,
Thanks to Miguel's careful review, could you please help to finish the PR
merge?
Then I and Miguel can continue the development with more PRs and refinement.
Gregory Nutt <sp...@gmail.com> 于2020年1月31日周五 下午11:30写道：
>
> > I have just added the CI scripts and testlist files in
> > https://github.com/apache/incubator-nuttx-testing/pull/3
> > And asked XiaoXiang help to create Nuttx Nightly build job in
> > builds.apache.org before. But it's just a freestyle CI job
> > which call cibuild.sh simply. Once CI script and testlist file verify
> well.
> > It would be better if it could be integrated into the
> > pipeline definition you mentioned. You could help to use the pipeline way
> > if access granted. Let's work together on it :)
>
> Perhaps Miguel could provide the review of PR3. I will be happy to
> finish the merge after some review is provided.
>
> Once it has been merged, both you and Miguel can continue the
> development with PRs. But we need to find someone competent to provide
> review.
>
> Greg
>
>
>
Re: Jenkins CI
Posted by Miguel Ángel Herranz <mi...@midokura.com.INVALID>.
Thanks for your summary. Some comments:
On Thu, Feb 13, 2020 at 5:16 AM Haitao Liu <li...@gmail.com> wrote:
> Apache Jenkins
> | Github action
>
> ============================================================================================================================
> ...
> Restrict accounts access to update jenkins configs
> | Free to Make PRs to update githubaction workflow
>
That is not exactly true: accounts access is restricted to create the
_Jenkins jobs_. The logic executed in these jobs, once created, is stored
in a Jenkinsfile or in Groovy pipeline script, so the workflow can be
updated by a PR as in Github. Moreover, if using shared libraries, the
logic can reside in another repository, as Greg requested. I don't know if
that is possible with Github.
> So, I think we could consider to use apache jenkins ci as nightly build
> and use github action ci as PR check build accordingly.
>
I agree with that, but mostly due to apparently saturation of AFS CI
service.
1. Apache Jenkins CI for Nightly Build
>
> There is an existing freestyle NuttX Nightly Build do cron job at
> midnight each day. However, it's very simple. And there is still flock
> issue to resolve to make parallel build all pass.
>
>
> https://builds.apache.org/view/Incubator%20Projects/job/NuttX-Nightly-Build/
>
> Thanks to @Maht, he made multibranch jenkins pipeline PR which would make
> the Nightly build more robust. But it may still need some time to switch to
> it because of Apache Jenkins access control. The PRs as below:
>
> https://github.com/apache/incubator-nuttx-testing/pull/2
>
> https://github.com/apache/incubator-nuttx/pull/174
>
>
Well, my multibranch proposal is originally intended to manage PR triggers
(each PR is a branch). For nightly, as long there is only one branch now
(master), there is less added value. Still, the pipeline logic could be
modified to be more "parallel" (to be done, though).
Also, there is still travis CI pull request form @yamt, we could integrate
> it into incubator-nuttx-testing as maht do.
>
> https://github.com/apache/incubator-nuttx/pull/183
>
> <
> https://builds.apache.org/view/Incubator%20Projects/job/NuttX-Nightly-Build/
> >
>
Yes, I think the idea of having the CI/testing stuff in a separate repo is
good, as Greg requested, despite maybe some minimal file info has to be
kept in the main repo. But I won't consider it a strong requirement for the
short term if achieving it would take a lot of effort to implement outside
of the typical usage of Github Actions/CircleCI/TravisCI/etc.
2. Github action CI for Check Build
>
> I have just make two PRs to incubator-nuttx and inucbator-nuttx-apps
> accordingly as below:
>
> https://github.com/apache/incubator-nuttx/pull/261
>
> https://github.com/apache/incubator-nuttx-apps/pull/65
>
> The two PRs are not intended to be merged right now. They still need align
> to check testlist update under incubator-nuttx-testing.
>
> For example, Consider have a minimal check testlist set that always run.
>
> As for github action check build CI, there are still more space to
> optimize:
>
> 1. Matrix builds to allow concurrent jobs and even multi-platorms
>
> To speed up the check build time, use multiple jobs in github action
> workflow to do parallel builds as @btashton and @davids suggest.
>
That would be a very nice addition, but I am not sure how to do it Github
Actions, and have no much time this week to explore.
> 2. Use docker images with tools preinstalled
>
> Now it takes several minitues to install all essential tools in ubuntu vm.
>
> we may build nuttx with docker images which tools preinstalled in the
> future.
>
@davids suggested to use PX4 Docker images, and I think they are very good
to use/reference: https://hub.docker.com/r/px4io/px4-dev-nuttx
> Anyway, These are just some thoughts from our previous discussions,
> welcome more reviews and feedbacks from community.
> Thanks!
>
Thanks for the effort and count on me if you need some feedback.
Cheers,
Miguel
Re: Jenkins CI
Posted by Haitao Liu <li...@gmail.com>.
After practise with Apache Jenkins CI and Github action CI for a time and
do some CI problems search in Apache maillist and google, let's sum up as
below:
Apache Jenkins
| Github action
============================================================================================================================
Provide Linux/Windows real Machines, but MacOS not available
| Provide Linux/MacOS/Windows VM
Apache jenkins slaves are limited and shared with other ASF projects
| The computing power can be accessed for each user
at first time
Github PRs may wait nearly hours to be scheduled, for nightly build it
works well | Integrate with the github workflow smoothly
Restrict accounts access to update jenkins configs
| Free to Make PRs to update github
action workflow
More info on Jenkins and Github action for reference:
https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-github-actions#usage-limits
https://knapsackpro.com/ci_comparisons/jenkins/vs/github-actions // sth
maybe obsolete for github action
So, I think we could consider to use apache jenkins ci as nightly build
and use github action ci as PR check build accordingly.
1. Apache Jenkins CI for Nightly Build
There is an existing freestyle NuttX Nightly Build do cron job at
midnight each day. However, it's very simple. And there is still flock
issue to resolve to make parallel build all pass.
https://builds.apache.org/view/Incubator%20Projects/job/NuttX-Nightly-Build/
Thanks to @Maht, he made multibranch jenkins pipeline PR which would make
the Nightly build more robust. But it may still need some time to switch to
it because of Apache Jenkins access control. The PRs as below:
https://github.com/apache/incubator-nuttx-testing/pull/2
https://github.com/apache/incubator-nuttx/pull/174
Also, there is still travis CI pull request form @yamt, we could integrate
it into incubator-nuttx-testing as maht do.
https://github.com/apache/incubator-nuttx/pull/183
<https://builds.apache.org/view/Incubator%20Projects/job/NuttX-Nightly-Build/>
2. Github action CI for Check Build
I have just make two PRs to incubator-nuttx and inucbator-nuttx-apps
accordingly as below:
https://github.com/apache/incubator-nuttx/pull/261
https://github.com/apache/incubator-nuttx-apps/pull/65
The two PRs are not intended to be merged right now. They still need align
to check testlist update under incubator-nuttx-testing.
For example, Consider have a minimal check testlist set that always run.
As for github action check build CI, there are still more space to optimize:
1. Matrix builds to allow concurrent jobs and even multi-platorms
To speed up the check build time, use multiple jobs in github action
workflow to do parallel builds as @btashton and @davids suggest.
2. Use docker images with tools preinstalled
Now it takes several minitues to install all essential tools in ubuntu vm.
we may build nuttx with docker images which tools preinstalled in the
future.
Anyway, These are just some thoughts from our previous discussions,
welcome more reviews and feedbacks from community.
Thanks!
Miguel Ángel Herranz <mi...@midokura.com.invalid> 于2020年2月4日周二 下午7:23写道：
> Hi Haitao,
>
> I will take a look.
>
> By the way, do Apache Jenkins slave includes Docker? Or alternatively, does
> Apache Jenkins have Docker Slave [1] plugin available?
>
> If we could use either launch Docker containers in the slave or launch the
> jobs in a Docker container we could use an image with all the tools pre
> installed.
>
> Please someone with read privileges on Jenkins check that whenever possible
> (Xiang? Justin?).
>
> Thanks,
> Miguel
>
> [1] https://plugins.jenkins.io/docker-slaves
>
> On Tue, Feb 4, 2020 at 12:04 PM Haitao Liu <li...@gmail.com> wrote:
>
> > Thanks Xiang!
> > Another PR https://github.com/apache/incubator-nuttx-testing/pull/4
> since
> > Apache jenkins slave haven't gperf preinstalled,
> > so switch to use our owned prebuilt gperf. Xiang and Miguel please help
> to
> > review.
> > Or in another way, may I ask Justin Apache Infra team to apt-get install
> > gperf instead?
> >
> > Xiang Xiao <xi...@gmail.com> 于2020年2月4日周二 下午2:36写道：
> >
> > > I will merge it.
> > >
> > > On Tue, Feb 4, 2020 at 12:50 PM Haitao Liu <li...@gmail.com> wrote:
> > > >
> > > > Hi, Greg,
> > > > Thanks to Miguel's careful review, could you please help to finish
> the
> > PR
> > > > merge?
> > > > Then I and Miguel can continue the development with more PRs and
> > > refinement.
> > > >
> > > > Gregory Nutt <sp...@gmail.com> 于2020年1月31日周五 下午11:30写道：
> > > >
> > > > >
> > > > > > I have just added the CI scripts and testlist files in
> > > > > > https://github.com/apache/incubator-nuttx-testing/pull/3
> > > > > > And asked XiaoXiang help to create Nuttx Nightly build job in
> > > > > > builds.apache.org before. But it's just a freestyle CI job
> > > > > > which call cibuild.sh simply. Once CI script and testlist file
> > verify
> > > > > well.
> > > > > > It would be better if it could be integrated into the
> > > > > > pipeline definition you mentioned. You could help to use the
> > pipeline
> > > > > > way
> > > > > > if access granted. Let's work together on it :)
> > > > >
> > > > > Perhaps Miguel could provide the review of PR3. I will be happy to
> > > > > finish the merge after some review is provided.
> > > > >
> > > > > Once it has been merged, both you and Miguel can continue the
> > > > > development with PRs. But we need to find someone competent to
> > provide
> > > > > review.
> > > > >
> > > > > Greg
> > > > >
> > > > >
> > > > >
> > >
> >
>
Re: Jenkins CI
Posted by Miguel Ángel Herranz <mi...@midokura.com.INVALID>.
Hi Haitao,
I will take a look.
By the way, do Apache Jenkins slave includes Docker? Or alternatively, does
Apache Jenkins have Docker Slave [1] plugin available?
If we could use either launch Docker containers in the slave or launch the
jobs in a Docker container we could use an image with all the tools pre
installed.
Please someone with read privileges on Jenkins check that whenever possible
(Xiang? Justin?).
Thanks,
Miguel
[1] https://plugins.jenkins.io/docker-slaves
On Tue, Feb 4, 2020 at 12:04 PM Haitao Liu <li...@gmail.com> wrote:
> Thanks Xiang!
> Another PR https://github.com/apache/incubator-nuttx-testing/pull/4 since
> Apache jenkins slave haven't gperf preinstalled,
> so switch to use our owned prebuilt gperf. Xiang and Miguel please help to
> review.
> Or in another way, may I ask Justin Apache Infra team to apt-get install
> gperf instead?
>
> Xiang Xiao <xi...@gmail.com> 于2020年2月4日周二 下午2:36写道：
>
> > I will merge it.
> >
> > On Tue, Feb 4, 2020 at 12:50 PM Haitao Liu <li...@gmail.com> wrote:
> > >
> > > Hi, Greg,
> > > Thanks to Miguel's careful review, could you please help to finish the
> PR
> > > merge?
> > > Then I and Miguel can continue the development with more PRs and
> > refinement.
> > >
> > > Gregory Nutt <sp...@gmail.com> 于2020年1月31日周五 下午11:30写道：
> > >
> > > >
> > > > > I have just added the CI scripts and testlist files in
> > > > > https://github.com/apache/incubator-nuttx-testing/pull/3
> > > > > And asked XiaoXiang help to create Nuttx Nightly build job in
> > > > > builds.apache.org before. But it's just a freestyle CI job
> > > > > which call cibuild.sh simply. Once CI script and testlist file
> verify
> > > > well.
> > > > > It would be better if it could be integrated into the
> > > > > pipeline definition you mentioned. You could help to use the
> pipeline
> > > > > way
> > > > > if access granted. Let's work together on it :)
> > > >
> > > > Perhaps Miguel could provide the review of PR3. I will be happy to
> > > > finish the merge after some review is provided.
> > > >
> > > > Once it has been merged, both you and Miguel can continue the
> > > > development with PRs. But we need to find someone competent to
> provide
> > > > review.
> > > >
> > > > Greg
> > > >
> > > >
> > > >
> >
>
Re: Jenkins CI
Posted by Haitao Liu <li...@gmail.com>.
Thanks Xiang!
Another PR https://github.com/apache/incubator-nuttx-testing/pull/4 since
Apache jenkins slave haven't gperf preinstalled,
so switch to use our owned prebuilt gperf. Xiang and Miguel please help to
review.
Or in another way, may I ask Justin Apache Infra team to apt-get install
gperf instead?
Xiang Xiao <xi...@gmail.com> 于2020年2月4日周二 下午2:36写道：
> I will merge it.
>
> On Tue, Feb 4, 2020 at 12:50 PM Haitao Liu <li...@gmail.com> wrote:
> >
> > Hi, Greg,
> > Thanks to Miguel's careful review, could you please help to finish the PR
> > merge?
> > Then I and Miguel can continue the development with more PRs and
> refinement.
> >
> > Gregory Nutt <sp...@gmail.com> 于2020年1月31日周五 下午11:30写道：
> >
> > >
> > > > I have just added the CI scripts and testlist files in
> > > > https://github.com/apache/incubator-nuttx-testing/pull/3
> > > > And asked XiaoXiang help to create Nuttx Nightly build job in
> > > > builds.apache.org before. But it's just a freestyle CI job
> > > > which call cibuild.sh simply. Once CI script and testlist file verify
> > > well.
> > > > It would be better if it could be integrated into the
> > > > pipeline definition you mentioned. You could help to use the pipeline
> > > > way
> > > > if access granted. Let's work together on it :)
> > >
> > > Perhaps Miguel could provide the review of PR3. I will be happy to
> > > finish the merge after some review is provided.
> > >
> > > Once it has been merged, both you and Miguel can continue the
> > > development with PRs. But we need to find someone competent to provide
> > > review.
> > >
> > > Greg
> > >
> > >
> > >
>
Re: Jenkins CI
Posted by Xiang Xiao <xi...@gmail.com>.
I will merge it.
On Tue, Feb 4, 2020 at 12:50 PM Haitao Liu <li...@gmail.com> wrote:
>
> Hi, Greg,
> Thanks to Miguel's careful review, could you please help to finish the PR
> merge?
> Then I and Miguel can continue the development with more PRs and refinement.
>
> Gregory Nutt <sp...@gmail.com> 于2020年1月31日周五 下午11:30写道：
>
> >
> > > I have just added the CI scripts and testlist files in
> > > https://github.com/apache/incubator-nuttx-testing/pull/3
> > > And asked XiaoXiang help to create Nuttx Nightly build job in
> > > builds.apache.org before. But it's just a freestyle CI job
> > > which call cibuild.sh simply. Once CI script and testlist file verify
> > well.
> > > It would be better if it could be integrated into the
> > > pipeline definition you mentioned. You could help to use the pipeline
> > > way
> > > if access granted. Let's work together on it :)
> >
> > Perhaps Miguel could provide the review of PR3. I will be happy to
> > finish the merge after some review is provided.
> >
> > Once it has been merged, both you and Miguel can continue the
> > development with PRs. But we need to find someone competent to provide
> > review.
> >
> > Greg
> >
> >
> >