You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nuttx.apache.org by "Matias N." <ma...@imap.cc> on 2020/09/27 02:53:52 UTC
parallelizing mkdep?
Hi,
as a test I changed the depend generation to main nuttx Makefiles to this:
%.dd: %.c
$(Q) $(MKDEP) $(DEPPATH) "$(CC)" -- $(CFLAGS) -- $< > $@
makedepfile: $(patsubst %.c,%.dd,$(SRCS))
@for f in $^; do cat $$f; done > Make.dep
$(call DELFILE, $^)
.depend: Makefile $(SRCS) $(TOPDIR)$(DELIM).config
$(Q) $(MAKE) makedepfile
$(Q) touch $@
This allows to parallelize each MKDEP call, which currently is done once with all $(SRCS) together.
Original:
make depend -j8 11.29s user 6.67s system 106% cpu 16.888 total
Parallel:
make depend -j8 14.34s user 5.73s system 286% cpu 6.994 total
So about 10 seconds less and much more CPU utilized.
What do you think of the change? Any reason not to do this?
Best,
Matias
Re: parallelizing mkdep?
Posted by Xiang Xiao <xi...@gmail.com>.
Matias, please make a patch, let's improve it step by step.
On Sun, Sep 27, 2020 at 10:48 PM Matias N. <ma...@imap.cc> wrote:
>
> With the code I shown here it would be still an all or nothing generation of Make.dep
> as I remove all .d files. I didn't want to change that to avoid breaking anything.
> Do you intend to create a PR in this regard? Otherwise I can create one with this code
>
> Best,
> Matias
>
> On Sun, Sep 27, 2020, at 10:42, Xiang Xiao wrote:
> > Yes, this is what we are considering to change. It desn't only improve
> > the parallelization, but also avoids generating the full library(e.g.
> > libc) dependency if the user just changes one source file.
> >
> > On Sun, Sep 27, 2020 at 10:55 AM Matias N. <ma...@imap.cc> wrote:
> > >
> > > Hi,
> > > as a test I changed the depend generation to main nuttx Makefiles to this:
> > >
> > > %.dd: %.c
> > > $(Q) $(MKDEP) $(DEPPATH) "$(CC)" -- $(CFLAGS) -- $< > $@
> > >
> > > makedepfile: $(patsubst %.c,%.dd,$(SRCS))
> > > @for f in $^; do cat $$f; done > Make.dep
> > > $(call DELFILE, $^)
> > >
> > > .depend: Makefile $(SRCS) $(TOPDIR)$(DELIM).config
> > > $(Q) $(MAKE) makedepfile
> > > $(Q) touch $@
> > >
> > > This allows to parallelize each MKDEP call, which currently is done once with all $(SRCS) together.
> > >
> > > Original:
> > > make depend -j8 11.29s user 6.67s system 106% cpu 16.888 total
> > > Parallel:
> > > make depend -j8 14.34s user 5.73s system 286% cpu 6.994 total
> > >
> > > So about 10 seconds less and much more CPU utilized.
> > >
> > > What do you think of the change? Any reason not to do this?
> > >
> > > Best,
> > > Matias
> >
Re: parallelizing mkdep?
Posted by "Matias N." <ma...@imap.cc>.
With the code I shown here it would be still an all or nothing generation of Make.dep
as I remove all .d files. I didn't want to change that to avoid breaking anything.
Do you intend to create a PR in this regard? Otherwise I can create one with this code
Best,
Matias
On Sun, Sep 27, 2020, at 10:42, Xiang Xiao wrote:
> Yes, this is what we are considering to change. It desn't only improve
> the parallelization, but also avoids generating the full library(e.g.
> libc) dependency if the user just changes one source file.
>
> On Sun, Sep 27, 2020 at 10:55 AM Matias N. <ma...@imap.cc> wrote:
> >
> > Hi,
> > as a test I changed the depend generation to main nuttx Makefiles to this:
> >
> > %.dd: %.c
> > $(Q) $(MKDEP) $(DEPPATH) "$(CC)" -- $(CFLAGS) -- $< > $@
> >
> > makedepfile: $(patsubst %.c,%.dd,$(SRCS))
> > @for f in $^; do cat $$f; done > Make.dep
> > $(call DELFILE, $^)
> >
> > .depend: Makefile $(SRCS) $(TOPDIR)$(DELIM).config
> > $(Q) $(MAKE) makedepfile
> > $(Q) touch $@
> >
> > This allows to parallelize each MKDEP call, which currently is done once with all $(SRCS) together.
> >
> > Original:
> > make depend -j8 11.29s user 6.67s system 106% cpu 16.888 total
> > Parallel:
> > make depend -j8 14.34s user 5.73s system 286% cpu 6.994 total
> >
> > So about 10 seconds less and much more CPU utilized.
> >
> > What do you think of the change? Any reason not to do this?
> >
> > Best,
> > Matias
>
Re: parallelizing mkdep?
Posted by Xiang Xiao <xi...@gmail.com>.
Yes, this is what we are considering to change. It desn't only improve
the parallelization, but also avoids generating the full library(e.g.
libc) dependency if the user just changes one source file.
On Sun, Sep 27, 2020 at 10:55 AM Matias N. <ma...@imap.cc> wrote:
>
> Hi,
> as a test I changed the depend generation to main nuttx Makefiles to this:
>
> %.dd: %.c
> $(Q) $(MKDEP) $(DEPPATH) "$(CC)" -- $(CFLAGS) -- $< > $@
>
> makedepfile: $(patsubst %.c,%.dd,$(SRCS))
> @for f in $^; do cat $$f; done > Make.dep
> $(call DELFILE, $^)
>
> .depend: Makefile $(SRCS) $(TOPDIR)$(DELIM).config
> $(Q) $(MAKE) makedepfile
> $(Q) touch $@
>
> This allows to parallelize each MKDEP call, which currently is done once with all $(SRCS) together.
>
> Original:
> make depend -j8 11.29s user 6.67s system 106% cpu 16.888 total
> Parallel:
> make depend -j8 14.34s user 5.73s system 286% cpu 6.994 total
>
> So about 10 seconds less and much more CPU utilized.
>
> What do you think of the change? Any reason not to do this?
>
> Best,
> Matias