You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@buildr.apache.org by Ittay Dror <it...@gmail.com> on 2008/10/16 17:13:37 UTC
can buildr avoid pre-invoking all projects?
Currently, in buildr:initialize (project.rb), all projects are invoked.
This is done so that if I write 'buildr <project task>', it will be
already known (where project task is something like foo:bar:build).
In large build environments, a developer will usually want to invoke
just a small subst of projects (build a component with only the modules
it requires). In contrast, such a large environment will contain many
modules and so invoking all projects becomes expensive (in my case, it
takes 50% of the time when building a single module). Is there a way to
avoid this?
I also tried this with a buildfile for the abdera project (a very simple
project with only jars). For a simple module the initialization is 1
second and the building of the module is 4 seconds (20% overhead), there
are 24 modules.
I'm using Windows, which may have a part in this.
Ittay
--
--
Ittay Dror <it...@gmail.com>
Re: can buildr avoid pre-invoking all projects?
Posted by "Ittay Dror (Freiman)" <it...@gmail.com>.
On Tue, Nov 4, 2008 at 10:13 PM, Ittay Dror (Freiman)
<it...@gmail.com>wrote:
>
>
> On Thu, Oct 16, 2008 at 7:49 PM, Assaf Arkin <ar...@intalio.com> wrote:
>
>> On Thu, Oct 16, 2008 at 9:35 AM, Ittay Dror <it...@tikalk.com> wrote:
>> >
>> >
>> > Assaf Arkin wrote:
>> >
>> >> On Thu, Oct 16, 2008 at 8:13 AM, Ittay Dror <it...@gmail.com>
>> wrote:
>> >>
>> >>>
>> >>> Currently, in buildr:initialize (project.rb), all projects are
>> invoked.
>> >>> This
>> >>> is done so that if I write 'buildr <project task>', it will be already
>> >>> known
>> >>> (where project task is something like foo:bar:build).
>> >>>
>> >>
>> >> Actually this is done to create a graph of all the tasks and wire them
>> >> together, so whatever task you invoke from the command line, all the
>> >> wired dependencies are invoked as well. There's no lazy evaluation
>> >> going on, which of course means everything gets computed whether you
>> >> need it or not. On the other hand, it keeps the code simple.
>> >>
>> >
>> > Since I'm writing the code once and it is used many times afterwards by
>> > developers, I don't mind paying a price for making their life easier.
>>
>> It keeps the Buildr code simple.
>
> with the 1.3.4 trunk i tried this:
>
> in raw_load_buildfile, comment out 'Buildr.projects' and instead:
> top_level_tasks.each do |task_name|
> if task_name.index(':')
> parts = []
> task_name.split(':').each do |part|
> parts << part
> begin
> Buildr.project(parts.join(':')).invoke
> rescue
> nil
> end
> end
> end
> end
>
> also,
> in Project#define
> comment out 'project.parent.enhance { project.invoke } if project.parent'
>
and also in #project:
if options && options[:scope]
# We assume parent project is evaluated.
project = options[:scope].split(':').inject([[]]) { |scopes,
scope| scopes << (scopes.last + [scope]) }.
reverse.
map do |scope|
name_parts = name.split(':')
parts = []
found = name_parts.all? do |part|
parts << part
ancestor = @projects[(scope + parts).join(':')]
ancestor.invoke if ancestor
true if ancestor
end
@projects[(scope + [name]).join(':')] if found
end.
compact.
first
end
>
>
> this seems to work for me, and is not very complicated.
>
> but, now it takes 0.5 to invoke a task instead of 2.5 seconds (on linux, on
> windows it will probably be much faster)
>
> what do you think?
>
> ittay
>
>>
>>
>> Assaf
>>
>> >
>> > Maybe add a way of "exporting" tasks (with a way to do it repeatedly for
>> all
>> > projects), so that these tasks can be run from the command line and only
>> > then will they invoke the projects they depend on?
>> >
>> > Ittay
>> >>
>> >> assaf
>> >>
>> >>
>> >>>
>> >>> In large build environments, a developer will usually want to invoke
>> just
>> >>> a
>> >>> small subst of projects (build a component with only the modules it
>> >>> requires). In contrast, such a large environment will contain many
>> >>> modules
>> >>> and so invoking all projects becomes expensive (in my case, it takes
>> 50%
>> >>> of
>> >>> the time when building a single module). Is there a way to avoid this?
>> >>>
>> >>>
>> >>> I also tried this with a buildfile for the abdera project (a very
>> simple
>> >>> project with only jars). For a simple module the initialization is 1
>> >>> second
>> >>> and the building of the module is 4 seconds (20% overhead), there are
>> 24
>> >>> modules.
>> >>>
>> >>>
>> >>> I'm using Windows, which may have a part in this.
>> >>>
>> >>>
>> >>> Ittay
>> >>>
>> >>>
>> >>> --
>> >>> --
>> >>> Ittay Dror <it...@gmail.com>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>> >
>> > --
>> > Ittay Dror <it...@tikalk.com>
>> > Tikal <http://www.tikalk.com>
>> > Tikal Project <http://tikal.sourceforge.net>
>> >
>> >
>>
>
>
Re: can buildr avoid pre-invoking all projects?
Posted by "Ittay Dror (Freiman)" <it...@gmail.com>.
On Thu, Oct 16, 2008 at 7:49 PM, Assaf Arkin <ar...@intalio.com> wrote:
> On Thu, Oct 16, 2008 at 9:35 AM, Ittay Dror <it...@tikalk.com> wrote:
> >
> >
> > Assaf Arkin wrote:
> >
> >> On Thu, Oct 16, 2008 at 8:13 AM, Ittay Dror <it...@gmail.com>
> wrote:
> >>
> >>>
> >>> Currently, in buildr:initialize (project.rb), all projects are invoked.
> >>> This
> >>> is done so that if I write 'buildr <project task>', it will be already
> >>> known
> >>> (where project task is something like foo:bar:build).
> >>>
> >>
> >> Actually this is done to create a graph of all the tasks and wire them
> >> together, so whatever task you invoke from the command line, all the
> >> wired dependencies are invoked as well. There's no lazy evaluation
> >> going on, which of course means everything gets computed whether you
> >> need it or not. On the other hand, it keeps the code simple.
> >>
> >
> > Since I'm writing the code once and it is used many times afterwards by
> > developers, I don't mind paying a price for making their life easier.
>
> It keeps the Buildr code simple.
with the 1.3.4 trunk i tried this:
in raw_load_buildfile, comment out 'Buildr.projects' and instead:
top_level_tasks.each do |task_name|
if task_name.index(':')
parts = []
task_name.split(':').each do |part|
parts << part
begin
Buildr.project(parts.join(':')).invoke
rescue
nil
end
end
end
end
also,
in Project#define
comment out 'project.parent.enhance { project.invoke } if project.parent'
this seems to work for me, and is not very complicated.
but, now it takes 0.5 to invoke a task instead of 2.5 seconds (on linux, on
windows it will probably be much faster)
what do you think?
ittay
>
>
> Assaf
>
> >
> > Maybe add a way of "exporting" tasks (with a way to do it repeatedly for
> all
> > projects), so that these tasks can be run from the command line and only
> > then will they invoke the projects they depend on?
> >
> > Ittay
> >>
> >> assaf
> >>
> >>
> >>>
> >>> In large build environments, a developer will usually want to invoke
> just
> >>> a
> >>> small subst of projects (build a component with only the modules it
> >>> requires). In contrast, such a large environment will contain many
> >>> modules
> >>> and so invoking all projects becomes expensive (in my case, it takes
> 50%
> >>> of
> >>> the time when building a single module). Is there a way to avoid this?
> >>>
> >>>
> >>> I also tried this with a buildfile for the abdera project (a very
> simple
> >>> project with only jars). For a simple module the initialization is 1
> >>> second
> >>> and the building of the module is 4 seconds (20% overhead), there are
> 24
> >>> modules.
> >>>
> >>>
> >>> I'm using Windows, which may have a part in this.
> >>>
> >>>
> >>> Ittay
> >>>
> >>>
> >>> --
> >>> --
> >>> Ittay Dror <it...@gmail.com>
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >
> > --
> > Ittay Dror <it...@tikalk.com>
> > Tikal <http://www.tikalk.com>
> > Tikal Project <http://tikal.sourceforge.net>
> >
> >
>
Re: can buildr avoid pre-invoking all projects?
Posted by Assaf Arkin <ar...@intalio.com>.
On Thu, Oct 16, 2008 at 9:35 AM, Ittay Dror <it...@tikalk.com> wrote:
>
>
> Assaf Arkin wrote:
>
>> On Thu, Oct 16, 2008 at 8:13 AM, Ittay Dror <it...@gmail.com> wrote:
>>
>>>
>>> Currently, in buildr:initialize (project.rb), all projects are invoked.
>>> This
>>> is done so that if I write 'buildr <project task>', it will be already
>>> known
>>> (where project task is something like foo:bar:build).
>>>
>>
>> Actually this is done to create a graph of all the tasks and wire them
>> together, so whatever task you invoke from the command line, all the
>> wired dependencies are invoked as well. There's no lazy evaluation
>> going on, which of course means everything gets computed whether you
>> need it or not. On the other hand, it keeps the code simple.
>>
>
> Since I'm writing the code once and it is used many times afterwards by
> developers, I don't mind paying a price for making their life easier.
It keeps the Buildr code simple.
Assaf
>
> Maybe add a way of "exporting" tasks (with a way to do it repeatedly for all
> projects), so that these tasks can be run from the command line and only
> then will they invoke the projects they depend on?
>
> Ittay
>>
>> assaf
>>
>>
>>>
>>> In large build environments, a developer will usually want to invoke just
>>> a
>>> small subst of projects (build a component with only the modules it
>>> requires). In contrast, such a large environment will contain many
>>> modules
>>> and so invoking all projects becomes expensive (in my case, it takes 50%
>>> of
>>> the time when building a single module). Is there a way to avoid this?
>>>
>>>
>>> I also tried this with a buildfile for the abdera project (a very simple
>>> project with only jars). For a simple module the initialization is 1
>>> second
>>> and the building of the module is 4 seconds (20% overhead), there are 24
>>> modules.
>>>
>>>
>>> I'm using Windows, which may have a part in this.
>>>
>>>
>>> Ittay
>>>
>>>
>>> --
>>> --
>>> Ittay Dror <it...@gmail.com>
>>>
>>>
>>>
>>>
>>
>>
>
> --
> Ittay Dror <it...@tikalk.com>
> Tikal <http://www.tikalk.com>
> Tikal Project <http://tikal.sourceforge.net>
>
>
Re: can buildr avoid pre-invoking all projects?
Posted by Ittay Dror <it...@tikalk.com>.
Assaf Arkin wrote:
> On Thu, Oct 16, 2008 at 8:13 AM, Ittay Dror <it...@gmail.com> wrote:
>
>> Currently, in buildr:initialize (project.rb), all projects are invoked. This
>> is done so that if I write 'buildr <project task>', it will be already known
>> (where project task is something like foo:bar:build).
>>
>
> Actually this is done to create a graph of all the tasks and wire them
> together, so whatever task you invoke from the command line, all the
> wired dependencies are invoked as well. There's no lazy evaluation
> going on, which of course means everything gets computed whether you
> need it or not. On the other hand, it keeps the code simple.
>
Since I'm writing the code once and it is used many times afterwards by
developers, I don't mind paying a price for making their life easier.
Maybe add a way of "exporting" tasks (with a way to do it repeatedly for
all projects), so that these tasks can be run from the command line and
only then will they invoke the projects they depend on?
Ittay
> assaf
>
>
>> In large build environments, a developer will usually want to invoke just a
>> small subst of projects (build a component with only the modules it
>> requires). In contrast, such a large environment will contain many modules
>> and so invoking all projects becomes expensive (in my case, it takes 50% of
>> the time when building a single module). Is there a way to avoid this?
>>
>>
>> I also tried this with a buildfile for the abdera project (a very simple
>> project with only jars). For a simple module the initialization is 1 second
>> and the building of the module is 4 seconds (20% overhead), there are 24
>> modules.
>>
>>
>> I'm using Windows, which may have a part in this.
>>
>>
>> Ittay
>>
>>
>> --
>> --
>> Ittay Dror <it...@gmail.com>
>>
>>
>>
>>
>
>
--
Ittay Dror <it...@tikalk.com>
Tikal <http://www.tikalk.com>
Tikal Project <http://tikal.sourceforge.net>
Re: can buildr avoid pre-invoking all projects?
Posted by Ittay Dror <it...@tikalk.com>.
Assaf Arkin wrote:
> On Thu, Oct 16, 2008 at 8:13 AM, Ittay Dror <it...@gmail.com> wrote:
>
>> Currently, in buildr:initialize (project.rb), all projects are invoked. This
>> is done so that if I write 'buildr <project task>', it will be already known
>> (where project task is something like foo:bar:build).
>>
>
> Actually this is done to create a graph of all the tasks and wire them
> together, so whatever task you invoke from the command line, all the
> wired dependencies are invoked as well. There's no lazy evaluation
> going on, which of course means everything gets computed whether you
> need it or not. On the other hand, it keeps the code simple.
>
Is there a way to use something like nailgun when running with cruby?
Ittay
> assaf
>
>
>> In large build environments, a developer will usually want to invoke just a
>> small subst of projects (build a component with only the modules it
>> requires). In contrast, such a large environment will contain many modules
>> and so invoking all projects becomes expensive (in my case, it takes 50% of
>> the time when building a single module). Is there a way to avoid this?
>>
>>
>> I also tried this with a buildfile for the abdera project (a very simple
>> project with only jars). For a simple module the initialization is 1 second
>> and the building of the module is 4 seconds (20% overhead), there are 24
>> modules.
>>
>>
>> I'm using Windows, which may have a part in this.
>>
>>
>> Ittay
>>
>>
>> --
>> --
>> Ittay Dror <it...@gmail.com>
>>
>>
>>
>>
>
>
--
Ittay Dror <it...@tikalk.com>
Tikal <http://www.tikalk.com>
Tikal Project <http://tikal.sourceforge.net>
Re: can buildr avoid pre-invoking all projects?
Posted by Assaf Arkin <ar...@intalio.com>.
On Thu, Oct 16, 2008 at 8:13 AM, Ittay Dror <it...@gmail.com> wrote:
> Currently, in buildr:initialize (project.rb), all projects are invoked. This
> is done so that if I write 'buildr <project task>', it will be already known
> (where project task is something like foo:bar:build).
Actually this is done to create a graph of all the tasks and wire them
together, so whatever task you invoke from the command line, all the
wired dependencies are invoked as well. There's no lazy evaluation
going on, which of course means everything gets computed whether you
need it or not. On the other hand, it keeps the code simple.
assaf
>
>
> In large build environments, a developer will usually want to invoke just a
> small subst of projects (build a component with only the modules it
> requires). In contrast, such a large environment will contain many modules
> and so invoking all projects becomes expensive (in my case, it takes 50% of
> the time when building a single module). Is there a way to avoid this?
>
>
> I also tried this with a buildfile for the abdera project (a very simple
> project with only jars). For a simple module the initialization is 1 second
> and the building of the module is 4 seconds (20% overhead), there are 24
> modules.
>
>
> I'm using Windows, which may have a part in this.
>
>
> Ittay
>
>
> --
> --
> Ittay Dror <it...@gmail.com>
>
>
>