You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@buildr.apache.org by Ittay Dror <it...@gmail.com> on 2008/11/05 08:10:22 UTC
lazy invoke projects
Hi,
The attached patch to buildr works for me (invoking without arguments,
with local task and with specific project tasks). What do you think? Can
this be included in Buildr?
The reason I'm requesting this is that I have a buildfile with 150
projects and the current way Buildr works, by invoking all projects
takes too much time when I want to build just a single project (and its
dependencies of course). On developer windows machine this "warmup" time
is ~15 seconds and is increasing as more modules are added. From my
tests, this patch can lower it to just a few seconds.
Ittay
--
--
Ittay Dror <it...@gmail.com>
Re: lazy invoke projects
Posted by Ittay Dror <it...@gmail.com>.
the attached update to the patch fixes recursive_task to work when the
subprojects have not already being invoked. it is not the best way to
implement it, but the quickest. note that it also makes 'recursive_task'
actually recursive, meaning you can call it on a specific project so it
will call the children of that project (the original implementation
would just make the parent task depend on it).
ittay
Ittay Dror wrote:
>
> Hi,
>
>
> The attached patch to buildr works for me (invoking without arguments,
> with local task and with specific project tasks). What do you think?
> Can this be included in Buildr?
>
>
> The reason I'm requesting this is that I have a buildfile with 150
> projects and the current way Buildr works, by invoking all projects
> takes too much time when I want to build just a single project (and
> its dependencies of course). On developer windows machine this
> "warmup" time is ~15 seconds and is increasing as more modules are
> added. From my tests, this patch can lower it to just a few seconds.
>
>
> Ittay
>
> --
> --
> Ittay Dror <it...@gmail.com>
>
--
--
Ittay Dror <it...@gmail.com>
Re: lazy invoke projects
Posted by lacton <la...@users.sourceforge.net>.
I usually have about 10 projects and 1000 files.
Maybe this question would be better on buildr-users instead of -dev,
to involve more of the community.
Lacton
On Wed, Nov 5, 2008 at 11:57 PM, Assaf Arkin <ar...@intalio.com> wrote:
> On Wed, Nov 5, 2008 at 1:10 AM, Ittay Dror <it...@gmail.com> wrote:
>>
>> The reason I'm requesting this is that I have a buildfile with 150 projects
>> and the current way Buildr works, by invoking all projects takes too much
>> time when I want to build just a single project (and its dependencies of
>> course). On developer windows machine this "warmup" time is ~15 seconds and
>> is increasing as more modules are added. From my tests, this patch can lower
>> it to just a few seconds.
>
> The reason I asked you to bring this back to the mailing list is the size issue.
>
> The projects I'm working on have, in the order of 1000~5000 files (not
> just source code), 5~25 projects, warmup around a second.
> Occasionally I would test things out against a larger corpus of code,
> ~50,000 files, I'm happy when that turns to go under 5 seconds. Only
> because I consider that at the top scale of what a single buildfile is
> expected to support.
>
> There are certain optimizations you can make that would benefits the
> smaller sizes (the 1000~5000 above), and would be acceptable on larger
> size (~50,000), but might be a drag on larger buildfiles. I never
> test with anything larger because I think the soft spot, the majority
> of uses fall in the smaller scale. Anything substantially larger you
> would want to (well, at least me) break up.
>
> So I'm wondering where do other people stand on this? What's the size
> range you expect buildfiles to be optimized for?
>
> Assaf
>
>
>>
>> Ittay
>>
>> --
>> --
>> Ittay Dror <it...@gmail.com>
>>
>
Re: lazy invoke projects
Posted by Assaf Arkin <ar...@intalio.com>.
On Thu, Nov 6, 2008 at 12:04 PM, Ittay Dror <it...@tikalk.com> wrote:
>
>
> Assaf Arkin wrote:
>>
>> If the patch works for you, you are more than welcome to use it. I
>> have to consider everything else that gets affected, beyond the needs
>> of a single project.
>>
>>
>
> can you please elaborate?
That would be the next step, if we decided that this path is worth pursuing.
Assaf
>
> Ittay
>>
>> Assaf
>>
>>
>>>
>>> ittay
>>>
>>>>
>>>> So I'm wondering where do other people stand on this? What's the size
>>>> range you expect buildfiles to be optimized for?
>>>>
>>>> Assaf
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Ittay
>>>>>
>>>>> --
>>>>> --
>>>>> Ittay Dror <it...@gmail.com>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> Ittay Dror <it...@tikalk.com>
>>> Tikal <http://www.tikalk.com>
>>> Tikal Project <http://tikal.sourceforge.net>
>>>
>>>
>>>
>>
>>
>
> --
> Ittay Dror <it...@tikalk.com>
> Tikal <http://www.tikalk.com>
> Tikal Project <http://tikal.sourceforge.net>
>
>
Re: lazy invoke projects
Posted by Ittay Dror <it...@tikalk.com>.
Assaf Arkin wrote:
> If the patch works for you, you are more than welcome to use it. I
> have to consider everything else that gets affected, beyond the needs
> of a single project.
>
>
can you please elaborate?
Ittay
> Assaf
>
>
>> ittay
>>
>>> So I'm wondering where do other people stand on this? What's the size
>>> range you expect buildfiles to be optimized for?
>>>
>>> Assaf
>>>
>>>
>>>
>>>
>>>> Ittay
>>>>
>>>> --
>>>> --
>>>> Ittay Dror <it...@gmail.com>
>>>>
>>>>
>>>>
>>>
>> --
>> Ittay Dror <it...@tikalk.com>
>> Tikal <http://www.tikalk.com>
>> Tikal Project <http://tikal.sourceforge.net>
>>
>>
>>
>
>
--
Ittay Dror <it...@tikalk.com>
Tikal <http://www.tikalk.com>
Tikal Project <http://tikal.sourceforge.net>
Re: lazy invoke projects
Posted by Assaf Arkin <ar...@intalio.com>.
On Wed, Nov 5, 2008 at 10:17 PM, Ittay Dror <it...@tikalk.com> wrote:
>
>
> Assaf Arkin wrote:
>>
>> On Wed, Nov 5, 2008 at 1:10 AM, Ittay Dror <it...@gmail.com> wrote:
>>
>>>
>>> The reason I'm requesting this is that I have a buildfile with 150
>>> projects
>>> and the current way Buildr works, by invoking all projects takes too much
>>> time when I want to build just a single project (and its dependencies of
>>> course). On developer windows machine this "warmup" time is ~15 seconds
>>> and
>>> is increasing as more modules are added. From my tests, this patch can
>>> lower
>>> it to just a few seconds.
>>>
>>
>> The reason I asked you to bring this back to the mailing list is the size
>> issue.
>>
>> The projects I'm working on have, in the order of 1000~5000 files (not
>> just source code), 5~25 projects, warmup around a second.
>> Occasionally I would test things out against a larger corpus of code,
>> ~50,000 files, I'm happy when that turns to go under 5 seconds. Only
>> because I consider that at the top scale of what a single buildfile is
>> expected to support.
>>
>>
>
> did you test on windows machines?
Yes. Give or take same performance.
> also, in my case, buildr is used to build c++ modules, where unlike java,
> you need to run the compilation frequently. so even 5 seconds is annoying.
>>
>> There are certain optimizations you can make that would benefits the
>> smaller sizes (the 1000~5000 above), and would be acceptable on larger
>> size (~50,000), but might be a drag on larger buildfiles. I never
>> test with anything larger because I think the soft spot, the majority
>> of uses fall in the smaller scale. Anything substantially larger you
>> would want to (well, at least me) break up.
>>
>
> majority of users in general, yes. but out of those users who will look into
> buildr? users with small scale projects, can probably manage with ant+ivy or
> maven. they will not need to learn a new language, there's a better
> community, more plugins, etc. so i think the sweet spot of buildr is in
> complex projects.
That's why I'm asking the community, and I'm looking for data points
about size not complexity.
> i do want to break the buildfile, but unfortunately i can't, the projects
> are very interconnected. it is enough to have one dependency to make two
> otherwise separate groups be in one file. doesn't make the code complex
>
> anyway, i think the patch i sent is fairly simple. it took me a few hours to
> create, so why not incorporate it?
If the patch works for you, you are more than welcome to use it. I
have to consider everything else that gets affected, beyond the needs
of a single project.
Assaf
>
> ittay
>>
>> So I'm wondering where do other people stand on this? What's the size
>> range you expect buildfiles to be optimized for?
>>
>> Assaf
>>
>>
>>
>>>
>>> Ittay
>>>
>>> --
>>> --
>>> Ittay Dror <it...@gmail.com>
>>>
>>>
>>
>>
>
> --
> Ittay Dror <it...@tikalk.com>
> Tikal <http://www.tikalk.com>
> Tikal Project <http://tikal.sourceforge.net>
>
>
Re: lazy invoke projects
Posted by Ittay Dror <it...@tikalk.com>.
Assaf Arkin wrote:
> On Wed, Nov 5, 2008 at 1:10 AM, Ittay Dror <it...@gmail.com> wrote:
>
>> The reason I'm requesting this is that I have a buildfile with 150 projects
>> and the current way Buildr works, by invoking all projects takes too much
>> time when I want to build just a single project (and its dependencies of
>> course). On developer windows machine this "warmup" time is ~15 seconds and
>> is increasing as more modules are added. From my tests, this patch can lower
>> it to just a few seconds.
>>
>
> The reason I asked you to bring this back to the mailing list is the size issue.
>
> The projects I'm working on have, in the order of 1000~5000 files (not
> just source code), 5~25 projects, warmup around a second.
> Occasionally I would test things out against a larger corpus of code,
> ~50,000 files, I'm happy when that turns to go under 5 seconds. Only
> because I consider that at the top scale of what a single buildfile is
> expected to support.
>
>
did you test on windows machines?
also, in my case, buildr is used to build c++ modules, where unlike
java, you need to run the compilation frequently. so even 5 seconds is
annoying.
> There are certain optimizations you can make that would benefits the
> smaller sizes (the 1000~5000 above), and would be acceptable on larger
> size (~50,000), but might be a drag on larger buildfiles. I never
> test with anything larger because I think the soft spot, the majority
> of uses fall in the smaller scale. Anything substantially larger you
> would want to (well, at least me) break up.
>
majority of users in general, yes. but out of those users who will look
into buildr? users with small scale projects, can probably manage with
ant+ivy or maven. they will not need to learn a new language, there's a
better community, more plugins, etc. so i think the sweet spot of buildr
is in complex projects.
i do want to break the buildfile, but unfortunately i can't, the
projects are very interconnected. it is enough to have one dependency to
make two otherwise separate groups be in one file. doesn't make the code
complex
anyway, i think the patch i sent is fairly simple. it took me a few
hours to create, so why not incorporate it?
ittay
> So I'm wondering where do other people stand on this? What's the size
> range you expect buildfiles to be optimized for?
>
> Assaf
>
>
>
>> Ittay
>>
>> --
>> --
>> Ittay Dror <it...@gmail.com>
>>
>>
>
>
--
Ittay Dror <it...@tikalk.com>
Tikal <http://www.tikalk.com>
Tikal Project <http://tikal.sourceforge.net>
Re: lazy invoke projects
Posted by Assaf Arkin <ar...@intalio.com>.
On Wed, Nov 5, 2008 at 1:10 AM, Ittay Dror <it...@gmail.com> wrote:
>
> The reason I'm requesting this is that I have a buildfile with 150 projects
> and the current way Buildr works, by invoking all projects takes too much
> time when I want to build just a single project (and its dependencies of
> course). On developer windows machine this "warmup" time is ~15 seconds and
> is increasing as more modules are added. From my tests, this patch can lower
> it to just a few seconds.
The reason I asked you to bring this back to the mailing list is the size issue.
The projects I'm working on have, in the order of 1000~5000 files (not
just source code), 5~25 projects, warmup around a second.
Occasionally I would test things out against a larger corpus of code,
~50,000 files, I'm happy when that turns to go under 5 seconds. Only
because I consider that at the top scale of what a single buildfile is
expected to support.
There are certain optimizations you can make that would benefits the
smaller sizes (the 1000~5000 above), and would be acceptable on larger
size (~50,000), but might be a drag on larger buildfiles. I never
test with anything larger because I think the soft spot, the majority
of uses fall in the smaller scale. Anything substantially larger you
would want to (well, at least me) break up.
So I'm wondering where do other people stand on this? What's the size
range you expect buildfiles to be optimized for?
Assaf
>
> Ittay
>
> --
> --
> Ittay Dror <it...@gmail.com>
>